Seif Sallam

I know this is a no brainer for most people, but it took me sometime to do it, so incase anyone having problems with connecting via SSH to Ubuntu here is a simple walkthrough.

1. Create or Copy Development machine’s public RSA located ~/.ssh/id_rsa.pub
2. Login to server and paste it in vim ~/.ssh/authorized_keys
3. Ensure that /etc/ssh/sshd_config contains PubkeyAuthentication yes and RSAAuthentication yes uncommented
4. Finally set chmod go-w ~/, chmod 700 ~/.ssh, and chmod 600 ~/.ssh/authorized_keys to make sure that permissions are set correctly

Update

For more information check this guide
http://library.linode.com/securing-your-server . Thanks to @kamasheto