Slow SSH authentication on Ubuntu servers / 30 second delay


#1

Since having my server switched to Ubuntu, I’ve noticed that establishing SSH connections is much slower than it used to be. Fortunately, the backups user machine is still running Debian, so I had a chance to compare the process and found that not only is there a long delay on the Ubuntu machine, but the delay is also right around 30 seconds. This length of time is significant for anyone who has been following the periodic slowdown threads whereby apache will sometimes wait a multiple of 30 seconds before responding to requests. I wonder if the two are related somehow?

Here’s a log of establishing an SSH connection (via SCP) to the backups server (running Debian):

[code]$ touch /tmp/test; time scp -v /tmp/test bxxxxxx@hanjin.dreamhost.com:/dev/null
Executing: program /usr/bin/ssh host hanjin.dreamhost.com, user bxxxxxx, command scp -v -t – /dev/null
OpenSSH_5.9p1 Debian-5ubuntu1.4, OpenSSL 1.0.1 14 Mar 2012
debug1: Connecting to hanjin.dreamhost.com [66.33.206.119] port 22.
debug1: Connection established.
debug1: identity file /home/user/.ssh/id_rsa type 1
debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048
debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.5p1 Debian-6+squeeze2
debug1: match: OpenSSH_5.5p1 Debian-6+squeeze2 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.9p1 Debian-5ubuntu1.4
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 zlib@openssh.com
debug1: kex: client->server aes128-ctr hmac-md5 zlib@openssh.com
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: RSA 0e:c2:f6:f4:d9:86:9d:4b:c4:3d:77:e7:a4:bb:59:14
debug1: Host ‘hanjin.dreamhost.com’ is known and matches the RSA host key.
debug1: Found key in /home/user/.ssh/known_hosts:10
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/user/.ssh/id_rsa
debug1: Server accepts key: pkalg ssh-rsa blen 277
debug1: Enabling compression at level 6.
debug1: Authentication succeeded (publickey).
Authenticated to hanjin.dreamhost.com ([66.33.206.119]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: Sending environment.
debug1: Sending command: scp -v -t – /dev/null
Sending file modes: C0664 0 test
Sink: C0664 0 test
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: channel 0: free: client-session, nchannels 1
debug1: fd 0 clearing O_NONBLOCK
debug1: fd 1 clearing O_NONBLOCK
debug1: fd 2 clearing O_NONBLOCK
Transferred: sent 3120, received 2296 bytes, in 0.9 seconds
Bytes per second: sent 3613.1, received 2658.9
debug1: Exit status 0
debug1: compress outgoing: raw data 593, compressed 335, factor 0.56
debug1: compress incoming: raw data 128, compressed 96, factor 0.75

real 0m2.967s
user 0m0.024s
sys 0m0.008s
[/code]

And here is one doing exactly the same thing to my shared server which is running Ubuntu (note the long delay in user authentication):



$ touch /tmp/test; time scp -v /tmp/test user@example.com:/dev/null
Executing: program /usr/bin/ssh host example.com, user dh_shell_user, command scp -v -t -- /dev/null
OpenSSH_5.9p1 Debian-5ubuntu1.4, OpenSSL 1.0.1 14 Mar 2012
debug1: Connecting to example.com [64.111.x.x] port 22.
debug1: Connection established.
debug1: identity file /home/user/.ssh/id_rsa type 1
debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048
debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.9p1 Debian-5ubuntu1.4
debug1: match: OpenSSH_5.9p1 Debian-5ubuntu1.4 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.9p1 Debian-5ubuntu1.4
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 zlib@openssh.com
debug1: kex: client->server aes128-ctr hmac-md5 zlib@openssh.com
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: RSA xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx
debug1: Host 'example.com' is known and matches the RSA host key.
debug1: Found key in /home/user/.ssh/known_hosts:52
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
<<<<<<<<<< LONG DELAY HERE >>>>>>>>>>
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/user/.ssh/id_rsa
debug1: Server accepts key: pkalg ssh-rsa blen 277
debug1: Enabling compression at level 6.
debug1: Authentication succeeded (publickey).
Authenticated to example.com ([64.111.x.x]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: Sending environment.
debug1: Sending command: scp -v -t -- /dev/null
Sending file modes: C0664 0 test
Sink: C0664 0 test
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: channel 0: free: client-session, nchannels 1
debug1: fd 0 clearing O_NONBLOCK
debug1: fd 1 clearing O_NONBLOCK
debug1: fd 2 clearing O_NONBLOCK
Transferred: sent 3048, received 2280 bytes, in 0.9 seconds
Bytes per second: sent 3498.0, received 2616.7
debug1: Exit status 0
debug1: compress outgoing: raw data 593, compressed 335, factor 0.56
debug1: compress incoming: raw data 128, compressed 96, factor 0.75

real    0m30.875s
user    0m0.016s
sys     0m0.016s

#2

I have noticed this on Ubuntu server as well. Actually, it does happen on a physical box inside our server room. It has a significant delay when compared to CentOS.

But try this quick fix and see how it goes for you.

http://www.westernwillow.com/cms/blog/franco/fix-slow-ssh-response-your-ubuntu-server


#3

I believe the DNS lookup is likely the culprit as my IP doesn’t resolve to a domain name which leads to the 30 second delay.

So, Dreamhost, please disable that option on your new Ubuntu servers. Thanks.


#4

Try using the -4 flag. I’ve been having issues with IPv6 since the upgrade.