A comprehensive guide to fixing slow SSH logins

The debug text that brought you here

Most of you are probably getting here just from frustratedly googling “slow ssh login”.  Those of you who got a little froggier and tried doing an ssh -vv to get lots of debug output saw things hanging at debug1: SSH2_MSG_SERVICE_ACCEPT received, likely for long enough that you assumed the entire process was hung and ctrl-C’d out.  If you’re patient enough, the process will generally eventually continue after the debug1: SSH2_MSG_SERVICE_ACCEPT received line, but it may take 30 seconds.  Or even five minutes.

You might also have enabled debug logging on the server, and discovered that your hang occurs immediately after debug1: KEX done [preauth]
and before debug1: userauth-request for user in /var/log/auth.log.

I feel your frustration, dear reader. I have solved this problem after hours of screeching head-desking probably ten times over the years.  There are a few fixes for this, with the most common – DNS – tending to drown out the rest.  Which is why I keep screeching in frustration every few years; I remember the dreaded debug1: SSH2_MSG_SERVICE_ACCEPT received hang is something I’ve solved before, but I can only remember some of the fixes I’ve needed.

Anyway, here are all the fixes I’ve needed to deploy over the years, collected in one convenient place where I can find them again.

It’s usually DNS.

The most common cause of slow SSH login authentications is DNS. To fix this one, go to the SSH server, edit /etc/ssh/sshd_config, and set UseDNS no.  You’ll need to restart the service after changing sshd_config: /etc/init.d/ssh restart, systemctl restart ssh, etc as appropriate.

If it’s not DNS, it’s Avahi.

The next most common cause – which is devilishly difficult to find reference to online, and I hope this helps – is the never-to-be-sufficiently damned avahi daemon.  To fix this one, go to the SSH client, edit /etc/nsswitch.conf, and change this line:

hosts:          files mdns4_minimal [NOTFOUND=return] dns

to:

hosts:          files dns

In theory maybe something might stop working without that mdns4_minimal option?  But I haven’t got the foggiest notion what that might be, because nothing ever seems broken for me after disabling it.  No services need restarting after making this change, which again, must be made on the client.

You might think this isn’t your problem. Maybe your slow logins only happen when SSHing to one particular server, even one particular server on your local network, even one particular server on your local network which has UseDNS no and which you don’t need any DNS resolution to connect to in the first place.  But yeah, it can still be this avahi crap. Yay.

When it’s not Avahi… it’s PAM.

This is another one that’s really, really difficult to find references to online.  Optional PAM modules can really screw you here.  In my experience, you can’t get away with simply disabling PAM login in /etc/ssh/sshd_config – if you do, you won’t be able to log in at all.

What you need to do is go to the SSH server, edit /etc/pam.d/common-session and comment out the optional module that’s causing you grief.  In the past, that was pam_ck_connector.so.  More recently, in Ubuntu 16.04, the culprit that bit me hard was pam_systemd.so. Get in there and comment that bugger out.  No restarts required.

#session optional pam_systemd.so

GSSAPI, and ChallengeResponse.

I’ve seen a few seconds added to a pokey login from GSSAPIAuthentication, whatever that is. I feel slightly embarrassed about not knowing, but seriously, I have no clue.  Ditto for ChallengeResponseAuthentication.  All I can tell you is that neither cover standard interactive passwords, or standard public/private keypair authentication (the keys you keep in ~/ssh/authorized_keys).

If you aren’t using them either, then disable them.  If you’re not using Active Directory authentication, might as well go ahead and nuke Kerberos while you’re at it.  Make these changes on the server in /etc/ssh/sshd_config, and restart the service.

ChallengeResponseAuthentication no
KerberosAuthentication no
GSSAPIAuthentication no

Host-based Authentication.

If you’re actually using this, don’t disable it. But let’s get real with each other: you’re not using it.  I mean, I’m sure somebody out there is.  But it’s almost certainly not you.  Get rid of it.  This is also on the server in /etc/ssh/sshd_config, and also will require a service restart.

# Don't read the user's ~/.rhosts and ~/.shosts files
IgnoreRhosts yes
# For this to work you will also need host keys in /etc/ssh_known_hosts
RhostsRSAAuthentication no
# similar for protocol version 2
HostbasedAuthentication no

Need frequent connections? Consider control sockets.

If you need to do repetitive ssh’ing into another host, you can speed up the repeated ssh commands enormously with the use of control sockets.  The first time you SSH into the host, you establish a socket.  After that, the socket obviates the need for re-authentication.

ssh -M -S /path/to/socket -o ControlPersist=5m remotehost exit

This creates a new SSH socket at /path/to/socket which will survive for the next 5 minutes, after which it’ll automatically expire.  The exit at the end just causes the ssh connection you establish to immediately terminate (otherwise you’d have a perfectly normal ssh session going to remotehost).

After creating the control socket, you utilize it with -S /path/to/socket in any normal ssh command to remotehost.

The improvement in execution time for commands using that socket is delicious.

me@banshee:/~$ ssh -M -S /tmp/demo -o ControlPersist=5m eohippus exit

me@banshee:/~$ time ssh eohippus exit
real 0m0.660s
user 0m0.052s
sys 0m0.016s

me@banshee:/~$ time ssh -S /tmp/demo eohippus exit
real 0m0.050s
user 0m0.005s
sys 0m0.010s

Yep… from 660ms to 50ms.  SSH control sockets are pretty awesome.

Published by

Jim Salter

Mercenary sysadmin, open source advocate, and frotzer of the jim-jam.

41 thoughts on “A comprehensive guide to fixing slow SSH logins”

  4. Thank you so much! I was banging me head against a wall trying to diagnose this, and stumbled across this post while looking for ideas to troubleshoot it! DNS was the issue for me! Thank you!

  5. Yet another one… DNS for the server has AAAA and A records. Client OS thinks it’s got IPv6 routing but actually it hasn’t – but you have to wait for the IPv6 attempt to fail before it falls back to “legacy” IPv4.

  8. DNS! Its always DNS… Slows everything! Found an imporperly configure server with slow and non existent dns servers. I was tasked into why website performance was so slow… first thing first though, why is ssh so slow…

  9. DNS! It’s always DNS… Slows everything! Found an improperly configured server with slow and non-existent DNS servers. I was tasked into why website performance was so slow… first thing first though, why is ssh so slow…

  12. Thank you. I encountered this problem before, but had forgotten the fix.
    This time I had to disable both UseDNS *AND* GSSAPIAuthentication (Centos 7.7)

  15. Beware that the PAM disabling could affect display manger, which could disable gdm3 and lightdm. This is because the /etc/pam.d/common-session is common for multiple services.

  16. Thank you very much. I have a few hundreds of concurrent SSH connection and pam_systemd slowed down the connection.

  18. Thank you! I’ve been living with the long login time for so long and finally decided I would fix it, you made it so quick!

    For me, I’m on Ubuntu 16.04, and commenting the line below in /etc/pam.d/common-session worked like a charm!

    #session optional pam_systemd.so

  19. I have to say that this article is awesome in every way. Professional, funny and sexi.

    Also, maybe something new. We had also problem with landscape-sysinfo process, which consumed whole CPU usage. We are suspicious that this is caused by mounted fileshares in our VM. landscape-sysinfo is monitor for resources. In our case was problem with monitoring of disks.

    To fix this is needed this configuration (wihtout this we were sometime not even able to connect.) .
    Change config in /etc/landscape/client.conf to :
    [sysinfo]
    exclude_sysinfo_plugins = Disk

    For us worked this, which drecrease time to 30 s in some cases.
    GSSAPIAuthentication no
    ChallengeResponseAuthentication no
    UseDNS no
    GSSAPICleanupCredentials yes

    And commented pam to max 8 seconds.
    #session optional pam_systemd.so

    Thanks, Jim Salter good job!

  20. Another additional: DNS had ipv4+ipv6, but ipv6 was not reachable because of firewall rule. My command was: ssh myhost, and this tries first ipv6, after a timeout from 5-10 seconds it tries ipv4.

    I checked it with:

    ssh -6 myhost (timeout, aha! ipv6 not reachable)
    ssh -4 myhost (immediately)

  22. Ubuntu 20.10 latest updates and I was running into slow ssh. Thanks for the article. Here it was the line below I had to comment:
    #session optional pam_systemd.so

  23. Very useful post.

    The avahi mdns4_minimal enables the domain .local to find hosts (i.e. my_computer.local) on the local network, using avahi lookup. It’s not something too many people do, but is useful for some setups. You can discover computers that you can locate this way using the tool “avahi-discover”

  25. This can also be caused by pam_krb5 (Kerberos) if using a local account to log in. It tries to authenticate to the domain first. Turning off this module in /etc/pam.d/common-auth works, if you’re not using it…

  27. Thank you for this awesome little article! I recently ran into problems with SSH connections that took about 2½ minutes before the password prompt showed up. In my case what finally worked was to add the option `AddressFamily inet` (to disable IPv6) for the relevant host in the client side config (`~/.ssh/config`).

    Since that isn’t covered above I wanted to share my findings. Thanx for an awesome guide!

  28. Hi,

    Thanks for the information! After updating to Debian bullseye I had slow SSH connection.
    Seems that #session optional pam_systemd.so was the cruelpit.

  29. This was very helpful, but did not quite solve my problem. The problem turned out to be related to dns. I am using hosts.allow and hosts.deny to limit connections to local networks. I had a LOCAL in the statement for hosts.allow. Removing that fixed things. I’m not sure I fully understand why, but according to the hosts.allow documentation, LOCAL stands for any hostname without a dot. I was doing this on a new router without fully functional dns, and I think that the LOCAL triggered a dns lookup. Either adding a host entry for the client into the servers /etc/hosts or deleting LOCAL worked.

  31. I had a huge /var/log/btmp on a Fedora system, which was read by pam_lastlog.so’s showfailed and so lead to a ~20s login time. I fixed it by fixing logrotation of that file.

    Sadly, this PAM stuff is rather difficult to debug so this was hard to find.

  34. Thank you, in my case, using Debian 11, it was the parameter : #session optional pam_systemd.so

  35. There can be one more thing (copied here https://serverfault.com/questions/792486/ssh-connection-takes-forever-to-initiate-stuck-at-pledge-network):

    This is probably an issue with D-Bus and systemd. If the dbus service is restarted for some reason, you will also need to restart systemd-logind.

    You can check if this is the issue by opening the ssh daemon log (on Ubuntu it should be /var/log/auth.log) and check if it has these lines:

    sshd[2721]: pam_systemd(sshd:session): Failed to create session: Connection timed out
    If yes, just restart systemd-logind service:

    systemctl restart systemd-logind
    I had this same issue on CentOS 7, because the messagebus was restarted (which is how the D-Bus service is called on CentOS).

  40. Thanks Hans Schou – your Redhat fix worked after the login suddenly slowed following a yum update.

  41. That was really helpful, thank you. It would have taken me ages to track down the PAM thing.

Leave a Reply

Your email address will not be published. Required fields are marked *