The debug text that brought you here
Most of you are probably getting here just from frustratedly googling “slow ssh login”. Those of you who got a little froggier and tried doing an ssh -vv to get lots of debug output saw things hanging at debug1: SSH2_MSG_SERVICE_ACCEPT received, likely for long enough that you assumed the entire process was hung and ctrl-C’d out. If you’re patient enough, the process will generally eventually continue after the debug1: SSH2_MSG_SERVICE_ACCEPT received line, but it may take 30 seconds. Or even five minutes.
You might also have enabled debug logging on the server, and discovered that your hang occurs immediately after debug1: KEX done [preauth]
and before debug1: userauth-request for user in /var/log/auth.log.
I feel your frustration, dear reader. I have solved this problem after hours of screeching head-desking probably ten times over the years. There are a few fixes for this, with the most common – DNS – tending to drown out the rest. Which is why I keep screeching in frustration every few years; I remember the dreaded debug1: SSH2_MSG_SERVICE_ACCEPT received hang is something I’ve solved before, but I can only remember some of the fixes I’ve needed.
Anyway, here are all the fixes I’ve needed to deploy over the years, collected in one convenient place where I can find them again.
It’s usually DNS.
The most common cause of slow SSH login authentications is DNS. To fix this one, go to the SSH server, edit /etc/ssh/sshd_config, and set UseDNS no. You’ll need to restart the service after changing sshd_config: /etc/init.d/ssh restart, systemctl restart ssh, etc as appropriate.
If it’s not DNS, it’s Avahi.
The next most common cause – which is devilishly difficult to find reference to online, and I hope this helps – is the never-to-be-sufficiently damned avahi daemon. To fix this one, go to the SSH client, edit /etc/nsswitch.conf, and change this line:
hosts: files mdns4_minimal [NOTFOUND=return] dns
hosts: files dns
In theory maybe something might stop working without that mdns4_minimal option? But I haven’t got the foggiest notion what that might be, because nothing ever seems broken for me after disabling it. No services need restarting after making this change, which again, must be made on the client.
You might think this isn’t your problem. Maybe your slow logins only happen when SSHing to one particular server, even one particular server on your local network, even one particular server on your local network which has UseDNS no and which you don’t need any DNS resolution to connect to in the first place. But yeah, it can still be this avahi crap. Yay.
When it’s not Avahi… it’s PAM.
This is another one that’s really, really difficult to find references to online. Optional PAM modules can really screw you here. In my experience, you can’t get away with simply disabling PAM login in /etc/ssh/sshd_config – if you do, you won’t be able to log in at all.
What you need to do is go to the SSH server, edit /etc/pam.d/common-session and comment out the optional module that’s causing you grief. In the past, that was pam_ck_connector.so. More recently, in Ubuntu 16.04, the culprit that bit me hard was pam_systemd.so. Get in there and comment that bugger out. No restarts required.
#session optional pam_systemd.so
GSSAPI, and ChallengeResponse.
I’ve seen a few seconds added to a pokey login from GSSAPIAuthentication, whatever that is. I feel slightly embarrassed about not knowing, but seriously, I have no clue. Ditto for ChallengeResponseAuthentication. All I can tell you is that neither cover standard interactive passwords, or standard public/private keypair authentication (the keys you keep in ~/ssh/authorized_keys).
If you aren’t using them either, then disable them. If you’re not using Active Directory authentication, might as well go ahead and nuke Kerberos while you’re at it. Make these changes on the server in /etc/ssh/sshd_config, and restart the service.
ChallengeResponseAuthentication no KerberosAuthentication no GSSAPIAuthentication no
If you’re actually using this, don’t disable it. But let’s get real with each other: you’re not using it. I mean, I’m sure somebody out there is. But it’s almost certainly not you. Get rid of it. This is also on the server in /etc/ssh/sshd_config, and also will require a service restart.
# Don't read the user's ~/.rhosts and ~/.shosts files IgnoreRhosts yes # For this to work you will also need host keys in /etc/ssh_known_hosts RhostsRSAAuthentication no # similar for protocol version 2 HostbasedAuthentication no
Need frequent connections? Consider control sockets.
If you need to do repetitive ssh’ing into another host, you can speed up the repeated ssh commands enormously with the use of control sockets. The first time you SSH into the host, you establish a socket. After that, the socket obviates the need for re-authentication.
ssh -M -S /path/to/socket -o ControlPersist=5m remotehost exit
This creates a new SSH socket at /path/to/socket which will survive for the next 5 minutes, after which it’ll automatically expire. The exit at the end just causes the ssh connection you establish to immediately terminate (otherwise you’d have a perfectly normal ssh session going to remotehost).
After creating the control socket, you utilize it with -S /path/to/socket in any normal ssh command to remotehost.
The improvement in execution time for commands using that socket is delicious.
me@banshee:/~$ ssh -M -S /tmp/demo -o ControlPersist=5m eohippus exit me@banshee:/~$ time ssh eohippus exit real 0m0.660s user 0m0.052s sys 0m0.016s me@banshee:/~$ time ssh -S /tmp/demo eohippus exit real 0m0.050s user 0m0.005s sys 0m0.010s
Yep… from 660ms to 50ms. SSH control sockets are pretty awesome.
33 thoughts on “A comprehensive guide to fixing slow SSH logins”
Thank you, this helped me.
I also had this 25 seconds timeout.
This work-around helped me on the server side:
systemctl restart systemd-logind.service
Red Hat 7.6
You are da best!!
Thank you so much! I was banging me head against a wall trying to diagnose this, and stumbled across this post while looking for ideas to troubleshoot it! DNS was the issue for me! Thank you!
Yet another one… DNS for the server has AAAA and A records. Client OS thinks it’s got IPv6 routing but actually it hasn’t – but you have to wait for the IPv6 attempt to fail before it falls back to “legacy” IPv4.
This Really Helped …
All of the solutions were already default on my debian ssh server, but it was still slow for me.
Your last solution was what took me in the right direction, but you can have that automatically done in the ssh client config.
Here’s a post explaining how:
DNS! Its always DNS… Slows everything! Found an imporperly configure server with slow and non existent dns servers. I was tasked into why website performance was so slow… first thing first though, why is ssh so slow…
DNS! It’s always DNS… Slows everything! Found an improperly configured server with slow and non-existent DNS servers. I was tasked into why website performance was so slow… first thing first though, why is ssh so slow…
This was useful. I have linked this page on slow ssh logins to the stack exchange/server fault thread that most searches for slow ssh seem to lead to (https://serverfault.com/questions/792486/ssh-connection-takes-forever-to-initiate-stuck-at-pledge-network/998761#998761).
For my particular case the the optional PAM module was the culprit.
For me, it was just an IPv6 problem.
Forcing the IPv4 of the server in my /etc/hosts file was enough to fix it.
Thank you. I encountered this problem before, but had forgotten the fix.
This time I had to disable both UseDNS *AND* GSSAPIAuthentication (Centos 7.7)
Thank you bro, you helped alot
Thank you so much, installing Centos 8
Beware that the PAM disabling could affect display manger, which could disable gdm3 and lightdm. This is because the /etc/pam.d/common-session is common for multiple services.
Thank you very much. I have a few hundreds of concurrent SSH connection and pam_systemd slowed down the connection.
Another issue (also pam related) is described here:
On archlinux open /etc/pam.d/system-auth and comment out every line with pam_systemd_home.so
Thank you! I’ve been living with the long login time for so long and finally decided I would fix it, you made it so quick!
For me, I’m on Ubuntu 16.04, and commenting the line below in /etc/pam.d/common-session worked like a charm!
#session optional pam_systemd.so
Another additional: DNS had ipv4+ipv6, but ipv6 was not reachable because of firewall rule. My command was: ssh myhost, and this tries first ipv6, after a timeout from 5-10 seconds it tries ipv4.
I checked it with:
ssh -6 myhost (timeout, aha! ipv6 not reachable)
ssh -4 myhost (immediately)
Thanks for this summary! This helped me (it was GSSAPI from Putty causing me problems)
Ubuntu 20.10 latest updates and I was running into slow ssh. Thanks for the article. Here it was the line below I had to comment:
#session optional pam_systemd.so
Very useful post.
The avahi mdns4_minimal enables the domain .local to find hosts (i.e. my_computer.local) on the local network, using avahi lookup. It’s not something too many people do, but is useful for some setups. You can discover computers that you can locate this way using the tool “avahi-discover”
Could also be something in /etc/profile: https://bugzilla.redhat.com/show_bug.cgi?id=1885090
This can also be caused by pam_krb5 (Kerberos) if using a local account to log in. It tries to authenticate to the domain first. Turning off this module in /etc/pam.d/common-auth works, if you’re not using it…
Hans Schou: That was the solution to my problem too, RHEL 7.8.
Thank you for this awesome little article! I recently ran into problems with SSH connections that took about 2½ minutes before the password prompt showed up. In my case what finally worked was to add the option `AddressFamily inet` (to disable IPv6) for the relevant host in the client side config (`~/.ssh/config`).
Since that isn’t covered above I wanted to share my findings. Thanx for an awesome guide!
Thanks for the information! After updating to Debian bullseye I had slow SSH connection.
Seems that #session optional pam_systemd.so was the cruelpit.
This was very helpful, but did not quite solve my problem. The problem turned out to be related to dns. I am using hosts.allow and hosts.deny to limit connections to local networks. I had a LOCAL in the statement for hosts.allow. Removing that fixed things. I’m not sure I fully understand why, but according to the hosts.allow documentation, LOCAL stands for any hostname without a dot. I was doing this on a new router without fully functional dns, and I think that the LOCAL triggered a dns lookup. Either adding a host entry for the client into the servers /etc/hosts or deleting LOCAL worked.
The SSH hanged for about a minute in “debug1: pledge: network”, and PAM was the culprit for me.
I had a huge /var/log/btmp on a Fedora system, which was read by pam_lastlog.so’s showfailed and so lead to a ~20s login time. I fixed it by fixing logrotation of that file.
Sadly, this PAM stuff is rather difficult to debug so this was hard to find.
I bow down to you. Where can I buy (digital) coffee for you or something? 😀
Just got the PAM problem on a newly installed debian 11 CT, thanks !
Thank you, in my case, using Debian 11, it was the parameter : #session optional pam_systemd.so