Heads up—Let’s Encrypt and Dovecot

Let’s Encrypt certificates work just dandy not only for HTTPS, but also for SSL/TLS on IMAP and SMTP services in mailservers. I deployed Let’s Encrypt to replace manually-purchased-and-deployed certificates on a client server in 2019, and today, users started reporting they were getting certificate expiration errors in mail clients.

When I checked the server using TLS checking tools, they reported that the certificate was fine; both the tools and a manual check of the datestamp on the actual .pem file showed that it had been updating just fine, with the most recent update happening in January and extending the certificate validation until April. WTF?

As it turns out, the problem is that Dovecot—which handles IMAP duties on the server—doesn’t notice when the certificate has been updated on disk; it will cheerfully keep using an in-memory cached copy of whatever certificate was present when the service started until time immemorial.

The way to detect this was to use openssl on the command line to connect directly to the IMAPS port:

you@anybox:~$ openssl s_client -showcerts -connect mail.example.com:993 -servername example.com

Scrolling through the connect data produced this gem:

---
Server certificate
subject=CN = mail.example.com

issuer=C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA
Server Temp Key: ECDH, P-384, 384 bits
---
SSL handshake has read 3270 bytes and written 478 bytes
Verification error: certificate has expired

So obviously, the Dovecot service hadn’t reloaded the certificate after Certbot-auto renewed it. One /etc/init.d/dovecot restart later, running the same command instead produced (among all the other verbiage):

---
Server certificate
subject=CN = mail.example.com

issuer=C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA
Server Temp Key: ECDH, P-384, 384 bits
---
SSL handshake has read 3269 bytes and written 478 bytes
Verification: OK
---

With the immediate problem resolved, the next step was to make sure Dovecot gets automatically restarted frequently enough to pick new certs up before they expire. You could get fancy and modify certbot’s cron job to include a Dovecot restart; you can find certbot’s cron job with grep -ir certbot /etc/crontab and add a –deploy-hook argument to restart after new certificates are obtained (and only after new certificates are obtained).

But I don’t really recommend doing it that way; the cron job might get automatically updated with an upgraded version of certbot at some point in the future. Instead, I created a new root cron job to restart Dovecot once every Sunday at midnight:

# m h dom mon dow   command
0 0 * * Sun /etc/init.d/dovecot restart

Since Certbot renews any certificate with 30 days or less until expiration, and the Sunday restart will pick up new certificates within 7 days of their deployment, we should be fine with this simple brute-force approach rather than a more efficient—but also more fragile—approach tying the update directly to restarting Dovecot using the –deploy-hook argument.

 

Published by

Jim Salter

Mercenary sysadmin, open source advocate, and frotzer of the jim-jam.

4 thoughts on “Heads up—Let’s Encrypt and Dovecot”

  1. “You could get fancy and modify certbot’s cron job to include a Dovecot restart; you can find certbot’s cron job with grep -ir certbot /etc/crontab and add a –deploy-hook argument to restart after new certificates are obtained (and only after new certificates are obtained).”

    FWIW, on systemd-based systems this will *not* work; certbot uses a systemd timer instead of a cron job on such systems. It still *installs* the cronjob though, the systemd timer just overrides it. This can lead to a frustrating half hour or so of staring at the screen wondering why the $?#*! cerbot is ignoring your updates to the cronjob.

    Incidentally, this also provides a way to solve the “cron job might get updated with an upgraded version of the cronjob” problem: you can create an override in (IIRC) /etc/systemd/system/certbot.service.d/override.conf and then specify only those directives that you want to override.

    Admittedly if they ever change the certbot API (which they have a habit of doing) this can still break, so you’re probably still better off with the brute-force weekly-restart option.

  2. What I usually do is I drop script to /etc/letsencrypt/renewal-hooks/post which handles service reload (be it nginx, dovecot, postfix, whatever).

    In situations where there are multiple certificates and not all require restart of the same service, then you can specify post_hook in /etc/letsencrypt/renewal configuration files.

  3. I’m so happy I found this, I’ve been dealing with it for almost a year. I was frustrated by the fact that all the SSL ‘checkers’ said everything was perfect and yet mail clients were rejecting the cert. I-Phones were particularly nasty as they pop up a message box about 5000 times a day. Okay, maybe not that much except to our I-Phone users.

    What seems like an easy working fix is we set up a cron job for the dovecot restart about 10 minutes after the certbot renew. Works fine.

  4. I use OpenBSD’s acme-client instead of certbot. It exits with 0 if and only if a certificate was renewed. This makes it trivial to perform actions after a cert renewal:

    acme-client example.com && nginx -s reload

    You can find the portable version of acme-client for non-OpenBSD systems by searching for graywolf/acme-client-portable.

Leave a Reply

Your email address will not be published. Required fields are marked *