Heartbleed SSL vulnerability

Last night (2014 Apr 7) a massive security vulnerability was publicly disclosed in OpenSSL, the library that encrypts most of the world’s sensitive traffic. The bug in question is approximately two years old – systems older than 2012 are not vulnerable – and affects the TLS “heartbeat” function, which is why the vulnerability has been nicknamed HeartBleed.

The bug allows a malicious remote user to scan arbitrary 64K chunks of the affected server’s memory. This can disclose any and ALL information in that affected server’s memory, including SSL private keys, usernames and passwords of ANY running service accepting logins, and more. Nobody knows if the vulnerability was known or exploited in the wild prior to its public disclosure last night.

If you are an end user:

You will need to change any passwords you use online unless you are absolutely sure that the servers you used them on were not vulnerable. If you are not a HIGHLY experienced admin or developer, you absolutely should NOT assume that sites and servers you use were not vulnerable. They almost certainly were. If you are a highly experienced ops or dev person… you still absolutely should not assume that, but hey, it’s your rope, do what you want with it.

Note that most sites and servers are not yet patched, meaning that changing your password right now will only expose that password as well. If you have not received any notification directly from the site or server in question, you may try a scanner like the one at http://filippo.io/Heartbleed/ to see if your site/server has been patched. Note that this script is not bulletproof, and in fact it’s less than 24 hours old as of the time of this writing, up on a free site, and under massive load.

The most important thing for end users to understand: You must not, must not, MUST NOT reuse passwords between sites. If you have been using one or two passwords for every site and service you access – your email, forums you post on, Facebook, Twitter, chat, YouTube, whatever – you are now compromised everywhere and will continue to be compromised everywhere until ALL sites are patched. Further, this will by no means be the last time a site is compromised. Criminals can and absolutely DO test compromised credentials from one site on other sites and reuse them elsewhere when they work! You absolutely MUST use different passwords – and I don’t just mean tacking a “2” on the end instead of a “1”, or similar cheats – on different sites if you care at all about your online presence, the money and accounts attached to your online presence, etc.

If you are a sysadmin, ops person, dev, etc:

Any systems, sites, services, or code that you are responsible for needs to be checked for links against OpenSSL versions 1.0.1 through 1.0.1f. Note, that’s the OpenSSL vendor versioning system – your individual distribution, if you are using repo versions like a sane person, may have different numbering schemes. (For example, Ubuntu is vulnerable from 1.0.1-0 through 1.0.1-4ubuntu5.11.)

Examples of affected services: HTTPS, IMAPS, POP3S, SMTPS, OpenVPN. Fabulously enough, for once OpenSSH is not affected, even in versions linking to the affected OpenSSL library, since OpenSSH did not use the Heartbeat function. If you are a developer and are concerned about code that you wrote, the key here is whether your code exposed access to the Heartbeat function of OpenSSL. If it was possible for an attacker to access the TLS heartbeat functionality, your code was vulnerable. If it was absolutely not possible to check an SSL heartbeat through your application, then your application was not vulnerable even if it linked to the vulnerable OpenSSL library.

In contrast, please realize that just because your service passed an automated scanner like the one linked above doesn’t mean it was safe. Most of those scanners do not test services that use STARTTLS instead of being TLS-encrypted from the get-go, but services using STARTTLS are absolutely still affected. Similarly, none of the scanners I’ve seen will test UDP services – but UDP services are affected. In short, if you as a developer don’t absolutely know that you weren’t exposing access to the TLS heartbeat function, then you should assume that your OpenSSL-using application or service was/is exploitable until your libraries are brought up to date.

You need to update all copies of the OpenSSL library to 1.0.1g or later (or your distribution’s equivalent), both dynamically AND statically linked (PS: stop using static links, for exactly things like this!), and restart any affected services. You should also, unfortunately, consider any and all credentials, passwords, certificates, keys, etc. that were used on any vulnerable servers, whether directly related to SSL or not, as compromised and regenerate them. The Heartbleed bug allowed scanning ALL memory on any affected server and thus could be used by a sufficiently skilled user to extract ANY sensitive data held in server RAM. As a trivial example, as of today (2014-Apr-08) users at the Ars Technica forums are logging on as other users using password credentials held in server RAM, as exposed by standard exploit test scripts publicly disclosed.

Completely eradicating all potential vulnerability is a STAGGERING amount of work and will involve a lot of user disruption. When estimating your paranoia level, please do remember that the bug itself has been in the wild since 2012 – the public disclosure was not until 2014-Apr-07, but we have no way of knowing how long private, possibly criminal entities have been aware of and/or exploiting the bug in the wild.

Understanding ctime

ctimes have got to be one of the most widely misunderstood features of POSIX filesystems. It’s very intuitive – and very wrong – to believe that just as mtime is the time the file was last modified and atime is the time the file was last accessed, ctime must be the time the file was created. Nope!

ctime is not the file creation time, it’s the inode creation time. Any time the inode for a file changes, the ctime changes. Which isn’t very intuitive, and happens much more frequent than you might think. In fact, I can’t think of a time that the mtime changes that the ctime won’t change with it… and in a lot of cases, the ctime will update when the mtime doesn’t! There’s a LOT of bad information floating around about this… so let’s examine it directly:

me@box:/tmp$ touch test
SHOWTIMES="%P \t ctime:  %Cb %Cd %CT \t mtime:  %Tb %Td %TT \t atime:  %Ab %Ad %AT \n" ;   export SHOWTIMES
me@box:/tmp$ find . -maxdepth 1 -name test -printf "$SHOWTIMES"
test ctime:  Jul 29 15:53:51.4814403150 mtime:  Jul 29 15:53:51.4814403150 atime:  Jul 29 15:53:51.4814403150
me@box:/tmp$ cat test > /dev/null
me@box:/tmp$ find . -maxdepth 1 -name test -printf "$SHOWTIMES"
test ctime:  Jul 29 15:53:51.4814403150 mtime:  Jul 29 15:53:51.4814403150 atime:  Jul 29 15:54:18.1014495830
me@box:/tmp$ touch test
me@box:/tmp$ find . -maxdepth 1 -name test -printf "$SHOWTIMES"
test ctime:  Jul 29 15:54:25.6522832920 mtime:  Jul 29 15:54:25.6522832920 atime:  Jul 29 15:54:25.6522832920
me@box:/tmp$ chmod 777 test
me@box:/tmp$ find . -maxdepth 1 -name test -printf "$SHOWTIMES"
test ctime:  Jul 29 15:54:32.7214485080 mtime:  Jul 29 15:54:25.6522832920 atime:  Jul 29 15:54:25.6522832920
me@box:/tmp$ mv test test2 ; mv test2 test
me@box:/tmp$ find . -maxdepth 1 -name test -printf "$SHOWTIMES"
test ctime:  Jul 29 15:54:54.6322825980 mtime:  Jul 29 15:54:25.6522832920 atime:  Jul 29 15:54:25.6522832920

OK, from top to bottom:

1. we create a file, and we check its ctime, mtime, and atime. No surprises.
2. we access the file, by cat’ing it to /dev/null. atime updates, ctime and mtime remain the same. No surprises.
3. we modify the file, by touching it. atime, mtime, and ctime update… not what you expected, amirite?
4. we change permissions on the file. ctime updates, but mtime and atime do not. Again… not what you expected, right right?
5. we mv the file a couple times. ctime updates again – mtime and atime still don’t.

This is usually the first answer that should be given to “how do I modify the ctime on a file?” (The second answer is: don’t. [i]By design[/b], there is no feature within a POSIX filesystem to set a ctime to anything other than the current system time, so the only way to do it is either to reset the system time, then mv the file, or to unmount the filesystem entirely and hexedit the metadata with a debugging tool. Neither is a good idea, and wanting to do so in the first place usually stems from a misunderstanding of what ctime is and does.)

SouthEast Linux Fest (SELF) 2012

I had a great time at SELF in Charlotte, NC this weekend. One of the highlights, for me, was for the first time meeting some fellow BSD types in the flesh – Kris Moore (founder of the PC-BSD distribution) and Dru Lavigne (Director of the FreeBSD Foundation), no less. While discussing the pain of doing FreeBSD installs onto RAIDZ, Kris told me about the new graphical installer in PC-BSD that lets you create new RAIDZ arrays of all types and install directly to them, all without ever having to leave the installer, which I found pretty exciting. I found it even more exciting when he told me that the procedures taken by the installer were based partly on my own work at freebsdwiki.net!

I set up a new VM with three virtual disks pretty much the minute I got home, and started a new PC-BSD 9.0 install. Sure enough, although the option is a little hard to discover, I managed to figure it out without having to go in search of any documentation – and without ever leaving the installer, and with a bare minimum of blood and chicken feathers, I got a brand new RAIDZ1 across my three virtual disks set up, and PC-BSD cheerfully installed onto it. (This is testing only, of course – in production, you should only do RAIDZ onto bare metal, not onto an abstraction like linux logical volumes or raw files accessed through a hypervisor.) Pretty heady stuff!

To the right – Tux dropped by the table while Dru and Kris and I were chatting, and posed for me with BSD’s horns.  How great is that?

Storing PHP sessions in memcached instead of in files

in php.ini:

session.save_handler = memcache
session.save_path = "tcp://serv01:11211,tcp://serv02:11211,tcp://serv03:11211"

Obviously, you need php5-memcache installed, replace “serv1” “serv2” and “serv3” with valid server address(es), and you’ll need to restart your Apache server after making the change.

Why would you need to do this? Well, this week I had to deal with a web application server pool that kept slowly increasing its number of children all the way up to MaxCli, no matter what. It was unpredictable, other than being a slow creep. Eventually, stracing the pids showed that they were getting stuck in FLOCK on files in /var/lib/php5/php_sess*. This turns out to be an endemic problem with PHP that the devs don’t seem inclined to fix: php’s garbage collector will delete session files dirtily if a php process (which in the case of mod_php, means an Apache process) violates any of the php limits, such as max_execution_time (among many, many others). So you end up with your php script trying to lock a session file (file descriptor 3) that php’s garbage collector already deleted, and therefore an infinitely hung process that will never go away on its own.

Changing over to using memcache to store php sessions eliminated this file lock issue and resulted in a much more stable situation – a server pool that had been creeping up to 800 children per server over the course of a couple hours has been running stable and sweet on less than 150 children per server for days now.