Could not complete SSL handshake, Socket timeout after 10 seconds with Nagios and NSClient++ 0.4x

Adding a couple of new Windows hosts to my monitoring network this morning, my NRPE plugin checks against them were failing.

me@nagios:~$ /usr/lib/nagios/plugins/check_nrpe -H monitoredwindowsserver.vpn
CHECK_NRPE: Error - could not complete SSL handshake.

The usual culprits – password set but not being used, or hosts_allowed not including the host doing the checking – were already set correctly.

Turns out the NSClient++ folks changed up the default configs. In order to get it working again with a relatively vanilla Nagios server on the other end, I needed to set two new directives under [/settings/NRPE/server]:

verify mode = none
insecure = true

With those directives set and a restart to the nsclient service on the Windows end, manual tests to NRPE worked properly:

me@nagios:~$ /usr/lib/nagios/plugins/check_nrpe -H monitoredwindowsserver.vpn
I (0.4.4.19 2015-12-08) seem to be doing fine...

One of these days I should figure out how to get the default modes, which use peer-to-peer certificate checks, working… but for the moment, I’m only allowing traffic over a VPN tunnel anyway, so it was more important to get it working in its existing (secured by VPN) configuration than to blow a few hours untangling the new defaults.

Wasn’t out of the woods yet, though – that got NRPE working, but not NSClientServer, which is what I’m actually using to monitor these Windows hosts for the most part. So I was still seeing “CRITICAL – Socket timeout after 10 seconds” on a lot of tests against the new hosts in Nagios. Doing a netstat -an on the Windows hosts themselves showed that they were listening on 5666 – the NRPE port – but nothing was listening on 12489, the NSClientServer port. This required another fix in the nsclient.ini file. Just underneath [/modules], you’ll need to add (not just uncomment!) this line:

NSClientServer = enabled

And restart the NSClient++ service again, either from Services applet or with net stop nscp ; net start nscp. Now we test the check_nt plugin against the host…

me@nagios:~$ /usr/lib/nagios/plugins/check_nt -H monitoredwindowsserver.vpn -v UPTIME -p 12489
System Uptime - 0 day(s) 1 hour(s) 34 minute(s)

Now, finally, all of my tests are working. Hope this saves somebody else from having the kind of morning I had!

libguestfs0 and ZFS on Linux in Ubuntu

Trying to get Kimchi installed this morning, I ran into a roadblock almost immediately: libguestfs-tools depends on libguestfs0, which, on Ubuntu at least, stupidly has a hard dependency on zfs-fuse. Which is a dead project, and which conflicts with zfsutils.

In the real world, you might want libguestfs-tools without ever wanting the first thing to do with ANY form of zfs, so this dependency is a really bad idea. Even if you DO want to use libguestfs-tools WITH zfs, it’s an incredibly bad idea because zfsutils – part of ZFS on Linux – provides all the functionality needed already. Unfortunately, the package maintainers don’t seem to quite understand the issues here – I’m guessing none of them are ZFS people – so that leaves you with the need to edit the dependencies yourself.

Luckily, that’s not too hard. First, you’ll need a script, which we’ll call debedit:

#!/bin/bash

EDITOR=nano

if [[ -z "$1" ]]; then
  echo "Syntax: $0 debfile"
  exit 1
fi

DEBFILE="$1"
TMPDIR=`mktemp -d /tmp/deb.XXXXXXXXXX` || exit 1
OUTPUT=`basename "$DEBFILE" .deb`.modified.deb

if [[ -e "$OUTPUT" ]]; then
  echo "$OUTPUT exists."
  rm -r "$TMPDIR"
  exit 1
fi

dpkg-deb -x "$DEBFILE" "$TMPDIR"
dpkg-deb --control "$DEBFILE" "$TMPDIR"/DEBIAN

if [[ ! -e "$TMPDIR"/DEBIAN/control ]]; then
  echo DEBIAN/control not found.

  rm -r "$TMPDIR"
  exit 1
fi

CONTROL="$TMPDIR"/DEBIAN/control

MOD=`stat -c "%y" "$CONTROL"`
$EDITOR "$CONTROL"

if [[ "$MOD" == `stat -c "%y" "$CONTROL"` ]]; then
  echo Not modfied.
else
  echo Building new deb...
  dpkg -b "$TMPDIR" "$OUTPUT"
fi

rm -r "$TMPDIR"

Save that, name it debedit, and chmod 755 it.

Now, you’ll need to download libguestfs0, which is the package that has the bad dependencies, which you’ll edit:

you@box:~$ apt-get download libguestfs0
you@box:~$ ./debedit libguest*deb

Remove the zfs-fuse dependency from the Depends: line in the deb file, and exit nano. Finally, install your modified libguestfs0 package:

you@box:~$ sudo dpkg -i *modified.deb ; sudo apt-get -f install

All done! At least, until and unless the next update to libguestfs0 downloads and attempts to install a new .deb that wants to put that dependency right back again, in which case you’ll need to lather-rinse-repeat.

I me-too’ed an existing bug at https://bugs.launchpad.net/ubuntu/+source/libguestfs/+bug/1053911 ; if you’re affected, you probably should too.

The SSLv3 “POODLE” attack in a (large) nutshell

A summary of the POODLE sslv3 vulnerability and attack:

A vulnerability has been discovered in a decrepit-but-still-widely-supported version of SSL, SSLv3, which allows an attacker a good chance at determining the true value of a single byte of encrypted traffic. This is of limited use in most applications, but in HTTPS (eg your web browser, many mobile applications, etc) an attacker in an MITM (Man-In-The-Middle) position, such as someone operating a wireless router you connect to, can capture and resend the traffic repeatedly until they manage to get a valuable chunk of it assembled in the clear. (This is done by manipulating cleartext traffic, to the same or any other site, injecting some Javascript in that traffic to get your browser to run it. The rogue JS function is what reloads the secure site, offscreen where you can’t see it happening, until the attacker gets what s/he needs out of it.)

That “valuable chunk” is the cookie that validates your user login on whatever secure website you happen to be browsing – your bank, webmail, ebay or amazon account, etc. By replaying that cookie, the attacker can now hijack your logged in session directly on his/her own device, and from there can do anything that you would be able to do – make purchases, transfer funds, change the password, change the associated email account, et cetera.

It reportedly takes 60 seconds or less for an attacker in a MITM position (again, typically someone in control of a router your traffic is being directed through, which is most often going to be a wireless router – maybe even one you don’t realize you’ve connected to) to replay traffic enough to capture the cookie using this attack.

Worth noting: SSLv3 is hopelessly obsolete, but it’s still widely supported in part because IE6/Windows XP need it, and so many large enterprises STILL are using IE6. Many sites and servers have proactively disabled SSLv3 for quite some time already, and for those, you’re fine. However, many large sites still have not – a particularly egregious example being Citibank, to whom you can still connect with SSLv3 today. As long as both your client application (web browser) and the remote site (web server) both support SSLv3, a MITM can force a downgrade dance, telling each side that the OTHER side only supports SSLv3, forcing that protocol even though it’s strongly deprecated.

I’m an end user – what do I do?

Disable SSLv3 in your browser. If you use IE, there’s a checkbox in Internet Options you can uncheck to remove SSLv3 support. If you use Firefox, there’s a plugin for that. If you use Chrome, you can start Chrome with a command-line option that disables SSLv3 for now, but that’s kind of a crappy “fix”, since you’d have to make sure to start Chrome either from the command line or from a particular shortcut every time (and, for example, clicking a link in an email that started up a new Chrome instance would fail to do so).

Instructions, with screenshots, are available at https://zmap.io/sslv3/ and I won’t try to recreate them here; they did a great job.

I will note specifically here that there’s a fix for Chrome users on Ubuntu that does fairly trivially mitigate even use-cases like clicking a link in an email with the browser not already open:


* Open /usr/share/applications/google-chrome.desktop in a text editor
* For any line that begins with "Exec", add the argument --ssl-version-min=tls1
* For instance the line "Exec=/usr/bin/google-chrome-stable %U" should become "Exec=/usr/bin/google-chrome-stable --ssl-version-min=tls1 %U

You can test to see if your fix for a given browser worked by visiting https://zmap.io/sslv3/ again afterwards – there’s a banner at the top of the page which will warn you if you’re vulnerable. WARNING, caching is enabled on that page, meaning you will have to force-refresh to make certain that you aren’t seeing the old cached version with the banner intact – on most systems, pressing ctrl-F5 in your browser while on the page will do the trick.

I’m a sysadmin – what do I do?

Disable SSLv3 support in any SSL-enabled service you run – Apache, nginx, postfix, dovecot, etc. Worth noting – there is currently no known way to usefully exploit the POODLE vulnerability with IMAPS or SMTPS or any other arbitrary SSL-wrapped protocol; currently HTTPS is the only known protocol that allows you to manipulate traffic in a useful enough way. I would not advise banking on that, though. Disable this puppy wherever possible.

The simplest way to test if a service is vulnerable (at least, from a real computer – Windows-only admins will need to do some more digging):

openssl s_client -connect mail.jrs-s.net:443 -ssl3

The above snippet would check my mailserver. The correct (sslv3 not available) response begins with a couple of error lines:

CONNECTED(00000003)
140301802776224:error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure:s3_pkt.c:1260:SSL alert number 40
140301802776224:error:1409E0E5:SSL routines:SSL3_WRITE_BYTES:ssl handshake failure:s3_pkt.c:596:

What you DON’T want to see is a return with a certificate chain in it:

CONNECTED(00000003)
depth=1 C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = PositiveSSL CA 2
verify error:num=20:unable to get local issuer certificate
verify return:0
---
Certificate chain
0 s:/OU=Domain Control Validated/OU=PositiveSSL/CN=mail.jrs-s.net
i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=PositiveSSL CA 2
1 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=PositiveSSL CA 2
i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root

On Apache on Ubuntu, you can edit /etc/apache2/mods-available/ssl.conf and find the SSLProtocol line and change it to the following:

SSLProtocol all -SSLv2 -SSLv3

Then restart Apache with /etc/init.d/apache2 restart, and you’re golden.

I haven’t had time to research Postfix or Dovecot yet, which are my other two big concerns (even though they theoretically shouldn’t be vulnerable since there’s no way for the attacker to manipulate SMTPS or IMAPS clients into replaying traffic repeatedly).

Possibly also worth noting – I can’t think of any way for an attacker to exploit POODLE without access to web traffic running both in the clear and in a Javascript-enabled browser, so if you wanted to disable Javascript completely (which is pretty useless since it would break the vast majority of the web) or if you’re using a command-line tool like wget for something, it should be safe.

Opening up SQL server in the Windows Server 2008 firewall

@echo ========= SQL Server Ports ===================
@echo Enabling SQLServer default instance port 1433
netsh firewall set portopening TCP 1433 “SQLServer”
@echo Enabling Dedicated Admin Connection port 1434
netsh firewall set portopening TCP 1434 “SQL Admin Connection”
@echo Enabling conventional SQL Server Service Broker port 4022
netsh firewall set portopening TCP 4022 “SQL Service Broker”
@echo Enabling Transact-SQL Debugger/RPC port 135
netsh firewall set portopening TCP 135 “SQL Debugger/RPC”
@echo ========= Analysis Services Ports ==============
@echo Enabling SSAS Default Instance port 2383
netsh firewall set portopening TCP 2383 “Analysis Services”
@echo Enabling SQL Server Browser Service port 2382
netsh firewall set portopening TCP 2382 “SQL Browser”
@echo ========= Misc Applications ==============
@echo Enabling HTTP port 80
netsh firewall set portopening TCP 80 “HTTP”
@echo Enabling SSL port 443
netsh firewall set portopening TCP 443 “SSL”
@echo Enabling port for SQL Server Browser Service’s ‘Browse’ Button
netsh firewall set portopening UDP 1434 “SQL Browser”
@echo Allowing multicast broadcast response on UDP (Browser Service Enumerations OK)
netsh firewall set multicastbroadcastresponse ENABLE

Review: ASUS “Transformer” Tablet

ASUS Transformer (tablet only)
Tablet only - 1.6 lbs

ASUS has entered the Android tablet market with a compelling new contender – the Eee TF-101 “Transformer.”  Featuring an Nvidia Tegra dual-core CPU at 1.0GHz, the device feels “snappier” than most relatively high-end desktop PCs – nothing lags; when you open an app, it pops onto the screen smartly.  If you’re accustomed to browsing on smartphones, you’ll feel the performance difference immediately – even notoriously heavy pages like CNN or ESPN render as quickly as they would if you were using a high-end desktop computer.

I first got my hands on a TF101 that one of my clients had purchased, sans docking station.  After I’d played with it for a few minutes, I knew I wanted one, but that left the $150 question – what will it be like when it’s docked?  The answer is “there’s a lot of potential here” – but there are problems to be worked through before you can give it an unqualified “hey, awesome!

CNET complains that the TF101 feels cheap, with poorly-rounded corners and a flimsy backplate.  After a week or so of ownership and something like 20 hours of active use, I do not agree on either issue.  I find the tablet nicely balanced, easy to grip, and solid feeling.  It weighs in at 1.6 pounds (tablet only) and 2.9 lbs (tablet and docking station), which puts it pretty much dead center in standard weight for both tablets and netbooks.  However, while the weight of the docked TF101 is an ounce heavier than the weight of my Dell Mini 10v, the TF101 feels much less cumbersome – probably because even though it’s slightly heavier, it’s much, much slimmer.

Docking the tablet feels easy and intuitive; line up the edges of the tablet with the edges of the docking station, and you’re in the right position for the sockets to mate. Pressing down first gently, then firmly produces good tactile feedback for whether it’s lined up properly, and whether it’s “clicked” all the way in. The hinge itself is very solid and doesn’t feel “loose” or sloppy at all – in fact, it’s stiff enough that most people would have trouble moving it at all without the tablet already inserted. Undocking the tablet is easy; there’s a release toggle that slides to the left (marked with an arrow POINTING to the left, which is a nice touch); the release toggle also has a solid, not-too-sloppy but not-too-stiff action.

The docking station offers more than just the keyboard.  There are also two USB ports (a convenience which is missing on the tablet itself), a full-size SD card slot, and an internal battery pack, roughly comparable to the battery in the tablet itself.  The extra battery life is a great feature; the tablet itself gets 9 hours or so of fully active use, and the docking station roughly doubles that.  In practical terms, most people will be able to go away for a long weekend with a fully-charged TF101 sans charger, use it for 4 hours a day without ever bothering to turn it off, and come home with a significant fraction of the battery left – especially if they’ve taken the time to set the “disconnect from wireless when screen is off” option in the Power settings. I used the docked tablet 2 to 4 hours a day for a full seven-day week, playing games, emailing, and browsing; at the end of the week I was at 15% charge remaining.

ASUS Transformer - with docking station
With docking station - 2.9 lbs

Polaris Office, the office suite shipped with the TF101, was a pleasant surprise – a client asked if I could display PowerPoint presentations on the tablet, and the answer turned out to be “yes, I certainly can.” I’ve only tried a few of them, none of which had any particularly fancy animations; but 40MB slideshows load and display just fine. Paired with an HDMI projector, the TF101 should make a pretty solid little presentation device, particularly since it feels just as “fast” running slideshows as it does browsing and playing games from the Market.

Moving on to the docking station itself, I quite liked the way the touchpad is integrated into the Android OS – instead of an arrow cursor, you get a translucent “bubble” roughly the size of a fingertip press, which felt much more intuitive to me. With the arrow, I tend to try to be just as precise as I would with a mouse – which can be frustrating. The “fingertip bubble” made it easier for me to relax and just “get what you want inside the circle” without trying to be overly finicky. Sensitivity for both tracking and tapping was also very good; the touchpad feels slick and responsive to use.

The keyboard, unfortunately, is a mixed bag – it’s better-suited to large hands than many netbook keyboards, but you won’t ever mistake it for the full-size keyboard on your desk.  The dimensions are almost exactly the same as the keyboard on my Dell Mini 10v; but I find that it feels significantly more cramped and awkward – probably because ASUS elected to go the trendy new route of “raised keys with space between them”, where the Mini’s keys are literally edge-to-edge with one another.  This should make the TF101 less likely to collect crumbs, skin flakes, and other kinds of “yuck” than the Mini 10v, but I personally would rather deal with more cleaning than less roominess.

Several applications don’t really play well with the keyboard – ConnectBot, which I use as an SSH client to operate remote servers, becomes completely unusable due to handling the shift key wrong – you can’t type anything from ! through + without resorting to re-enabling the onscreen keyboard.  In the Android Browser, typing URLs in the address bar works fine, but if you do any significant amount of typing in a form – for example, writing this post in the TinyMCE control WordPress uses – the up and down arrow keys frequently map to the wrong thing.  Sometimes up/down arrow would scroll through the text I was typing, sometimes they would tab me to different controls on the page, and some OTHER times they would simply scroll the entire page up and down.

In Polaris Office (the office suite shipped with the TF101), the keyboard itself worked perfectly – but the touchpad was too sensitive and placed too closely. It was difficult to type more than one sentence at a time without the heel of my hand brushing the touchpad and registering as a “tap”, causing the last half of a sentence to appear in the midst of the sentence before it.

Can these problems be mitigated?  Probably.  The ConnectBot issue was solved pretty simply by Googling “connectbot transformer”, which immediately leads you to a Transformer-specific fork of ConnectBot – after uninstalling the original ConnectBot, temporarily enabling off-Market app installation, and downloading and installing the fork directly from GitHub, my shift-key problems there were solved.  Presumably either Google or ASUS will eventually deal with the arrow key behavior in the Android Browser.  I tried using the Dolphin HD Browser in the meantime, but had no better luck with it – it is at least consistent in how it handles arrow key usage, but unfortunately it’s consistently wrong – it always scrolls the entire page up and down when you press up or down arrow keys, no matter where the focus on the page is.   Finally, you can toggle the touchpad completely on or off by using a function button at the top of the keyboard – but it would be nice to simply change the sensitivity instead, or automatically disable it for half a second or so after keypresses, the way you can on a traditional (non-Android) netbook.

In the end, though, you can’t really fix all the problems by yourself with “tweaking”; some of the frustrations with the poor integration of the physical keyboard into the Android environment are going to keep ambushing you until Google itself addresses them. ASUS and individual app developers can and likely will continue working to mitigate these issues, but it will be a never-ending game of whack-a-mole until Android itself takes adapting to the “netbook” environment more seriously.

Final verdict: The tablet looks, feels, and performs incredibly well; in most cases it “feels faster” than even high-end desktop computers. Even though my Atom-powered Dell Mini 10v has a Crucial C300 SSD (Solid State Drive), the TF101 spanks it thoroughly in pretty much every performance category possible and sends it home crying. Battery life is also phenomenal, at 9-ish active hours undocked or 18-ish hours docked. It looks and feels, on first blush, like it would make a truly incredible netbook when docked – but Android 3.2 and its apps clearly haven’t come to the party well-prepared for a physical keyboard – and it shows, which knocks the initial blush well off the device as a netbook competitor. If you really need physical keyboard and conventional data entry, this is probably not going to be the device for you – at least, not until the rest of the OS and its apps evolve to support it better.

If you want a tablet, I can recommend the TF-101 without reservation.  If you want a netbook, though, you should probably give the TF-101 a pass unless and until Google starts taking the idea of “Android Netbooks” seriously.

B&N Nook Color

So, I finally got an e-reader today. After getting my wife a Nook Color for her birthday, I found it intriguing enough to take the plunge and get my own. I still wasn’t sure I would really be into it, but the only way to find out for sure was to go ahead and take the plunge.

So far, so … well, OK. Some things I really like, others annoy me a lot. The color touchscreen is WORLDS better, for me, than the “e-ink” more typically found in e-readers. The “PC application” is Windows-only… but it does run fine, so far, under WINE in Linux, so there’s that. Battery life seems pretty sweet so far.

One thing that bothers me – the “lending” feature, which was something I heartily approved of, so far seems to require that you link the Nook to your Facebook account… and give it permission to post on your wall. NOT COOL, B&N. I am really, really not okay with applications which can pretend to be you by posting things as though they were you, ever, from pretty much anybody. And to be honest – I am looking at you, Mark Zuckerberg – the fact that this is even an option with Facebook apps drives me insane. There should never be a legitimate case for an application making a post as a human being without that human’s express consent, expressed beforehand, for that particular post. Anyway. Back to the actual device:

The feel of the device in my hands – which was a really big concern for me – is pretty nice so far. Part of how nice it is to hold is the leather “book” cover I got for it, which I am frankly kind of in love with – it’s glossy, nice-smelling black leather, with reverse-embossed classical authors’ names in big all caps serif text all over. I wasn’t sure when I went into B&N today whether I would get the Nook or not – I was really leaning more towards a Samsung Galaxy android tablet. I’m still not sure if I would have actually taken the plunge, without that cover sitting there all seductive-like. Having seen it though… had to have it.

My biggest gripe so far is the interface of the shop. The Nook store is frankly AWFUL – it’s almost impossible to navigate effectively. If you just want to buy whatever is selling well, you’re in luck, and you’ll be very happy. If you have more specific tastes… prepare for some pain. You can search for author name or book title, which is great if you know EXACTLY what you want – and by “great” I mean “OK”, because all you have is a simple, single-level search with no sorting or grouping. Better hope your favorite author has an unusual name, because you can’t limit searches by genre; for example, searching for “David Drake” got me both the military sci-fi author and some young gay dude who wrote a tell-all book. The lack of sorting or grouping is even worse; should you actually find the author you’re looking for, you can expect to find a complete mish-mash of crap: in a series of novels you’ll likely see #5 first, followed by three unrelated books, followed by #7, followed by more unrelateds, followed by #2… you get the idea.

You are also ridiculously likely to see the SAME book multiple times, with a different cover image. It’s even worse in the “free books” section – some dude wrote his own Star Wars book and it’s listed, I kid you not, AT LEAST ten different times. Which wouldn’t be so bad if it was SORTED or grouped in any way, but… did I mention that you can’t sort, or group, and your searches are single-level simple searches only?

Still, so far I’m enjoying the experience of actually *reading* on the device, and with any luck eventually B&N will sort out their godawful navigation issues on the store.

10 PRINT “HELLO WORLD” : GOTO 10

Today I was telling my friend Chris about setting up Xrdp on Ubuntu Linux, and he said “you know, you really ought to write a blog for all the business stuff you do.” At first, it seemed redundant – I’ve been running technical wiki sites for years now – but after I thought about it for a while, it struck me as a really good idea. Wikis work well as a repository of knowledge, when you already know what you’re looking for and where to look, but introducing new ideas isn’t one of the format’s strengths.

So what will you see here? Day-to-day problems and solutions, covering most of the major platforms, with an emphasis on the needs you run across servicing power users and small-to-medium businesses.

Thanks for stopping by!