Windows Server 2012 / Windows 8 activation boondoggle

On my VERY FIRST activation of Windows Server 2012 Standard today, I got the incredibly unhelpful error message “The filename, directory name or volume label syntax is incorrect.”

My first reaction, of course, was “PC Load Letter?!”

My second was to google the error. Unfortunately, but probably not all that surprisingly, INCREDIBLE amounts of weird issues that have nothing to do with each other can spring this error on you. Eventually, I found the one that was actually related to activation of Windows 8, which is the same issue that Server 2012 has. The problem is that MS has configured Windows 8 and Server 2012 by default to look for a Key Management Server… which it isn’t going to find, if you aren’t in an enterprise that maintains a KMS. And it’s not bright enough to fallback to just asking you for an old-style MAK product key, which is almost certainly what you have.

The fix is to go to the command prompt. Which can ALSO be confusing… what you do is, hit the Windows key and type cmd (there’s no visible search box UNTIL you start typing. But just start typing. Yay, Metro interface.) then press enter.  This brings up the familiar old “dos” console.  From there, you want to type in:

slmgr.vbs /ipk  XXXXX-XXXXX-XXXXX-XXXXX-XXXXX

… use your actual product key instead of all those Xs, of course.  You should very quickly get a popup Windows Script Host window that says “Installed Product Key XXXXX-XXXXX-XXXXX-XXXXX-XXXXX successfully”.  Now when you go back to activate your Server 2012 or Windows 8 installation, it will activate successfully.

Getting SPICE working in Ubuntu 12.04.1-LTS

Well, I finally got Spice working with KVM in Ubuntu 12.04.1 LTS today. It… was not pleasant. But it can be done (and without any PPAs, with some ugly hackery).

THESE STEPS ARE NECESSARY AS OF OCTOBER 2012. THEY MAY NOT BE BY THE TIME YOU READ THIS. IF YOU’RE FROM THE FUTURE, DO NOT HACK YOUR APPARMOR PROFILE OR MOVE SYSTEM BINARIES AROUND UNTIL AFTER YOU HAVE CONFIRMED THAT SPICE DOESN’T WORK WITHOUT YOU DOING THESE THINGS.

First, there’s a problem in the AppArmor profile for KVM that prevents it getting read access to /proc/*/auxv, which is necessary for QXL/Spice stuff to work. So let’s fix that: nano /etc/apparmor.d/abstractions/libvirt-qemu, find the line @{PROC}/*/status r, and add the following line:

  @{PROC}/*/auxv r,

NOTE THE TRAILING COMMA – it’s not a bug, it’s a (necessary) feature! Now sudo /etc/init.d/apparmor restart. This fixes the apparmor problem.

Now, let’s add all the packages you’ll need (assuming you’re already successfully running KVM, and just want to add Spice support):

sudo apt-get update ; sudo apt-get install qemu-kvm-spice python-spice-client-gtk

Another bit of REALLY dirty hackery is necessary here – you’ll need to replace your /usr/bin/kvm with the spice-enabled binary you just installed, which nothing in your system knows how to find. Sigh.

sudo mv /usr/bin/kvm /usr/bin/kvm.dist ; sudo ln -s /usr/bin/qemu-system-x86_64 /usr/bin/kvm

Alright, now – assuming you’re running a Windows guest – you’ll need to boot into that guest, download the Windows spice-guest-tools from http://spice-space.org/download.html, and install them. THINGS GET REALLY, REALLY UGLY HERE: YOU NEED TO MAKE VERY SURE YOU HAVE RDP (or some other client side remote control tool) INSTALLED AND WORKING BEFORE GOING ANY FURTHER. Seriously, TEST. IT. NOW.

Okay, tested? You can RDP or otherwise get into your Windows guest WITHOUT use of virt-manager, or the VNC channel virt-manager provides? Good. Shut down your Windows guest, and change the Display to Spice (say yes to “install Spice channels”) and the Video to QXL. Reboot your guest. YOU WILL GET NO VIDEO, EITHER FOR BIOS OR FOR THE GUEST (reference bug https://bugs.launchpad.net/ubuntu/+source/seabios/+bug/958549 ). Remote into your guest using RDP or other client-side remote tool as mandated in last step. Open Device Manager, find “Standard VGA Adapter”, right-click and select “Update Driver Software…” Choose “Search automatically for updated driver software” and it will find the “RedHat QXL Driver” and install it, after which it will need a reboot.

After rebooting, virt-manager will now successfully connect to the guest using Spice, and you will have a nice wicked-cool uber-accelerated control window, with working sound, to your Windows guest. HOWEVER cut and paste, unfortunately, still won’t work. You also won’t have any display during the SeaBIOS boot sequence or the guest’s boot sequence – video won’t start happening until you get all the way to the login screen for Win7.

—–

From the “don’t blame me” dept: if you don’t follow these instructions exactly, it can get REALLY annoying – in particular, if you have Display VNC set while you have Video QXL *and the spice-guest-tools installed on your guest*, kvm will generate crash after crash, REPEATEDLY, and manage to expose a bug in apport-gtk as well so that you’re constantly asked to submit problem reports but then informed that the problem reports are corrupt (incorrect padding) and therefore you can neither see them nor send them. Once you get tired of that, force your guest shut in virt-manager and change its video to QXL and you should be fine. I learned this one the hard way while figuring this crap out today.

Cloning an OSX disk using Linux, dd, and parted

Tonight I needed to upgrade a customer’s Macbook. She had a 160GB HDD, and wanted to upgrade to a 500GB HDD. No problem, right? Slap the old drive and the new one into an Ubuntu Linux workstation, dd if=/dev/sdc bs=4M conv=sync,noerror | pv -s 160G | dd of=/dev/sdd bs=4M and, 45 minutes or so later, we’re good to go.

Well, not quite. It turns out that, unlike Windows, Linux, or FreeBSD, OSX doesn’t really know how to deal with a partition TABLE that says it’s smaller than the physical disk is – so the cloned larger drive boots, and you can get into Disk Utility just fine, and it shows the drive as being 500GB… but when you try to expand the partition to use the new space, you get the error MediaKit reports partition (map) too small. Googling this error message leads you to some pretty alarming instructions for booting into an OSX install DVD, hitting the Terminal, and manually deleting and C-A-R-E-F-U-L-L-Y custom recreating the gpt partition table. Scary.

But no fear – you were using Linux in the first place, remember? Turns out, Linux’s parted command will automatically fix the gpt table. So after cloning the disk with dd, all you need to do is this:

root@box:/# parted /dev/sdd
GNU Parted 2.3
Using /dev/sdd

Welcome to GNU Parted! Type ‘help’ to view a list of commands.
(parted) print
Error: The backup GPT table is not at the end of the disk, as it should be. This might mean that another operating system believes the disk is smaller.
Fix, by moving the backup to the end (and removing the old backup)?
Fix/Ignore/Cancel? Fix
Warning: Not all of the space available to /dev/sdd appears to be used, you can fix the GPT to use all of the space (an extra 664191360 blocks) or
continue with the current setting?
Fix/Ignore? Fix
Model: ATA WDC WD5000BPKT-7 (scsi)
Disk /dev/sdd: 500GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt

Number Start End Size File system Name Flags
1 20.5kB 210MB 210MB fat32 EFI system partition boot
2 210MB 160GB 160GB hfs+ Customer

(parted) quit
root@box:/#

All fixed. NOW, when you boot that disk into OSX, go into Disk Utility and resize the partition, it will Just Work.

… unless, of course, you’ve got errors in the filesystem, which my customer did. To fix these, I needed to boot to single-user mode in the Mac by HOLDING DOWN command and S while the system booted (holding command down and pressing S rapidly won’t do the trick!) to get into single user mode, then I could fsck the drive with fsck -fy. After THAT, I could resize the partition normally from Disk Utility. 🙂

Understanding ctime

ctimes have got to be one of the most widely misunderstood features of POSIX filesystems. It’s very intuitive – and very wrong – to believe that just as mtime is the time the file was last modified and atime is the time the file was last accessed, ctime must be the time the file was created. Nope!

ctime is not the file creation time, it’s the inode creation time. Any time the inode for a file changes, the ctime changes. Which isn’t very intuitive, and happens much more frequent than you might think. In fact, I can’t think of a time that the mtime changes that the ctime won’t change with it… and in a lot of cases, the ctime will update when the mtime doesn’t! There’s a LOT of bad information floating around about this… so let’s examine it directly:

me@box:/tmp$ touch test
SHOWTIMES="%P \t ctime:  %Cb %Cd %CT \t mtime:  %Tb %Td %TT \t atime:  %Ab %Ad %AT \n" ;   export SHOWTIMES
me@box:/tmp$ find . -maxdepth 1 -name test -printf "$SHOWTIMES"
test ctime:  Jul 29 15:53:51.4814403150 mtime:  Jul 29 15:53:51.4814403150 atime:  Jul 29 15:53:51.4814403150
me@box:/tmp$ cat test > /dev/null
me@box:/tmp$ find . -maxdepth 1 -name test -printf "$SHOWTIMES"
test ctime:  Jul 29 15:53:51.4814403150 mtime:  Jul 29 15:53:51.4814403150 atime:  Jul 29 15:54:18.1014495830
me@box:/tmp$ touch test
me@box:/tmp$ find . -maxdepth 1 -name test -printf "$SHOWTIMES"
test ctime:  Jul 29 15:54:25.6522832920 mtime:  Jul 29 15:54:25.6522832920 atime:  Jul 29 15:54:25.6522832920
me@box:/tmp$ chmod 777 test
me@box:/tmp$ find . -maxdepth 1 -name test -printf "$SHOWTIMES"
test ctime:  Jul 29 15:54:32.7214485080 mtime:  Jul 29 15:54:25.6522832920 atime:  Jul 29 15:54:25.6522832920
me@box:/tmp$ mv test test2 ; mv test2 test
me@box:/tmp$ find . -maxdepth 1 -name test -printf "$SHOWTIMES"
test ctime:  Jul 29 15:54:54.6322825980 mtime:  Jul 29 15:54:25.6522832920 atime:  Jul 29 15:54:25.6522832920

OK, from top to bottom:

1. we create a file, and we check its ctime, mtime, and atime. No surprises.
2. we access the file, by cat’ing it to /dev/null. atime updates, ctime and mtime remain the same. No surprises.
3. we modify the file, by touching it. atime, mtime, and ctime update… not what you expected, amirite?
4. we change permissions on the file. ctime updates, but mtime and atime do not. Again… not what you expected, right right?
5. we mv the file a couple times. ctime updates again – mtime and atime still don’t.

This is usually the first answer that should be given to “how do I modify the ctime on a file?” (The second answer is: don’t. [i]By design[/b], there is no feature within a POSIX filesystem to set a ctime to anything other than the current system time, so the only way to do it is either to reset the system time, then mv the file, or to unmount the filesystem entirely and hexedit the metadata with a debugging tool. Neither is a good idea, and wanting to do so in the first place usually stems from a misunderstanding of what ctime is and does.)

SouthEast Linux Fest (SELF) 2012

I had a great time at SELF in Charlotte, NC this weekend. One of the highlights, for me, was for the first time meeting some fellow BSD types in the flesh – Kris Moore (founder of the PC-BSD distribution) and Dru Lavigne (Director of the FreeBSD Foundation), no less. While discussing the pain of doing FreeBSD installs onto RAIDZ, Kris told me about the new graphical installer in PC-BSD that lets you create new RAIDZ arrays of all types and install directly to them, all without ever having to leave the installer, which I found pretty exciting. I found it even more exciting when he told me that the procedures taken by the installer were based partly on my own work at freebsdwiki.net!

I set up a new VM with three virtual disks pretty much the minute I got home, and started a new PC-BSD 9.0 install. Sure enough, although the option is a little hard to discover, I managed to figure it out without having to go in search of any documentation – and without ever leaving the installer, and with a bare minimum of blood and chicken feathers, I got a brand new RAIDZ1 across my three virtual disks set up, and PC-BSD cheerfully installed onto it. (This is testing only, of course – in production, you should only do RAIDZ onto bare metal, not onto an abstraction like linux logical volumes or raw files accessed through a hypervisor.) Pretty heady stuff!

To the right – Tux dropped by the table while Dru and Kris and I were chatting, and posed for me with BSD’s horns.  How great is that?

Storing PHP sessions in memcached instead of in files

in php.ini:

session.save_handler = memcache
session.save_path = "tcp://serv01:11211,tcp://serv02:11211,tcp://serv03:11211"

Obviously, you need php5-memcache installed, replace “serv1” “serv2” and “serv3” with valid server address(es), and you’ll need to restart your Apache server after making the change.

Why would you need to do this? Well, this week I had to deal with a web application server pool that kept slowly increasing its number of children all the way up to MaxCli, no matter what. It was unpredictable, other than being a slow creep. Eventually, stracing the pids showed that they were getting stuck in FLOCK on files in /var/lib/php5/php_sess*. This turns out to be an endemic problem with PHP that the devs don’t seem inclined to fix: php’s garbage collector will delete session files dirtily if a php process (which in the case of mod_php, means an Apache process) violates any of the php limits, such as max_execution_time (among many, many others). So you end up with your php script trying to lock a session file (file descriptor 3) that php’s garbage collector already deleted, and therefore an infinitely hung process that will never go away on its own.

Changing over to using memcache to store php sessions eliminated this file lock issue and resulted in a much more stable situation – a server pool that had been creeping up to 800 children per server over the course of a couple hours has been running stable and sweet on less than 150 children per server for days now.

Enabling core dumps on Apache2.2 on Debian

It was quite an adventure today, figuring out how to get a segfaulting Apache to give me core dumps to analyze on Debian Squeeze. What SHOULD have been easy… wasn’t. Here’s what all you must do:

First of all, you’ll need to set ulimit -c unlimited in your /etc/init.d/apache2 script’s start section.

case $1 in
start)
log_daemon_msg "Starting web server" "apache2"
# set ulimit for debugging
# ulimit -c unlimited

Now make a directory for core dumps – mkdir /tmp/apache2-dumps ; chmod 777 /tmp/apache2-dumps – then you’ll need to apt-get install apache2-dbg libapr1-dbg libaprutil1-dbg

And, the current (Debian Squeeze, in May 2012) version of Debian does not have PIE support in the default gdb, so you’ll need to install gdb from backports. So, add deb http://backports.debian.org/debian-backports squeeze-backports main to /etc/apt/sources.list, then apt-get update && apt-get install -t squeeze-backports gdb

Now add “CoreDumpDirectory /tmp/apache2-dumps” to /etc/apache2/apache2.conf (or its own file in conf.d, whatever), then /etc/init.d/apache2 stop ; /etc/init.d/apache2 start

And once you start getting segfaults, you’ll get a core in /tmp/apache2-dumps/core.

Finally, now that you have your core, you can gdb apache2 /tmp/apache2-dumps/core, bt full, and debug to your heart’s content. WHEW.

Opening up SQL server in the Windows Server 2008 firewall

@echo ========= SQL Server Ports ===================
@echo Enabling SQLServer default instance port 1433
netsh firewall set portopening TCP 1433 “SQLServer”
@echo Enabling Dedicated Admin Connection port 1434
netsh firewall set portopening TCP 1434 “SQL Admin Connection”
@echo Enabling conventional SQL Server Service Broker port 4022
netsh firewall set portopening TCP 4022 “SQL Service Broker”
@echo Enabling Transact-SQL Debugger/RPC port 135
netsh firewall set portopening TCP 135 “SQL Debugger/RPC”
@echo ========= Analysis Services Ports ==============
@echo Enabling SSAS Default Instance port 2383
netsh firewall set portopening TCP 2383 “Analysis Services”
@echo Enabling SQL Server Browser Service port 2382
netsh firewall set portopening TCP 2382 “SQL Browser”
@echo ========= Misc Applications ==============
@echo Enabling HTTP port 80
netsh firewall set portopening TCP 80 “HTTP”
@echo Enabling SSL port 443
netsh firewall set portopening TCP 443 “SSL”
@echo Enabling port for SQL Server Browser Service’s ‘Browse’ Button
netsh firewall set portopening UDP 1434 “SQL Browser”
@echo Allowing multicast broadcast response on UDP (Browser Service Enumerations OK)
netsh firewall set multicastbroadcastresponse ENABLE

Testing Ubuntu Precise (12.04 LTS beta)

Precise Pangolin beta installerOK, this is the coolest thing ever – I decided to download the beta for the upcoming Ubuntu Precise Pangolin (12.04 LTS) release and do some testing.  And I start installing it in a VM, and *while* it’s installing, I see it populate a dialog with Ubuntu-relevant links, and curiously, I click one… and BAM, instant fully-functional, working Firefox!

So, while you’re installing your operating system, you can goof around on the internet.  Seriously, how cool is that?!