Routing between wg interfaces with WireGuard

Aha! This was the last piece I was really looking for with WireGuard. It gets a bit tricky when you want packets to route between WireGuard clients. But once you grok how it works, well, it works.

This also works for passing traffic between WireGuard clients on the same interface – the trick is in making certain that AllowedIPs in the client configs includes the entire IP subnet services by the server, not just the single IP address of the server itself (with a /32 subnet)… and that you not only set up the tunnel on each client, but initialize it with a bit of data as well.

Set up your server with two WireGuard interfaces:

root@server:~# touch /etc/wireguard/keys/server.wg0.key
root@server:~# chmod 600 /etc/wireguard/keys/server.wg0.key 
root@server:~# wg genkey > /etc/wireguard/keys/server.wg0.key root@server:~# wg pubkey < /etc/wireguard/keys/server.wg0.key > /etc/wireguard/keys/server.wg0.pub

root@server:~# touch /etc/wireguard/keys/server.wg1.key
root@server:~# chmod 600 /etc/wireguard/keys/server.wg1.key 
root@server:~# wg genkey > /etc/wireguard/keys/server.wg1.key root@server:~# wg pubkey < /etc/wireguard/keys/server.wg1.key > /etc/wireguard/keys/server.wg1.pub

Don’t forget to make sure ipv4 forwarding is enabled on your server:

root@server:~# sed -i 's/^#net\.ipv4\.ip_forward=1/net.ipv4.ip_forward=1/' /etc/sysctl.conf
root@server:~# sysctl -p

Now set up your wg0.conf:

# server.wg0.conf

[Interface] 
   Address = 10.0.0.1/24 
   ListenPort = 51820
   PrivateKey = WG0_SERVER_PRIVATE_KEY
   SaveConfig = false

[Peer] 
   # client1 
   PublicKey = PUBKEY_FROM_CLIENT_ONE
   AllowedIPs = 10.0.0.2/32

And your wg1.conf:

# server.wg1.conf

[Interface] 
 Address = 10.0.1.1/24 
 ListenPort = 51821
 PrivateKey = WG1_SERVER_PRIVATE_KEY
 SaveConfig = false

[Peer] 
 # client2 
 PublicKey = PUBKEY_FROM_CLIENT_TWO
 AllowedIPs = 10.0.1.2/32

Gravy. Now enable both interfaces, and bring them online.

root@server:~# systemctl enable wg-quick@wg0 ; systemctl enable wg-quick@wg1
root@server:~# systemctl start wg-quick@wg0 ; systemctl start wg-quick@wg1

Server’s done. Now, set up your clients.

Client install, multi-wg server:

Client one will connect to the server’s wg0, and client two will connect to the server’s wg1. After creating your keys, set them up as follows:

# /etc/wireguard/wg0.conf on Client1
#    connecting to server/wg0
 
[Interface]
   Address = 10.0.0.2/24
   PrivateKey = PRIVATE_KEY_FROM_CLIENT1
   # set up routing from server/wg0 to server/wg1
   PostUp = route add -net 10.0.1.0/24 gw 10.0.0.1 ; ping -c1 10.0.0.1
   PostDown = route delete -net 10.0.1.0/24 gw 10.0.0.1
   SaveConfig = false

[Peer]
   PublicKey = PUBKEY_FROM_SERVER
   AllowedIPs = 10.0.0.1/24, 10.0.1.1/24
   Endpoint = wireguard.yourdomain.tld:51820

Now set up wg0 on Client2:

# /etc/wireguard/wg0.conf on Client2
#   connecting to server/wg1 

[Interface]
   Address = 10.0.1.2/24
   PrivateKey = PRIVATE_KEY_FROM_CLIENT2
   # set up routing from server/wg1 to server/wg0
   PostUp = route add -net 10.0.0.0/24 gw 10.0.1.1 ; ping -c1 10.0.1.1
   PostDown = route delete -net 10.0.0.0/24 gw 10.0.1.1
   SaveConfig = false

[Peer]
   PublicKey = PUBKEY_FROM_SERVER
   AllowedIPs = 10.0.0.1/24, 10.0.1.1/24
   Endpoint = wireguard.yourdomain.tld:51821

Now start up the client interfaces. First, Client1:

root@client1:~# systemctl start wg-quick@wg0
root@client1:~# wg
interface: wg0
 public key: MY_PUBLIC_KEY
 private key: (hidden)
 listening port: some-random-port

peer: SERVER_PUBLIC_KEY
 endpoint: server-ip-address:51820
 allowed ips: 10.0.0.0/24, 10.0.1.0/24
 latest handshake: 3 seconds ago
 transfer: 2.62 KiB received, 3.05 KiB sent

Now Client2:

root@client2:~# systemctl start wg-quick@wg0
root@client2:~# wg
interface: wg0
 public key: MY_PUBLIC_KEY
 private key: (hidden)
 listening port: some-random-port

peer: SERVER_PUBLIC_KEY
 endpoint: server-ip-address:51821
 allowed ips: 10.0.0.0/24, 10.0.1.0/24
 latest handshake: 4 seconds ago
 transfer: 2.62 KiB received, 3.05 KiB sent

Now that both clients are connected, we can successfully send traffic back and forth between client1 and client2.

root@client1:~# ping -c3 10.0.1.2
PING 10.0.1.2 (10.0.1.2) 56(84) bytes of data.
64 bytes from 10.0.1.2: icmp_seq=1 ttl=63 time=80.4 ms
64 bytes from 10.0.1.2: icmp_seq=2 ttl=63 time=83.5 ms
64 bytes from 10.0.1.2: icmp_seq=3 ttl=63 time=83.2 ms

--- 10.0.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 80.492/82.410/83.501/1.380 ms

Sweet.

The step some of you undoubtedly missed:

There’s a little trick that will frustrate you to no end if you glossed over the client config files without paying attention.

PostUp = route add -net 10.0.0.0/24 gw 10.0.1.1 ; ping -c1 10.0.1.1

If you left out that ping command in one or both of the clients, then you won’t be able to ping client1 from client2, or vice versa, until each client has sent some data down the tunnel. This frustrated the crap out of me initially – ping client2 from client1 would be broken, but then ping client1 from client2 would work and then ping client2 from client1 would work. The solution, as shown, is just to make sure we immediately chuck a packet down the tunnel whenever each client connects; then everything works as intended.

What if I don’t want multiple interfaces?

If all you want to do is pass traffic from one client to the next, you don’t need two interfaces on the server, and you don’t need PostUp and PostDown route commands, either. The trick is that you do need to make sure that AllowedIPs on each client is set to a range that includes the other client, and you do need the PostUp ping -c1 server-ip-addresscommand on each client as well. Without that PostUp ping, you’re going to get frustrated by connectivity that “sometimes works and sometimes doesn’t”.

If you want to give access to some clients but not all clients, you can do that by setting multiple AllowedIPs arguments on the clients, like so:

[Peer]
   PublicKey = PUBKEY_FROM_SERVER
   # this stanza allows access from the server (.1), client one (.2),
   # and client two (.3) - but not from any clients at .4-.254.
   AllowedIPs = 10.0.0.1/32, 10.0.0.2/32, 10.0.0.3/32
   Endpoint = wireguard.yourdomain.tld:51820

That is a sample [Peer] stanza of a client wg config, not a[Peer] stanza of the server wg config! The[Peer] stanzas of the server config should only allow connection to a single IP (using a /32 subnet) for each individual[Peer] definition.

If you try to set AllowedIPs 10.0.0.0/24 on both client1 and client2’s[Peer] stanzas in the server’s wg config, you’ll break one or the other client – they can’t BOTH be allowed the entire subnet.

This bit’s a little confusing, I know, but that’s how it works.

As always, caveat imperator.

Just like the last couple of posts, everything here is tested and working. Also just like the last couple of posts, this is day zero of my personal experience with Wireguard. While everything here works, I still have not delved into any of the actual crypto components’ configurability or personally evaluated/researched how sane the defaults are.

These configs absolutely will encrypt the data sent down the tunnel – I’ve verified that much! – but I cannot offer a well-established and researched opinion on how well they encrypt that data, or what weaknesses there might be. I have no particular reason to believe they’re not secure, but I haven’t done any donkey-work to thoroughly establish that they are secure either.

Caveat imperator.

I may come back to edit out these dire warnings in a few weeks or months as I hammer at this stuff more. Or I might forget to! Feel free to tweet @jrssnet if you want to ask how or if things have changed; I am definitely going to continue my evaluation and testing.

Working VPN-gateway configs for WireGuard

Want to set up a simple security VPN, that routes all your internet traffic out of a potentially hostile network through a trusted VM somewhere? Here you go. Note that while all this is tested and working, this is still literal day zero of my personal experience with Wireguard; in particular while Wireguard claims to use only the most secure crypto (the best, everybody says that!) I not only have not really investigated that, I don’t know how to configure that part of it, so this is just using whatever the WG defaults are. Caveat imperator.

Installing Wireguard, generating keys:

This first set of steps is the same for all machines. Substitute the actual machine name as appropriate; you want to make sure you know which of these keys is which later on down the line, so actually name them and don’t be sloppy about it.

root@machine:~# apt-add-repository ppa:wireguard/wireguard ; apt update ; apt install wireguard-dkms wireguard-tools

root@machinename:~# mkdir /etc/wireguard/keys
root@machinename:~# chmod 700 /etc/wireguard/keys

root@machinename:~# touch /etc/wireguard/keys/machinename.wg0.key
root@machinename:~# chmod 600 /etc/wireguard/keys/machinename.wg0.key

root@machinename:~# wg genkey > /etc/wireguard/keys/machinename.wg0.key
root@machinename:~# wg pubkey < /etc/wireguard/keys/machinename.wg0.key > /etc/wireguard/keys/machinename.wg0.pub

OK, you’ve installed wireguard on your server VM and one or two clients, and you’ve generated some keys.

Setting up your server VM:

Create your config file on the server, at /etc/wireguard/wg0.conf:

[Interface] 
   Address = 10.0.0.1/24 
   ListenPort = 51820 
   PrivateKey = YOUR_SERVER_PRIVATE_KEY
   SaveConfig = false
 
   # Internet Gateway config: nat wg1 out to the internet on eth0 
   PostUp = iptables -A FORWARD -i wg1 -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE 
   PostDown = iptables -D FORWARD -i wg1 -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE

[Peer] 
   # Client1
   PublicKey = PUBLIC_KEY_FROM_CLIENT1
   AllowedIPs = 10.0.0.2/32

[Peer] 
   # Client2 
   PublicKey = PUBLIC_KEY_FROM_CLIENT2
   AllowedIPs = 10.0.0.3/32

Now you’ll need to enable ipv4 forwarding in /etc/sysctl.conf.

root@server:~# sed -i 's/^#net\.ipv4\.ip_forward=1/net.ipv4.ip_forward=1/' /etc/sysctl.conf
root@server:~# sysctl -p

Enable your wg0 interface to start automatically at boot, and bring it up:

root@server:~# sysctl enable wg-quick@wg0
root@server:~# sysctl start wg-quick@wg0

Server should be good to go now.

Setting up your clients:

Client setup is a bit simpler; all you really need is the /etc/wireguard/wg0.conf file itself.

[Interface] 
   # CLIENT1 
   Address = 10.0.0.2/24 
   PrivateKey = CLIENT1_PRIVATE_KEY
   SaveConfig = false

   # the DNS line is broken on 18.04 due to lack of resolvconf 
   # DNS = 1.1.1.1

[Peer] 
   # SERVER 
   PublicKey = PUBLIC_KEY_FROM_SERVER
   Endpoint = wireguard.yourdomain.fqdn:51820

   # gateway rule - send all traffic out over the VPN
   AllowedIPs = 0.0.0.0/0

Note that I have the DNS = 1.1.1.1 line commented out above – its syntax is correct, and it works fine on Ubuntu 16.04, but on 18.04 it will cause the entire interface not to come up due to a lack of installed resolvconf.

You can use sysctl enable wg-quick@wg0 to have the wg0 interface automatically start at boot the way we did on the server, but you likely won’t want to. Without enabling it to start automatically at boot, you can use sysctl start wg-quick@wg0 by itself to manually start it, and sysctl stop wg-quick@wg0 to manually disconnect it. Or if you’re not in love with systemd, you can accomplish the same thing with the raw wg-quick commands: wg-quick up wg0 to start it, and wg-quick down wg0 to bring it down again. Your choice.

What about Windows? Android? Etc?

You can use TunSafe as a Windows client, and the WireGuard app on Android. Setup steps will basically be the same as shown above. On a Mac, you can reportedly brew install wireguard-tools and have everything work as above (though you’ll need to invoke wg-quick directly; systemd isn’t a thing there).

If you’ve rooted your Android phone, you can build a kernel that includes the Wireguard kernel module; if you haven’t, stock kernels work fine – the Android app just runs in userspace mode, which is somewhat less efficient. (You’re currently stuck in userspace mode on a Mac no matter what, AFAIK; not sure what the story is with TunSafe on Windows.)

If you’re using iOS, there’s a Git repository that purports to be a Wireguard client for iPhone/iPad; but good f’n luck actually doing anything with it unless you’re pretty deep into the iOS development world already.

Some testing notes on WireGuard

I got super, super interested in WireGuard when Linus Torvalds heaped fulsome praise on its design (if you’re not familiar with Linus’ commentary, then trust me – that’s extremely fulsome in context) in an initial code review this week. WireGuard aims to be more secure and faster than competing VPN solutions; as far as security goes, it’s certainly one hell of a lot more auditable, at 4,000 lines of code compared to several hundred thousand lines of code for OpenVPN/OpenSSL or IPSEC/StrongSwan.

I’ve got a decade-and-a-half of production experience with OpenVPN and various IPSEC implementations, and “prettiness of code” aside, frankly they all suck. They’re not so bad if you only work with a client or ten at a time which are manually connected and disconnected; but if you’re working at a scale of hundred+ clients expected to be automatically connected 24/7/365, they’re a maintenance nightmare. The idea of something that connects quicker and cleaner, and is less of a buggy nightmare both in terms of security and ongoing usage, is pretty strongly appealing!

WARNING:  These are my initial testing notes, on 2018-Aug-05. I am not a WireGuard expert. This is my literal day zero. Proceed at own risk!

Alright, so clearly I wanna play with this stuff.  I’m an Ubuntu person, so my initial step is apt-add-repository ppa:wireguard/wireguard ; apt update ; apt install wireguard-dkms wireguard-tools .

After we’ve done that, we’ll need to generate a keypair for our wireguard instance. The basic commands here are wg genkey and wg pubkey. You’ll need to pipe private key created with wg genkey into wg pubkey to get a working private key.  You don’t have to store your private key anywhere outside the wg0.conf itself, but if you’re a traditionalist and want them saved in nice organized files you can find (and which aren’t automagically monkeyed with – more on that later), you can do so like this:

root@box:/etc/wireguard# touch machinename.wg0.key ; chmod 600 machinename.wg0.key
root@box:/etc/wireguard# wg genkey > machinename.wg0.key
root@box:/etc/wireguard# wg pubkey < machinename.wg0.key > machinename.wg0.pub

You’ll need this keypair to connect to other wireguard machines; it’s generated the same way on servers or clients. The private key goes in the [Interface] section of the machine it belongs to; the public key isn’t used on that machine at all, but is given to machines it wants to connect to, where it’s specified in a [Peer] section.

From there, you need to generate a wg0.conf to define a wireguard network interface. I had some trouble finding definitive information on what would or wouldn’t work with various configs on the server side, so let’s dissect a (fairly) simple one:

# /etc/wireguard/wg0.conf - server configs

[Interface]
   Address = 10.0.0.1/24
   ListenPort = 51820
   PrivateKey = SERVER_PRIVATE_KEY
   
   # SaveConfig = true makes commenting, formattting impossible
   SaveConfig = false
   # This stuff sets up masquerading through the server's WAN,
   # if you want to route all internet traffic from your client
   # across the Wireguard link. 
   #
   # You'll also need to set net.ipv4.ip_forward=1 in /etc/sysctl.conf
   # if you're going this route; sysctl -p to reload sysctl.conf after
   # making your changes.
   #
   PostUp = iptables -A FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE; ip6tables -A FORWARD -i wg0 -j ACCEPT; ip6tables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
   PostDown = iptables -D FORWARD -i wg0 -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE; ip6tables -D FORWARD -i wg0 -j ACCEPT; ip6tables -t nat -D POSTROUTING -o eth0 -j MASQUERADE

OK, so far so good. Note that SERVER_PRIVATE_KEY above is not a reference to a filename – it’s the server’s private key itself, pasted directly into the config file!

With the above server config file (and a real private key on the private key line), wg0 will start, and will answer incoming connections. The problem is, it’ll answer incoming connections from anybody who has the server’s public key – no verification of the client necessary. (TESTED)

Here’s a sample client config:

# client config - client 1 - /etc/wireguard/wg0.conf

[Interface]
   Address = 10.0.0.2/20
   SaveConfig = true
   PrivateKey = MY_PRIVATE_KEY

   # Warning: setting DNS here won't work if you don't
   # have resolvconf installed... and if you're running
   # Ubuntu 18.04, you probably don't have resolvconf
   # installed. If you set this without resolvconf available,
   # the whole interface will fail to come up.
   #
   # DNS = 1.1.1.1

[Peer]
   PublicKey = SERVER_PUBLIC_KEY
   Endpoint = wireguard.mydomain.wtflol:51820

   # this restricts tunnel traffic to the VPN server itself
   AllowedIPs = 172.29.128.1/32

   # if you wanted to route ALL traffic across the VPN, do this instead:
   # AllowedIPs = 0.0.0.0/0

Notice that we set SaveConfig=true in wg0.conf here on our client. This may be more of a bug than a feature. See those nice helpful comments we put in there? And notice how we specified an FQDN instead of a raw IP address for our server endpoint? Well, with SaveConfig=true on, those are going to get wiped out every time the service is restarted (such as on boot). The comments will just get wiped, stuff like the random dynamic port the client service uses will get hard-coded into the file, and the FQDN will be replaced with whatever IP address it resolved to the last time the service was started.

So, yes, you can use an FQDN in your configs – but if you use SaveConfig=true you might as well not bother, since it’ll get immediately replaced with a raw IP address anyway. Caveat imperator.

If we want our server to refuse random anonymous clients and only accept clients who have a private key matching a pubkey in our possession, we need to add [Peer] section(s):

[Peer]
PublicKey = PUBLIC_KEY_OF_CLIENT_ONE
AllowedIPs = 10.0.0.2/32

This works… and with it in place, we will no longer accept connections from anonymous clients. If we haven’t specifically authorized the pubkey for a connecting client, it won’t be allowed to send or receive any traffic. (TESTED.)

We can have multiple peers defined, and they’ll all work simultaneously, on the same port on the same server: (TESTED)

# appended to wg0.conf on SERVER

[Peer]
PublicKey = PUBKEY_OF_CLIENT_ONE
AllowedIPs = 10.0.0.2/32

[Peer]
PublicKey = PUBKEY_OF_CLIENT_TWO
AllowedIPs = 10.0.0.3/32

Wireguard won’t dynamically reload wg0.conf looking for new keys, though; so if we’re adding our new peers manually to the config file like this we’ll have to bring the wg0 interface down and back up again to load the changes, with wg-quick down wg0 && wg-quick up wg0. This is definitely not a good way to do things in production at scale, because it means approximately 15 seconds of downtime for existing clients before they automatically reconnect themselves: (TESTED)

64 bytes from 172.29.128.1: icmp_seq=10 ttl=64 time=35.0 ms
64 bytes from 172.29.128.1: icmp_seq=11 ttl=64 time=39.3 ms
64 bytes from 172.29.128.1: icmp_seq=12 ttl=64 time=37.6 ms

[[[       client disconnected due to server restart      ]]]
[[[  16 pings dropped ==> approx 15-16 seconds downtime  ]]]
[[[ client automatically reconnects itself after timeout ]]]

64 bytes from 172.29.128.1: icmp_seq=28 ttl=64 time=51.4 ms
64 bytes from 172.29.128.1: icmp_seq=29 ttl=64 time=37.6 ms

A better way to do things in production is to add our clients manually with the wg command itself. This allows us to dynamically add clients without bringing the server down, and that doing so will also add those clients into wg0.conf for persistence across reboots and what-have-you.

If we wanted to use this method, the CLI commands we’ll need to run on the server look like this: (TESTED)

root@server:/etc/wireguard# wg set wg0 peer CLIENT3_PUBKEY allowed-ips 10.0.0.4/24

The client CLIENT3 will immediately be able to connect to the server after running this command; but its config information won’t be added to wg0.conf, so this isn’t a persistent addition. To make it persistent, we’ll either need to append a [Peer] block for CLIENT3 to wg0.conf manually, or we could use wg-quick save wg0 to do it automatically. (TESTED)

root@server:/etc/wireguard# wg-quick save wg0

The problem with using wg-quick save (which does not require, but shares the limitations of, SaveConfig = true in the wg0.conf itself) is that it strips all comments and formatting, permanently resolves FQDNs to raw IP addresses, and makes some things permanent that you might wish to keep ephemeral (such as ListenPort on client machines). So in production at scale, while you will likely want to use the wg set command to directly add peers to the server, you probably won’t want to use wg-quick save to make the addition permanent; you’re better off scripting something to append a well-formatted [Peer] block to your existing wg0.conf instead.

Once you’ve gotten everything working to your liking, you’ll want to make your wg0 interface come up automatically on boot. On Ubuntu Xenial or later, this is (of course, and however you may feel about it) a systemd thing:

root@box:/etc/wireguard# systemctl enable wg-quick@wg0

This is sufficient to automatically bring up wg0 at boot; but note that since we’ve already brought it up manually with wg-quick up in this session, an attempt to systemctl status wg-quick@wg0 will show an error. This is harmless, but if it bugs you, you’ll need to manually bring wg0 down, then start it up again using systemctl:

root@box:/etc/wireguard# wg-quick down wg0
root@box:/etc/wireguard# systemctl start wg-quick@wg0

At this point, you’ve got a working wireguard interface on server and client(s), that’s persistent across reboots (and other disconnections) if you want it to be.

What we haven’t covered

Note that we haven’t covered getting packets from CLIENT1 to CLIENT2 here – if you try to communicate directly between two clients with this setup and no additional work, you’ll see the following error: (TESTED)

root@client1:/etc/wireguard# ping -c1 CLIENT2
From 10.0.0.2 icmp_seq=1 Destination Host Unreachable
ping: sendmsg: Required key not available
--- 10.0.0.3 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

We also haven’t looked around at any kind of crypto configuration yet; at this point we’re blindly accepting whatever defaults for algorithms, key sizes, and so forth and hoping for the best. Make sure you understand these (and I don’t, yet!) before deploying in production.

At this point, though, we’ve at least got something working we can play with. Happy hacking, and good luck!