Online Resizing of KVM drive

Don’t try to resize a disk of a running VM with qemu-img directly. Use virsh instead. NB: This can’t be done via the GUI, either. You have to use the shell.

Gather info:

virsh # domblkinfo <domain> --all --human

Resize virtual disk:

virsh # blockresize <domain> --path <vd?> --size 300GB

Then inside the VM, assuming you’re using LVM and XFS as filesystem, re-read the partition table of the resized disk, then resize the physical volume, then the logical volume and finally the filesystem:

# blockdev --rereadpt /dev/vd?
# pvresize /dev/vd?
# lvextend -l +<no. of extends> /dev/mapper/<lv-name>
# xfs_growfs /dev/mapper/<lv-name>

That’s all, folks!

Valhalla NIC

As it seems, the NIC in current valhalla has a hardware bug, causing hardware unit hangs. To work around this, disable scatter-gather, TCP segmentation offloading and generic receive offload with this command:

# ethtool -K wan sg off tso off gro off

I did that 11 days ago, and didn’t get a single unit hang yet. Keep your fingers crossed!

Networking with Hetzner

Setting the scene

Recently I found out the hard way that Hetzner frowns upon MAC addresses on their switch ports they don’t know, so they blocked my server.

What happened: The wan interface was a member of the bridge connecting my virtual machines, because from what I knew back then that was the only way to provide the VMs with IPv6 addresses from the Hetzner subnet. So their MAC addresses (52:54:…) went out to the switch. This was so bad that they didn’t even allow me to unlock it temporarily to fix the problem!

What I did: Reboot the server via the robot to shut down. They aren’t started automatically on boot. Then notify Hetzner via the robot that I fixed the problem by reconfiguring the network configuration. They replied very quickly by checking for foreign MAC addresses and, since they found none, unlocked my server. Now I had a chance to reconfigure the network without using their hardware KVM.

The fix

First thing to do, was remove the wan interface from the bridge. That was a remnant from a completely bridged setup from long ago, anyway. Since all IP4 traffic was routed, that wasn’t a problem. IPv6 was a problem, though. The router announcements from radvd worked, but neighbor discovery didn’t, so the VMs didn’t have IPv6 connectivity.

I finally found this article: It boils down to:

  • Give the wan interface a single IPv6 address from your assigned network (netmask /128) to make it a part of it.
  • Assign the bridge the same address, only with a /64 netmask!

The article does it for Debian. Translated to systemd-networkd I ended up with this:

The wan interface

/etc/systemd/network/wan.network

[Match] 
Name=wan <- The interface name is set by an udev rule

[Network] 
Address=2001:db8::2/128 <- /128 is the clue!!!
Gateway=<replace.with.ipv4.gateway>
Gateway=fe80::1 <- This is literal!!!

[Address] 
Address=<your.ipv4.address>
Peer=<the.peer>/32

Create the bridge

[NetDev] 
Name=br0 
Kind=bridge

Do not attach the wan interface to the bridge!

Configure the bridge

[Match] 
Name=br0 

[Network] 
Address=192.168.0.1/24 <- Your private IPv4 VM network
Address=2001:db8::2/64 <- YES, this is the same address as wan!

Don’t be alarmed if br0 is not configured and down on reboot. Once you start a VM using the bridge it will be taken up and have all the right addresses. If you add the IPv6 address manually with ip, you will loose IPv6 connectivity, since the kernel creates a useless default route! You have been warned!

Works like a charm (for now)!

DDNS with bind and perl

Problem

You have a crappy internet provider who hands out dynamic IPv4 addresses and IPv6 prefixes, but you can’t remember all those numbers and want to use DNS as intended. Actually, I have 2 internet providers, Vodafone and Telekom, each with its own problems . Vodafone’s IPv6 prefix seems to be static for now, but they blessed me with DS-Lite after the update to 400 Mbit. Telekom has a dual stack, but the public IPv4 address and the delegated IPv6 prefixes are highly dynamic.

What you need

  1. A DNS zone and control over the zone file, so you can delegate a subdomain to another name server
  2. The “another name server” with a static IP. If you don’t have that, stop reading!
  3. a script for the dynamic updates

I have a root server and a DNS zone at Hetzner. The root Server has a static IPv4 address and IPv6 prefix. Also, they let you edit the pure bind zone file via a text box, not a form! Since I’m a perl guy, I did the script in my favorite perfect and easy readable language.

Configure your “another name server”

Install bind on your box with the static IP. Then generate a key for dynamic updates. Do not use dnssec-keygen as described in various tutorials. You will fail! Use ddns-confgen instead and follow the instructions in the comment. Your zone file should look something like this:

...
// THE key
key "ddns-key.dyn.d-tor.org" { 
       algorithm hmac-sha256; 
       secret "base64stufffromddns-confgen="; 
};
// THE dynamic zone
zone "dyn.d-tor.org." { 
       update-policy { grant ddns-key.dyn.d-tor.org zonesub ANY; }; 
       type master; 
       file "dyn.d-tor.org.hosts"; 
};
...

The above snippet configures the zone dyn.d-tor.org and allows updates to anyone who has THE key. Next, create the initial zone file in the directory specified by the “directory” directive in named.conf (on ArchLinux it’s the default /var/named, thus /var/name/dyn.d-tor.org.hosts):

$ORIGIN . 
$TTL 600        ; 10 minutes 
dyn.d-tor.org           IN SOA  valhalla.d-tor.org. me.d-tor.org. ( 
  1          ; serial 
  500        ; refresh (8 minutes 20 seconds) 
  500        ; retry (8 minutes 20 seconds) 
  86400      ; expire (1 day) 
  500        ; minimum (8 minutes 20 seconds) 
) 
  NS      valhalla.d-tor.org.

Then restart/reconfigure bind and check for errors:

# systemctl restart named
# journalctl -u named

If you have ip(6)tables running, allow access to port 53 UDP and TCP for good measure:

# iptables -I INPUT 2 -p udp --dport 53 -j ACCEPT
# iptables -I INPUT 2 -p tcp --dport 53 -j ACCEPT
# iptables -I INPUT 2 -p udp --dport 53 -j ACCEPT
# iptables -I INPUT 2 -p tcp --dport 53 -j ACCEPT

Copy the key to the box where you want to update from, e.g. /etc/ddns.key and set permissions to 600. It should contain only the key section:

key "ddns-key.dyn.d-tor.org" { 
       algorithm hmac-sha256; 
       secret "base64stufffromddns-confgen="; 
};

Now test it:

# nsupdate -k /etc/ddns.key <<EOF
> server 2a02:4f8:2a:254e::2
> update delete blub.dyn.d-tor.org A
> update add blub.dyn.d-tor.org 200 A 191.139.50.71
> send
> EOF

If you’ve done everything right:

# host blub.dyn.d-tor.org 2a02:4f8:2a:254e::2
blub.dyn.d-tor.org has address 191.139.50.71

Delegate the Zone

Now we need to delegate dyn.d-tor.org to 2a02:4f8:2a:254e::2. Add this to the zone file on your authoritative name server (in my case, Hetzner):

...
dyn IN NS dyndns ; delegation to dyndns.d-tor.org
dyndns IN AAAA 2a02:4f8:2a:254e::2 ; IPv6 glue record for dyndns.d-tor.org
dyndns IN A <IPv4.address.of.dyndns> ; IPv4 glue record
...

Wait for the information to spread. Check with dnstracer:

# dnstracer -s . dyn.d-tor.org

If you get a result, we can continue 🙂

THE script

Write a script that extracts the IP addresses from your interfaces and uses nsupdate to update the records. It shouldn’t be that hard. Sorry, I can’t provide mine, because it is very specific to my setup.

Multi-homed: Telekom, Vodafone and Hetzner

Problem

I refined my network configuration a bit, because there was a problem: When I pinged the Vodafone IPv6 address of hadante from my new, shiny Hetzner box, it wouldn’t answer, because I got the policy routing wrong.

Initially I routed everyting to Hetzner via the Vodafone interface, but that’s plain wrong. This way hadante even sent packets originating from Telekom IP’s via the Vodafone interface, with the Telekom IP as source. What I really wanted:

  1. answer requests to the Vodafone interface via Vodafone
  2. make Telekom the default route

Answer requests to the Vodafone IP

The solution was easy: create a rule to send everything from the Vodafone interface out there. Unfortunately, nothing is as easy as it seems. Because the Vodafone-IPv6-Prefix is semi-static, systemd-networkd policy routing doesn’t work. The routing table can be filled automatically:

[Match] 
Name=ext 

[Network] 
DHCP=yes 
IPv6Token=::dead:b0a1 

[DHCP] 
RouteMetric=4096 
RouteTable=199 

[IPv6AcceptRA] 
RouteTable=199

The RouteTable directive adds the routes acquired by DHCP and Router Announcments to the routing table 199 (aka kd, see /etc/iproute2/rt_tables), but without a rule it doesn’t do anything. The IPv6Token directive sets the IPv6 address to <prefix>::dead:b0a1, by the way.

The rule is added by a perl script written by yours truly. It does something like this (pseudo perl code):

...
$old = <old IPv6 address>;
$new = <new IPv6 address>;
# match old prefix
$old =~ m#^([[:xdigit:]]{1,4}:[[:xdigit:]]{1,4}:[[:xdigit:]]{1,4}:[[:xdigit:]]{1,4}:)#;
# delete old rule
system("/usr/bin/ip -6 rule dele from $1:/64 table kd");
# match new prefix
$new =m#^([[:xdigit:]]{1,4}:[[:xdigit:]]{1,4}:[[:xdigit:]]{1,4}:[[:xdigit:]]{1,4}:)#;
# add new rule
system("/usr/bin/ip -6 rule add from $1:/64 lookup kd");
...

This way both the Telekom IP and the Vodafone IP work from anywhere. As a bonus, IPv6 requests to Hetzner from the delegated Telekom IPv6 network now work, too 🙂

Configuring the Telekom interface

During my network configuration spree I tried to configure the Telekom interface with systemd-networkd instead of dhcpcd, but that didn’t work, unfortunately. I couldn’t get the prefix delegation to the internal interface to work. Supposedly systemd-networkd can do it, but the documentation is, let’s say, sparse at best. After several attempts I gave up and reverted to dhcpcd, as described in this post.

Hetzner – kVkm

How to get the vKVM console @Hetzner to work? It ain’t much of a surprise that it doesn’t work out of the box, because the advertised link directs you to a Java applet. That only displayed the header for me, on Linux and Windows 10. But fret not: there is a solution.

Fortunately, when vKVM is running, you can access it via SSH on Port 47772 with the given password. VNC should be listening on Port 47774, but it’s stunnled, so you can’t access it directly. QEMU-VNC is actually listening on Port 5901/tcp, so you have to tunnel you way in.

# ssh -L 5901:<remote_ip>:5901 -l root <remote_ip>

That should forward remote 5901/tcp to something you can access. Now run:

# vncviewer 127.0.0.1::5901

And no, the double colon is no typo! Now go, fix your problems and have fun!

IPv6 configuration Hetzner

(obsolete, superseded by https://tollana.d-tor.org/notes-to-self/?p=585)

Well, another issue I just noticed after the recent reboot of valhalla. When bridging, do never, ever use IPv6 autoconfiguration on the actual ethernet interface or the bridge itself. That will totally screw up the routing!

Disable it by adding the following lines somewhere in /etc/sysctl.d:

net.ipv6.conf.wan.use_tempaddr = 0 
net.ipv6.conf.wan.autoconf = 0 
net.ipv6.conf.br0.use_tempaddr = 0 
net.ipv6.conf.br0.autoconf = 0

You can change it directly by echoing the values to the respective proc files. Unfortunately, the changes only take effect after shutting down and taking the interface up again. So be really, really careful! Be warned: The interface won’t have an IPv6 address any more, so make sure that you have IPv4 connectivity!

You can do this with e.g. screen:

# screen
# ip link set down wan ; sleep 1; ip link set up wan

 

Kernel 4.4.1-2-ARCH and libvirtd

Well, well, well… What a fuckup! As of this writing [2016-03-03] the current versions of the Archlinux Kernel and libvirtd are not working at all. Took me almost a week to figure this out.

There are two issues:

  1. The Kernel package 4.4.1-2-ARCH (which is actually 4.4.3 with patches) crashes silently if you try to start a VM from libvirtd (if you even get so far, but that’s covered below). Honestly, I have no idea why, because it doesn’t sputter an OOPS. What I know: Downgrading to 4.3.3-3-ARCH fixes that issue.
  2. libvirt-1.3.2-1 is broken, or rather libvirtd.service is… It says that libvirtd is of type notify. That may be true, but it doesn’t work. Don’t even bother to fiddle around with polkit rules: It doesn’t work. IMHO polkit is over-engineered.

To fix (1.): Downgrade the Kernel to 4.3.3. If you don’t have a copy, download it here. Install it with:

# pacman -U linux-4.3.3-3-x86_64.pkg.tar.xz
# systemctl reboot

When the older kernel is running (confirm it with uname -a), we can fix (2.) by turning libvirtd.service from Type=notify to Type=simple. Be warned: It may have serious side effects! You’re on your own when you do this!

# cp /usr/lib/systemd/system/libvirtd.service /etc/systemd/system

Now edit /etc/systemd/system/libvirtd.service and comment out Type=notify. This turns libvirtd into a “simple” service, not forking, no authentication, nothing. Then reload everything and start libvirtd, virtlockd and virtlogd:

# systemctl daemon-reload
# systemctl start libvirtd virtlockd virtlogd

Finally (re-)start your virtual machines. Share and enjoy!

Disk Change @ Hetzner

1. Get serial of defective disk

# for i in a b; \
 do echo $i ; smartctl -i /dev/sd$i | grep -i serial ;\
done

The disk not returning a serial is most likely the defective one.

2. Create a Ticket @ Hetzner Robot

Log in to the Hetzner Robot and create a Ticket: Left menu:

Anfragen -> Serverprobleme ausklappen -> Festplatte defekt

If the serial of the defective drive can’t be obtained, enter the serial of the working one and stress that the said serial is the working drive. Otherwise they’ll swap the wrong one and you end up with nothing!

3. Copy partition table

Once the drive is replaced, copy the partition table from the working drive to the new one. sgdisk comes to rescue (THINK BEFORE YOU COPY & PASTE, WON’T WORK ANYWAY):

# sgdisk -R=/dev/sd[new] /dev/sd[old]

In doubt, RTFM twice: man sgdisk

4. Resync the RAID

mdadm comes to rescue:

# mdadm --manage /dev/md0 --add /dev/sd[new][part]

If you can’t get the former command to work, again: RTFM! It’s your data!

5. Stare at the progress

# watch cat /proc/mdstat

6. Be delighted!