Tag: debian

Upgrading check-mk and Debian

1. Overview

Upgrading Debian from 8 (jessie) to 9 (stretch) with check-mk installed isn’t as easy as it seems. You have to:

  1. Upgrade check-mk to 1.4.0 and fix all issues
  2. Backup all sites
  3. Purge check-mk
  4. Upgrade debian
  5. Reinstall check-mk 1.4.0
  6. Restore the check-mk-sites from backup

2. Upgrade check-mk and back it up

Download the .deb package and install it. Follow the official the official guide and upgrade all sites. After fixing all issues, create a backup of each site:

# su - <sitename>
$ omd backup site.name.tar.gz

Repeat this for all sites.

3. Remove check-mk and upgrade to stretch

Stop sites:

# su - <sitename>
$ omd stop

Repeat this for all sites. Then remove (purge) check-mk:

# dpkg -P check-mk-raw-1.4.0p17

Once this is done, update the distribution to debian (you really should know how to do that!). Autoremove all obsolete packages and reboot.

4. Reinstall check-mk and restore from backup

Download the .deb package for stretch and install it. Since you autoremoved dependent packages earlier, the install will most likely fail. Fix it with:

# apt --fix-broken install

Now we can restore the sites from our backup (as root!):

# omd restore <site-name.tar.gz>
# su - sitename
$ omd start

Repeat for all sites and fix all remaining issues.

5. Notes

Of course you don’t want to do this without a security net. Take a snapshot and destroy that instead of the real VM. How to do that with KVM and libvirt is explained here.

IPv6 connectivity of security.debian.org

The Problem

Have been hunting this down for quite some time now: several virtual hosts weren’t able to connect to security.debian.org. First I thought it was me, even though I had all the ingredients for IPv6-forwarding to work (this is the host):

*filter 
:FORWARD DROP [0:0]
-A FORWARD -p ipv6-icmp -j ACCEPT
-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

Of course, net.ipv6.conf.*.forwarding was set on the host. That should be enough to forward all outgoing connections and drop incoming, right? And it does, for pretty much any host, except security.debian.org (AKA as lobos.debian.org and villa.debian.org). There may be more, but that one caught my attention, because apt update hung just there (ftp.de.debian.org worked, btw).

First I thought that it was the MTU, but that was pretty much a red herring. After a while I realized that it was working when the FORWARD policy was ACCEPT, but of course that wasn’t a viable solution. So I dug deeper: Strangely enough, with the policy back to DROP and this rule:

-A FORWARD -d <VM-IPv6> -p tcp -m multiport \
   --sports 80,443 -j ACCEPT

it also worked, but this wasn’t enough:

-A FORWARD -s <VM-IPv6> -j ACCEPT

WTF? Fortunately I had a working virtual machine (also debian 8.6, same kernel), so I ended up comparing the IPv6-sysctl values (sysctl -a | grep ipv6).

The solution

As it turned out, the only difference was that the working virtual machine had net.ipv6.conf.*.forwarding enabled. So I added

net.ipv6.conf.all.forwarding=1
net.ipv6.conf.default.forwarding=1

to /etc/sysctl.conf of the failing virtual machine, rebooted and then it finally worked ™! I don’t have the slightest clue why this is necessary, though. The VM is the final receiver, the end of the chain, but certainly not a router! Maybe it’s a kernel bug, I don’t know… I’m just glad it works 🙂

Just calling sysctl -w doesn’t do it, btw. You have to take the interface down and up again to take effect, hence the reboot…

Updating check-mk

It’s actually surprisingly easy! Just download the latest .deb from here to the server. Then install it with:

# dpkg -i <latest.deb>

This by itself does nothing. It just installs the new version in parallel to the old one. All instances must be updated separately with these commands:

# su - <instance_user>
$ omd stop
$ omd update
$ omd start

Now check for new/missing/vanished services and update the agents (it’s not a must, though). Acknowledge all incompatibilities (also not a must) and you’re done!

Upgrade Mailscanner

The easy part:

  • Download the tarball from the official website
  • Unpack to somewhere
  • execute <unpack-dir>/install.sh

The hard part:

Do not agree to install the missing perl modules from CPAN! Install them via the package manager. If almost every module is missing, perldoc is not installed. Debian is especially nasty here. They have a perldoc executable, even if it isn’t installed, and it only prints “the package perl-doc is not installed” or something and instructions how to install it.

Unfortunately the installer uses perldoc to figure out which modules are missing, so when it calls the stub everything is missing…

If there is a version mismatch, check if you have a CPAN-Version installed in /usr/local/<something>.

# perl -V

shows @INC at the bottom. Remove the offending package and install it via your favorite package manager.

The config part:

Generate a new config in the new etc-Directory as described in the docs. If you’re using MailWatch, don’t forget to copy

lib/MailScanner/CustomFunctions/MailWatch.pm

to the new directory. Stop mailscanner, link the new version to /opt/MailScanner or something and start it. You applaud 🙂

 

Upgrade wheezy to jessie

All in all, it was easy. Just one big caveat regarding apache2. Can’t say if it was PEBKAC any more. But read below.

1. Check for obsolete packages

As stated in the official Upgrade Guide: Do an audit!

# dpkg --audit

Really, do it! And fix the non-compliant packages.

2. Install the Distribution Kernel

Again, really do it. Otherwise the dist-upgrade will fail on udev, if your custom kernel doesn’t have CONFIG_SIGNALFD, CONFIG_FHANDLE and whatnot built in. You can override it, but then it won’t boot. So better be safe.

3. Update and dist-upgrade

Change /etc/apt/sources.list to use jessie. Then

# apt-get update
# apt-get dist-upgrade

Carefully watch for configfile-conflicts. Esp. Apache and cyrus! Keep the config for:

  • /etc/cyrus.conf
  • /etc/imapd.conf

Don’t overwrite them! We’ll fix them later!

4. Reboot

Reboot into the new system, get systemd running. Again, don’t even think to go on without a reboot

5. Fix cyrus and postfix

The command for cyr_* has changed. Now it’s so cyrus <*>, so move the old config out of the way and replace it with .dpkg-dist. Then diff it and change the relevant line(s). That would be especially:

lmtpunix cmd="lmtpd" \
   listen="/var/spool/postfix/cyrus-socket/lmtp" \
   prefork=0 maxchild=20

with

lmtp cmd="lmtpd -a" listen="localhost:lmtp" prefork=0 maxchild=20

So don’t listen on an AF_UNIX socket any more but on a network socket. For this to work you need to add this line to /etc/services:

lmtp        24/tcp

LMTP doesn’t seem to be a well known port, whatever…

Now for postfix: Change /etc/postfix/main.cf accordingly:

mailbox_transport = lmtp:inet:localhost

I.e. listen on a network socket instead of AF_UNIX. This way you don’t have to bother with permissions on the socket file, and it’s IPv6-enabled as a bonus 🙂

6. Fix apache2

Make sure all your files in sites-available end in *.conf! Otherwise they won’t be loaded! I learned the hard way, wondering why SSL-Connections failed because there was no SSL. Check what VHosts are loaded with:

# apache2ctl -t -D DUMP_VHOSTS

If that shows no Vhosts, you don’t have any! Fix it. BTW, NameVirtualHost is obsolete. Remove it. So seems to be gnutls…

Create an IPv6-enabled-VHost like this:

<VirtualHost _default_:443>
...
</VirtualHost>