Author: arno

IPv6 connectivity of security.debian.org

The Problem

Have been hunting this down for quite some time now: several virtual hosts weren’t able to connect to security.debian.org. First I thought it was me, even though I had all the ingredients for IPv6-forwarding to work (this is the host):

*filter 
:FORWARD DROP [0:0]
-A FORWARD -p ipv6-icmp -j ACCEPT
-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

Of course, net.ipv6.conf.*.forwarding was set on the host. That should be enough to forward all outgoing connections and drop incoming, right? And it does, for pretty much any host, except security.debian.org (AKA as lobos.debian.org and villa.debian.org). There may be more, but that one caught my attention, because apt update hung just there (ftp.de.debian.org worked, btw).

First I thought that it was the MTU, but that was pretty much a red herring. After a while I realized that it was working when the FORWARD policy was ACCEPT, but of course that wasn’t a viable solution. So I dug deeper: Strangely enough, with the policy back to DROP and this rule:

-A FORWARD -d <VM-IPv6> -p tcp -m multiport \
   --sports 80,443 -j ACCEPT

it also worked, but this wasn’t enough:

-A FORWARD -s <VM-IPv6> -j ACCEPT

WTF? Fortunately I had a working virtual machine (also debian 8.6, same kernel), so I ended up comparing the IPv6-sysctl values (sysctl -a | grep ipv6).

The solution

As it turned out, the only difference was that the working virtual machine had net.ipv6.conf.*.forwarding enabled. So I added

net.ipv6.conf.all.forwarding=1
net.ipv6.conf.default.forwarding=1

to /etc/sysctl.conf of the failing virtual machine, rebooted and then it finally worked ™! I don’t have the slightest clue why this is necessary, though. The VM is the final receiver, the end of the chain, but certainly not a router! Maybe it’s a kernel bug, I don’t know… I’m just glad it works 🙂

Just calling sysctl -w doesn’t do it, btw. You have to take the interface down and up again to take effect, hence the reboot…

Updating check-mk

It’s actually surprisingly easy! Just download the latest .deb from here to the server. Then install it with:

# dpkg -i <latest.deb>

This by itself does nothing. It just installs the new version in parallel to the old one. All instances must be updated separately with these commands:

# su - <instance_user>
$ omd stop
$ omd update
$ omd start

Now check for new/missing/vanished services and update the agents (it’s not a must, though). Acknowledge all incompatibilities (also not a must) and you’re done!

How to crop and split a movie

Cropping

I just leaned about a very valuable feature of mplayer: you can graphically determine the crop region with -vf rectangle!

To do so, create a new config file with this:

RIGHT change_rectangle 2  10
LEFT  change_rectangle 2 -10
UP    change_rectangle 3 -10
DOWN  change_rectangle 3  10
KP6   change_rectangle 0  10
KP4   change_rectangle 0 -10
KP8   change_rectangle 1  10
KP2   change_rectangle 1 -10

Then view the movie with

$ mplayer -vf rectangle -input conf=</path/to/conf> <movie_file>

You’ll see a white rectangle in the view area. Change the size with the keypad and the position with the cursor keys. The keypad down key enlarges the height, keypad down reduces it. Keypad left reduces the width, keypad right enlarges it.

Once you’re done, quit mplayer and use the rectangle geometry as crop parameter for ffmpeg:

$ ffmpeg -vf crop=<rectangle_data> ...

Splitting

Splitting isn’t as easy as it seems. You need 2 parameters:

  • -ss hh:mm:ss
  • -t hh:mm:ss

The latter is not a position in the file, but a duration! So, if you want to cut out everything from position 00:33:42 to 00:46:43, use -ss 00:33:42 -t 00:13:01 (33:42 + 13:02 = 46:43).

Also, -ss is a positional parameter. Use it as an input parameter, i.e. before -i if you don’t want silence and a black screen up front!

Example

Split out 13:02 minutes from position 33:42:

$ mplayer -ss 00:33:42 -t 00:13:02 -i <source> -acodec copy -vcodec copy out.file

Use the rectangle feature:

$ mplayer -vf rectangle -input conf=</path/to/conf> in.file

Reencode the split movie to mkv with the rectangle data:

$ ffmpeg -i <in.file> -acodec copy -vcodec libx264 -preset slow -threads 0 -x264opts fast_pskip=0:crf=21 <out.mkv>

Windows 10 Update

I had another fight with my Windows 10 VM after realizing that it didn’t get all the updates my other Win10 box got. Turns out I was stuck at Build No. 10240.

Why? Because the VM had the wrong Processor! <sarcasm attr=”biting”>Obvious, isn’t it?</sarcasm>

After Downloading the installer Windows10Upgrade28084.exe (WinVer 1607, build 14393) for a manual upgrade, it failed with a blue screen:

SYSTEM_THREAD_EXCEPTION_NOT_HANDLED

Quite self-explanatory, right? Well, all you have to do is to set the virtual CPU type to core2duo and limit the VM to 1 socket. Silly me! Why didn’t I think of that in the first place? It’s so obvious!

Once the upgrade is installed, you can change it back to anything you want. Thank you so much, Microsoft!

This update orgy is getting worse every time. ‘Nuff said 🙁

Remote git from Windows

To push your latest code ejaculations to a remote repository, e.g. for making it available via cgit:

Create a bare remote repository:
$ cd /path/to/repository
$ git init --bare
Add the remote repository
c:\repo> git remote add <name> ssh://server.tld.domain:/path/to/repository
c:\repo> git push --set-upstream <name> <branch>

You must have write permissions on /path/to/repository, and you have enter the password manually, so do it from cmd.exe. I guess you could setup a windows SSH agent, but I don’t trust it. Who knows what it sends where…

And then…

… it’s just as easy as:

c:\repo> git push

That’s all, folks!

Printing troubles

In a painful, tedious quest to make my OKI B431dn actually print from a Windows VM I learned several things:

  1. First and foremost: It really, really helps if your printer doesn’t share the IPv4 address with your TV (even if it’s turned off!)
  2. Thinking that you can get the IPv6-stacks on embedded devices such as said printer to work is just wishful thinking
  3. That I (fortunately) didn’t set an admin password for my printer
  4. That my SAMSUNG TV is still online even on standby

To elaborate: My quest started, because I wanted my Windows 10 VM to print. Easy enough, you’d think, but nothing is as easy as it seems 🙁

Adventure Levels:
  1. Fight with cups and Windows and encryption (http vs. https). That was a red herring.
  2. Fight with Samba, shared printers and Windows: another red herring
  3. Fight with different drivers or PPDs
  4. Find out that printing via localhost cups is also painstakingly slow
  5. Eventually figure out that the printer shares the IP with my TV
Solution:
  • Change the IPv4-adress of the printer, turn off IPv6 and only use the (now unique) IPv4-adress.
  • Use the URLs provided by the printer web page
Remarks:

Still don’t know why printing via IPv6 didn’t work as it should, because the printer’s IPv6-address was pretty unique, but what do I know… Anyway, after applying the solution using the generic cups postscript driver and the installed windows postscript driver, printing started after seconds instead of minutes, so problem solved 🙂

Adding a custom icon to an application – Windows 10

Wow, it just took me the better part of an hour to figure out how to pin a custom program to the taskbar with a custom icon! If the .exe doesn’t contain an application icon, the task bar icon is replaced by the default icon on exit.

Well, my program does contain an icon (it’s a QT App), but Winblows doesn’t care. Maybe because it’s a PNG instead of ICO, but who knows.

The Würgaround: create a shortcut for the executable, then right-click for properties and voila, there’s a “Change icon…” button! Change the icon to whatever, start the App via the shortcut and pin it to the taskbar!

This is wrong in so, so many ways… Don’t even get me started 🙁

Dusting off the Array! (Part 3)

And the story continues… The spare drive I bought on 2016/06/27 was defective as well. As it turned out, it wasn’t even new! The Seagate Warranty Check said: “Out of Warranty” 🙁

Z1F142XH-2

I contacted Amazon and they immediately forwarded my request to the retailer (2016/09/03 4:44pm). Let’s what happens…

I ordered a new drive on 2016/08/27 6:50pm, this time a Hitachi 4TB drive (HGST 0S03665 4TB Deskstar), but I made a mistake: I chose a Packstation as delivery address, even though I don’t have an account (yet), so the parcel was returned to sender (Amazon). At first I couldn’t make sense of the delivery status: Amazon said that the parcel was successfully delivered, but DHL said that it had been returned to sender. A short phone call cleared things up: The drive was indeed returned and I received a credit note (2016/09/02 about 1:40pm).

Later that day I ordered another Hitachi 4TB drive with the same retailer which arrived early next day (2016/09/03 about 9:00am). Unfortunately there wasn’t much time to waste: I had to fail the spare drive hard, because it hung the SATA bus during rebuild:

# mdadm --manage /dev/md1 --fail /dev/sdi

At first I thought that munin -> smartctl -a caused the hangs, but disabling it didn’t help.

While replacing the failed drive I burnt my fingers from the heat, so I set the fan to maximum when I turned Hadante on again. Rebuild is 42% done, still 11 hours to go  as of 2016/09/03 5:25pm. No issues yet, keeping my fingers crossed 🙂

Anyway, this is a photo of the anti-static bag the Hitachi drive came in (SN: P4HU95KB):

P4HU95KB

(Update 2016/09/04 06:56AM): Yeah! The rebuild is done! Hopefully safe again! The obnam LV shut down due to xfs errors, but that’s something I can live with. Maybe it’s the aftermath for force-assembling the array…

Part 1
Part 2
Part 4

Dusting off the Array! (Part 2)

Craptastic^2! Another drive failed as of Thursday morning during backup (2016/08/25). The box hung hard, the SATA bus was completely b0rked, so the process list was filling up with defunct smartctl commands, driving the load towards 100…

OK, no problem, one hard reset later the array was rebuilding. So far, so good, but during the next backup the array failed again, which was kinda expected. In hindsight I should have disabled the job, though. Anyway, Friday morning the box was locked up hard again. Poweroff hung at unmounting the array, no progress at all, so I just turned it off.

Friday afternoon I replaced the failed disk, booted up and was in deep shit! mdadm told me that it cannot start a dirty degraded array. FUCK! There goes my data, I thought… But Google came to rescue!

Fortunately mdadm allows you to force-assemble a dirty, degraded array with:

# mdadm --assemble --force /dev/md1 /dev/sd[ghj] missing

Or so I thought. That command exited with an I/O-Error, because the drives were for busy for some reason.

# cat /sys/block/md1/md/array_state  
inactive

As turned out, inactive is kinda still active. You have to stop the array first to get it working again:

# mdadm -S /dev/md1

Only then it can be force-assembled with the aforementioned command. Once it’s up and running (degraded), add the new disk:

# mdadm --manage --add /dev/md1 /dev/sdi

Now it should be rebuilding. Cross your fingers and pray to whatever god you worship 🙂 Of course the array was shut down Saturday morning, because I still didn’t disable the backup job, but this time it shut down cleanly. One reboot later the rebuild continued…

I guess I was very, very, very lucky: As far as I can tell there was mostly read access up to the 2nd failure (backup). The file systems (all XFS) mounted after recovering from the transaction logs, and the data seems to be OK, but I’ll see…

Lessons learned

  • Always shut down the array cleanly at the first sign of trouble! Don’t wait until the drive fails completely!
  • Don’t think that the failing drive will recover during rebuild. It won’t! It’ll only make things worse.
  • SEAGATE Barracuda drives, esp. ST3000DM001, are, to put it mildly, crap! I didn’t keep track of the history, but I think I replaced each of them at least once. So I ordered a  HGST 0S03665 Deskstar NAS 4TB 6Gb/s SATA as replacement instead of the cheaper (and smaller) SEAGATE drive. Let’s see how that turns out…
  • An inactive array can still be busy, e.g. active and has to be stopped before you can force anything…
  • Keep an up-to-date list of drives, their serials and position in the external SATA casing, so you don’t have to guess which drive failed!

Update (2016/08/27 5:23pm): Fuck SEAGATE! Once again a supposedly new drive almost failed me! At 99.9% rebuild the array shut down and I had to reboot, due to:

Aug 27 16:43:50 hadante kernel: ata5.02: exception Emask 0x100 SAct 0x7fffbfff SErr 0x0 action 0x6 frozen 
Aug 27 16:43:50 hadante kernel: ata5.02: failed command: WRITE FPDMA QUEUED 
Aug 27 16:43:50 hadante kernel: ata5.02: cmd 61/40:00:a0:9b:71/05:00:5c:01:00/40 tag 0 ncq 688128 out 
                                         res 40/00:ff:00:00:00/00:00:00:00:00/40 Emask 0x4 (timeout) 
Aug 27 16:43:50 hadante kernel: ata5.02: status: { DRDY }

After the reboot, the array rebuilt successfully, though. I’ll replace the failing (new) drive with the HITACHI when it arrives, and if that works, I’ll replace all drives, I think…

Part 1
Part 3
Part 4