check_http

The Name field in a check_http-rule is the name of the rule, not the host!

I tripped on this when I tried to check an URL and the certificate age of the same host. The certificate check failed with TCP Connection failed, since there already was a rule by that name. Not very helpful error message, BTW…

Upgrading check-mk and Debian

1. Overview

Upgrading Debian from 8 (jessie) to 9 (stretch) with check-mk installed isn’t as easy as it seems. You have to:

  1. Upgrade check-mk to 1.4.0 and fix all issues
  2. Backup all sites
  3. Purge check-mk
  4. Upgrade debian
  5. Reinstall check-mk 1.4.0
  6. Restore the check-mk-sites from backup

2. Upgrade check-mk and back it up

Download the .deb package and install it. Follow the official the official guide and upgrade all sites. After fixing all issues, create a backup of each site:

# su - <sitename>
$ omd backup site.name.tar.gz

Repeat this for all sites.

3. Remove check-mk and upgrade to stretch

Stop sites:

# su - <sitename>
$ omd stop

Repeat this for all sites. Then remove (purge) check-mk:

# dpkg -P check-mk-raw-1.4.0p17

Once this is done, update the distribution to debian (you really should know how to do that!). Autoremove all obsolete packages and reboot.

4. Reinstall check-mk and restore from backup

Download the .deb package for stretch and install it. Since you autoremoved dependent packages earlier, the install will most likely fail. Fix it with:

# apt --fix-broken install

Now we can restore the sites from our backup (as root!):

# omd restore <site-name.tar.gz>
# su - sitename
$ omd start

Repeat for all sites and fix all remaining issues.

5. Notes

Of course you don’t want to do this without a security net. Take a snapshot and destroy that instead of the real VM. How to do that with KVM and libvirt is explained here.

Updating check-mk

It’s actually surprisingly easy! Just download the latest .deb from here to the server. Then install it with:

# dpkg -i <latest.deb>

This by itself does nothing. It just installs the new version in parallel to the old one. All instances must be updated separately with these commands:

# su - <instance_user>
$ omd stop
$ omd update
$ omd start

Now check for new/missing/vanished services and update the agents (it’s not a must, though). Acknowledge all incompatibilities (also not a must) and you’re done!

How to get rid of Adobe Flash

How to remove

It’s not as easy as it seems. First of all, do not use the official Uninstaller from Adobe. I don’t know what it does other than showing a reverse progress bar, but it certainly does not remove the Flash-Crap from your computer.

To remove it for good, you have to jump through several hoops:

  1. Close every program using Flash, like IE (if you’re using this piece of shit) or Firefox. AFAIK Chrome has its own plugin, but it can’t hurt to close it, too
  2. Open the Explorer and go to C:\Windows\System32. Select the Macromed-Folder and take ownership. Then grant yourself full access including all subdirectories.
  3. Delete the Folder
  4. Do the same in C:\Windows\SysWOW64.
  5. Take ownership of C:\Windows\SysWOW64\FlashPlayerApp.exe and C:\Windows\SysWOW64\FlashPlayerCPLApp.cpl, grant yourself full access and delete the files
  6. Go to %appdata% and remove the Macromedia\Flash directory
  7. Reboot (surprise!)
  8. Check that the Flash-Crap is gone from the control panel
  9. Open this link in Edge or IE to see if Flash is gone for good
  10. If you’re using Chrome, disable the plugin in about:plugins

What else can I do?

If you’re monitoring your Windows-Box with check_mk, you can use these scripts to check for Flash.

Put this into C:\Program Files (x86)\check_mk\local\check_flash.ps1 or wherever your check_mk agent is installed

$items = "C:\Windows\System32\Macromed",`
"C:\Windows\SysWOW64\Macromed",`
"C:\Windows\SysWOW64\FlashPlayerApp.exe",`
"C:\Windows\SysWOW64\FlashPlayerCPLApp.cpl"

echo "<<<flash_gordon>>>"

foreach ($i in $items){
 if(Test-Path $i){
  echo ("{0} 1" -f $i)
 }else{
  echo ("{0} 0" -f $i)
 }
}

Now put this script into ~siteuser/local/share/check_mk/checks/flash_gordon:

#!/usr/bin/python 
 
def inventory_flash_gordon(info): 
    inventory = [] 
    for line in info: 
        inventory.append((line[0], line[1])) 
 
    return inventory 
 
def check_flash_gordon(items, params, info): 
    retstring = "Good, no Flash-Crap found!" 
    found = [] 
    retval = 0 
    for line in info: 
        if int(line[1]) > 0: 
            found.append(line[0]) 
 
    if found: 
        retstring = "Crap, found %s" % ', '.join(found) 
        retval = 2 
 
    return (retval, retstring) 
 
check_info["flash_gordon"] = { 
    "check_function":   check_flash_gordon, 
    "inventory_function":       inventory_flash_gordon, 
    "service_description":      "Check for Adobe Flash Crap", 
    "has_perfdata":             False, 
}

Reinventorize. Now the host goes CRITICAL if any part of Flash is found.

[Update 2015-12-09]

I hate this crap! Windows Update decided that I need a security update for Flash, even though it isn’t installed any more. What the fuck? Of course the update fails, but in their wisdom they decided to remove the option to hide updates from the Windows 10 dialog. Now you have to use this program 🙁

Anyway, I only noticed it because of check_mk. At least now I know it works 🙂 Let’s see when this bites my ass…

[Update 2015-12-30]

What a clusterfuck! Another Windows update for flash. Thanks to check_mk I noticed it right away. Shouldn’t be too difficult to check for this crap before installing the update, right?

[Update 2016-11-10]

Finally got tired of removing this crap manually again and again! Since you can’t prevent Windows Update from installing this shit, I wrote a script to put it where it belongs. You need the Powershell Community Extensions (PSCX) for it to work. Install then, download the txt, remove the txt-Extension and replace the owner with whatever is appropriate.

Have fun!

Check_MK: Send notifications to contact group

This is kinda convoluted and not very intuitive.

  1. Create a contact group: WATO → Contact Groups → New Contact group
  2. Create a User with the E-Mail-Address you want the notifications sent to: WATO → Users → New Users and add him to the contact group! The tab may be collapsed by default, so it’s easily overlooked. It’s the third panel from the top, right under “Security”
  3. Create a notification rule: WATO → Notifications → New Rule. Then add the newly created contact group. Expand the tab “Contact Selection” and check “The members of certain contact group”. Next, click “Add Element” and select the newly created contact group.
  4. Save and activate changes.

Custom Checks with check_mk_agent

1. Install xinetd and check_mk_agent

Install xinetd first. The check_mk_agent package drops a configuration snippet into /etc/xinetd.d/, which makes xinetd listen on port 6556 for the agent.

2. Write the custom check (Service to check)

The server-side check can be written in any language, as long as it can print to STDOUT. The output format is quite simple:

<<<check_name>>>
data1 data2 ... dataN
data1 data2 ... dataN
...

check_name is the identifier. It can be anything, but it has to be enclosed in <<<>>>. Data[1-] are the things you want to check, including performance counters, anything you like. Server-side the data will be split at white-spaces.

Simple example: check if bacula-director and bacula-storage-daemon is running (in perl):

#!/usr/bin/perl -w 
 
use strict; 
 
my @instances = `ps o cmd ax`; 
my %count; 
foreach my $i (@instances){ 
        if($i =~ m#bacula-sd#){ 
                $count{"storage_daemon"}++; 
        } 
        if($i =~ m#bacula-dir#){ 
                $count{"director"}++; 
        } 
} 
 
print "<<<bacula_system>>>\n"; 
foreach my $k (keys %count){ 
        print "$k $count{$k}\n"; 
}

Output:

<<<bacula_system>>> 
storage_daemon 1 
director 1

One storage daemon process and one director process running. Perfect! Drop this script in /usr/lib/check_mk/plugins and make it executable.

3. The check_mk-Server-Part

Now we have to create the server-side counterpart. This has to be in python, because check_mk uses introspection extensively. Create a script named exactly as the identifier in local/share/check_mk/checks starting in the home directory of your check_mk-site-user. In the example given: local/share/check_mk/checks/bacula_system

The first thing we need is the dictionary check_info[] with at least these entries:

  • check_function: Name of the function actually performing the check
  • inventory_function: Name of the inventory function (called by WATO, the Web Administration)
  • service_description: Name of the service, can be a format string

Example:

#!/usr/bin/python 
 
def inventory_bacula_instances(info): 
    inventory = [] 
    inventory.append( ("storage_daemon", None) ) 
    inventory.append( ("director", None) ) 
    return inventory 
 
def check_bacula_instances(item, params, info): 
    count = 0 
    for each in info: 
        if each[0] == item: 
            count = int(each[1]) 
 
    retval = 3 
    perfdata = [ (item, count) ] 
    if count == 0: 
       retstring = "No instance found for %s" % item 
       return (2, retstring, perfdata) 
    else: 
        retstring = "%d instances found for %s" % (int(count), item) 
        return (0, retstring, perfdata) 
 
    return (3, "Checking %s failed" % item, perfdata) 
 
check_info["bacula_system"] = { 
    "check_function":       check_bacula_instances, 
    "inventory_function":   inventory_bacula_instances, 
    "service_description":  'Bacula: %s', 
    "has_perfdata":         True, 
}

3. The inventory function

The inventory function returns an array of Tuples with the items to check, in our case the items storage_daemon and directory returned by our perl script. The Tuples are fed to the check_function, in our case check_bacula_instances(item, params, info).

4. The check_function

The check_function does the actual work. The return values are:

  • 0: OK
  • 1: Warning
  • 2: Critical
  • 3: Unknown

The parameters:

  • item: The name of the item to check, i. e. the first item of the Tuple returned by the inventory_function. In our case “storage_daemon” or “director”
  • params: no idea (TODO)
  • info: a dictionary as key and an array with the values returned from the server-side script. In the example given something like this:
{'director': ['director', 1], 'storage_daemon': ['storage_daemon', 1]}

So info[‘director’][1] gives you the count of director processes.

5. Configure the check

  1. Login to WATO, click WATO-Configuration->Hosts->New Host.
  2. Enter hostname
  3. Click Save & Test, check that the agent is available
  4. Click Save & Exit
  5. Click Save & go to Services
  6. Save and activate the new configuration
  7. Have FUN!

6. Debugging

Sprinkle the python script with “print <somethingorother>” and call

$ cmk <hostname>

from the command line.

Ich habe fertig 🙂