I love OpenVZ, I think its one of the easiest to use virtualisation technologies on the market and it adds almost no overhead compared to other technologies.

I’ve been using it for a couple of years now and I always wanted to have a nicer way to automate container creation, configuration or actions than writing shell scripts. There are already a couple of webinterfaces around, but none of them I liked.

Another possibility would be to use libvirt - but libvirt always felt a bit too complex, since its a general API implementation for several hypervisors.

So I started to implement my own API, which should rather be a simple and minimalistic approach. The project is hosted on GitHub but you can as well install it by Rubygems.

Installing

 $ gem install openvz

Example: Restart

 require 'rubygems'
 require 'openvz'

 container = OpenVZ::Container.new('109')
 container.restart

Example: Provisioning

Here is the example of how a whole provisioning for a new container could look like:

 require 'rubygems'
 require 'openvz'

 container = OpenVZ::Container.new('110')

 container.create( :ostemplate => 'debain-6.0-boostrap',
                   :config     => 'vps.basic' )

 container.deboostrap( :dist   => 'squeeze',
                       :mirror => 'http://cdn.debian.net/debian' )

 container.set( :nameserver => '8.8.8.8',
                :ipadd      => '10.0.0.2',
                :hostname   => 'foo.ono.at' )

 container.start

 # Update the system
 container.command('aptitude update ; aptitude -y upgrade ; apt-key update')

 # Install puppet
 container.command('aptitude -o Aptitude::Cmdline::ignore-trust-violations=true -y install puppet')

 # Install puppet
 container.command('puppetd -t --server=puppet.ono.at')

The first version is basically able to replace the provisioning scripts I’ve been using for a while. In future versions I’d probably like to integrate this into mcollective and puppet to orchestrate my machines. :-)

Lately we bought a new HP blade chassis to replace a customer’s old database server. All it’s services run on ~15 blades, splitted cross two HP C7000 chassis.

The Proliant BL460 G6 we bought came with much newer firmware revisions than all the existing G1 – part of the infrastructure didn’t receive much sysadmin love over quite some time. :-)

Blades, ILO, chassis and controllers where all running way outdated firmware and upgrading was highly recommended. The arising firmware combinations haven’t been tested and the new blade wouldn’t even be detected, so HP. They offered us an upgrade for about $2000 and 6 hours of downtime per chassis.

Here are some handsome findings, to do the upgrade on your own:

HP Firmware Comapatibility Matrix

HP tested certain sets of firmware for compatibility. Take a look at their compatibility matrix and try to stay within the tested boundaries. This could mean to upgrade in more than one step, if you are running an older release.

(http://h18004.www1.hp.com/products/blades/components/c-class.html)

Hp-Firmware-Catalog

There is Christian Hofstedtlers great firmware upgrade script, which automatically downloads the latest and greatest HP firmware installation packages. Its even creating softlinks, to reference cryptic firmware package names to their corresponding hardware components.

(https://github.com/zeha/hp-firmware-catalog)

You can run them from your OS as an online upgrade. Certain components still might require rebooting, to finish the “delayed upgrade”.

I would love to see HP maintaining this, since the approach provides a good example of providing customers with a modern and automated way to upgrade and monitor firmware for more recent releases.

ILO Shell

When upgrading many machines it will save you a lot of time, if you just use the SSH shell for configuring a boot device and rebooting the server.

Connect to ILO by SSH

Make sure you send the right username, AFAIK it’s case sensitive on the ILO:

     ssh phx-vnode03.oob.ono.at -l Administrator
   

Set an ILO Advanced Licence key

     cd /map1
     set license=YOUR-LICENCE-KEY
   

The advanced licence key is required to enable virtual device firmware features. Eg. to make use of the remote console or a virtual disk boot drive.

Mount and configure a network hosted ISO image as boot device

     cd /map1
     vm cdrom insert http://10.0.10.21/FW920B.2010_1129.2.iso
     vm cdrom set boot_always
   

…be it a firmware upgrade or an OS installation disk. Make sure you run the following command to “eject” it again:

     cd /map1
     vm cdrom eject
   

Monitoring

To please your monitoring system as well, check out check_mk. They wrote a couple of good SNMP checks for your HP or IBM bladecenter.

In the end I can highly recommend to keep your hardware firmware up to date. At least HP, my vendor of choice, they add a lot of useful bug fixes.

HP currently informs customers by a e-mail newsletter about updates, I would love to see this in my monitoring system too, like all the other security upgrades.

Try to plan the upgrade a bit or use existing downtimes to boot the HP Firmware Maintenance image.

Recently we decided upon a new host naming convention for our infrastructure at my $dayjob. So I will soon be renaming a few hosts.

I noticed that munin still doesn’t provide an easy way, to rename an existing node without losing historical data. So I wrote a small shell script which does that for me and might be of use to everybody else in the same situation.

Puppet really shines at automating infrastructures. You will notice a sudden change of working methodology, once you manage the first systems with it.

Instead of manually logging on to each single system for updating a certain part of configuration by issuing shell commands, you will stop to repeat yourself and just update a single piece of code, which describes the desired config state for all systems.

As recommended in the Puppet documentation you are well advised to keep your Puppet manifests under revision control.

I wrote a small script which will come in handy, to ease your life with keeping your repository and the manifests on the master in sync and should fit to most of the environments out there.

Once installed, you can store the manifests for each Puppet environment in its own GIT branch and every time you commit a new version to one of your branches, it will automatically sync the most recent version and inform the Puppet master process.

BTW. this could also be used to keep the manifests on multiple Puppet instances in sync.

Puppet-Sync

Puppet-sync is a Ruby based command line tool to synchronize every commit from a central GIT repository to your Puppet master instance. You should install it on your Puppet master and configure a GIT hook which calls the script over ssh.

Puppet-sync takes some parameters to specify how the environment on the master looks like. Run it with ’–help’ to get a list of available options. Here is an example:

puppet-sync --branch master \
            --passenger     \
            --destination /etc/puppet/environments \
            --repository ssh+git://git.ono.at/srv/git/puppet.git

By running the command above, the script connects to the git repository, fetches the manifests from the master branch and puts it into /etc/puppet/environments/production. Since we use the master branch for production, I added the logic to translate “master”-branch to “production”-environment.

Installation on the master

Instead of listing each and every shell command needed to install the environment for the script, I’d like to provide a simple manifest instead. I think this is more readable and you can either use it or figure out the appropriate shell commands. ;-)

file { "/usr/local/bin/pupept-sync":
    ensure  => present,
    source  => "file:///puppet-sync",
}

file { "/home/psync/.ssh":
    ensure  => directory,
    owner   => "psync",
    mode    => 700,
    require => User["psync"],
}

file { "/etc/puppet/environments":
    ensure  => directory,
    owner   => "psync",
    mode    => 775,
    require => User["psync"],
}

user { "psync":
    ensure     => present,
    home       => "/home/psync"
    managehome => true,
}

ssh_authorized_key { "puppet-sync-ssh-key":
    ensure  => present,
    key     => "AAAAB3.....lVBp0nPLNcs=",
    type    => "ssh-rsa"
    user    => "psync",
    require => File["${homeroot}/$name/.ssh"],
}

I have the following configuration in my puppet.conf to make puppet aware of each of the directories in /etc/environments:

...
[master]
.....
templatedir = /etc/puppet/environments/$environment/
modulepath  = /etc/puppet/environments/$environment/modules/
manifest    = /etc/puppet/environments/$environment/manifests/site.pp
manifestdir = /etc/puppet/environments/$environment/manifests

Git Hook

The only thing left is to create a Git Hook in your repository. Here is the one i use. I also created a psync user on the master, so i just need to store the private ssh key in psync’s home directory.

#!/bin/sh
#
# An example hook script to prepare a packed repository for use over
# dumb transports.
#
# To enable this hook, make this file executable by "chmod +x post-update".

branch=`echo $1 | awk -F/ {'print $3'}`

sudo -u psync ssh puppet.ono.at /usr/local/bin/puppet-sync \
                                       --passenger         \
                                       --branch $branch

exec git-update-server-info

Perfect its installed and you are ready to use it. Go ahead an try to commit a new version.

Rescently I blogged about my CheckMK APT plugin, capable of checking for upgradeable packages.

Meanwhile I wrote two more plugins, both adopting existing checks and migrated all plugins into one single ‘checkmk’ repository over at GitHub.

bacula

This check adapts the idea of Bernhard Miklautz bacula-utils. The plugin will define four different services to be monitored on each Bacula server:

• bacula.errorvols

  Checks the state of available backup volumes and will report critical, if errounos volumes are found. In case of an error, it will as well return the names of these volumes.

  Use the bacula-clear-errvols script to resolve these issues.

• bacula.freshness

  Checks whether all clients have an associated backup within the last 30 hours.

• bacula.fullbackups

  Checks whether all clients have an associated full-backup.

• bacula.fullbackupspool

  Checks whether any volumes used for full-backups come from a full-backup pool.

puppet

The second plugin will monitor the status of the puppet agent. It integrates nicely with puppetstatus.py. This script was initially written by TMZ from the Fedora Infrastructure Team and can be used to enable or disable the puppet agent on a specific host.

You are able to disable Puppet on a server, by running the following command:

 sudo puppetstatus -d "Interesting things are about to happen."

The monitoring system will change the state of the Puppet check to WARNING and will display this message and the username of the person who was disabling the agent.

When the agent is not disabled on a host, the check will just change to WARNING after 3 and to CRITICAL after 4 hours.

I’d be glad to get some feedback on my plugins, please report any bugs by sending me an e-mail or leave a comment below.