Testing puppet with Jenkins before deploying

During a rather boring conference a few weeks back, I decided to convert my own infrastructure from 'standalone puppet' (that is, a set of standalone puppet manifests that were executed by a basic shell script on each server I managed) to the 'client -> server' or 'puppetmaster' model (whereby a central puppet daemon controls the manifests, and servers connect to it for updates as 'clients'.)

You can read more about the different models here.

It actually didn't take long to convert my Puppet manifests to being puppetmaster-friendly: mainly it was a matter of rejigging the directories a bit, and also changing file definitions using the 'source' attribute from having an absolute path to the file (/opt/puppet/puppet/modules/foobar/files/somefile) to the puppet:/// syntax.

Couple this with cafuego's useful check_puppetmaster Nagios command to complain if clients haven't checked in with the puppetmaster for a while, and I had a pretty damn good setup going on!

However, I had overlooked one element: I tend to make a lot of typos.

In the past, this came to my attention pretty quick: running the 'standalone puppet' shell script on each server would fail pretty miserably if I had left out a curly bracket, or a comma, or missed an apostrophe. Even when I grew up a bit and ran this 'standalone' script in bulk via Jenkins, having it poll my puppet git repo, the matter would be brought to my attention pretty quick.

With puppetmaster, however, I was basically keeping all of /etc/puppet in a git repo, and had a 5 minute cronjob coming around and doing a git pull to fetch updates. The problem was, there was no testing phase occuring here: my changes were going live almost immediately, and the clients were
sporadically checking in and trying to run those changes.

To make matters worse, the fact that there were errors was almost a silent affair: I had to less my logs on the puppetmaster to see that there were syntax problems blocking those clients from updating. Time to go back to Jenkins!

I wrote a simple puppet-test script that uses the command puppet --parseonly to check for explicit syntax problems. I also use a combination of erb and ruby commands that a fellow sysadmin provided me, to parse the templates for any errors.

Enter Jenkins again:

1. I have an intermediate 'puppet-test' Jenkins job that polls the git repo for changes I push up.
2. It runs this puppet-test command on the repo
3. If it fails, I get slapped in the face
4. If it succeeds, it invokes a 'child' job in Jenkins called 'puppet-pull' which actually goes ahead and pulls down the new changes over on the puppetmaster server itself (the puppetmaster is a slave node in Jenkins, reachable over SSH)

Token diagram explaining how it works - apparently when Bieber is the one applying changes to your puppet manifests. Thanks yEd diagram editor!

It sounds simple, and it is; I'm not trying to pretend I've done anything clever here. It's just a reminder that it is all too easy to sidestep the testing process and get yourself in trouble. Whenever you're planning for change in your infrastructure, be it a small change or an overhaul of an entire system like puppet-standalone to puppetmaster, remember to factor in the importance of continuous testing and integration: not one-time, but make it part of the regular cycle of change.

You can download the puppet-test script at github. It isn't perfect, and far too simplistic, but it's a work in progress. The next step is to incorporate puppet-lint for style guideline testing, but unfortunately I haven't got it to report 'Warnings' in STDOUT with the filename included properly yet.

How are you testing your puppet manifests and modules prior to deployment? If there's more elegant tools around for achieving this end than hacky little shell scripts, please do tell me :)

Tags: