Posts Tagged ‘Puppet’

I’ve been working with puppet a lot in the last few weeks. I really enjoy this style of administration compared to “ssh in a for loop”. Its great to script things and I still do it but for maintaining configuration I don’t know if it gets much better than puppet. That being said I ran into a few issues today that had me splitting my head open on my desk, keyboard, pop cans and just about everything else I could reach.

It’s standard practice to keep your puppet configuration in version control and I am doing this with subversion. It provides a central authority with log history so we can audit why changes were made. It’s also handy for my co-workers to be able to use tortisesvn on their windows clients to update configuration. Be careful with that! I do all of my editing in vim and neglected to take into account what using various editors might do to my puppet configuration. Today as we were testing some puppet configurations and I was showing co-workers how to add new nodes to the puppet cluster this bit me right in the soft bits.

My co-worker added a new node by copy and pasting a previous node definition and updating the node name. Of course he did this with notepad in windows. We noticed when we ran puppetd –test on the new node that it failed with a parser error. It complained about a missing } at line one of our nodes.pp. This was very odd since he was editing the bottom of the file. I opened the file in vim and everything looked right. I could find no such missing curly braces. I went ahead and removed his node definition and ran puppetd –test again. Not surprisingly the node complained of having no definition (I did not have a default node setup). I re-added the node definition and again the node was successful. I initially chalked this up to some error I must have missed in the node definition. But when my co-worker added more nodes and we saw the same behavior it was obvious there was a different issue going on. Now you would think this would be an easy conclusion to draw but that was not the case for me, at least it wasn’t today.

I noticed that after he added a node if I called in with an already configured node that node would complain about the same parse error. However if I called in again it would receive its configuration just fine and it would say it was ignoring the cache. I saw nothing exceptional in the puppetmaster logs. Of course the same parser error was there the first time a node called in to get its configuration after he altered it with a windows editor but all subsequent call ins of previously configured nodes succeeded. After I noticed this pattern things started coming together. To complicate my troubleshooting I started trying to restart puppetmasterd. This at first sent me for a tail spin. After puppet master was restarted with the file that had been updated with notepad on windows ALL node call ins were failing. No longer was it only the first time. Eventually this lead me to the conclusion. Puppetmasterd works very hard to make sure that your clients get served a valid configuration. If you update the config and do not restart puppetmasterd and that new config is invalid puppetmasterd appears to serve up the previous version of the config that it knew worked. This was all well and dandy and probably a desirable behavior. My issue is that the logs didn’t appear to say “Hey by the way I see you jacked up your puppet config. This config wont work dummy, Im going to go ahead and serve up your last config since it works.”. I saw no mention of this behavior in the docs but perhaps I missed something.

The moral of the story. Don’t use windows to edit your puppet configs and if you do use vim for windows. Secondary to that test! test! and test some more. Its pretty easy to screw up some syntax and have things start breaking on you.

Matt Simmons recently had another post that mentioned structured systems management. I find myself re-reading Ad-Hoc vs Structured Systems Management from time to time and I figure its time for me to chime in.

First off, Michael Jankes’ post is one of my favorite blog post reads. Few posts do I ever re-read unless its tutorialesque. For a quick sidetrack one of those posts that I do return to when I deal with developers is The Joel Test: 12 Steps to Better Code. Back on subject …

One of the most frequent things that comes up when people start talking about good administration practices is scripting things. If you can’t script it make a checklist is a common sentiment. It’s not that I don’t think that scripting things is a good idea because I do. I would go so far to say that scripting things is an important part of documentation. You should be able to reproduce a system from documentation. Any configuration that can be done programatically should be and it should be in your documentation so in case of catastrophic failure someone can reproduce your work relatively easily. If I like scripting so much why is this post titled “Don’t script it”?

I think there is a better way. Use a configuration management system. Declare your configuration and allow the configuration management system to make it happen. This has many advantages over simply scripting something. Configuration can be enforced over time. I know I have been guilty of making small changes to configurations to see how things go. While those small changes seem to trickle into production systems they never (or at least rarely) seem to trickle into the documentation. Before you know it you have a pile of undocumented small changes that really encompass an entirely different configuration from what you started with.

A well placed configuration management system helps avoid these issues. Store your configuration in version control. I might even go so far as to say to automate the update of the checkout your configuration management server uses so that you don’t allow yourself to make small changes to the management system configuration without checking it into your version control. A few tweaks being automatically removed will go a long way to drive home the point that your changes need to be in version control. Placing your config in version control also allows you to see your changes over time. You can look back through the log and your comments to see when and why changes were made. This can be especially important if you are not the only person making changes to configuration (ah hem Matt, I know your hiring someone you better version your config before your no longer the lone admin).

This all comes up because I have been doing more work with puppet recently. It takes a while to wrap your head around but I don’t really think I have heard any arguments against that type of system being the “right” way to do systems administration. If I have they sure didn’t register. I like puppet because it works on a variety of systems and abstracts things like package management. I just tell my configuration system to make sure nodeX has package x installed and the puppet client figures out if its RH or Debian or SUSE or Solaris or whatever and runs the proper commands to get the package installed.

I am interested in spacewalk as well. But right now it only runs on Oracle databases so until it runs on postgresql there really isn’t even a point in me looking as the places I would be able to use it would be very limited.