Don’t script it, just make it so

Matt Simmons recently had another post that mentioned structured systems management. I find myself re-reading Ad-Hoc vs Structured Systems Management from time to time and I figure its time for me to chime in.

First off, Michael Jankes’ post is one of my favorite blog post reads. Few posts do I ever re-read unless its tutorialesque. For a quick sidetrack one of those posts that I do return to when I deal with developers is The Joel Test: 12 Steps to Better Code. Back on subject …

One of the most frequent things that comes up when people start talking about good administration practices is scripting things. If you can’t script it make a checklist is a common sentiment. It’s not that I don’t think that scripting things is a good idea because I do. I would go so far to say that scripting things is an important part of documentation. You should be able to reproduce a system from documentation. Any configuration that can be done programatically should be and it should be in your documentation so in case of catastrophic failure someone can reproduce your work relatively easily. If I like scripting so much why is this post titled “Don’t script it”?

I think there is a better way. Use a configuration management system. Declare your configuration and allow the configuration management system to make it happen. This has many advantages over simply scripting something. Configuration can be enforced over time. I know I have been guilty of making small changes to configurations to see how things go. While those small changes seem to trickle into production systems they never (or at least rarely) seem to trickle into the documentation. Before you know it you have a pile of undocumented small changes that really encompass an entirely different configuration from what you started with.

A well placed configuration management system helps avoid these issues. Store your configuration in version control. I might even go so far as to say to automate the update of the checkout your configuration management server uses so that you don’t allow yourself to make small changes to the management system configuration without checking it into your version control. A few tweaks being automatically removed will go a long way to drive home the point that your changes need to be in version control. Placing your config in version control also allows you to see your changes over time. You can look back through the log and your comments to see when and why changes were made. This can be especially important if you are not the only person making changes to configuration (ah hem Matt, I know your hiring someone you better version your config before your no longer the lone admin).

This all comes up because I have been doing more work with puppet recently. It takes a while to wrap your head around but I don’t really think I have heard any arguments against that type of system being the “right” way to do systems administration. If I have they sure didn’t register. I like puppet because it works on a variety of systems and abstracts things like package management. I just tell my configuration system to make sure nodeX has package x installed and the puppet client figures out if its RH or Debian or SUSE or Solaris or whatever and runs the proper commands to get the package installed.

I am interested in spacewalk as well. But right now it only runs on Oracle databases so until it runs on postgresql there really isn’t even a point in me looking as the places I would be able to use it would be very limited.

5 Comments

  • PB Linux Firefox 3.0.11 wrote:

    Did you happen to take a look at anything other than puppet? It’s not clicking for me like bcfg did when I took a look at that.

    I think that some big players are looking at puppet and it’s always important to at least consider that when making a long term decision.

  • I did look bcfg2 and cfengine. I didn’t like cfengine because it didn’t seem to provide the abstraction that puppet provided. bcfg2 I liked but I found the xml configs a bit obtuse. That combined with a few installation issues kind of turned me off of bcfg2. I really like that its python as opposed to ruby though. How large of an infrastructure are you managing with bcfg? Also this was not meant to be as much of a props for puppet as a general methodology. I think the most important thing is that you version your configs. It makes the most sense to store them and manage them centrally. There are a few systems that allow you to do this and I just find puppet to be the right fit for me? What did you like about bcfg compared to puppet?

  • Sol Jerome Linux Unknow wrote:

    Nick,

    The xml configuration in bcfg2 isn’t as bad as it seems. Most of the xml present in your configuration can be generated using tools which are shipped with Bcfg2. There are also advantages to using xml. You are able to easily verify your configuration using xmllint (which is nice since you don’t end up having to maintain your configuration checker). Also, having the configuration in xml means you can transport it seamlessly via the most popular transport protocol (http). IMO, the benefits far outweigh the fact that xml is used for the configuration.

  • A configuration management tool certainly would qualify as a ‘script’, not a ‘click’ in my world, and if it’s a decent tool, would have the additional properties of handling version control and maybe even configuration auditing.

    I also like the idea of storing the configuration of the configuration tool in version control.

    And ‘config creep’ is a problem on any system. One of the simplest ways that we’ve battled it (on Solaris) is to manage the apps in a way that permits re-installation from scratch easily and consistently. When we upgraded from Solaris 10 u4 to Solaris 10 u6, we re-installed each server OS from a golden image, re-applied any per-server customizations and re-installed the apps from a master copy in an SVN repository (the data is untouched, on separate LUN’s). We did not do any in-place upgrades. That method is faster and more consistent than an in-place upgrade.

    Can’t do that with Windows though.

  • Indeed, I never do OS upgrades on anything (as long as I have a say). I always do a fresh install and drop the needed data in place. The configuration management tools like puppet make it very consistent and easy to either stand up a new server or to push a firewall change either to a specific node or to a group of nodes.
    I also agree it would be almost pointless to not store your configuration managements tool outside of version control. You would never have a record of what has changed over time or why. Both of those aspects are very important especially when looking at configurations you have not touched in a while.

Leave a Reply

Your email is never shared.Required fields are marked *

To submit your comment, click the image below where it asks you to...
Clickcha - The One-Click Captcha