Puppet works hard to make sure nodes are in compliance

I’ve been working with puppet a lot in the last few weeks. I really enjoy this style of administration compared to “ssh in a for loop”. Its great to script things and I still do it but for maintaining configuration I don’t know if it gets much better than puppet. That being said I ran into a few issues today that had me splitting my head open on my desk, keyboard, pop cans and just about everything else I could reach.

It’s standard practice to keep your puppet configuration in version control and I am doing this with subversion. It provides a central authority with log history so we can audit why changes were made. It’s also handy for my co-workers to be able to use tortisesvn on their windows clients to update configuration. Be careful with that! I do all of my editing in vim and neglected to take into account what using various editors might do to my puppet configuration. Today as we were testing some puppet configurations and I was showing co-workers how to add new nodes to the puppet cluster this bit me right in the soft bits.

My co-worker added a new node by copy and pasting a previous node definition and updating the node name. Of course he did this with notepad in windows. We noticed when we ran puppetd –test on the new node that it failed with a parser error. It complained about a missing } at line one of our nodes.pp. This was very odd since he was editing the bottom of the file. I opened the file in vim and everything looked right. I could find no such missing curly braces. I went ahead and removed his node definition and ran puppetd –test again. Not surprisingly the node complained of having no definition (I did not have a default node setup). I re-added the node definition and again the node was successful. I initially chalked this up to some error I must have missed in the node definition. But when my co-worker added more nodes and we saw the same behavior it was obvious there was a different issue going on. Now you would think this would be an easy conclusion to draw but that was not the case for me, at least it wasn’t today.

I noticed that after he added a node if I called in with an already configured node that node would complain about the same parse error. However if I called in again it would receive its configuration just fine and it would say it was ignoring the cache. I saw nothing exceptional in the puppetmaster logs. Of course the same parser error was there the first time a node called in to get its configuration after he altered it with a windows editor but all subsequent call ins of previously configured nodes succeeded. After I noticed this pattern things started coming together. To complicate my troubleshooting I started trying to restart puppetmasterd. This at first sent me for a tail spin. After puppet master was restarted with the file that had been updated with notepad on windows ALL node call ins were failing. No longer was it only the first time. Eventually this lead me to the conclusion. Puppetmasterd works very hard to make sure that your clients get served a valid configuration. If you update the config and do not restart puppetmasterd and that new config is invalid puppetmasterd appears to serve up the previous version of the config that it knew worked. This was all well and dandy and probably a desirable behavior. My issue is that the logs didn’t appear to say “Hey by the way I see you jacked up your puppet config. This config wont work dummy, Im going to go ahead and serve up your last config since it works.”. I saw no mention of this behavior in the docs but perhaps I missed something.

The moral of the story. Don’t use windows to edit your puppet configs and if you do use vim for windows. Secondary to that test! test! and test some more. Its pretty easy to screw up some syntax and have things start breaking on you.


  • I’m hopefully going to get some time to start dabbling in system configuration management soon. What made you decide on Puppet vs Chef and BCFG2? How is the documentation for Puppet?

  • I had not heard of Chef so I’ll be reading up on that shortly. I tried bcfg2 for a bit. There were things I liked about it but I really don’t like writing xml and iirc the config was xml based. I did really like that it was python and I would have had a better shot at extending it since I already dabble with python (not that I can’t do ruby, I just prefer python).
    Puppet does have a fairly decent sized community which I think is somewhat important and there are a lot of recipies out floating around to help you get started. Chef looks a lot like puppet so Ill have to read up on it.

    As for puppet documentation. Overall its pretty good. Personally I find it somewhat difficult to navigate their wiki but I just attribute that to poor searching on my part. Once I figure out the right term I tend to find the information pretty quickly.

  • damm Mac OS X Firefox 3.5.1 wrote:

    Truthfully, I’ve been using chef for several months now and I find the documentation; the examples; and the support to be far superior to anything puppet ever provided.

    Not to mention chef scales a heck of alot easier then puppet ever did. With Chef 0.8 up and coming in early August you may want to re-evaluate your time… as well as not use ‘notepad’…

  • Chef looks nice. However for my environment I think the knowledge of requiring ruby knowledge would not go over well. I’m still drawn to bcfg2 but for now I just can’t seem to get over the xml based configs. I’d love to see a good writeup on chef though if you have done anything. I have a feeling that Chef would be a better fit where you have more developer types taking care of systems but maybe thats just a mis-conception on my part.

  • This webcast was an interesting watch: http://www.ducea.com/2009/06/30/osbridge-configuration-management-panel/. BCFG2, Chef, Puppet, and CFEngine were all present for a panel discussion. Before watching it, you should know that Chef was written by a long-time Puppet user who never contributed much back to Puppet, so there’s some “tension” there.

    At one point, the panel speaker asks, who comprises the majority of your user base? Sysadmins was the answer for Puppet, and Devs was the answer for Chef. One point for Puppet. Google uses Puppet, which is another point for Puppet.

    However, the author of Puppet talks about how there’s code in there that is still around from when he didn’t know Ruby so well, which makes me a bit nervous. Looking at some of my perl programs from when I was first learning it makes me shudder.

    The one thing that I think is a huge positive is that Chef makes you learn a bit of Ruby to do the configuration. Puppet has it’s own DSL, but you can’t use that knowledge anyplace else except for within Puppet. What you learn about Chef’s “language”, you can use later for scripts, Rails, whatever.

    In the end, I think I’ll go with Puppet, but I hope that the author has his eyes open and incorporates some of the things that Chef does well into future releases.

    Good discussion!

  • Thanks for the link, that sounds interesting. Im all for being able to re-use knowledge and Im sure that learning Chef would help with ruby things in the future but Im more of a python kind of guy. There are definitely some things with puppet that could improve but personally I am not nervous about some old code that was written when Luke was still learning Ruby. Everyone has that, there are deployments that I did in the past that still work well but I cringe at the thought of doing it that way again. I suppose to me its a good sign, the developers skills are improving and they are critiquing their own code.

  • Zed Mac OS X Firefox 3.5.2pre wrote:

    FYI, notepad is the worst editor ever (use notepad2, notepad++ or even wordpad).

    The issue you were seeing is because of notepad’s poor handling of end of lines.

  • Indeed I am aware that notepad is horrible. I was just surprised because I did not see any ^M characters on any of the lines.

  • Mark Carey Windows Vista Firefox 3.5.2 wrote:

    As you’ve seen there are other things than bad line endings that can happen when using “bad” windows editors on files that will be parsed in various linux daemons. The most notorious I’ve encountered is where garbage meta characters are added to the first line of a php file. This causes php to fail to parse the file and returns an obscure error. Unfortunately vi/vim does not show the meta chars. This is where a quick check in a tool like od is handy.

  • Thanks for the comment Mark.

    You learn something new every day. I had never used od before, it will be a handy tool to add to my collection.

    If someone wants a short rundown on the tool see this article http://www.linuxjournal.com/article/1326

  • Chuck Sharp Windows XP Firefox 3.5.2 wrote:

    Another simple way to check text file line terminations (what a horrible troubleshooting time-waster) is the file command. Here’s how it looks:

    $ echo -e ‘test\ntest\n’ > testme
    $ file testme
    testme: ASCII text
    $ unix2dos testme
    unix2dos: converting file testme to DOS format …
    $ file testme
    testme: ASCII text, with CRLF line terminators

  • “Before watching it, you should know that Chef was written by a long-time Puppet user who never contributed much back to Puppet, …”

    This is not true. I’m not sure what name to place on the course of events that has led anyone to believe this. Whatever would convey it, disappointing is a word I would associate.

    Theres a telling trail of crumbs here:

  • Puppet vs Chef is a debate which seems to provoke strong reactions (on both sides). Whatever the history betwen the developers, it’s the technical merits of each tool which will weigh most with people choosing their configuration management tool.

    I’ve written a summary of the respective merits (in my opinion, of course) of Puppet and Chef here:


Leave a Reply

Your email is never shared.Required fields are marked *

To submit your comment, click the image below where it asks you to...
Clickcha - The One-Click Captcha