Puppet works hard to make sure nodes are in compliance

I’ve been work­ing with pup­pet a lot in the last few weeks. I really enjoy this style of admin­is­tra­tion com­pared to “ssh in a for loop”. Its great to script things and I still do it but for main­tain­ing con­fig­u­ra­tion I don’t know if it gets much bet­ter than pup­pet. That being said I ran into a few issues today that had me split­ting my head open on my desk, key­board, pop cans and just about every­thing else I could reach.

It’s stan­dard prac­tice to keep your pup­pet con­fig­u­ra­tion in ver­sion con­trol and I am doing this with sub­ver­sion. It pro­vides a cen­tral author­ity with log his­tory so we can audit why changes were made. It’s also handy for my co-workers to be able to use tor­tis­esvn on their win­dows clients to update con­fig­u­ra­tion. Be care­ful with that! I do all of my edit­ing in vim and neglected to take into account what using var­i­ous edi­tors might do to my pup­pet con­fig­u­ra­tion. Today as we were test­ing some pup­pet con­fig­u­ra­tions and I was show­ing co-workers how to add new nodes to the pup­pet clus­ter this bit me right in the soft bits.

My co-worker added a new node by copy and past­ing a pre­vi­ous node def­i­n­i­tion and updat­ing the node name. Of course he did this with notepad in win­dows. We noticed when we ran pup­petd –test on the new node that it failed with a parser error. It com­plained about a miss­ing } at line one of our nodes.pp. This was very odd since he was edit­ing the bot­tom of the file. I opened the file in vim and every­thing looked right. I could find no such miss­ing curly braces. I went ahead and removed his node def­i­n­i­tion and ran pup­petd –test again. Not sur­pris­ingly the node com­plained of hav­ing no def­i­n­i­tion (I did not have a default node setup). I re-added the node def­i­n­i­tion and again the node was suc­cess­ful. I ini­tially chalked this up to some error I must have missed in the node def­i­n­i­tion. But when my co-worker added more nodes and we saw the same behav­ior it was obvi­ous there was a dif­fer­ent issue going on. Now you would think this would be an easy con­clu­sion to draw but that was not the case for me, at least it wasn’t today.

I noticed that after he added a node if I called in with an already con­fig­ured node that node would com­plain about the same parse error. How­ever if I called in again it would receive its con­fig­u­ra­tion just fine and it would say it was ignor­ing the cache. I saw noth­ing excep­tional in the pup­pet­mas­ter logs. Of course the same parser error was there the first time a node called in to get its con­fig­u­ra­tion after he altered it with a win­dows edi­tor but all sub­se­quent call ins of pre­vi­ously con­fig­ured nodes suc­ceeded. After I noticed this pat­tern things started com­ing together. To com­pli­cate my trou­bleshoot­ing I started try­ing to restart pup­pet­mas­terd. This at first sent me for a tail spin. After pup­pet mas­ter was restarted with the file that had been updated with notepad on win­dows ALL node call ins were fail­ing. No longer was it only the first time. Even­tu­ally this lead me to the con­clu­sion. Pup­pet­mas­terd works very hard to make sure that your clients get served a valid con­fig­u­ra­tion. If you update the con­fig and do not restart pup­pet­mas­terd and that new con­fig is invalid pup­pet­mas­terd appears to serve up the pre­vi­ous ver­sion of the con­fig that it knew worked. This was all well and dandy and prob­a­bly a desir­able behav­ior. My issue is that the logs didn’t appear to say “Hey by the way I see you jacked up your pup­pet con­fig. This con­fig wont work dummy, Im going to go ahead and serve up your last con­fig since it works.”. I saw no men­tion of this behav­ior in the docs but per­haps I missed something.

The moral of the story. Don’t use win­dows to edit your pup­pet con­figs and if you do use vim for win­dows. Sec­ondary to that test! test! and test some more. Its pretty easy to screw up some syn­tax and have things start break­ing on you.

13 Comments

  • I’m hope­fully going to get some time to start dab­bling in sys­tem con­fig­u­ra­tion man­age­ment soon. What made you decide on Pup­pet vs Chef and BCFG2? How is the doc­u­men­ta­tion for Puppet?

  • I had not heard of Chef so I’ll be read­ing up on that shortly. I tried bcfg2 for a bit. There were things I liked about it but I really don’t like writ­ing xml and iirc the con­fig was xml based. I did really like that it was python and I would have had a bet­ter shot at extend­ing it since I already dab­ble with python (not that I can’t do ruby, I just pre­fer python).
    Pup­pet does have a fairly decent sized com­mu­nity which I think is some­what impor­tant and there are a lot of recip­ies out float­ing around to help you get started. Chef looks a lot like pup­pet so Ill have to read up on it.

    As for pup­pet doc­u­men­ta­tion. Over­all its pretty good. Per­son­ally I find it some­what dif­fi­cult to nav­i­gate their wiki but I just attribute that to poor search­ing on my part. Once I fig­ure out the right term I tend to find the infor­ma­tion pretty quickly.

  • damm Mac OS X Firefox 3.5.1 wrote:

    Truth­fully, I’ve been using chef for sev­eral months now and I find the doc­u­men­ta­tion; the exam­ples; and the sup­port to be far supe­rior to any­thing pup­pet ever provided.

    Not to men­tion chef scales a heck of alot eas­ier then pup­pet ever did. With Chef 0.8 up and com­ing in early August you may want to re-evaluate your time… as well as not use ‘notepad’…

  • Chef looks nice. How­ever for my envi­ron­ment I think the knowl­edge of requir­ing ruby knowl­edge would not go over well. I’m still drawn to bcfg2 but for now I just can’t seem to get over the xml based con­figs. I’d love to see a good writeup on chef though if you have done any­thing. I have a feel­ing that Chef would be a bet­ter fit where you have more devel­oper types tak­ing care of sys­tems but maybe thats just a mis-conception on my part.

  • This web­cast was an inter­est­ing watch: http://www.ducea.com/2009/06/30/osbridge-configuration-management-panel/. BCFG2, Chef, Pup­pet, and CFEngine were all present for a panel dis­cus­sion. Before watch­ing it, you should know that Chef was writ­ten by a long-time Pup­pet user who never con­tributed much back to Pup­pet, so there’s some “ten­sion” there.

    At one point, the panel speaker asks, who com­prises the major­ity of your user base? Sysad­mins was the answer for Pup­pet, and Devs was the answer for Chef. One point for Pup­pet. Google uses Pup­pet, which is another point for Puppet.

    How­ever, the author of Pup­pet talks about how there’s code in there that is still around from when he didn’t know Ruby so well, which makes me a bit ner­vous. Look­ing at some of my perl pro­grams from when I was first learn­ing it makes me shudder.

    The one thing that I think is a huge pos­i­tive is that Chef makes you learn a bit of Ruby to do the con­fig­u­ra­tion. Pup­pet has it’s own DSL, but you can’t use that knowl­edge any­place else except for within Pup­pet. What you learn about Chef’s “lan­guage”, you can use later for scripts, Rails, whatever.

    In the end, I think I’ll go with Pup­pet, but I hope that the author has his eyes open and incor­po­rates some of the things that Chef does well into future releases.

    Good dis­cus­sion!

  • Thanks for the link, that sounds inter­est­ing. Im all for being able to re-use knowl­edge and Im sure that learn­ing Chef would help with ruby things in the future but Im more of a python kind of guy. There are def­i­nitely some things with pup­pet that could improve but per­son­ally I am not ner­vous about some old code that was writ­ten when Luke was still learn­ing Ruby. Every­one has that, there are deploy­ments that I did in the past that still work well but I cringe at the thought of doing it that way again. I sup­pose to me its a good sign, the devel­op­ers skills are improv­ing and they are cri­tiquing their own code.

  • Zed Mac OS X Firefox 3.5.2pre wrote:

    FYI, notepad is the worst edi­tor ever (use notepad2, notepad++ or even wordpad).

    The issue you were see­ing is because of notepad’s poor han­dling of end of lines.

  • Indeed I am aware that notepad is hor­ri­ble. I was just sur­prised because I did not see any ^M char­ac­ters on any of the lines.

  • Mark Carey Windows Vista Firefox 3.5.2 wrote:

    As you’ve seen there are other things than bad line end­ings that can hap­pen when using “bad” win­dows edi­tors on files that will be parsed in var­i­ous linux dae­mons. The most noto­ri­ous I’ve encoun­tered is where garbage meta char­ac­ters are added to the first line of a php file. This causes php to fail to parse the file and returns an obscure error. Unfor­tu­nately vi/vim does not show the meta chars. This is where a quick check in a tool like od is handy.

  • Thanks for the com­ment Mark.

    You learn some­thing new every day. I had never used od before, it will be a handy tool to add to my collection.

    If some­one wants a short run­down on the tool see this arti­cle http://www.linuxjournal.com/article/1326

  • Chuck Sharp Windows XP Firefox 3.5.2 wrote:

    Another sim­ple way to check text file line ter­mi­na­tions (what a hor­ri­ble trou­bleshoot­ing time-waster) is the file com­mand. Here’s how it looks:

    $ echo –e ‘test\ntest\n’ > testme
    $ file testme
    testme: ASCII text
    $ unix2dos testme
    unix2dos: con­vert­ing file testme to DOS for­mat …
    $ file testme
    testme: ASCII text, with CRLF line terminators

  • Before watch­ing it, you should know that Chef was writ­ten by a long-time Pup­pet user who never con­tributed much back to Puppet, …”

    This is not true. I’m not sure what name to place on the course of events that has led any­one to believe this. What­ever would con­vey it, dis­ap­point­ing is a word I would associate.

    Theres a telling trail of crumbs here:
    http://projects.reductivelabs.com/issues/1010

  • Pup­pet vs Chef is a debate which seems to pro­voke strong reac­tions (on both sides). What­ever the his­tory betwen the devel­op­ers, it’s the tech­ni­cal mer­its of each tool which will weigh most with peo­ple choos­ing their con­fig­u­ra­tion man­age­ment tool.

    I’ve writ­ten a sum­mary of the respec­tive mer­its (in my opin­ion, of course) of Pup­pet and Chef here:

    http://bitfieldconsulting.com/puppet-vs-chef

Leave a Reply

Your email is never shared.Required fields are marked *

To submit your comment, click the image below where it asks you to...
Clickcha - The One-Click Captcha