Archive for July, 2009

Congratulations to everyone in the club. You have successfully navigated another trip around the sun. Today is not the day to worry about your infrastructure falling apart today is a day for celebration. Ok maybe thats not exactly true, celebrate after you get your network and systems in check .

I’ve been working with puppet a lot in the last few weeks. I really enjoy this style of administration compared to “ssh in a for loop”. Its great to script things and I still do it but for maintaining configuration I don’t know if it gets much better than puppet. That being said I ran into a few issues today that had me splitting my head open on my desk, keyboard, pop cans and just about everything else I could reach.

It’s standard practice to keep your puppet configuration in version control and I am doing this with subversion. It provides a central authority with log history so we can audit why changes were made. It’s also handy for my co-workers to be able to use tortisesvn on their windows clients to update configuration. Be careful with that! I do all of my editing in vim and neglected to take into account what using various editors might do to my puppet configuration. Today as we were testing some puppet configurations and I was showing co-workers how to add new nodes to the puppet cluster this bit me right in the soft bits.

My co-worker added a new node by copy and pasting a previous node definition and updating the node name. Of course he did this with notepad in windows. We noticed when we ran puppetd –test on the new node that it failed with a parser error. It complained about a missing } at line one of our nodes.pp. This was very odd since he was editing the bottom of the file. I opened the file in vim and everything looked right. I could find no such missing curly braces. I went ahead and removed his node definition and ran puppetd –test again. Not surprisingly the node complained of having no definition (I did not have a default node setup). I re-added the node definition and again the node was successful. I initially chalked this up to some error I must have missed in the node definition. But when my co-worker added more nodes and we saw the same behavior it was obvious there was a different issue going on. Now you would think this would be an easy conclusion to draw but that was not the case for me, at least it wasn’t today.

I noticed that after he added a node if I called in with an already configured node that node would complain about the same parse error. However if I called in again it would receive its configuration just fine and it would say it was ignoring the cache. I saw nothing exceptional in the puppetmaster logs. Of course the same parser error was there the first time a node called in to get its configuration after he altered it with a windows editor but all subsequent call ins of previously configured nodes succeeded. After I noticed this pattern things started coming together. To complicate my troubleshooting I started trying to restart puppetmasterd. This at first sent me for a tail spin. After puppet master was restarted with the file that had been updated with notepad on windows ALL node call ins were failing. No longer was it only the first time. Eventually this lead me to the conclusion. Puppetmasterd works very hard to make sure that your clients get served a valid configuration. If you update the config and do not restart puppetmasterd and that new config is invalid puppetmasterd appears to serve up the previous version of the config that it knew worked. This was all well and dandy and probably a desirable behavior. My issue is that the logs didn’t appear to say “Hey by the way I see you jacked up your puppet config. This config wont work dummy, Im going to go ahead and serve up your last config since it works.”. I saw no mention of this behavior in the docs but perhaps I missed something.

The moral of the story. Don’t use windows to edit your puppet configs and if you do use vim for windows. Secondary to that test! test! and test some more. Its pretty easy to screw up some syntax and have things start breaking on you.

I have to use windows for a few things at my new employeer. I’ve found a few programs that I just cant survive using windows without.

• Launchy – Gnome-do like launcher
• Quotefix – Properly bottom post when replying to emails
• IsoRecorder – Right click iso and burn
• Cygwin – Linux-like environment for Windows

I still spend very little time in windows so its not too bad but when I am there not having access to those tools drives me batty. What tools do you use to make windows bearable when you are being abused?

Matt Simmons recently had another post that mentioned structured systems management. I find myself re-reading Ad-Hoc vs Structured Systems Management from time to time and I figure its time for me to chime in.

First off, Michael Jankes’ post is one of my favorite blog post reads. Few posts do I ever re-read unless its tutorialesque. For a quick sidetrack one of those posts that I do return to when I deal with developers is The Joel Test: 12 Steps to Better Code. Back on subject …

One of the most frequent things that comes up when people start talking about good administration practices is scripting things. If you can’t script it make a checklist is a common sentiment. It’s not that I don’t think that scripting things is a good idea because I do. I would go so far to say that scripting things is an important part of documentation. You should be able to reproduce a system from documentation. Any configuration that can be done programatically should be and it should be in your documentation so in case of catastrophic failure someone can reproduce your work relatively easily. If I like scripting so much why is this post titled “Don’t script it”?

I think there is a better way. Use a configuration management system. Declare your configuration and allow the configuration management system to make it happen. This has many advantages over simply scripting something. Configuration can be enforced over time. I know I have been guilty of making small changes to configurations to see how things go. While those small changes seem to trickle into production systems they never (or at least rarely) seem to trickle into the documentation. Before you know it you have a pile of undocumented small changes that really encompass an entirely different configuration from what you started with.

A well placed configuration management system helps avoid these issues. Store your configuration in version control. I might even go so far as to say to automate the update of the checkout your configuration management server uses so that you don’t allow yourself to make small changes to the management system configuration without checking it into your version control. A few tweaks being automatically removed will go a long way to drive home the point that your changes need to be in version control. Placing your config in version control also allows you to see your changes over time. You can look back through the log and your comments to see when and why changes were made. This can be especially important if you are not the only person making changes to configuration (ah hem Matt, I know your hiring someone you better version your config before your no longer the lone admin).

This all comes up because I have been doing more work with puppet recently. It takes a while to wrap your head around but I don’t really think I have heard any arguments against that type of system being the “right” way to do systems administration. If I have they sure didn’t register. I like puppet because it works on a variety of systems and abstracts things like package management. I just tell my configuration system to make sure nodeX has package x installed and the puppet client figures out if its RH or Debian or SUSE or Solaris or whatever and runs the proper commands to get the package installed.

I am interested in spacewalk as well. But right now it only runs on Oracle databases so until it runs on postgresql there really isn’t even a point in me looking as the places I would be able to use it would be very limited.

Sorry for the spam just testing modifications to a plugin

A while back I wrote about using Apache as a dynamic reverse proxy. Anyone who has done even minimal research into web servers knows that Apache is the swiss army knife. It trys to be everything for everyone, and like a swiss army knife may not be as good as a more refined too at least as far as efficiency is concerned.

Here is the situation. You have a single pinhole into your private network. You have a single ip at your gateway. You want to serve multiple websites on your lan that may be running on multiple physical servers. Rather than opening up multiple ports and pinholling to all the different spots you want to serve, or getting more external ips and doing 1to1 NAT you can use a reverse proxy to be your single entrance point. The reverse proxy will fetch the content from the backend server and serve it up.

nginx is a HTTP server and mail proxy server. One of its features basic HTTP features is accelerated reverse proxying.

nginx should be available through your package manager so just aptitude (or whatever your package manager is yum, emerge, pacman) install it.

The config file paths shown are Debian specific but the config itself should work on any distro.

Edit /etc/nginx/sites-available/default and make it look like this

server {
     listen  :80;
     server_name  _;
     access_log  /var/log/nginx/proxy.access.log;

     location / {
     resolver        127.0.0.1;
     proxy_pass      http://$host$uri;
     proxy_redirect off;
     proxy_set_header        Host    $host;
     proxy_set_header        X-Real-IP $remote_addr;
     proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
     }

     error_page   500 502 503 504  /50x.html;
     location = /50x.html {
          root   /var/www/nginx-default;
     }
}

So this config causes nginx to listen on all interfaces/ips. server_name _; matches on anything so essentially this is a catchall now. You can tail proxy.access.log in order to see the requests are they come in and are served.

The location section is where the actual proxying happens. Since this is a dynamic configuration you need to set a resolver where the requested names can be looked up (and overridden for the local lan address). dnsmasq reads is dns configuration right out of /etc/hosts. It’s easy to install and configure so I reccomend using it. We will install and configure it shortly but for now just leave resolver as 127.0.0.1. proxy_pass does the requesting of the page we are proxying. Since this is a transparent dynamic proxy we just have it request the same thing that was requested of the proxy. proxy_redirect should be set to off since we are just passing on the same request. We need to set a few headers for logfiles on the backend servers as well as making sure that Host is set to the requesting host in case your using name based virtual hosts on your backend servers. I have left the error page in the default config (at least on debian its default). This provides a nice error message in case your proxy is working but one of the backend servers is not. It just serves the index.html that is located in /var/www/nginx-default. Feel free to change that path to something else, modify the index.html or omit the error_page and error page location section all together as they aren’t needed for this to work.

Now we need to get that local resolver (dnsmasq) installed so we can take our reverse proxy for a spin. Go ahead and aptitude (or whatever) install dnsmasq.

At least on debian dnsmasq comes out wanting to serve dhcp. You probably do not want this behavior. There is also the question of needing access to these same services by the same name on your LAN. If you need this you might need to do some slight adjusting of your dns. I might reccomend pointing your main dns to this dnsmask proxy or pointing all of your clients at this dnsmasq install since it will look up other requested names other than those in /etc/hosts. For this example I will assume you will be wanting to access these same web services internally with the same names and bypass the proxy. So I will assume you have either changed your primary dns cacher/resolver (think soho router or whatnot) to the address of the proxy server (since its running dnsmasq as well), or set all of your clients to point directly at the proxy server for dns. We need to edit the dnsmasq config to disable dhcp.

Edit /etc/dnsmasq.conf and add no-dhcp-interface=ethx. Do that for every interface on your system so that your not accidentally serving out dhcp to anyone. If somone has a more generic way to disable dhcp in dnsmasq without specifying each interface I would love to know but from reading the man this was the only way I could find. So you may have something like the following in you /etc/dnsmasq.conf.

no-dhcp-interface=eth0
no-dhcp-interface=eth1

After making the change you should be ready to add entries to the proxy servers /etc/hosts for dnsmasq to use and then test your reverse proxy.

Lets say you have www.test.com served off of a machine with the ip 192.168.1.2 and you have tickets.office.test.com served off of 192.168.1.3. Lets also assume that your world routeable ip is 123.123.123.123. You will need to make sure that your authoritative dns (the real one that servs for test.com has A records for both www.test.com and tickets.office.test.com pointing to 123.123.123.123. Now on the machine running dnsmasq (in this example also your proxy server) add the following entries to /etc/hosts.

192.168.1.2 www.test.com
192.168.1.3 tickets.office.test.com

Go ahead and restart dnsmasq (from making changes to the config, subsequent changes to /etc/hosts should not require dnsmasq restart to pick up changes) and nginx.

Now tail your proxy.access.log file and start making requests to www.test.com and tickets.office.test.com from both the inside of your lan as well as outside against your world ip. It should all magically serve up the same content.

This type of config can be useful in many situations. You have a small office and budget that reflects that not being able to afford multiple ips but needing to provide web services behind the firewall. You work in a large corporation where someone else manages the firewall and you would like to bring up more web services without waiting for the other person to make the necessary changes to the firewall.

One of the other benefits this provides is being relatively self documenting  with regard to what web services you host behind the firewall. (you should be able to see all of them in /etc/hosts since you have to override the dns)

Ha! I have trapped you with my sensationalized headline. Or you didn’t read this just like I wouldn’t have .

I don’t know about anyone else but I had to turn off some of my feeds over the past several days. I was sick and tired of all the Chrome OS posts filtering about the net. It’s not that I don’t think having google behind another linux platform is a bad idea. Really I think that will be great for Linux in general. The more average people who are exposed to well packaged Linux distributions the better. What I was tired of was all of the sensationalism surrounding it. Here are some of my favorite titles.

• Chrome OS: But Will it Run Photoshop?
• Google Drops A Nuclear Bomb On Microsoft. And It’s Made of Chrome.
• Google could kneecap Microsoft with Chrome OS
• So, Google announced a new OS using the Linux Kernel. Opinions?
• Is Google Stealing Ubuntu’s Thunder?
• Google Chrome OS: A Free Alternative To Windows 7
• Instant-on Linux vendors put on a brave face against Google Chrome OS
• Microsoft Worldwide Partner Conference: Will Google Chrome OS Steal the Show?
• Are Google and Microsoft Switching Roles?
• What does Red Hat, Ubuntu and openSUSE think about Google Chrome OS?

Actually I didn’t read all of these because I couldn’t get past the headlines. For example “Chrome OS: But Will it Run Photoshop?”. Seriously is there an actual readership for these stories? Were they written to incite flame-wars? Riddled with in-accuracies and presumptions. I don’t know I’ve just had it up to my ears with with these stories. At any rate just reading the headlines is worth a chuckle.

Apparently I have had a caching issue on the blog for a while now. It should be fixed but if anyone notices that your feed is updating but the site is not feel free to drop me a line. Thanks to Warren Guy from planetsysadmin.com for letting me know.

I recently had to write an init script for our continuous integration systems remote build agent. Of course this agent is a java jar and does not write its own pidfile. On debian or gentoo (yes I used to rice it) this would not have been an issue. Both have and use start-stop-daemon in their init scripts. start-stop-daemon actually understands that not every process you might want to daemonize writes its own pidfile and thus gives you the ability to daemonize a process, capture its pid, and write a pidfile.

Oh, but not on redhat/centos. No, they have daemon(), which really doesnt do much of anything. <pompus voice> Each process must write its own pidfile </pompus voice>. Well I already have a custom init script, I really don’t want to write a custom wrapper, nor do I want to add yet another dependency to what ought to be a simple task. Perhaps I just don’t see the proper way to do this so I welcome solutions. This feature lack causes lines like this to show up in init scripts.

PID=$(pgrep -n -f "$LAUNCH_CMD")

I don’t know about you but that is horribly ugly and error prone in my eyes.

chkconfig, the other suck.

Yes its documented that the tags chkconfig and description are required or your init script will not support chkconfig. That’s all fine and dandy. I copied my init template from /usr/share/doc/initscripts-8.45.25/sysvinitfiles. It has doc at the end of the initscript after exit is run. This is perfectly valid. There is nothing that is going to cause issues with the execution of the script. I left that doc there as I was developing my script. When it came time to chkconfig –add my script it continually failed. Granted it only took me a few minutes to figure out but what fixed it? Oh yes by removing all the doc at the end of the file. Why is this conflicting with chkconfig no reason that I can see to make it just not work. It would make a bit more sense if the example tags below just overrode the ones at the top of the file.

Recently I have been working on cleaning up our continuous integration enviornment. Continuous integration is pretty cool but don’t shoot yourself in the foot by making bad build plans and not being very specific about your build requirements. If you have a package dependancy for a build make sure you put it in your requirements. If you need specific access to a database to do fixture tests SPECIFY it.

In the short term you can see the benefits of continuous integration  without doing these things. But as people cycle in and out of your orginization having a clear record of all the requirements for a build is crucial.