administration

I know I’ve mentioned how much I love the sysstat package before. I use sar regularly to help with performance diagnostics (Analyzing Linux System Performance And Finding Bottle Necks, CPU Performance Analysis In Linux, Baseline Analysis Is Important, CPU Performance Analysis In Linux Revisited). I wrote this little Nagios plugin to collect the performance metrics that sar collects.

I use this plugin with Zenoss and I set any performance thresholds there, more important to me was collecting the information for historical graphing. I searched around and didn’t really find any existing solutions thats why anyone wanting to do similar perhaps with cacti is stuck with my craptastic code (or please point me to a better implementation).  Anyway if you want to grab the plugin and check it out its on github.

check_sar_perf

If you hadn’t already guessed I am a big fan of the Xen hypervisor. Lately I have been using the Citrix XenServer release because it makes it quite palatable for my co-workers. One annoyance that I do have about XenServer is the requirement that you license it (with a free license) every year. If you fail to license it the GUI stops working. Now I hate relying on GUIs but the fact of the matter is others in my team expect to have a working GUI when they need to do something. And I dont know about you but I don’t really log onto the management console very often. Really I only log on to it if I need to provision a new server so its entirely plausible that a license would expire and I wouldn’t know about it until I really needed to do something.

I ended up writing a little Nagios plugin that checks the license expiration date using XenAPI. I don’t know that it’s 100% compliant with the plugin specification but it does work for me. I actually don’t prefer to use the warn and critical states with the Nagios (I use the performance data with Zenoss and apply thresholds there. I find that to be a bit more flexible.) but I did implement them. The plugin can be executed on the XenServer (you may want to reference how to install nrpe on XenServer) or on from your monitoring host as long as the host performing the check has the python XenAPI installed.

The plugin check_citrix_xenserver_license can be found on github.

I hope someone can find it useful.

Have you ever had a machine that was a bit flaky? You know those ones that occasionally crash and don’t write anything useful into the log file. Sometimes you can capture those messages with netconsole. Just revisiting a small walk-through I wrote a while back.

Remote kernel logging with netconsole for fun and profit

I like to have as little run in dom0 as possible. However some things you really need checked from dom0, like the status of your raid perhaps. Just some quick instructions on getting Nagios NRPE running in XenServer.

1. Install EPEL repository and disable it by default (remember we don’t want to accidentally install unnecessary packages)

   wget http://download.fedora.redhat.com/pub/epel/5/$(uname -i)/epel-release-5-3.noarch.rpm
   rpm -hiv epel-release*.rpm
   sed -i 's/enabled=1/enabled=0/g' /etc/yum.repos.d/epel.repo
   

2. Install nrpe and configure it to start on boot

   yum install --enablerepo=epel nrpe
   chkconfig nrpe on
   

3. Modify the firewall to allow NRPE connections. Add the following before the REJECT line in /etc/sysconfig/iptables

   -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 5666 -j ACCEPT
   

4. Restart your firewall and start nrpe

   restart your firewall , and start nrpe
   /etc/init.d/iptables restart && /etc/init.d/nrpe start
   

5. Configure nrpe like normal and have fun

Another day, another flashback. I don’t know about you but at times I have had some very odd and uncomfortable requests from vendors. It’s not wholly  uncommon for them to want to log into your system to diagnose an issue for themselves. Or perhaps for whatever reason your manager says hey give this random outside person access to some important box. As a SA you get used to extremely odd requests and figure out how to just solve the problem. This flashback outlines what I did the last time I had to let an untrusted (note I had no reason to _not_ trust this person, but then again I also had no reason _to_ trust them) person have elevated privileges on a box of mine.

Automatic session logging and monitoring with GNU screen for the paranoid

Matt Simmons is trying to dust off some old articles he think some people may have missed. So I figure why not. A while back I talked about how to move a website (read DNS sucks , you need a reverse proxy). In fact I have talked about reverse proxies a few times since I tend to find them so useful. At any rate, this article stemmed from the first hand experience that opened my eyes to how bad DNS infrastructure really was.

Trust not DNS or how to properly move a website

The last few days I have been having a pretty good debate with a friend about the virtues of open source vs Appliances. At times its gotten pretty heated but its all in good fun. The current debate centers around email infrastructure. There are options on the table to use an appliance, or a 3rd party service to control the spam. Of course I was appalled that SpamAssassin and brethren were not on the table. (more…)

Ewwww, scary isn’t it. No Its not Halloween, but you may have entered the twilight zone. Right, I never touch Microsoft products. Well in actuality sometimes I do (I just don’t brag about it). Some of the development at $work uses Microsofts Mediaroom, and I have a “Personal Server” (great name right?) that the developers use. I was trying to install the Mediaroom service pack yesterday and took some notes on the process. Some of my friends found it quite entertaining. I found it quite aggravating as you might imagine. (more…)

Gah so I was futzing with the acl map on our subversion server. Organizing things into groups. I wasn’t thinking and started uppercasing the users ids when moving them into groups (yes i hate uppercase but its easy to highlight paste them). This of course stopped authentication from working for people who have cached auths returning a 403 error. So how to quickly lowercase a huge swath of ids? (more…)

I am sure you are aware of my affinity for the Xen hypervisor. In the last year I have switched over to Citrix Xenserver. With other people managing VMs as well having a nice GUI is helpful. One of my complaints is that the GUI (XenCenter) is a windows only app. Common Citrix, please release a cross platform management console. I’ve got to run a windows VM just to use the GUI (granted I don’t have to use the gui, there is a nice API and console utilities). At any rate today I noticed that I could no longer pull up the console for a windows VM nor could I pull up the performance metrics available in XenCenter. A few searches turned up Kenneth Hunts blog and a post that showed me where to fix it. (more…)