howto

Have you ever had a machine that was a bit flaky? You know those ones that occasionally crash and don’t write anything useful into the log file. Sometimes you can capture those messages with netconsole. Just revisiting a small walk-through I wrote a while back.

Remote kernel logging with netconsole for fun and profit

I like to have as little run in dom0 as possible. However some things you really need checked from dom0, like the status of your raid perhaps. Just some quick instructions on getting Nagios NRPE running in XenServer.

1. Install EPEL repository and disable it by default (remember we don’t want to accidentally install unnecessary packages)

   wget http://download.fedora.redhat.com/pub/epel/5/$(uname -i)/epel-release-5-3.noarch.rpm
   rpm -hiv epel-release*.rpm
   sed -i 's/enabled=1/enabled=0/g' /etc/yum.repos.d/epel.repo
   

2. Install nrpe and configure it to start on boot

   yum install --enablerepo=epel nrpe
   chkconfig nrpe on
   

3. Modify the firewall to allow NRPE connections. Add the following before the REJECT line in /etc/sysconfig/iptables

   -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 5666 -j ACCEPT
   

4. Restart your firewall and start nrpe

   restart your firewall , and start nrpe
   /etc/init.d/iptables restart && /etc/init.d/nrpe start
   

5. Configure nrpe like normal and have fun

I’ve been wiping a lot of hard drives recently. I use shred to do the job.

shred -n6 -z -v /dev/sdx

What do you do to your drives before disposing of them?

A while back I wrote about using Apache as a dynamic reverse proxy. Anyone who has done even minimal research into web servers knows that Apache is the swiss army knife. It trys to be everything for everyone, and like a swiss army knife may not be as good as a more refined too at least as far as efficiency is concerned.

Here is the situation. You have a single pinhole into your private network. You have a single ip at your gateway. You want to serve multiple websites on your lan that may be running on multiple physical servers. Rather than opening up multiple ports and pinholling to all the different spots you want to serve, or getting more external ips and doing 1to1 NAT you can use a reverse proxy to be your single entrance point. The reverse proxy will fetch the content from the backend server and serve it up.

nginx is a HTTP server and mail proxy server. One of its features basic HTTP features is accelerated reverse proxying.

nginx should be available through your package manager so just aptitude (or whatever your package manager is yum, emerge, pacman) install it.

The config file paths shown are Debian specific but the config itself should work on any distro.

Edit /etc/nginx/sites-available/default and make it look like this

server {
     listen  :80;
     server_name  _;
     access_log  /var/log/nginx/proxy.access.log;

     location / {
     resolver        127.0.0.1;
     proxy_pass      http://$host$uri;
     proxy_redirect off;
     proxy_set_header        Host    $host;
     proxy_set_header        X-Real-IP $remote_addr;
     proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
     }

     error_page   500 502 503 504  /50x.html;
     location = /50x.html {
          root   /var/www/nginx-default;
     }
}

So this config causes nginx to listen on all interfaces/ips. server_name _; matches on anything so essentially this is a catchall now. You can tail proxy.access.log in order to see the requests are they come in and are served.

The location section is where the actual proxying happens. Since this is a dynamic configuration you need to set a resolver where the requested names can be looked up (and overridden for the local lan address). dnsmasq reads is dns configuration right out of /etc/hosts. It’s easy to install and configure so I reccomend using it. We will install and configure it shortly but for now just leave resolver as 127.0.0.1. proxy_pass does the requesting of the page we are proxying. Since this is a transparent dynamic proxy we just have it request the same thing that was requested of the proxy. proxy_redirect should be set to off since we are just passing on the same request. We need to set a few headers for logfiles on the backend servers as well as making sure that Host is set to the requesting host in case your using name based virtual hosts on your backend servers. I have left the error page in the default config (at least on debian its default). This provides a nice error message in case your proxy is working but one of the backend servers is not. It just serves the index.html that is located in /var/www/nginx-default. Feel free to change that path to something else, modify the index.html or omit the error_page and error page location section all together as they aren’t needed for this to work.

Now we need to get that local resolver (dnsmasq) installed so we can take our reverse proxy for a spin. Go ahead and aptitude (or whatever) install dnsmasq.

At least on debian dnsmasq comes out wanting to serve dhcp. You probably do not want this behavior. There is also the question of needing access to these same services by the same name on your LAN. If you need this you might need to do some slight adjusting of your dns. I might reccomend pointing your main dns to this dnsmask proxy or pointing all of your clients at this dnsmasq install since it will look up other requested names other than those in /etc/hosts. For this example I will assume you will be wanting to access these same web services internally with the same names and bypass the proxy. So I will assume you have either changed your primary dns cacher/resolver (think soho router or whatnot) to the address of the proxy server (since its running dnsmasq as well), or set all of your clients to point directly at the proxy server for dns. We need to edit the dnsmasq config to disable dhcp.

Edit /etc/dnsmasq.conf and add no-dhcp-interface=ethx. Do that for every interface on your system so that your not accidentally serving out dhcp to anyone. If somone has a more generic way to disable dhcp in dnsmasq without specifying each interface I would love to know but from reading the man this was the only way I could find. So you may have something like the following in you /etc/dnsmasq.conf.

no-dhcp-interface=eth0
no-dhcp-interface=eth1

After making the change you should be ready to add entries to the proxy servers /etc/hosts for dnsmasq to use and then test your reverse proxy.

Lets say you have www.test.com served off of a machine with the ip 192.168.1.2 and you have tickets.office.test.com served off of 192.168.1.3. Lets also assume that your world routeable ip is 123.123.123.123. You will need to make sure that your authoritative dns (the real one that servs for test.com has A records for both www.test.com and tickets.office.test.com pointing to 123.123.123.123. Now on the machine running dnsmasq (in this example also your proxy server) add the following entries to /etc/hosts.

192.168.1.2 www.test.com
192.168.1.3 tickets.office.test.com

Go ahead and restart dnsmasq (from making changes to the config, subsequent changes to /etc/hosts should not require dnsmasq restart to pick up changes) and nginx.

Now tail your proxy.access.log file and start making requests to www.test.com and tickets.office.test.com from both the inside of your lan as well as outside against your world ip. It should all magically serve up the same content.

This type of config can be useful in many situations. You have a small office and budget that reflects that not being able to afford multiple ips but needing to provide web services behind the firewall. You work in a large corporation where someone else manages the firewall and you would like to bring up more web services without waiting for the other person to make the necessary changes to the firewall.

One of the other benefits this provides is being relatively self documenting  with regard to what web services you host behind the firewall. (you should be able to see all of them in /etc/hosts since you have to override the dns)

Have you ever gotten a pdf with one of those annoying passwords? A while back I bought an e-book and it came with a password. Its really annoying especially if I want to read it on a mobile device. Anyway if you are annoyed as much as I am fear no more.

Install qpdf

aptitude install qpdf

Decrypt your pdf

qpdf --password=password --decrypt input.pdf output.pdf

Over the last couple days I’ve briefly talked about revisioning configs and making your home directory portable. It seems to have stirred up a bit of discussion over at Matt Simmons Standalone Sysadmin, and Hugh Browns mentioned he uses mercurial for this task.. As I noted in the post about managing /etc with version control I do revision my home directory, or at least pieces of it. (more…)

Have you ever experienced hard lockups and seen no trace of the cause in your log files? Those situations can be even more of a pain if you do not have physical access to the machine since you will not be able to look for kernel oops on the console. You could buy a serial console or an ip kvm but if you don’t have the need for remote control, but would really like to be able to debug without being physically present you need to check out netconsole. Netconsole sends printk messages over UDP. (more…)

Ever run into a situation where passing the option single to the kernel wasn’t enough to get your root password reset? This is not Debian specific but some distros (including Debian) require that you still enter the root password when booting to single user mode. This is just a quick run through of how to reset your root password without a live cd. (more…)

Recently I had to move a website that gets a fair amount of traffic. I prepared for this by lowering the TTL on the domains associated several weeks in advance to 600 seconds. Originally my plan was to toss up a maintenance page on the old server, change the DNS, and figured that within a few hours max the vast majority of DNS servers would have the update being that TTL had been set at 600 seconds for several weeks and prior to that it was set at 48 hours. This was all planned for the middle of the night on a weekend (very slow traffic time). I was in for a rude awakening the next morning.
(more…)

I hope you are already using ssh keys, but just in case your not go ahead and generate one with ssh-keygen -t rsa (you should do this on your local box)
You may as well go ahead and copy your publick key to your git server now as well.

scp ~/.ssh/id_rsa.pub gitserver:/tmp/

Log into your gitosis server

ssh gitserver

* Make sure setuptools is installed

sudo aptitude install python-setuptools

git clone git://eagain.net/gitosis.git
cd gitosis
sudo python setup.py install

sudo adduser --system --shell /bin/bash --gecos 'git version control' --group --disabled-password --home /home/git git

sudo -H -u git gitosis-init < /tmp/id_rsa.pub

We need to fix up a few things that don’t seem to get proper permissions.

sudo -H -u git chmod 600 ~/.ssh/authorized_keys
sudo -H -u git chmod 755 ~/repositories/gitosis-admin.git/hooks/post-update
sudo rm /tmp/id_rsa.pub

Thats all for now on your git server. You manage gitosis with git of course and you can do that from your local machine or any machine that you have the match to the public key you installed with gitosis.

git clone git@gitserver:gitosis-admin.git
cd gitosis-admin

Creating new repositories

New repositories are created by authorizing a user to write and pushing to it.

[group someproject]
members = jack jill
writeable = someproject

This definition would create the group someproject and allow jack and jill write access to the repository someproject (which you would reference as someproject.git)

For a bit more explanation …

[group developers]
members = jack john frank
writable = project1 project2 project3

This definition would create the group developers, and allow jack, john, and frank write access to the repositories project1, project2, and project3

Once you have defined your repositories commit and push your changes to gitosis admin.

git commit -a -m "Created inital gitosis config"
git push

Now if you have an existing repository that you would like to add to your git server simply change into the repository and do

git remote add origin git@gitserver:repo.git
git push

That will push the current branch of the repo over to the repo you set up on your git server. So thats about it.

Anonymous access
To allow anonymous read access touch git-daemon-export-ok inside ~/repositories/repo.git of each repo you wish to allow anonymous access (read only) to.

Once you have done that launch git-daemon with base path /home/git/repositories.