cmdln.org (a sysadmin blog)

Have you ever expe­ri­enced hard lock­ups and seen no trace of the cause in your log files? Those sit­u­a­tions can be even more of a pain if you do not have phys­i­cal access to the machine since you will not be able to look for ker­nel oops on the con­sole. You could buy a ser­ial con­sole or an ip kvm but if you don’t have the need for remote con­trol, but would really like to be able to debug with­out being phys­i­cally present you need to check out net­con­sole. Net­con­sole sends printk mes­sages over UDP.

Set­ting up net­con­sole is not dif­fi­cult but the syn­tax can be a bit tire­some. Net­con­sole needs sev­eral bits of infor­ma­tion in order to func­tion properly.

dev_name — Local net­work inter­face name

local_port — Source UDP port to use

remote_port — Remote agent’s UDP port

local_ip — Source IP address to use

remote_ip — Remote agent’s IP address

local_mac — Local interface’s MAC address

remote_mac — Remote agent’s MAC address

Of those remote_mac tends to be the tricky one. Not because its hard to get but because it is slightly mis-leading. If the remote agent is in the same sub­net its the mac of the remote agent, but if the remote agent is not in the same sub­net (think log­ging over inter­net) then you really need the mac address of the gate­way that will han­dle the traf­fic (if you have mul­ti­ple wans). Typ­i­cally your look­ing for the mac of your default gateway.

Find MAC of remote agent in same subnet

REMOTE_AGENT=172.16.0.1
MAC=$(ping -c 1 $REMOTE_AGENT > /dev/null ; arp -n $REMOTE_AGENT | grep ^$REMOTE_AGENT | awk '{print $3}')
echo Remote MAC: $MAC

Find MAC of default gw

GATEWAY=$(netstat -rn | awk '/^0.0.0.0/ {print $2}')
MAC=$(ping -c 1 $GATEWAY > /dev/null ; arp -n $GATEWAY | grep ^$GATEWAY | awk '{print $3}')
echo Remote MAC: $MAC

Ini­tial­ize net­con­sole
Now you should have enough infor­ma­tion to go ahead and ini­tal­ize net­con­sole so lets give it a test

modprobe netconsole netconsole=local_port@local_ip/dev_name,remote_port@remote_ip/remote_mac

Now we still need to get some­thing lis­ten­ing on the remote and test if it actu­ally works. Log into your remote machine and run

nc -l -p remote_port -u | tee  somelogfile.log

For a more per­ma­nent setup you might want to use sys­log but this will suf­fice for now. If it’s a short term but long run­ning test you might be well advised to run that from a screen session.

Good now we have the remote lis­ten­ing on udp with net­cat. We should make sure that the mes­sages are get­ting logged. Log back into the machine thats run­ning net­con­sole (local_ip) and run the following.

dmesg -n 8

This will increase the num­ber of things that get logged.
Now find an innocu­ous ker­nel mod­ule that you can load and unload (i like to use floppy)

rmmod floppy (in case its already loaded)
modprobe floppy

You should have seen some out­put on your remote machine that looks some­thing like

Floppy drive(s): fd0 is 1.44M
FDC 0 is a post-1991 82077

Great now you have net­con­sole work­ing! If you get ker­nel oops your remote box should dis­play it and log it to a file as well.

Want to make net­con­sole active through reboots? No prob­lem we just need to edit a few files.

First lets get net­con­sole load­ing on boot by adding the mod­ule to /etc/modules

echo "netconsole" >> /etc/module

That was easy enough, but we need to make sure it has the proper options as well so lets add the mod­ule options to /etc/modprobe.d/netconsole

echo "options netconsole netconsole=local_port@local_ip/dev_name,remote_port@remote_ip/remote_mac" > /etc/modprobe.d/netconsole

That should do it. Go ahead and try reboot­ing the machine run­ning net­con­sole and watch your remote to see the boot msgs that hap­pen after net­con­sole loads.

Note: there is a dynamic way to spec­ify how net­con­sole is con­fig­ured but you need to have CONFIG_NETCONSOLE_DYNAMIC in your ker­nel and since debian etch does not have this by default I wont cover it here. For more infor­ma­tion check out the net­con­sole doc in the ker­nel source /usr/src/linux/Documentation/networking/netconsole.txt.

Now if you would like to make the remote side a bit more per­ma­nent thats pretty easy as well. Lets install and con­fig­ure syslog-ng.

aptitude install syslog-ng

append the fol­low­ing to your /etc/syslog-ng/syslog-ng.conf
Note: make sure your set remote_port as you did above

source net { udp(ip("0.0.0.0") port(remote_port)); };
destination netconsole { file("/var/log/$HOST/netconsole.log"); };
log { source(net); destination(netconsole); };

Now restart syslog-ng

/etc/init.d/syslog-ng restart

Now you should be able to find the logs in /var/log/local_ip/netconsole.log on your remote machine. Note: local_ip is the ip of the machine that was run­ning netconsole