9. Don’t be afraid to be wrong
Learning from your mistakes is extremely important. No one can be perfect, you can strive to be, but everyone makes mistakes. Recognize your mistakes and figure out how to not repeat them.
8. Give credit where credit is due
If a colleague solved one of your problems, or had a great idea be sure to give them the credit. This is beneficial in several ways. First, it makes that person feel good. Second, they will continue to bring you solutions and ideas. Third, you won’t look like an ass for always trying to be “the man” (or woman as the case may be). Fourth, it will (or should) make you feel good to praise someones good idea after all you like your good ideas praised.
7. Don’t be afraid to be right
You are an expert. You may not be THE expert but you are an expert. There are (or should be) valid reasons for your decisions. Don’t back down for politics sake or because you (in the back of your mind are unsure).
6. Be open minded
Just because you are an expert doesn’t mean you should not listen to others. If an idea is brought up at least consider it (even if only momentarily).
5. Be cordial
Be kind to your users. They are the reason you have a job, can afford to drive to work (or own the fancy bicycle to save the trees). Yes that means don’t immediately say NO when a user asks if they can have a sticky note on their screen with their password on it. Explain why it’s not a good idea, offer to help them come up with a secure AND memorable password. Ask them if they understand why you have asked them to do something a certain way. This makes them feel happy about you being around, makes you (or should) feel happy about helping, and ultimately makes your job easier in the long run. Your users will be more willing to accept change even if it scares them because they know your there to back them up and their past experience should prove that it’s for the best.
4. Share information
Sharing information is important. It allows everyone to be on the same page. If it is not company secret (like the fact that you have slave pidgin labor) or your root password then share it. Share it with your co-workers, share it with the community. Good information is hard to find, help someone get home before 2am and post how to rebuild a software raid. Have a blog, write documentation! Never forget the Bus Factor.
3. Be discrete
You are entrusted with sensitive information. As a sys admin you may have access to everything from payroll to, hiring and firing, to passwords, and even other general personal and company data that needs to remain private. Do not break this trust. As Uncle Ben said, “With great power comes great responsibility.”
2. Work Hard
Have a great work ethic. Be there when the job needs done. Spend the extra time to get things working as best you can. In the long run it will help you with the next trait on my list.
1. Be Lazy
Thats right, “Be Lazy”. Good sysadmins are lazy sysadmins. They want to sit and not be bothered by fires. Good sysadmins go to great lengths to be lazy. They write scripts to automate their jobs in the never ending quest of automating themselves out of a job.
0. Be passionate and Never Stop Learning
This I consider to be one of the most important traits of a successful Systems Administrator (or anyone for that matter). Be passionate about what you do. When you love your work its not hard to get out of bed in the morning, its not hard to stay up writing until 2:56am. Its not hard to constantly be on top of emerging technologies. Just because they are new and should not be implemented does not mean you should not follow the progress. Work can begin now on a project that may not come to fruition for years, but when the time comes you will be familiar and will not have to spend months getting a basic understanding of a system only to deploy and have no idea how to deal with the inevitable failure of a system. Read, Listen, Watch, Talk to people who know more than you.

mesg y

To initiate a shared session just run

/usr/share/doc/libexpect-perl/examples/kibitz user

where user is the user that you would like to share a session with. Kibitz will then prompt that user to run a kibitz command that will connect them to the shared session.

Obviously the first option is less helpful in a multi user environment as one has to wheel up to a different user to share a screen. At first glance screens multiuser mode seems to be a great option. However multiuser mode requires that screen be setuid. Hopefully you are well aware of the dangers of running setuid processes. I will not drive the point home any more. If you still wish to use multiuser mode on screen by all means proceed.

I assume you have already installed screen. To enable multiuser mode log in to your machine

sudo setuid /usr/bin/screen
sudo chmod 755 /usr/bin/screen

now run screen as your normal user. Then enable multi user mode with C-a :multiuser on. And allow the user joe to connect to your session with C-a :addacl joe (optional password)

Now user joe can see what screens you have available with
screen -ls youruser/
and joe could connect to a screen with
screen -x youruser/(optional name of session)

For those of you who yearn for a better way to share a session without the setuid stay tuned for the next post where we actually address this issue.

Ultradns and Dnsmadeeasy are two leading hosted DNS providers. The model is simple. You pay to have your dns hosted on their network and servers. They ensure DNS propagation between their servers is fast and they have the capacity to protect against DOS attacks. Typically you get some base package of queries per month. For example the Business Membership is 59.95/yr and you get 10 million queries/month. That is a lot of queries when you consider that many queries for your domain will be served by caching servers. And overage charges are minimal at $6.00/ 1 million queries (if you don’t purchase blocks ahead of time). I was thinking boy I hope there is some kind of throttling in place to prevent some unsavory competitor from looping a dig against their name servers for my domain. So on a whim i looked around and actually found a domain (on the first try I might add) which uses Dnsmadeeasy. Oh in case you were wondering how I found out, I just did a whois and looked at the authoritative name servers and wow ns0.dnsmadeeasy.com was listed. So I ran a quick loop for 100 lookups on the domain.

time for i in $(seq 100);do dig redacted.tld @ns0.dnsmadeeasy.com;done

While I expected to get off a few lookups and then just wait for some throttle timeout to shut me down I was supprised to get all 100 lookups done in 11 seconds, subsequent tests showed similar times mostly faster. So conservativly say you can do 10 lookups a second. If my math serves me correct you can do 10 million lookups in just about 5 hours. After that you have broken the 10 million limit for the month. Holding steady at the same rate thats 864000 queries in 24 hours and 25920000 in 30 days. Yeah so not a bank breaker at $6/ million but this was from a single PC and I doubt network was the bottle neck. A distributed attack could end up costing a company thousands upon thousands. Refusal to pay could result in DNS being shut off, and effectively creating a DOS. For fun I tried 100 lookups against Ultradns for some of their banner customers and also received no throttling. Still a bit surprised at this seemingly overlooked hole I called Dnsmadeeasy and asked the sales department what protections were in place to prevent or mitigate malicious lookups. His response was do you mean DOS? When I explained the issue he said we can not block that, as we do not know if there are 1000 people behind your company firewall that are really interested in that website.

It does not seem unreasonable to provide a throttling mechanism. Oh you queried 10 times in the last 2 seconds? I think I will block you for 5 mintues. Happens again within x time increase block time to 10 minutes and so on. So who wants to loop while true dig amazon @udns1.ultradns.net for a month or so and see what happens. Will they report being hacked? Will the cops bust down your door? Will amazon just eat the cost (probably). But why not just provide simple throttling for obviously either misconfigured or malicious lookups?

In KDE4 the panel takes its configuration from the global settings so you need to change the time format under System Settings.

System Settings -> Regional & Language -> Time & Dates -> Time Format

You need to aptitude install libapache2-mod-proxy-html apache2, and a2enmod proxy proxy_connect proxy_html proxy_http rewrite.

Then comment out the the line

from /etc/apache2/sites-available/000-default.

Then add these lines outside a Directory directive.

ProxyRequests Off
<Proxy *>
AddDefaultCharset off
Order deny,allow
Allow from all #
</Proxy>

RewriteEngine on
RewriteRule ^(.+) $1 [P]
ProxyPassReverse / $1

Thats basically it. So if you have a dns setup where something.lan.tld.com resolves to your main firewall from the outside. But on the inside resolves to a local webserver, and you have defaulted port 80 to this new gateway machine you should be able to access the internal machine from outside.

The thing to note is that remotely something.lan.tld.com will resolve to your public ip. And locally it will resolve to a local lan ip. That allows the rewrite and proxy rule to work correctly. Since it just rewrites the same thing and proxies for it the gateway server has to be able to resolve the internal names correctly.

Example:

ticket.lan.somecompany.com resolves to a world routable address like 151.164.1.8. (externally)

ticket.lan.somecompany.com resolves to a local ip like 192.168.1.5 (internally)

Now you can access that internal resource with the same domain name either internally or externally.

It scales well because you do not have to add a new proxy rule for each specific internal resource, all you have to do is add dns both externally and internally. On top of that you could wildcard your external dns for *.lan.somecompany.com and then all that has to be done is add internal dns for each resource you want to access.

Changing the boot dealy for vmware is pretty easy. One line needs to be added to the vmx configuration file.

bios.bootDelay = "boot delay in milliseconds"


I can tell you from experience don’t even try. The install docs are not that great. All you need to do (if you have bootstrapped with rinse) is install a few packages so that the cpanel installer can complete successfully. I had to install tar, gzip, wget, and perl. The error that I got when first trying to install was related to uncompressing. I contacted Cpanel about the issue and one of their techs logged into my vm to see what was going on. All we did was install the above packages then run the installer. We did not remove any other packages, and I have yet to have an issue (granted its been hours, and it is cpanel, so I expect something to rear its ugly head). But if anyone else is trying to get cpanel installed on a fresh xen-create-image with rinse and centos 5.1 is having issues try installing the above packages before doing anything else to your system and see how you fair.

Running Xen is an easy read. Easy in that it can actually be read cover to cover without becoming tired of mundane drivel. However it is not a glossy overview of Xen. Its 500 plus pages cover everything from using prebuilt images that can be downloaded from jailtime.org, rpath, virtualappliances.net as well as other resources. I do believe this is the first book I have come across that actually explains how to use the different image types (disk and partition). Ok so there are not many Xen books around, and I have not actually read Xen Virtualization: A Practical Handbook or Virtualization with Xen but I will not be surprised if they cover what prebuilt images are in less detail than Running Xen.

65 pages are dedicated to networking, covering bridging, routing, and Nat modes of operation. If you want to include the pages on fire walling there are more than 65 pages of networking related material. Storage backends including LVM, file, partition, nfs, and iscsi are also covered in detail. I was particularly impressed with the coverage of LVM being that many authors would consider it beyond the scope of the book. It is a great resource for any new Xen administrator, and I can wager that it will be valuable for mid-level Xen administrators as well. I defiantly will be revisiting several sections as I get deeper into Xen (read more hardware to test with). For those of you wanting to virtualize Windows, you have not been forgotten. The first several pages of chapter 7 Populating Guest Images shows how to install a windows (or linux) system using an iso or physical rom drive.

What was missing? One backend that was omitted was DRDB. I found this to be somewhat disappointing as live migration can be obtained with less than 3 machines and for those of us on a budget like me scrimping pennies while trying to make a scalable system is a prime directive. Also there was no mention of how to use multiple iso files for installing, how to change them out or if its possible. I have seen that question several times in the #Xen channel on freenode so I know its something people want to do. Other than that I am streched to think of something I would like to do with Xen that is not in some way shape or form covered. Perhaps a section on backups would be a good addition. How to save state and image the operating system with little downtime. I say little because to my knowledge its not yet possible to do hot backups. (if your wondering how to do it think xm save; lv snapshot, xm restore; dd snapshot to image file).

What is the verdict? If you are a new Xen administrator, or thinking about heading down the Xen path this book will be well worth your dime. If you have read The Definitive Guide to the Xen Hypervisor and felt overwhelmed not to worry, this book was written for us mere mortals. (The Definitive Guide to Xen Hypervisor, was a good book but most of it went over my head)

bonnie++ -r  $(free -m | awk '/^Mem/{print $2}') -s $(echo $(free -m | awk '/^Mem/{print $2}')*2| bc)

We use subcommands to expand and automattically use the right parameters for bonnie. Would be easy enough to stick that into a shell script to make a small wrapper and pass along any other arguments to bonnie like user and directory.