Archive for April, 2010

I’m curious what anyone who reads this blog thinks. My first reaction when someone mentions Ubuntu server is to grab the nearest trout and start slapping. Don’t get me wrong I like Ubuntu. It’s very nice on a workstation, and suitable for my wife, mother, aunt, etc …. But do you really think its good enough for prime time in the data center? According to a server-survey conducted by the Ubuntu marketing team almost 80% of users see Ubuntu as ready for mission critical use.

I was quite shocked when I first saw that. Then I dug a bit deeper. Turns out they announced surveys “variety of Ubuntu forums, websites and other channels”. They do note that “consequently, the results are focuses on Ubuntu users and not general Linux users”. I am not saying that I would _never_ run Ubuntu as a server. I just don’t think its proven itself yet. Perhaps in another LTS release or two I may consider it. I also know there is a chicken and egg paradox there. It of course won’t prove itself until people start putting it to the test.

I think this also brings up another interesting point. The survey noted that “95% of respondents considered hardware support as important to very important”. Included in the survey was a chart that listed Ubuntu users hardware preferences. Not surprisingly Tower/Desktop PC topped the list, but Dell servers, HP/Compaq x86 Servers, IBM x86 servers, and SUN x86_64 servers were also on the list. At the time of the report “Ubuntu Server Edition did not come pre-installed on machines from any major provider”. What happened to the desire for “Enterprise Support”?

I have long been greatly annoyed by “Enterprise Support”. More often than not it just gets in my way and to stay supported I have to do things in a less than optimal way. It would not hurt my feelings if the demand for “Enterprise Support” decreased and the demand for “knowledgeable individuals” increased. I would love to take the money spent on “Enterprise Support” and channel it back into training. Unfortunately I just don’t see that happening.

So what do you think? Do you think Ubuntu is ready for production? What does the lack of vendor support say about Ubuntu or the direction of “Enterprise Support” needs? Are you running Ubuntu in production (for anything serious, not your home server …)? Have any good or bad Ubuntu server stories to share?

Monitoring and analyzing performance is an important task for any sysadmin. Disk I/O bottlenecks can bring applications to a crawl. What are IOPS?  Should I use SATA, SAS, or FC? How many spindles do I need? What RAID level should I use? Is my system read or write heavy? These are common questions for anyone embarking on an disk I/O analysis quest. Obligatory disclaimer: I do not consider myself an expert in storage or anything for that mater. This is just how I have done I/O analysis in the past. I welcome additions and corrections. I believe it’s also important to note that this analysis is geared toward random operations than sequential read/write workloads.

Let’s start at the very beginning … a very good place to start. Hey it worked for Julie Andrews … So what are IOPS? They are input output (I/O) operations measured in seconds. It’s good to note that IOPS are also referred to as transfers per second (tps). IOPs are important for applications that require frequent access to disk. Databases, version control systems, and mail stores all come to mind.

Great so now that I know what IOPS are how do I calculate them? IOPS are a function of rotational speed (aka spindle speed), latency and seek time. The equation is pretty simple, 1/(seek + latency) = IOPS. Scott Lowe has a good example on his techreplublic.com blog.

Sample drive:

• Model: Western Digital VelociRaptor 2.5″ SATA hard drive
• Rotational speed: 10,000 RPM
• Average latency: 3 ms (0.003 seconds)
• Average seek time: 4.2 (r)/4.7 (w) = 4.45 ms (0.0045 seconds)

Calculated IOPS for this disk: 1/(0.003 + 0.0045) = about 133 IOPS

It’s great to know how to calculate a disks IOPS but for the most part you can get by with commonly accepted averages. Of course sources vary but from what I have seen.

Rotational Speed (rpm)	IOPS
5400	50-80
7200	75-100
10k	125-150
15k	175-210

Should I use SATA, SAS or FC? That’s a loaded question. As with most things the answer is “depends”. I don’t want to get into the SATA vs SAS debate you can do your own research and make your own decisions based on your needs, but I will point out a few things.

• SATA only gets up to 10k (at the time of this writing)
• SATA is only 1/2 duplex (From Tomak in comments)
• Differences in reliability (MTBF, BER) interesting article on Real Life Raid Reliability
• See differences in Native Command Queuing (NCQ) and Command Tag Queuing (CTQ)

These factors are  key considerations when choosing what kind of drives to use.

What RAID level should I use? You know what IOPS are, how to calculate them and determined what kind of drives to use, the next logical question is commonly RAID 5 vs RAID 10. There is difference in reliability, especially as the number of drives in your raid-set increases but that is outside the scope of this post.

As you can see in the table above, writes are where you take your performance hit. Now that the penalty or RAID factor is known for different raid levels we can get a good estimate of the theoretical maximum IOPS for a RAID set (excluding caching of course). To do this you take the product of the number of disks and IOPS per disk divided by the sum of the %read workload and the product of the raid factor (see write operations column) and %write workload.

Here is the equation:

d = number of disks
dIOPS = IOPS per disk
%r = % of read workload
%w = % of write workload
F = raid factor (write operations column)

Wait a second, where am I supposed to get %read and %write from?

You need to examine your workload. I usually turn to my favorite statistics collector, sysstat.  sar -d -p will report activity for each block device and pretty print the device name. I am assuming you already know what block device you are looking to analyze but if your looking for the busiest device just look in the tps column.  the rd_sec/s and wr_sec/s columns display number of sectors read/written from/to the device. To get the percentage of read or writes divide rd_sec/s by the sum of rd_sec/s and wr_sec/s.

The equations:

An example from my workstation:

Average for sdb rd_sec/s = 1150.80
Average for sdb wr_sec/s = 1166.53

As you can see my workstation read/write workload is pretty balanced at 49.6% read, and 50.3% write. Compare that to a cvs server (don’t get me started on how bad cvs is, its just something I have to deal with).

Average for sdb rd_sec/s = 27.78k
Average for sdb wr_sec/s = 2.07k

This server workload is extremely high on reads. Ok time to analyze the performance.

In and of itself being a heavy read workload is not a problem. My problem is user complaints of slowness. I note (again from sysstat collected metrics) that the tps or average IOPS on this device is about 574. Again thats not an issue in and of itself, we need to know what we can expect from its subsystem. This device happens to be SAN based storage. The raid set its on is comprised of 4 10kRPM FC drives in a raid 10. Remember from the table above that IOPS for a 10kRPM drive are in the 125-150ish range. We need to calculate the expected IOPS from that raid set using the IOPS equation above, our measured workloads for read/write, the number of disks, and the raid level (10 and 1 are treated the same).

Using the high end of the scale for 10kRPM IOPS per drive results in a maximum theoretical IOPS of 561.79, thats pretty close to what I am observing (remember cache is not taken into account). So based on these numbers it looks like my storage subsystem is saturated. I guess I better add some spindles. Unfortunately there is no historical data for this system so I have no way of knowing how many tps I need to aim for.

Don’t get stuck where I am and have to guess how many spindles need to be added to reduce the pain, start recording your trends now! Even better, once you start collecting your statistical information go ahead and set an alert for 65% or 70% utilization of theoretical max IOPS for an extended period as well as increasingly bothersome alerts going up from there. It’s never good to have to react to performance issues, always better to be proactive. There was absolutely nothing wrong with the sizing of this example raid set 2-4 years ago. Had it been under monitoring the entire time with proper thresholds set a proper plan could have been made, and spindles could have been added before causing users any pain.

If you want to use sysstat like I did, you might find this Nagios plug-in that I wrote helpful check_sar_perf. I use it with Zenoss, but it could be tied into any NMS that records the performance data from a Nagios plug-in.

Go forth, collect, analyze and plan so your users aren’t calling you with issues.

• http://wiki.horde.org/HardwareRequirements
• http://don.blogs.smugmug.com/2007/10/08/hdd-iops-limiting-factor-seek-or-rpm/
• http://blogs.techrepublic.com.com/datacenter/?p=2182
• http://www.sqlservercentral.com/blogs/sqlmanofmystery/archive/2009/12/07/fundamentals-of-storage-systems-raid-an-introduction.aspx
• http://blog.aarondelp.com/2009/10/its-now-all-about-iops.html
• http://adamstechblog.com/2009/02/10/how-to-calculate-iops-ios-per-second/
• http://www.performancewiki.com/diskio-tuning.html
• http://vmtoday.com/2009/12/storage-basics-part-i-intro/
• http://vmtoday.com/2009/12/storage-basics-part-ii-iops/
• http://vmtoday.com/2010/01/storage-basics-part-iii-raid/
• http://vmtoday.com/2010/01/storage-basics-part-iv-interface/
• http://vmtoday.com/2010/03/storage-basics-part-v-controllers-cache-and-coalescing/
• http://vmtoday.com/2010/04/storage-basics-part-vi-storage-workload-characterization/
• http://www.codecogs.com/components/equationeditor/equationeditor.php

As anyone in the search industry can tell you, getting website traffic for your primary keywords and phrases in organic search can mean the difference between the success or failure of your website.  If you are a site owner, website failure can have a detrimental impact on the profitability of your company.  If instead you are an SEO or web consultant, this can make or break your reputation in the industry.

There are numerous steps that can be employed to steer us toward success in search, with a few listed here.  Start by selecting the keywords that your customers likely search for.  Target those keywords in the content of your website.  Build links to your website, using keyword anchor text where possible.  Monitor your search results to see where you are gaining position on the results page, where you are losing position, and use these insights to hone your processes and allocate resources.  In other words, it can take a lot of work!

For this reason, automation tools can be invaluable for these tasks.  Caphyon (http://www.caphyon.com) has just released version 7.2 of their Advanced Web Ranking (AWR) and Advanced Link Manager (ALM) suite, intended for this very purpose.

Starting Out – Advanced Web Ranking

Let’s pretend we are a small sysadmin consulting firm – we have started an initiative to improve our website ranking (and hopefully traffic!), but we face stiff website competition from larger, established sites, both locally and nationwide.  Where do we stand?  Do we currently rank for any of the keywords relating to our core products or services?  Are there other keywords that we should be watching?  What about our competition?

We can work out all of these answers by hand, or we can setup Advanced Web Ranking and have it do much of the work for us – saving a lot of time and resources.  To get started we just need to step through a quick Project Wizard.  Select which search engines you want, which keywords you want to keep an eye on, which websites you want to track, and your communication settings.  AWR will then take off and start collecting stats!

The result is a collection of data which shows us how we rank for each of our keywords (per search engine), the top sites for a given keyword, and whether we moved up or down since our last check.  These results can be viewed in various charts, graphs, and reports – all complete with filtering and sorting options that make it easy to parse out the information that is most useful to us.  These results are stored on your computer for comparison with future data.

One feature I like is that as you type your keywords into the project wizard, a dropdown box populates with possible completions of your word or phrase, as well as the number of google results for each. This keyword research tool alone can give you valuable keyword ideas on-the-fly without any switching between apps.

Automation?

“Spectacular,” you may be saying, “but remembering to open Advanced Web Ranking and click ‘Start Update’ once a week or twice a month really sounds like a lot of work!”  Fortunately, Advanced Web Ranking has features that can handle this as well.  I can schedule this update to run once a week at a given time, e-mail selected reports to myself (custom reports, if I so desire), and trigger special notifications if certain incidents occur (for example, a ranking drop of 5 or more since last time).

Now What?

Now that we have some baseline for how our site is performing against our competition, what can we do to leverage this information and improve our site’s search engine ranking?  If we are not ranking as well as we hoped, our first reaction should be to check the content of our pages to see how they perform.  Once again, Advanced Web Ranking has a tool that can help.  The Keyword Analysis tool looks at basic page metrics (title, links, image ALTs, headings, etc) and shows us how well they perform for a number of keywords.  This is excellent groundwork for starting a more in-depth manual review later.

Maybe we rank fine for our initial set of keywords, but wonder what else we might be missing.  Are there other, related keywords that we should be targeting?  The built-in Keywords Research Tool allows us to select from a number of suggestion sources including Google Suggest, WordTracker, Yahoo Related Keywords Search, and more.  From the full list of suggestions, we can select the ones we desire and add them directly to our project – quick and easy.

So far, Advanced Web Ranking has shown us how well we are doing, and has given us tools to help target the on-site factors of our website that need work (or need to be created altogether!).  But what about our off-site factors?  What about links?  On-site content is only one piece of the puzzle, so Caphyon steps in yet again with today’s second application – Advanced Link Manager.

Starting Out – Advanced Link Manager

For years, the major search engines have used links as a primary ranking factor for website search results.  Ignoring web spam and other complexities for the moment, we can consider the quantity of links as indicative of popularity our site, the authority of the linking domains as indicative of the importance of our site, and the text of the links as indicative of what our site is about.  This leads to new questions – How many links does our website have?  Who links to us?  What link text do other sites use for us?  And again… What about our competition?

Advanced Link Manager is meant to help answer those questions and give us the information that we need to beat the competition and stay competitive on the web.  Starting a new project in ALM is a simple process – we first enter the domain of the site we want to monitor, then select the search engines that we want to use for results – that’s it..  ALM generates results using a multi-step process which begins by using search results to find linking pages.  Each linking page is then verified (does it still exist?) and classified (text of the link? authority of the page? etc).

Automation and Other Features

Much like Advanced Web Ranking discussed above, ALM allows you to gather, parse, and view your data all in one place.  You can create custom reports, schedule tasks, and e-mail reports for yourself.  The information gathered from each linking page (referrer) includes alexa rank, outbound links and backlinks as well as link text, URL title, and whether the links are no-followed. Serving as an excellent link popularity tool, ALM gathers many aspects of link data, not just one or two.

Advanced Link Manager’s link results are initially gathered as “Unverified Referrers” until each domain (along with specs and details) is checked and verified – only then are they moved to the final “Referrers” list.

Now What?

With this newfound knowledge of our inbound link status, as well as the link status of our competitors, we find a place to start with our link building.  We can look at our competitors links and start trying to duplicate them, starting with the highest quality links first.  Maybe there is an article site where they have been making posts.  Maybe they have been participating in several forums.  Advanced Link Manager makes it easy to discover these trends and use them to your own benefit.

Another aspect of backlink profiling is using it to discover what the public thinks your site is about.  Perhaps we use Advanced Link Manager’s Referrer report to go through the backlinks of our command-line sysadmin blog, only to discover a surprisingly large amount of link text referring to “virtualization” or “virtualization blog”.  This could indicate that a previously unknown portion of our visitor base sees us primarily as an expert on virtualization.  This information could prompt new types of posts, or even new types of advertising, targeting this visitor segment.

Conclusion

This combination of software by Caphyon (web ranking and link manager) can lend powerful capabilities to your internet marketing arsenal.  The ability to monitor the ranks and backlinks of our websites (as well as those of our competitors) can save a massive amount of time versus the old alternative of parsing such information by hand.

Advanced Web Ranking and Advanced Link Manager are both easy to use, with help files and tutorials that walk you through the basics of starting a project and making use of the data.  The look and feel of each is very similar, so becoming acquainted with one makes learning the other even easier.  You can expect to have your first project  ready within minutes of starting the program.

A 30-day trial of each is available, giving ample time to start a few projects and familiarize yourself with the feature-set.  Purchase options are broken down into 4 tiers, from Standard to Server.  This makes the software equally viable for small businesses or professional search marketing firms.

And finally, one last feature that is big for myself (and likely most of the readers here) is that this software is all cross-platform (Win/Mac/Linux). So in other words – you can have it while still avoiding Windows. Always a good thing.

Caveats

As always, any discussion of rank checking software and backlink profile tools comes with a caveat – Google officially frowns on such software, so exercise a bit of caution when performing your updates.  For example, set your delay times to “human” values and schedule automatic updates (to help avoid that pesky urge to check your rankings obsessively).

Credits

This post was written by Mark Bosma of LeapfrogSEO.com – small business seo and web consulting. Feel free to drop a comment here with any additional feedback or questions you have on my experience with AWR and ALM. Thanks!

I don’t think I bothered to complain here but I sure sent my fair share of nasty grams. In fact Dell became one of my four letter words after I heard they were firmware locking Gen11 servers to only dell drives. Of course it was a mistake but I loved to unashamedly repeat the famous quote from Howard Shoobe.

“There are a number of benefits for using Dell qualified drives in particular ensuring a positive experience and protecting our data.”

While I wish they had not tried to do this in the first place, I do want to be sure to give kudos to Dell for listening to reason. Matt Simmons is reporting that Dell is reversing its position on 3rd party drives. From the post:

DELL H-SERIES CONTROLLER BLOCKING OF 3RD PARTY DRIVES
TO BE REMOVED IN Q2FY11

At Dell, we are committed to our customers. We listen to our customers. And, to ensure our customers have a positive experience with our products, we validate and certify all of our PowerEdge servers and components.

Previously, Dell prevented the use of non-Dell hard drives attached to PERC H700 and H800 controllers in our 11th generation of PowerEdge servers as we could not certify these non-Dell drives. However, based on customer feedback, we are now reversing this decision and allowing their use (without support).

Dell will no longer prevent customer use of non-Dell hard drives attached to PERC H700 and H800 controllers on our 11th generation of PowerEdge servers. A Q2FY11 PERC Firmware Update is planned to allow non-Dell hard drives to be used with these controllers.

We strongly recommend choosing Dell drives as we have validated and certified our drives and can support any issues that may arise.

FOR MORE INFORMATION:

• www.dell.com/poweredge
• Dell Hard Drives Landing Page
• Why Dell Drives Paper

NOTE: There is no change to the current hard drive carrier policy. Hard drive carriers will not be sold separately (3rd party drives are still not supported on Dell Enterprise systems).

Dell, welcome back to my approved vendor list.

Recently a developer came to me and said they are starting to see failed builds apparently due to open file handle limitations on the build server. In case your not aware, by default there are limitations on users to ensure they don’t hog the entire resources of a system. Sometimes these limitations need to be adjusted.

In my case the “bamboo” user needed more than 1024 open files on occasion. I determined my system had a maximum number of open files of 1572928.

$ cat /proc/sys/fs/file-max
1572928

And my bamboo user has a limit of 1024 based on the output of the ulimit -a command run as the bamboo user.

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 139264
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 139264
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

It seems to be an intermittent problem so I’m pretty sure just doubling the number of open files the bamboo user can have will resolve the issue. To make the change you just need to edit /etc/security/limits.conf I added new hard and soft limits by adding these lines.

bamboo        hard    nofile    2048
bamboo        soft    nofile    2048

Now lets just make sure the new limits are in place. No need to reboot just log in as the bamboo user again and run “ulimit -a”.

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 139264
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 2048
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 139264
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

As you can see open files is now 2048.