Posts Tagged ‘centos’
Recently a developer came to me and said they are starting to see failed builds apparently due to open file handle limitations on the build server. In case your not aware, by default there are limitations on users to ensure they don’t hog the entire resources of a system. Sometimes these limitations need to be adjusted.
In my case the “bamboo” user needed more than 1024 open files on occasion. I determined my system had a maximum number of open files of 1572928.
$ cat /proc/sys/fs/file-max 1572928
And my bamboo user has a limit of 1024 based on the output of the ulimit -a command run as the bamboo user.
$ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 139264 max locked memory (kbytes, -l) 32 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 139264 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
It seems to be an intermittent problem so I’m pretty sure just doubling the number of open files the bamboo user can have will resolve the issue. To make the change you just need to edit /etc/security/limits.conf I added new hard and soft limits by adding these lines.
bamboo hard nofile 2048 bamboo soft nofile 2048
Now lets just make sure the new limits are in place. No need to reboot just log in as the bamboo user again and run “ulimit -a”.
$ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 139264 max locked memory (kbytes, -l) 32 max memory size (kbytes, -m) unlimited open files (-n) 2048 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 139264 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
As you can see open files is now 2048.
Don’t even start with me about how telnet is horrid. Out side of my control but I recently had issues trying to enable telnet on a server. Typically its pretty straightforward.
- yum install telnet-server
- chkconfig telnet on
- chkconfig xinetd on
- service xinetd start
Unfortunately for me this was not working. Every time I tried to telnet to the host after enabling it I would get an error message.
telnet host Trying 203.0.113.10... Connected to host (203.0.113.10). Escape character is '^]'. getaddrinfo: localhost Name or service not known Connection closed by foreign host.
I tried everything I could think of, selinux disabled, ensure localhost in /etc/hosts, connect to ip instead of hostname. Nothing was working. All of my searching was just turning up the obligatory “Don’t use telnet, use ssh”.
While that is generally good advice, in the event you are restricted to using telnet it’s not very helpful. Obviously is something related to name resolution. From both sides the fqdn was resolvable. Then it dawned on me. This environment also has the standard of not using the fqdn as the hostname as set in /etc/sysconfig/network. I had not ensured that the shorthand hostname was resolvable. I resolved the error by adding the hostname in /etc/hosts, but adding a default search domain in /etc/resolv.conf would work just as well.
This leads me back to the error message. Really it had nothing to do with “localhost” or “127.0.0.1″. Had it said “host” Name or service not known I would have chased down the issue much sooner.
I recently had to write an init script for our continuous integration systems remote build agent. Of course this agent is a java jar and does not write its own pidfile. On debian or gentoo (yes I used to rice it) this would not have been an issue. Both have and use start-stop-daemon in their init scripts. start-stop-daemon actually understands that not every process you might want to daemonize writes its own pidfile and thus gives you the ability to daemonize a process, capture its pid, and write a pidfile.
Oh, but not on redhat/centos. No, they have daemon(), which really doesnt do much of anything. <pompus voice> Each process must write its own pidfile </pompus voice>. Well I already have a custom init script, I really don’t want to write a custom wrapper, nor do I want to add yet another dependency to what ought to be a simple task. Perhaps I just don’t see the proper way to do this so I welcome solutions. This feature lack causes lines like this to show up in init scripts.
PID=$(pgrep -n -f "$LAUNCH_CMD")
I don’t know about you but that is horribly ugly and error prone in my eyes.
chkconfig, the other suck.
Yes its documented that the tags chkconfig and description are required or your init script will not support chkconfig. That’s all fine and dandy. I copied my init template from /usr/share/doc/initscripts-8.45.25/sysvinitfiles. It has doc at the end of the initscript after exit is run. This is perfectly valid. There is nothing that is going to cause issues with the execution of the script. I left that doc there as I was developing my script. When it came time to chkconfig –add my script it continually failed. Granted it only took me a few minutes to figure out but what fixed it? Oh yes by removing all the doc at the end of the file. Why is this conflicting with chkconfig no reason that I can see to make it just not work. It would make a bit more sense if the example tags below just overrode the ones at the top of the file.
I needed to install cpanel inside of one of my Centos 5.1 domU. The centos install was bootstrapped by the default rinse configuration (see xen-tools with my patch that you can see from a previous post). If you follow the install docs you will have tried to uninstall openldap etc …. (more…)
After more testing I have narrowed the issue that I left off with to a problem with the yum conf inside of the chroot. (more…)
ABOUT
a system administrators mutterings
Identica
Twitter
LinkedIn