The introduction gives an overview of automation, why you would choose CFEngine over or to complement another tool and a brief history of CFEngine. CFEngine has a strong computer science background. It’s nice to know what ideas lead to different design decisions. The introduction gives this background and Diego does a great job of making it a very easy read.

Chapter 2 covers installing CFEngine, bootstraping with an initial configuration , and the obligatory “hello world”. If you have the electronic version of the book (I highly recommend it, but you should probably buy the dead tree version too so you can read the book during air-plane take-off and landing) this is where you will see an outstanding feature.

Coloured syntax highlighting as well as fine little bullet dots that make the very detailed description of the policy easy to reference.  This is a great feature, it makes the policy infinitely easier to read and digest. But just like a late night info-mercial, “Wait, that’s not all folks!”.

Blow up that picture and see near point 3 “In the files:”. Yes that is a hyper-link and it gives you quick access to a cross-referenced index of all the section heading in the CFEngine Reference Manual. All CFEngine keywords in the electronic versions of the book are linked to this index to provide easy access to the official reference documentation about that concept. That is awesome, let me see you do that with the dewy decimal system! (No, I never did learn it. I recall telling my teachers in grade school that I would always have a computer to be able to look that stuff up. Hah, I remember the librarian scoffed and my teacher chuckled) But WAIT! There’s MORE! Yes. Really there is, I’m not kidding.

You can use Diegos’ index as a search provider in your browser. You can use searchplugins.net to create a search for it. Just enter http://cf-learn.info/ref/TEST in the Search URL box. Set the title to “CFEngine Reference Manual”, hit create plug-in and click the link to install the search plug-in. Once that’s done you can create a keyword to use in the address bar as well by clicking your search provider drop-down icon, going to “Manage Search Engines”, selecting the provider and finally “Edit Keyword”. Now you can use the search box in your browser or you can use your keyword in you address bar to quickly and efficiently search the CFEngine Reference Manual. Ok lets get out of chapter two already, I don’t want to completely turn into the shamwow guy.

Chapter 3 covers some more basics including desired-state configuration, promise theory, and convergent configuration which again I think are important to understand why CFEngine works the way it does. The components CFEngine is made of are explained in enough detail to understand the purpose of each, but not so much as to get fatiguing. A high level language structure overview and a section covering other sources of information are also included. I particularly like that early in the book a multitude of resources are laid out. If you get tired of reading the book you already know where to go to take a break and read about the same subject.

Chapter 4 gets down into actually using CFEngine to get things done. There are some good practical examples that are directly useful like configuring sshd configuration parameters and restarting sshd if its config gets changed as well as an example of how to do local user management. More importantly the examples shown in this chapter show techniques that can be used for many different applications. This chapter also covers the use of classes to specify policy that applies to nodes with specific characteristics or conditions. I should mention here that the book mentions that classes will be renamed to contexts so as to not confuse people with the Object-Oriented Programming definition of class which is not the same as CFEngines use of the term. I was able to get this confirmed by Mark Burgess (original author, founder, and Chief Thinking Officer) of CFEngine. I personally am still warming up to this upcoming change, I understand the desire to change it to better communicate the meaning to people who may be more familiar with the OOP use of class and that does fall in line with a core principal of knowledge management but I am still a little sad inside. I always thought the OOP use of the word class was awkward. Anyway, moving on.

Chapter 5 really starts showing more patterns of how to use the declarative policy language to describe intentions and help you “think like CFEngine”. It’s a very good build on Chapter 4 expanding the ideas and getting a little more generic to help you apply the language in different situations. There is a good explanation of Hierarchical Copying much better than others I have read. Also there is pretty good coverage of using arrays for passing configuration to bundles and a very good explanation of how to set default values for bundle parameters. I think this is very important for writing re-usable policy so be sure to read that section twice.

There are two places I think this chapter is slightly lacking. The coverage of arrays doesn’t cover using multi-dimensional arrays and how there are different techniques for dealing with them. I find myself using multi-dimensional arrays frequently when I want to do a similar thing multiple times like define multiple nagios checks and make a single usebundle call. Also I found the section on controlling promise execution order to be a little short. It’s true if you think order matters, you should step back and think if the order really matters or not, and then if you still think order matters, think about it some more. But there are times when order does matter, I tend to find this with variable definitions. I commonly will need to get variables set on the second pass after a class has been defined, and then set another class once that variable has been defined so that I know I can finally run a command. A few practical examples in this section would have been appreciated.

Chapter 6 is about advanced topics. It covers running CFEngine in multiple environments, and unit testing bundles. These are both great and I think there is room for a book to expand on chapters 5 and 6 just about policy writing techniques. Who wants to write that one?

Finally there is an appendix that covers Emacs configuration for editing CFEngine policy files. Well, I don’t have much to say about this section, I am a vim user. I am trying out orgmode but my vim habits are deeply ingrained. If you haven’t used Emacs before and you want to take your brain for a spin go check out Organize Your Life In Plain Text! by Bernt Hansen. He has a pretty amazing orgmode configuration, I barely understand how to use it, let alone much of the configuration. I would have appreciated an appendix B for vim. Neil Watson has a CFEngine 3 syntax highlighting plug-in for vim that I use.

Well, that’s it. So basically I had 3 wants after reading this book, and all centred around more information. I say that’s pretty good. After all you cant please everyone, and I have been known to be highly critical. It really is one of the best technical books I have read in a while. I didn’t find it a dry read at all.

I hope you didn’t make it this far in my diatribe, hopefully you bailed out early on went and bought book and are reading it instead of this. If not, you should head on over to the O’Reilly store and buy a copy. Don’t forget to use the discount code above to save a few bucks.

©2012 cmdln.org (a sysadmin blog). All Rights Reserved.

Just over a week ago I noticed it had been bought by Corel and its been re-branded to AfterShotPro. I really hope they continue to develop it and provide Linux support.

©2012 cmdln.org (a sysadmin blog). All Rights Reserved.

I thought it would be fun to write a CFEngine policy to set it up so here it is. Just install CFEngine 3, configure your settings in the policy file and kick it off with cf-agent –KIf ~/.cfagent/inputs/boxcryptor_dropbox_encfs. (You will need the standard library, and the policy is classed for ubuntu, but it should be easy enough to add support for another distro).

body common control {

    bundlesequence  => {
                        "main",
                        };

    inputs          => {
                        "cfengine_stdlib.cf",
                        };
}

bundle agent main {
# Setup encfs on one of your dropbox folders for use with boxcryptor
    vars:
        "settings[user]"      string => "user";
        "settings[group]"     string => "group";
        "settings[encfs]"     string => "/home/user/Dropbox/encfs";
        "settings[mount]"     string => "/home/user/Documents/Safe";
        "settings[password]"  string => "supersecret";

    methods:
        "required_software" 
            usebundle   => install_boxcryptor,
            action      => if_elapsed("360"),
            comment     => "Install software to work with boxcryptor, but only
                            verify and try once every 6 hours";

        "encfs" 
            usebundle => encfs_init_boxcryptor("main.settings"),
            comment   => "If no encfs is found, initalize one compatible with boxcryptor";

        "encfs" 
            usebundle => encfs_mounted("main.settings"),
            comment => "Ensure the encfs is mounted somewhere we can write to it";
}

bundle agent install_boxcryptor{
# This is just a meta to make sure the deps for boxcryptor are all installed

    packages:
        ubuntu::
            "python-pexpect"
                package_policy  => "add",
                package_method  => apt,
                comment         => "python-expect is needed for the custom 
                                    scripts to initalize and mount encfs";

    methods:
        # would be nice to find a dropbox fuse implimentation that works so we
        # dont have to rely on the nautilus dropbox plugin and thus be able to 
        # this on a headless machine
        #"dropbox"    usebundle => install_dropfuse;
        "dropbox" 
            usebundle => install_dropbox;

        "encryption" 
            usebundle => install_encfs;
}

bundle agent encfs_init_boxcryptor(config){

 vars:
    "temp_script"
        string => "/tmp/init_boxcryptor_encfs.py";

    "init_boxcryptor_encfs_template" 
        comment => "boxcryptor only supports a subset of encfs options, 
                    this script initalizes an encfs filesystem with those 
                    options. Its an interactive process so we needed expect.",
        string => "#!/usr/bin/env python
# encoding: utf-8

import pexpect
import sys

child = pexpect.spawn('encfs $($(config)[encfs]) $($(config)[mount])')
child.logfile = sys.stdout
child.expect('>')
child.sendline('x')
child.expect('The following cipher algorithms are available')
child.sendline('1')
child.expect('Selected key size')
child.sendline('256')
child.expect('filesystem block size')
child.sendline('1024')
child.expect('The following filename encoding algorithms are available')
child.sendline('3')
child.expect('Enable filename initialization vector chaining')
child.sendline('n')
child.expect('Enable per-file initialization vectors')
child.sendline('n')
child.expect('Enable block authentication code headers')
child.sendline('n')
child.expect('Add random bytes to each block header')
child.sendline('0')
child.expect('Enable file-hole pass-through')
child.sendline('y')
child.expect('New Encfs Password')
child.sendline('$($(config)[password])')
child.expect('Verify Encfs Password')
child.sendline('$($(config)[password])')
child.expect(pexpect.EOF, timeout=None)
child.close()
sys.exit(child.exitstatus)";

    classes:
        "encfs_exists" 
            expression  => fileexists("$($(config)[encfs])/.encfs6.xml"),
            comment     => "Determine if there is an existing encfs";
        "mount_exists"  
            expression  => isdir("$($(config)[mount])"),
            comment     => "Make sure there is a place to mount the encfs";
 
 files:
   !encfs_exists::
        "$($(config)[mount])/."
            create => "true",
            perms => mog("700", "$($(config)[user])", "$($(config)[group])"),
            comment => "Make sure that there is a place to mount the encfs";

        "$(temp_script)"
            create => "true",
            perms => mog("700", "$($(config)[user])", "$($(config)[group])"),
            edit_line => append_if_no_line("$(init_boxcryptor_encfs_template)"),
            edit_defaults => empty,
            classes => if_repaired("script_installed"),
            comment => "If no encfs filesystem is found place the initalization script
                        so we can execute it.";

    encfs_exists::
        "$(temp_script)"
            delete => tidy,
            classes => if_repaired("performed_cleanup"),
            comment => "Delete the temporary initalization script if an encfs already exists";

    commands:
        !(encfs_exists|initalized_boxcryptor_encfs)::
            "$(temp_script)",
                contain => setuidgid_silent("$($(config)[user])", "$($(config)[group])"),
                classes => "initalized_boxcryptor_encfs",
                comment => "If no encfs is detected and we havent yet successfully initalized
                            one fire the expect script to do so.";

    reports:

        script_installed::
            "boxcryptor_initalize: I installed a script to do my work";

        initalized_boxcryptor_encfs::
            "boxcryptor_initalize: I couldn't find an existing encfs at $($(config)[encfs]) so I initalized one";

        performed_cleanup::
            "boxcryptor_initalize: I cleaned up after myself";


}

bundle agent encfs_mounted(config){
    vars:
    "temp_script"
        comment => "Mounting an encfs filesystem requires a password, this script uses expect so we can supply one interactively",
        string => "/tmp/mount_encfs.py";

        "mount_encfs_tpl" string => "#!/usr/bin/env python
# encoding: utf-8

import pexpect
import sys

child = pexpect.spawn('/usr/bin/encfs $($(config)[encfs]) $($(config)[mount])')

child.logfile = sys.stdout
child.expect('EncFS Password')
child.sendline('$($(config)[password])')
child.expect(pexpect.EOF, timeout=None)
child.close()
sys.exit(child.exitstatus)";

    classes:
        "encfs_mounted" 
            expression  => returnszero("/bin/grep --silent -P ^encfs\s$($(config)[mount]) /etc/mtab", "noshell"),
            comment     => "Determine if the encfs filesystem is currently mounted or not"; 

    files:
        !encfs_mounted::
            "$(temp_script)"
                create          => "true",
                perms           => mog("700", "$($(config)[user])", "$($(config)[group])"),
                edit_line       => append_if_no_line("$(mount_encfs_tpl)"),
                edit_defaults   => empty,
                classes         => if_repaired("placed_script"),
                comment         => "If we dont detect that our desired encfs filesystem is mounted
                                    drop the expect script in place. We use the expect script
                                    because we need to provide a password";

        encfs_mounted::
            "$(temp_script)"
                delete  => tidy,
                classes => if_repaired("performed_cleanup"),
                comment => "If the encfs filesystem is mounted we have no need for the mount script to be in place, delete it";

    commands:
        !encfs_mounted::
            "$(temp_script)",
                contain => setuidgid_silent("$($(config)[user])", "$($(config)[group])"),
                classes => if_else("repaired_mount", "failed_repair_mount"),
                comment => "If the enfcs filesystem is not mounted execute our mount script";

    reports:
        placed_script::
            "boxcryptor_mounted: I installed a script";

        repaired_mount::
            "boxcryptor_mounted: $($(config)[encfs]) was not mounted on $($(config)[mount]), but we fixed it";

        performed_cleanup::
            "boxcryptor_mounted: I cleaned up aftermyself";

        !encfs_mounted.failed_repair_mount::
            "boxcryptor_mounted: I dunno what happened, but I couldnt mount the encfs volume check your settings!";

        !repaired_mount.encfs_mounted::
            "boxcryptor_mounted: Everything was as expected";
 
}




bundle agent install_dropbox{
    vars:
        ubuntu::
            "packages"  slist => { "nautilus-dropbox" };

    packages:
        ubuntu::
            "$(packages)"
                package_policy  => "add",
                package_method  => apt,
                comment         => "Install dropbox packages with apt";

}

bundle agent install_encfs {
# Install encfs
    vars:
        ubuntu::
            "packages" slist => { "encfs", "fuse-utils" };
 
    packages:
        ubuntu::
            "$(packages)"
                package_policy  => "add",
                package_method  => apt,
                comment         => "Install encfs with apt";
    
}

###########################################################################
#                               body parts                                #
###########################################################################

body contain setuidgid_silent(x,y){
    exec_owner => "$(x)";
    exec_group => "$(y)";
    no_output => "true";
}



###########################################################################
#                            Not really useful                            #
###########################################################################
# I couldnt get dropfuse to work, nor did I try very hard
bundle agent install_dropfuse{
    vars:
        "dropfuse_src" 
            string => "https://raw.github.com/arekzb/dropfuse/master/dropfuse.py",
            comment => "Location to get dropfuse binary from since its not packaged";

        "dropfuse_bin"
            string  => "/usr/local/bin/dropfuse.py",
            comment => "Location to install the dropfuse library";
    
        ubuntu::
            # TODO: find out if fuse-utils is really required for this
            "packages" 
                slist   => { "fuse-utils", "python-fuse" },
                comment => "Packages required";

    files:
        "$(dropfuse_bin)"
            perms       => mog("755", "root", "root"),
            ifvarclass  => "dropfuse_installed",
            comment     => "If dropfuse is installed ensure that 
                            it is executable";
    packages:
        ubuntu::
            "$(packages)"
                package_policy => "add",
                package_method => apt, 
                comment        => "Install the selected package with apt if
                                   its not installed";
    classes:
        "dropfuse_installed" 
            expression  => fileexists("$(dropfuse_bin)"),
            comment     => "Determine if the dropfuse binary is installed";

    commands:
        !dropfuse_installed::
            "/usr/bin/wget $(dropfuse_src) -O $(dropfuse_bin)",
            comment => "Install the dropfuse binary, its not currently 
                        packaged so just get it from the authors github account";

}

edit: February 27, 2012 at 2:52 pm

Thanks @highdraw for pointing out the mac installer.

©2012 cmdln.org (a sysadmin blog). All Rights Reserved.

The biggest improvement is you no longer have to specify the interface you want the route on. Instead you specify a regular expression of the ip that would be on that interface and the specific interface is found.

Second major improvement is the direct use of ip route del/add to update currently running system instead of restarting all networking.

Also a small change in the declaration of the routes that is less repetitive and requires less typing.

Take care in crafting your ip regular expression especially if you are not using a standard A, B or C class network.

Now lets see how it could be used.

Consider that you have multiple interfaces on a host, one interface connects to a “management” network, another connects to an application network, and another connects to a database network. Your default route is for the application network because you need to talk to the rest of the world.

Now consider what other networks that your host might need to connect to for “management”. Maybe you have a network that your backup servers sit on, and maybe you have another network that is for vpn clients, and you want vpn clients to be able to ssh to this host on the management interface.

We don’t need to add any routes for the db network or for the application network but for these other two networks we have to add the routes on the host.

172.16.208.0/25 is the network your backup servers sit on. 192.168.5.0/24 is your vpn client access network. Your management network on the host we are considering is 192.168.35.0/24.

You could define a common bundle to define these variables.

bundle common site {

vars:
ipv4_192_168_35::
"mgmt_ip_regex"
string  => "192\.168\.35\..*",
comment => "Regex to match the ip that is on the management network, we use this
to figure out which is the management interface";

"mgmt_routes[172.16.208.0/25]"
string  => "192.168.35.254",
comment => "Needed to talk to the backup servers over the management network";

"mgmt_routes[192.168.5.0/24]"
string  => "192.168.35.254",
comment => "Needed to talk to vpn clients on the vpn client network";

}

This is how you would call the rh_update_routes bundle.

bundle agent main{

methods:
"routing"       usebundle   => rh_update_routes("$(site.mgmt_ip_regex)", "site.mgmt_routes"),
comment => "Setup Management routes defined in bundle common site";

}

Lets walk through the decision tree a little bit. In bundle common site mgmt_ip_regex and mgmt_routes is only defined if the client matches the class ipv4_192_168_35. We wouldn’t want to define the same routes for a host on the 192.168.5.0 network because the gateway would be wrong, so that’s why you need to be careful to restrict your variable definition based on classes.

The mgmt_ip_regex is needed to dynamically determine which interface these routes will apply to. And really we only need the interface for this so that we can update the proper file for persistent routes (in redhat|centos its /etc/sysconfig/network-scripts/route-interfacename).

Lets look at rh_update_routes to see what’s going on.

bundle agent rh_update_routes(ipregex, routes) {
# Expects string, array
# Note: This bundle depricates rh_add_interface_routes. I see no good reason to continue using it.
#
# ipregex is a regular expression that matches the ip on the interface you want these routes added
#   hint: ipregex should match an ip that can communicate with the specified gateway
#         for example if your routing a network via 192.168.0.1 and the network is a /24 network
#         (255.255.255.0 netmask) then you should have an ip in the range 192.168.0.1-254 on the
#         host your trying to add this route on. So a regex of 192\.168\.0\.[0-9]++ would work.
# routes is an array keyed on the network you want to route to with the string value being the gateway to use.
#
# NOTE: Unfortunately the only way I could think of to automatically determine the interface a route
#       needs added for is to use regcmp to compare the ipregex to the array of addresses. It would be
#       better if there was some way to use the iprange function to determine which nic an ipaddress
#       is on, but that does not currently work, or I am thus far to dense to figure out how.
#       So right now I am stuck with using ugly ipaddress regular expressions which can be error prone
#       in construction especially when you start networks that dont fall into octet boundaries
#
#       This causes there to be a limitation of usage on this bundle, you MUST NOT MIX
#       routes that go on seperate interfaces in the same route configuration array. I believe
#       this limitation could be surpassed if we could use the iprange or similar function.
#
# vars:
#    "ipregex_mgmt" string => "192\.168\.0\.[0-9]++";
#    "management[CIDRNETWORK]"
#        string => "GATEWAY",
#        comment => "What do you need this for";
#
#    "management[10.119.156.0/26]"
#        string => "192.168.0.1",
#        comment => "Needed for talking to the special network used for backup servers";
#
# methods:
#    "any" usebundle => rh_add_routes("192\.168\.0\.[0-9]++", "context.management");
#
    vars:
        "nics"          slist => getindices("sys.ipv4");
        #"route_file"    string => "/etc/sysconfig/network-scripts/route-$(interface)";
        "route_index" slist => getindices("$(routes)");

    classes:
        "supported_os" or => { "centos_5", "redhat_5" };

        "$(nics)_matches_ipregex" expression    => regcmp("$(ipregex)", "$(sys.ipv4[$(nics)])"),
            comment => "Determine which network interface has an ip that we are adding routes for.
                        We need to know this so that we can insert the route in the proper
                        file for reboot persistence.";

    files:
        (centos_5|redhat_5)::
            "/etc/sysconfig/network-scripts/route-$(nics)"
                create      => "true",
                perms       => mog("644", "root", "root"),
                edit_line   => replace_or_add("$(route_index).*", "$(route_index) via $($(this.routes)[$(route_index)])"),
                classes     => if_repaired("persistent_route_updated_for_$(route_index)"),
                ifvarclass  => "$(nics)_matches_ipregex",
                comment     => "Replace any conflicting routes and ensure persistent across reboots";

    commands:
        # We only attempt to delete a route if we have modified the persistent route file
        "/sbin/ip route del $(route_index)"
            ifvarclass  => canonify("persistent_route_updated_for_$(route_index)"),
            classes     => "attempted_route_removal_for_$(route_index)",
            comment     => "Delete any possibly conflicting old route before adding the new one";

        # We only attempt to add a route if we have modified the persistent route file
        "/sbin/ip route add $(route_index) via $($(routes)[$(route_index)])"
            ifvarclass  => canonify("persistent_route_updated_for_$(route_index)"),
            classes     => "attempted_route_addition_for_$(route_index)",
            comment     => "Add the new route";

    reports:
        cfengine::
            "Persistent route updated for $(route_index) via $($(routes)[$(route_index)]) on dev $(nics)"
                ifvarclass => canonify("persistent_route_updated_for_$(route_index)");

        !supported_os::
            "Sorry I don't know how to work with this OS";

}

First we get a list of the interfaces on the system using getindices on sys.ipv4. Then we get a list of networks to route for from the configuration array that we passed into the bundle. Next we use the ipregex that we passed into the bundle and we iterate over the network interfaces looking for a match. When a match is found it defines the class $(nics)_matches_ipregex where nics expands to the current iterated nic from the list we built earlier. Thanks to Diego for this pattern. I found it in his upcoming book Learning CFEngine 3, if you haven’t you should go get a copy. The pre-release is available now, and even in its early stage I recommend it.

Now we get to some actual action. In the files section we edit the interface file for the current iterated value of $(nics) but only if we have a match on the regex “$(nics)_matches_ipregex” and we replace any routes for the same network  with the desired route entry or add it if one does not exist using the replace_or_add edit_line bundle from the standard library. If we made any repairs to the file we define the class persistent_route_updated_for_$(route_index) which is used to control when the commands promises are made.

In the commands section we first delete  any route for a matching network. This command may fail if there is no matching route but that’s ok. It’s far easier to implement delete add than to account for the current routing table to avoid the possible unnecessary delete and use replace in stead. (that being said I welcome patches)

Finally we just report if we have made any updates.

I think we are lacking in explained useful examples so I hope someone finds this useful. Any suggestions for improvements or additions are welcome.

UPDATE: 2.19.2012

I had an omission in the bundle, I forgot to restrict the files edit on the $(nics)_matches_ipregex class, It caused every interface to have the specified routes added so please use the updated bundle.

©2012 cmdln.org (a sysadmin blog). All Rights Reserved.

©2012 cmdln.org (a sysadmin blog). All Rights Reserved.

I have recently added these bundles to my library

lib_rh.cf

• rh_add_interface_routes - manage routes on an interface

• create_update_yum_repo - create a yum repo and update the metadata if files change

• set_selinux_disabled - disable selinux, yes its sad but its common

• config_yum_client_repos — configure yum client configs in /etc/yum.repos.d/

lib_local_user_management.cf

• local_users_enforce_password — enforce a local users password, supports updating last day changed for password expiration

©2012 cmdln.org (a sysadmin blog). All Rights Reserved.

http://sopastrike.com/

I encourage you to voice your opposition to SOPA and PIPA as well.  Follow the link below for an easy way to get your representatives contact information, and send them an email.

https://wfc2.wiredforchange.com/o/9042/p/dia/action/public/

©2012 cmdln.org (a sysadmin blog). All Rights Reserved.

I saw that feedback button glaring at me so decided to give it a shot, left a “very dissatisfied” about download experience review. Within an hour I had an email that my access had been changed, and I now have access to download the mibs.

Good job NetApp, in my experience those feedback forms usually go unanswered.

©2012 cmdln.org (a sysadmin blog). All Rights Reserved.

There’s still room to register, but spots are going fast!

For more information see:

http://blog.xen.org/index.php/2011/12/02/xen-day-presenters/

©2012 cmdln.org (a sysadmin blog). All Rights Reserved.

If you’re going to LISA in Boston this year consider registering for Xen Day on Friday Dec, 9. Registration is free and the schedule looks great.

©2012 cmdln.org (a sysadmin blog). All Rights Reserved.