I am very excited to be joining such a great team and very honoured to have this opportunity.

% cf-agent -KIf ./example_canonify_list.cf
R: canonified site: www_example_com
R: canonified site: www_example2_com
R: alternate canonified site: www_example_com
R: alternate canonified site: www_example2_com
 !! Method invoked repairs

body common control
{
  bundlesequence => { "main" };
  inputs => { "cfengine_stdlib.cf", };
}
 
bundle agent main
{
  vars:
      "apache_config[www.example.com][DocumentRoot]" string => "/var/www/www.example.com";
      "apache_config[www.example.com][ServerAdmin]" string => "webmaster@example2.com";
      "apache_config[www.example2.com][DocumentRoot]" string => "/var/www/www.example2.com";
      "apache_config[www.example2.com][ServerAdmin]" string => "webmaster@example3.com";
 
  methods:
      "Testing" usebundle => test("main.apache_config");
      
}
 
bundle agent test(config)
{
  vars:
      # build a list from the first index on the config array
      "site_names" slist => getindices("$(config)");
      # Build a new array mapping the original index to a canonified version of itself
      "site_names_canonified_map[$(site_names)]" string => canonify("$(site_names)");
      # Build a list of just the canonified versions
      "site_names_canonified_list" slist => maplist("$(site_names_canonified_map[$(this)])", "site_names");
 
      # alternate way to build list of canonified values
      # due to a bug in the way getindices/getvalues works this needs to happen on the pass after the map array is generated
      "site_names_canonified_list2" slist => getvalues("site_names_canonified_map"), policy => "free";
 
  reports:
    cfengine::
      "canonified site: $(site_names_canonified_list)";
      "alternate canonified site: $(site_names_canonified_list2)";
 
}

In lines 9–12 of the example policy we build an array with values that need to be canonified. We use getindices to get the list of the first index. You could just as easily start with a list of values that need canonified. John used the maplist function to construct a new array with the original index values and their canonified versions. Another way to do this is to use getvalues. Currently (as of CFEngine 3.4.4) getindices and getvalues are not able to get the index of a generated array on the same pass, so its something to be aware of if your using that approach. However Johns approach using maplist is able to work on the same pass because it does not rely on getting the index of the newly generated map array.

What I found interesting was that the default masterfiles policy has been updated. body executor control now calls $(sys.workdir)/inputs/update.cf instead of failsafe.cf. And $(sys.workdir)/failsafe/failsafe.cf (which was new as of 3.4.0) is gone. I think its a good change. As far as users are concerned failsafes main task is updating policy. Its also what is run if something in the policy fails (hence failsafe). I think we have conflated the two by using a failsafe mechanism for updates as a general practice. This means we will touch the failsafe file more, and there is more opportunity for us to shoot ourselves in the foot. I have not tested, but I believe that failsafe.cf should be generated by CFEngine. This is probably very similar if not identical to the bootstrap policy. I suspect if you have your own failsafe.cf it will be used but I believe the intention is for us to let CFEngine generate that unless we have a good reason to override it.

Any way, log in to your zenoss box as the zenoss user at a terminal and run zendmd. We will select a device, and then get a value from one of the RRDs.

d = dmd.Devices.findDevice('myjavaserver')
d.getRRDValue('HeapMemory_max')

1394606080.0

You can look up other things from here as well for example zproperties, try d.zDeviceTemplates or d.zCommandCycleTime.

The default failsafe.cf update policy is simple in nature. (We could probably debate what is simple or complex, but I am comfortable with the label in this case.) Agents copy policy from /var/cfengine/masterfiles on the policy hub, to /var/cfengine/inputs. This is the same for all agents, even the agent that runs on the policy hub, the only difference is that since the files are already local on the policy hub they don’t have to go over the network, but they are still copied from the same source, to the same destination.

The 3.1.2 release was an efficiency related release. One of the enhancements was the introduction of cf_promises_validated. Eystein wrote a great extended change log on his blog covering the release including a section on cf_promises_validated which is where I first learned of the feature and how to use it. Again, the cf_promises_validated mechanism is simple in nature. From his post “this file (/var/cfengine/masterfiles) is created by cf-agent or any other CFEngine component after it has successfully verified the policy with cf-promises.” What I think is missing from this description is that /var/cfengine/masterfiles is created/updated when policy has been verified after the policy has changed (so it’s not supposed to update this file every execution, there seems to be a bug with this but that is not expected behaviour). I do not know what constitutes change but I suspect it’s some variation of a tripwire policy similar to the following. Remember this is a simple mechanism and is the same for any agent, policy hub or not.

files:
  any::
    "/var/cfengine/masterfiles"
      changes      => detect_all_change,
      depth_search => recurse("inf"),
      classes      => if_repaired("cf_promises_validated");

  cf_promises_validated::
    "/var/cfengine/masterfiles/cf_promises_validated"
      create => "true",
      touch  => "true";

The difference between a policy hub (am_policy_hub) and a non policy hub as I understand it is determined by comparing the contents of /var/cfengine/policy_server.dat to the ips/hostnames associated with the interfaces on the system. If the policy server found in policy_server.dat file resolves to an ip on the current system, it raises the am_policy_hub class. This am_policy_hub class is used in the default failsafe.cf update policy to determine when to copy files from /var/cfengine/masterfiles on the policy hub to /var/cfengine/inputs (locally to the executing agent).

Initially cf_promises_validated was an empty file, and mtime was used to determine if the file was newer. This was problematic for hosts that had time skews and a time stamp was introduced in 3.3.0 so that digest could be used to determine difference more accurately. The fact that a time value is now stored in the file is only relevant to a human reading the file.

Spend a few minutes reading this snippet from the default update policy.

01 files:
02  
03  !am_policy_hub::  # policy hub should not alter inputs/ uneccessary
04
05   "$(inputs_dir)/cf_promises_validated"
06        comment => "Check whether a validation stamp is available for a new policy update to reduce the distributed load",
07         handle => "update_files_check_valid_update",
08      copy_from => u_rcp("$(master_location)/cf_promises_validated","$(sys.policy_hub)"),
09         action => u_immediate,
10        classes => u_if_repaired("validated_updates_ready");
11 
12   "$(modules_dir)"
13          comment => "Always update modules files on client side",
14           handle => "update_files_update_modules",
15        copy_from => u_rcp("$(modules_dir)","$(sys.policy_hub)"),
16     depth_search => u_recurse("inf"),
17            perms => u_m("755"),
18           action => u_immediate;
19
20  am_policy_hub|validated_updates_ready::  # policy hub should always put masterfiles in inputs in order to check new policy
21
22   "$(inputs_dir)"
23           comment => "Copy policy updates from master source on policy server if a new validation was acquired",
24            handle => "update_files_inputs_dir",
25         copy_from => u_rcp("$(master_location)","$(sys.policy_hub)"),
26      depth_search => u_recurse("inf"),
27      file_select  => u_input_files,
28            action => u_immediate,
29           classes => u_if_repaired("update_report");

The first thing that happens is for non policy hubs (line 3 starts the context class restriction, and line 5 begins the promiser). /var/cfengine/inputs/cf_promises_validated is checked against /var/cfengine/masterfiles/cf_promises_validated on the policy hub. If the file is different it is copied down and the validated_updates_ready class is defined. Skipping down to line 20 a new context class is defined for policy hubs or for agents which have validated that updates are ready (their cf_promises_validated in /var/cfengine/inputs was different from the cf_promises_validated file in /var/cfengine/masterfiles on the policy hub). If either of those classes are defined the agent recursively scans /var/cfengine/masterfiles on the policy hub and copies files that are different to /var/cfengine/inputs locally on the executing agent.

So, policy hubs always perform this update and copy files that are different from /var/cfengine/masterfiles to /var/cfengine/inputs. and non policy hubs only update /var/cfengine/inputs from /var/cfengine/masterfiles on the policy hub if cf_promises_validated has changes. The hub must always perform this update if you recall how cf_promises_validated is created/updated. New policy that successfully validates in /var/cfengine/inputs triggers cf_promises_validated to be updated in /var/cfengine/masterfiles. Agents need to see that file be different from the cf_promises_validated file in /var/cfengine/inputs in order to trigger a full policy update.

If its still not clear, read it a few more times. Things are usually pretty hard until they aren’t . I recall reading the extended change log from 3.1.2 several times, as well as the example policy until I thought I had a good grasp on the flow. I hope you find this useful and I expect that this post will become less useful in the near future. I have filed a bug requesting better documentation coverage of the default update policy.

You could run something like this to see what classes are raised during execution.

cf-agent -KIv | grep "cf3>\s*+" | grep -v :

But that isn’t really a clear picture under normal operation. You can use the allclassesreport in body agent control to write out a list of classes defined during execution in /var/cfengine/state/allclasses.txt. It can be handy to have. I was recently wanting to do something when policy was executed by cf-execd but not when executed manually. Based on all the other automatic classes defined I had a suspicion there was an automatic class for that, and I had no need to manually add a class to exec_command in body executor control. I turned on the allclassesreport and sure enough, I found from_cfexecd was being raised.

Of course all of this information is readily available for remote agents in the dashboard of the enterprise version. And with the 25free program to get started, you should definitely take a look.

I just want to share an example of calling bundles dynamically. It’s a contrived example, but I thought it was neat so here it is.

R: in bundle agent handler1: I got value1
R: in bundle agent handler1: I got value2
 !! Method invoked repairs
 !! Method invoked repairs
R: in bundle agent handler2: I got value1
R: in bundle agent handler2: I got value2
 !! Method invoked repairs
 !! Method invoked repairs

body common control {

    bundlesequence => {"main",};

}

bundle agent main{
vars:
  "mybundles" slist => { "handler1","handler2" };
  "myvalues" slist => { "value1", "value2" };

methods:
  "$(mybundle)" usebundle => handler_iterator("$(mybundles)", "@(main.myvalues)");

}

bundle agent handler_iterator(handler, values)
# This expects a single value
{
methods:
  "$(handler)" usebundle => $(handler)("@(handler_iterator.values)");

}
bundle agent handler1(value1)
{
reports:
  cfengine::
    "in bundle agent handler1: I got $(value1)";
}

bundle agent handler2(value1)
{
reports:
  cfengine::
  "in bundle agent handler2: I got $(value1)";
}

Breakdown

bundle agent main

Here we define two lists, a list of bundle names and a list of values. Because of implicit list iteration we then call “handler_iterator” 2 times. Once for each value of the mybundles list. Each activation also passes in the myvalues list.

Bundle agent handler_iterator

handler_iterator is where the neat part happens. You can see that we call the bundle $(handler) (outside of quotes) with the parameter @(handler_iterator.values). This is what I found so interesting. I Have called bundles dynamically in the past, but I have always put the variable inside of quotes. That worked fine but it prevented me from using parameters when calling because the parameters were seen as part of the bundle name.  Here is an example of the handler_iterator bundle trying to use a parametrized value inside of quotes.

bundle agent handler_iterator(handler, values)
# This expects a single value
{
methods:
  "$(handler) $(values)" usebundle => "$(handler)($(values))";

}

 !! A method attempted to use a bundle "handler1(value1)" that was apparently not defined!
I: Report relates to a promise with handle ""
I: Made in version 'not specified' of './test.cf' near line 21
 !! A method attempted to use a bundle "handler1(value2)" that was apparently not defined!
I: Report relates to a promise with handle ""
I: Made in version 'not specified' of './test.cf' near line 21
 !! Method failed in some repairs or aborted
 !! A method attempted to use a bundle "handler2(value1)" that was apparently not defined!
I: Report relates to a promise with handle ""
I: Made in version 'not specified' of './test.cf' near line 21
 !! A method attempted to use a bundle "handler2(value2)" that was apparently not defined!
I: Report relates to a promise with handle ""
I: Made in version 'not specified' of './test.cf' near line 21
 !! Method failed in some repairs or aborted

Since $(handler) is a single value (we iterated over the list of bundles in bundle agent main) only one method activation will happen for each activation of handler_iterator.

bundle agent handlerx

The handlers themselves just report once for each item in the list passed to them.

Specific Use Case

Well, I don’t have one. I can however imagine that based on some variable event you might want to call a bundle with some variable parameter. Something like the nagios_plugin_agent sketch comes to mind. (It can call a variable bundle right now).

Special thanks to CFEngine for sponsering me!

Here are my picks for the minute.

• Building the Open Cloud — Mark Hinkle, Citrix
• The OpenNebula Cloud Platform for Data Center Virtualization — Carlos Martin, OpenNebula
• Cloning SysAdmins with the Cloud and Juju — Clint Byrum, Canonical
• A Collaborative Approach to Managing Linux Configurations — Diego Zamboni, CFEngine
• Building FOSS Clouds — Joe Brockmeier, Citrix
• Automation and DevOps Best Practices — Rob Hirschfeld, Dell
• Scaling Configuration Management Across Data Center and Cloud — Mark Burgess, CFEngine
• Editing with vi: Advanced Topics, Part I — Aleksey Tsaloikhin, Vertical Sysadmin, Inc.