Over the past several months 2 years (since 3.1.2 release, wow time flys when your having fun. I checked and 3.1.2
was released on Dec 9th 2010.) I have seen a few questions regarding how the default CFEngine policy update works, and more specifically how cf_promises_validated plays into the update process. This is my stab at describing the history and behaviour. I welcome any corrections.
The default failsafe.cf update policy is simple in nature. (We could probably debate what is simple or complex, but I am comfortable with the label in this case.) Agents copy policy from /var/cfengine/masterfiles on the policy hub, to /var/cfengine/inputs. This is the same for all agents, even the agent that runs on the policy hub, the only difference is that since the files are already local on the policy hub they don’t have to go over the network, but they are still copied from the same source, to the same destination.
The 3.1.2 release was an efficiency related release. One of the enhancements was the introduction of cf_promises_validated. Eystein wrote a great extended change log on his blog covering the release including a section on cf_promises_validated which is where I first learned of the feature and how to use it. Again, the cf_promises_validated mechanism is simple in nature. From his post “this file (/var/cfengine/masterfiles) is created by cf-agent or any other CFEngine component after it has successfully verified the policy with cf-promises.” What I think is missing from this description is that /var/cfengine/masterfiles is created/updated when policy has been verified after the policy has changed (so it’s not supposed to update this file every execution, there seems to be a bug with this but that is not expected behaviour). I do not know what constitutes change but I suspect it’s some variation of a tripwire policy similar to the following. Remember this is a simple mechanism and is the same for any agent, policy hub or not.
files:
any::
"/var/cfengine/masterfiles"
changes => detect_all_change,
depth_search => recurse("inf"),
classes => if_repaired("cf_promises_validated");
cf_promises_validated::
"/var/cfengine/masterfiles/cf_promises_validated"
create => "true",
touch => "true";
The difference between a policy hub (am_policy_hub) and a non policy hub as I understand it is determined by comparing the contents of /var/cfengine/policy_server.dat to the ips/hostnames associated with the interfaces on the system. If the policy server found in policy_server.dat file resolves to an ip on the current system, it raises the am_policy_hub class. This am_policy_hub class is used in the default failsafe.cf update policy to determine when to copy files from /var/cfengine/masterfiles on the policy hub to /var/cfengine/inputs (locally to the executing agent).
Initially cf_promises_validated was an empty file, and mtime was used to determine if the file was newer. This was problematic for hosts that had time skews and a time stamp was introduced in 3.3.0 so that digest could be used to determine difference more accurately. The fact that a time value is now stored in the file is only relevant to a human reading the file.
Spend a few minutes reading this snippet from the default update policy.
01 files:
02
03 !am_policy_hub:: # policy hub should not alter inputs/ uneccessary
04
05 "$(inputs_dir)/cf_promises_validated"
06 comment => "Check whether a validation stamp is available for a new policy update to reduce the distributed load",
07 handle => "update_files_check_valid_update",
08 copy_from => u_rcp("$(master_location)/cf_promises_validated","$(sys.policy_hub)"),
09 action => u_immediate,
10 classes => u_if_repaired("validated_updates_ready");
11
12 "$(modules_dir)"
13 comment => "Always update modules files on client side",
14 handle => "update_files_update_modules",
15 copy_from => u_rcp("$(modules_dir)","$(sys.policy_hub)"),
16 depth_search => u_recurse("inf"),
17 perms => u_m("755"),
18 action => u_immediate;
19
20 am_policy_hub|validated_updates_ready:: # policy hub should always put masterfiles in inputs in order to check new policy
21
22 "$(inputs_dir)"
23 comment => "Copy policy updates from master source on policy server if a new validation was acquired",
24 handle => "update_files_inputs_dir",
25 copy_from => u_rcp("$(master_location)","$(sys.policy_hub)"),
26 depth_search => u_recurse("inf"),
27 file_select => u_input_files,
28 action => u_immediate,
29 classes => u_if_repaired("update_report");
The first thing that happens is for non policy hubs (line 3 starts the context class restriction, and line 5 begins the promiser). /var/cfengine/inputs/cf_promises_validated is checked against /var/cfengine/masterfiles/cf_promises_validated on the policy hub. If the file is different it is copied down and the validated_updates_ready class is defined. Skipping down to line 20 a new context class is defined for policy hubs or for agents which have validated that updates are ready (their cf_promises_validated in /var/cfengine/inputs was different from the cf_promises_validated file in /var/cfengine/masterfiles on the policy hub). If either of those classes are defined the agent recursively scans /var/cfengine/masterfiles on the policy hub and copies files that are different to /var/cfengine/inputs locally on the executing agent.
So, policy hubs always perform this update and copy files that are different from /var/cfengine/masterfiles to /var/cfengine/inputs. and non policy hubs only update /var/cfengine/inputs from /var/cfengine/masterfiles on the policy hub if cf_promises_validated has changes. The hub must always perform this update if you recall how cf_promises_validated is created/updated. New policy that successfully validates in /var/cfengine/inputs triggers cf_promises_validated to be updated in /var/cfengine/masterfiles. Agents need to see that file be different from the cf_promises_validated file in /var/cfengine/inputs in order to trigger a full policy update.
If its still not clear, read it a few more times. Things are usually pretty hard until they aren’t 
. I recall reading the extended change log from 3.1.2 several times, as well as the example policy until I thought I had a good grasp on the flow. I hope you find this useful and I expect that this post will become less useful in the near future. I have filed a bug requesting better documentation coverage of the default update policy.
Identica
Twitter
LinkedIn
GooglePlus
Thanks for that explanation. I was wondering what all the hubbub about this was for, and now I understand. I personally don’t use the default update or failsafe config, as we have a separate Q/A process to verify a policy before it goes in place (and we use a staged mechanism to deploy). This is certainly an interesting way to ensure that invalid policy doesn’t get propagated out, though I prefer my “use Subversion and don’t promote to a deployable location until someone has validated the code” method better.
I think using the cf_promises_validated file is a good pattern. It reduces the load on the hub from clients performing full scans on masterfiles unnecessarily. It also functions as a gate to stop broken policy from propagating, though it really shouldn’t matter if broken policy does propagate, failsafe.cf should fix it as soon as a correct version is available.
I use a slightly modified default failsafe.cf that has hubs maintain a clean svn checkout, this could be a specific branch or tag. As policy is committed a post-recieve hook does a sanity check to try and keep broken policy from hitting the policyhub. This still allows for some other process chain for policy validation. You could commit and the policy could be automatically validated, then you could require a human to review policy and tag it which the policy hub(s) would see and update their masterfiles.
The flexibility is there to do what you think works best for your environment, but if you are using cf-serverd for policy distribution to agents (instead of each agent doing its own svn checkout) I would encourage you to consider the extra work the clients are doing scanning masterfiles at whatever interval pointlessly looking for updates, when they could use cf_promises_validated to decided if its appropriate to do a full scan.
It would be great if you could do a write-up of your process. I think its great for the community to see multiple approaches. To me one of the hardest parts about getting started with any technology is not knowing what you don’t know.
Well, I do pretty much what you’re describing, with the policy in svn and a post-commit hook on the /tags directory (so developers are encouraged to save their work often).
There’s a good point made about the computational expense of comparing the files, but in my mind, it’s more important to confirm that the policy has not changed. A malicious or ignorant local administrator could potentially modify a policy file — intentionally or inadvertently — and that would not be corrected or possibly even observed until there was another policy update. In my environment, where ignorant people obtain local root access and inadvertently break all sorts of things, this risk is greater than the cost of calculating a few checksums four times per hour.
I’ve been talking to our legal department for a while about open-sourcing some of the tools I’ve written to manage this, and about writing some of the design down for the purposes of bettering the community. But it’s a bit of a process getting “internal” information approved for disclosure.
Yeah, I totally understand the ignorant people obtaining root and wreaking havoc. I’ve been thinking about adding a time class to bypass cf_promises_validated check and do a full scan at some minimum interval for the same reason.
Great post, Nick, thanks for writing it!