Managing a hundred servers with imperative shell scripts does not scale: every script takes for granted a starting state that, after a few months of manual edits, is no longer the same across the hundred nodes. Puppet — written in Ruby by Luke Kanies and maintained by Reductive Labs — turns the problem around by changing what you write. Instead of the sequence of commands to reach a state, you describe the state the machine should be in. The 0.24 series, current in 2009 under the GPLv2 licence, is the stable base under most installations today.

Context

Automating the configuration of machine fleets is not a new problem. CFEngine, which Mark Burgess started writing in 1993, comes before it: version 3, out in 2008, rebuilds the model on promise theory, the idea that each component declares what it guarantees rather than receiving orders. Puppet appeared in 2005 along the same line, with a different choice of language: a dedicated domain-specific language (DSL) that people who administer systems can read without being full-time Ruby developers.

Meanwhile the field is filling up. In January 2009 Opscode, Adam Jacob’s company, shipped the first release of Chef, also in Ruby but with its recipes written as actual Ruby code, closer to an imperative style. There is also bcfg2, developed at Argonne National Laboratory, less widely used. For now Puppet stays ahead: the larger community and the greater number of documented installations.

The declarative model

The basic unit is the resource: a package, a file, a service, a user. Each resource has a type, a title and a set of attributes describing the state it should be in. A manifest fragment:

package { 'nginx':
    ensure => installed,
}

file { '/etc/nginx/nginx.conf':
    ensure  => file,
    owner   => 'root',
    group   => 'root',
    mode    => '0644',
    content => template('nginx/nginx.conf.erb'),
    require => Package['nginx'],
    notify  => Service['nginx'],
}

service { 'nginx':
    ensure  => running,
    enable  => true,
    require => Package['nginx'],
}

The manifest does not say how to install nginx: there is no apt-get, no yum. It says the package must end up present, and Puppet picks the right provider for the system — apt on Debian, yum on Red Hat, pkgadd on Solaris. The same manifest reaches the intended state across different distributions, because translating to native commands is left to the providers of the resource abstraction layer.

The require and notify relationships are not a reading order: they declare dependencies. Puppet resolves them by building a graph, not by running resources in the order they appear in the file.

Catalog and idempotency

When a node requests its configuration, the Puppet master compiles the manifests into a catalog: a document for that single node, where abstract resources are resolved and the dependency graph is topologically ordered. The agent receives the catalog and applies it one resource at a time.

What makes the model workable is idempotency. Before touching a resource, the agent looks at its current state: if /etc/nginx/nginx.conf already has the required content, owner and permissions, the agent does nothing. Applying the same catalog twice, or a hundred times, leaves the system in the same state. Here lies the difference between a declarative resource and a script: the script echo "..." >> file appends a line on every run, the file resource converges to a single state.

Idempotency brings a sharp operational consequence: the agent can run continuously. The default configuration starts it every 30 minutes. If someone touches a managed file, on the next pass Puppet restores it to the declared state and notes it in the report. Configuration drift — nodes born identical that diverge over time — is corrected at each cycle instead of accumulating.

Agent/master architecture

In 2009 the typical install is client/server:

  • the master is the central server that compiles catalogs. It runs as a Ruby application served by WEBrick for testing or, in production, by Apache with the Passenger module;
  • the agent is the daemon on each managed node. It contacts the master over HTTPS with certificate authentication, downloads the catalog, applies it, sends back a report;
  • Facter is the introspection tool. It collects the node’s facts — operating system, kernel version, interface addresses, RAM, architecture — and exposes them as variables in manifests. That is how the same manifest branches on $operatingsystem or $memorytotal without the administrator sorting nodes one by one.

Certificates are the delicate point of the architecture. The master acts as certificate authority: a fresh agent generates a signing request, the administrator approves it on the master, and only from that moment does the node receive catalogs. It is the mechanism that stops any machine on the network from getting itself configured as a legitimate node, but it is also the first real snag when automating the provisioning of new instances — on EC2, say, where machines come and go constantly and manual signing becomes a bottleneck.

Manifests as code

The least visible consequence, and perhaps the most durable, is that manifests are text files. They sit in a Subversion or Git repository, have a commit history, can be compared with a diff, go through review before reaching production. An infrastructure’s configuration stops living in the head of whoever set it up and in files scattered across individual hosts, and becomes an artefact versioned like any other source.

Modules organise these files: an Apache module, a MySQL one, a Postfix one wrap up a component’s manifests, templates and any static files, and are exchanged over mailing lists or public repositories. By 2009 there are already dozens of community modules for the most common services, though a central catalogue is still missing and installing one means copying directories.

Limits

The declarative model is not free. Resource order has to be declared by hand with require and notify: missing one produces a catalog that sometimes works and sometimes does not, depending on the order in which the graph is walked. Debugging a missing dependency is less immediate than reading a linear script.

The master is a point of centralisation: it compiles the catalog for every node on every pass, and on large fleets it becomes itself a component to size and to make highly available. Certificate handling adds friction exactly where you would want smoothness, namely in automated provisioning.

There is, finally, the state Puppet reports, which is only the one the resources model. Whatever is not in a manifest is not managed, and on a real system there is always a grey area of hand-made configuration that no resource claims — and that on the next pass Puppet simply ignores.

Cover image: Diagram of a Puppet catalog: nodes representing resources (packages, files, services) linked by dependency arrows in an ordered chain — diagram by MarcelButucea, CC BY-SA 3.0 — https://commons.wikimedia.org/wiki/File:Expanded_relationships.svg