Jack Henschel's Blog

Self-Contained Puppet Control Repository

Posted on
#git #puppet

For some time I have been meaning to automate the configuration of my servers and PCs. At work we use Puppet for this, however Puppet is quite a heavy and “enterprisy” tool. Puppet really assumes to own your system when you install and run it. In that regard it is the total opposite of Ansible which seems much more lightweight - and there isn’t even any installation required!

However, I have been unable to devote the additional effort and time to learn another Configuration Management Tool, especially since I already know one quite well.

Therefore I ultimately went for Puppet on my personal systems, too. Luckily, the Debian packages are quite up to date (4.8 in Stretch, 5.5 in Buster). Big thanks to the Debian Puppet Team for that!

As mentioned above, Puppet really likes to deeply creep into your system. It expects a configuration file here, places caching files there, creates logs over there and fetches information therefrom. But I wanted to have a setup where I can put all these necessary file into one location and commit them to Git. This keeps it easy to deploy the system from scratch: just clone the repository, install Puppet and run it. No need manually to copy this file here and create another file there before you can automate your server.

So I created a configuration that allowed me to do pretty much exactly that.

I should also note at this point that I’m running Puppet in the “Masterless” mode. This means I don’t have a Puppet Master-Agent / Server-Client architecture but rather apply all manifests on each node individually.

#  Repository Layout

The basic structure of the repository looks as follows:

$ tree
├── environment.conf
├── etc
│   ├── hiera
│   │   ├── common.yaml
│   │   └── nodes
│   │       └── server1.example.com.yaml
│   ├── hiera.yaml
│   └── puppet.conf
├── Makefile
├── manifests
│   └── site.pp
├── modules
│   ├── stdlib
│   └── ...
├── production -> .
├── puppet.sh
├── README.md
└── site
    └── profile
        ├── data
        │   └── common.yaml
        ├── hiera.yaml
        ├── manifests
        │   ├── example.pp
        │   └── tor_relay.pp
        └── templates
            └── torrc.erb

You can compare this against the control repository template Puppetlabs provides: in addition to the environment.conf, manifests/, hiera.yaml and site/ paths my repository most notably features the main Puppet configuration file puppet.conf, the Hiera configuration data in etc/hiera/, external modules under modules and internal modules under site/. More specifically, I only use one internal module called profile.

#  Puppet Configuration

The Puppet configuration in etc/puppet.conf is the heart of this repository. It sets everything up so Puppet will only read and write stuff (for itself) within this repository (this does not include the configuration it applies to the system, obviously).

[user]
confdir = ~/.puppet/etc
codedir = ~/.puppet/
vardir = ~/.puppet/cache
logdir = ~/.puppet/var/log
rundir = ~/.puppet/var/run
environmentpath = $codedir
hiera_config = $confdir/hiera.yaml
config = $confdir/puppet.conf
storeconfigs = false

As you have probably already guessed, the entire repository is supposed to be put into ~/.puppet though you can easily find&replace this path in the file with a simple sed script. Quite a few variables need to be adjusted to tame Puppet. Luckily a lot of the other variables in the puppet.conf (not shown here) simply use the ones configured above as their base path.

Tip: you can generate a comprehensive puppet.conf file (including comments) with puppet apply --genconfig.

#  Environment

The environment configuration (in environment.conf) sets the load path for Puppet modules and is a bit special in my case. I put my own modules into site/profile/ (like profile::tor_relay) and external modules (like puppetlabs/stdlib) into modules/. This makes it much easier keep track of where modules are coming from, since I directly commit (internal and external) modules to Git. If you don’t want to do that you can just use a standard Puppetfile and deploy your modules through r10k, of course.

modulepath = site:modules:$basemodulepath
# disable catalog caching (e.g. for developing)
# environment_timeout = 0

To reduce another one of Puppet’s abstraction layers, I also create a symlink for the default deployment environment production to the main directory of the repository.

$ ln -s . production
$ file production
production: symbolic link to .

I simply install external modules via puppet module install --target 'modules' 'puppetlabs-apt' --version '6.2.1'.

Tip: always explicitly state in your commit message which version of which module you are committing!

#  Main Puppet Manifest

The control repo holds the basic configuration of each node in manifests/site.pp:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

node default {
  $description = lookup('description')
  notice("${::fqdn}: ${description}")
  contain profile::example
}

node 'server1.example.com' {
  $description = lookup('description')
  notice("${::fqdn}: ${description}")
  contain profile::tor_relay
}

Each node declaration in manifests/site.pp refers to a specific node (what exactly identifies the node can be configured in the puppet.conf, though it defaults to the hostname). If none of the nodes matches, Puppet simply falls back to default. Depending on which node is selected Puppet will execute different action. In this case it will include the class example from profile profile (profile::example) for the default node, and class tor_relay from profile profile (profile::tor_relay).

Puppet experts will have noticed another little quirk here: Classes with different functionality are part of the same profile. Essentially what I’m doing here is removing one layer of abstraction from the Puppet Role-Profile Pattern. I’m not using any Role classes because I can do that directly in manifests/site.pp and I’m also not creating a profile for each class because “my classes” are a) either very simple (like profile/tor_relay) or b) just super-thin wrappers over other modules (like puppetlabs/docker).

#  Hiera

Hiera (as in hierarchy) is Puppet’s built-in key-value configuration lookup system. It is an excellent approach to providing highly-specific and individual configuration to any number of machines while keeping the code as short as possible.

As we have declared our main Hiera configuration file to be stored under etc/hiera.yaml in the main Puppet configuration section, this is where we’ll put the file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

---
version: 5

defaults:
  data_hash: yaml_data

hierarchy:
  - name: "eYaml backend"
    lookup_key: eyaml_lookup_key
    datadir: "hiera"
    paths:
      - "nodes/%{::fqdn}.yaml"
      - "common.yaml"
    options:
      pkcs7_private_key: "/etc/puppet/keys/eyaml_private_key.pkcs7.pem"
      pkcs7_public_key: "/etc/puppet/keys/eyaml_public_key.pkcs7.pem"

This examples configures quite a basic hierarchy: when searching for a specific key Puppet will first look into the file nodes/${hostname}.yaml and use the value assigned to the key. Otherwise Puppet will look into common.yaml. If it doesn’t find the key there either, it will simply leave the value as undef.

The common.yaml (stored under etc/hiera/) could look like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

description: "Common host"
profile::ssh::keys:
  'jacks private key':
    ensure: present
    type: 'ssh-ed25519'
    key: >
      ENC[PKCS7,MIIBuQYJKoZIhvcNAQcDoIIBqjCCAaYCAQAxggEhMIIBHQIBADAFMAACAQEw
      DQYJKoZIhvcNAQEBBQAEggEADn+xwabUmNeEpBhvXjfRJYAEfyklg05HADc0
      nbUehiiudWISS//68wmdhRYBzYRnq6qig8reVW9cn56EnRaArQLGCbJc8Eyy
      GUFQSlCR0LesU48DdtjJlMUNfmxOz0kBA6/0HZGTtrOwSK9BWOzTwSsPtHMq
      gNk6ocKr1o3E4Us+R7OGfyj5/Scs70KeeBzuxQHGVG9BbkhhJ53EBSr4OsmX
      c5EuZ7ZFHGX7ICCZB4g3dgMMaEueyuGKl7Eqcu5HbiaoE7A0xZXfUKCP+Qsk
      aj814VEKnUHbMFlp2U+N0T0AOxd+kfoT8ceEyEezLwbpbN9qpStt9mOaiHL6
      yBvhWDB8BgkqhkiG9w0BBwEwHQYJYIZIAWUDBAEqBBCbXVE5EZ/4w+QojkwE
      45z0gFBco95IZN4dneoUly/4U9WpypKZ+8jVdNotIFCP8F92GIFneMdKTkLo
      vGn+B1qZR/PhHukYHWX6SvZTfROT570vmUZW2DFrsq2WxvBW5eUY2Q==]

In this example we see a value encrypted with Hiera-eYaml – definitely one of my favorite features about Puppet! Hiera-eYaml makes it very easy to commit secrets (e.g. passwords, private keys, tokens, …) to Git because you can encrypt only parts of your files (“per-value encryption”) and it doesn’t require any additional setup (since Puppet handles the decryption transparently with the configuration shown above).

I use the following wrapper script to en- and decrypt snippets of eYaml data. This is required because I need to tell the eyaml tool where the public and private key are located.

1
2

#!/bin/sh
eyaml $@ --pkcs7-private-key /etc/puppet/keys/eyaml_private_key.pkcs7.pem --pkcs7-public-key /etc/puppet/keys/eyaml_public_key.pkcs7.pem

#  Makefile

I use the following Makefile to run various actions on the repository:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

#!/usr/bin/make

PHONY: clean provision validate yamllint puppetlint

# executes a puppet run to provision the machine
provision: puppetlint validate yamllint
	puppet apply --config etc/puppet.conf manifests/site.pp

# checks that puppet manifests conform to the style guide
puppetlint:
	puppet-lint --no-documentation-check --no-parameter_order-check manifests/
	puppet-lint --no-documentation-check site/

# checks puppet syntax
validate:
	puppet parser validate manifests/site.pp site/profile/manifests/

# checks yaml syntax
yamllint:
	yamllint etc/ site/profile/

# deletes caches and runtime directories
clean:
	rm -rf cache/ var/

This way I can check all my manifests and run them with make provision. Unfortunately, one can not really pass arguments through a Makefile (for example the --noop or --debug Puppet options), therefore I still keep this puppet.sh shell script around:

1
2

#!/bin/sh
puppet apply --config etc/puppet.conf manifests/site.pp $@

With this script I can invoke a verbose Puppet run by typing ./puppet.sh --debug.

#  Conclusion

The Makefile completes the repository. It contains everything that’s needed to execute a Puppet run: custom Puppet code, internal and external Puppet modules, Hiera YAML configuration and all the boilerplate around it.

Well, almost everything: I intentionally put the eYaml public and private key into /etc/puppet/keys/... (and not in the repository at ~/.puppet) to avoid accidentally committing them to Git.

#  Apply it!

Get the entire repository at https://git.cubieserver.de/jh/puppet-control-example and apply the example profile to your server:

git clone https://git.cubieserver.de/jh/puppet-control-example ~/.puppet
cd ~/.puppet
puppet apply --config etc/puppet.conf

#  References