Locking down Traefik systemd service
On one of my server I’m running two bare instances of Traefik and Minio. Traefik is used as the HTTP reverse proxy, mainly because of its ability to automatically retrieve certificates from Let’s Encrypt for any domain it is hosting. Minio is a self-hosted, S3-compatible object storage.
The usual approach to deploy these two applications is Docker (especially since Traefik has the awesome ability to configure itself automatically through Docker), but for this setup I decided to go down a different path: plain, old system services.
Nevertheless, these services shall be as secure and fail-safe as possible. While the programming language these two applications are written already helps toward this goal, from a systems administrator and security perspective that is not enough.
Therefore, I’m using the Traefik service to show how to lock down a service with systemd, so that even in case of an exploit or breach the system integrity is not undermined.
For reference, I’m running systemd v232 on Debian Stretch.
The initial, standard service file for Traefik looks like this:
| |
The first enhancement is in the Unit section:
assuring whether files and directories required to (properly) run the service exist.
This can be done by means of the AssertFile* and AssertPath* settings.
These settings add assertion checks to the start-up of the unit.
Any assertion setting that is not met results in failure of the job.
Assertion expressions are for units that cannot operate when specific requirements are not met.
| |
In this case, the executable .../bin/traefik needs to be present (which is implicitly checked) and additionally the executable bit needs to be set for the file.
Also, the configuration file /etc/traefik/traefik.toml should be present, otherwise the service cannot operate.
Next, we’ll look at the Service section.
| |
When using service types “simple” or “notify”, it is expected that the process launched with ExecStart is the main process of the service (i.e. it does not fork into the background).
Behavior of “notify” is similar to “simple”; however, it is expected that the daemon sends a notification message via sd_notify(3) or an equivalent call when it has finished starting up. systemd will proceed with starting follow-up units after this notification message has been sent.
This means that the process tells systemd when it is ready (to accept external connections), therefore systemd can wait before launching subsequent units which depend on this service. This is especially useful when an applications immediately tries to connect the database upon launching but the database takes a few seconds before it is ready to handle requests.
The watchdog additionally monitors the status of the service. The service has to emit a “ping"1 at least once in the specified interval (here: one second). Otherwise systemd assumes the process crashed and will start a new one.
Traefik has to run as root since it binds to ports 80 and 443 (though I’ll have to look into CAP_NET_BIND_SERVICE a bit more). And you know what they say:
With great power comes great responsibility.
By restricting the available, readable and writeable directories we can ensure that even if the service gets hijacked, the damaged is contained as much as possible by sandboxing.
| |
The ProtectSystem setting ensures that any modification of the operating system and its configuration is prohibited for the service.
If set to “true”, mounts the /usr and /boot directories read-only for processes invoked by this unit. If set to “full”, the /etc directory is mounted read-only, too. If set to “strict” the entire file system hierarchy is mounted read-only, except for the API file system subtrees /dev, /proc and /sys.
The systemd.exec man page even contains this little advice:
It is recommended to enable this setting for all long-running services, unless they are involved with system updates or need to modify the operating system in other ways.
But since Traefik still needs to read its own configuration file (/etc/traefik/traefik.toml) and write the acquired certificates onto disk (/etc/traefik/acme.json), we set the ReadOnlyPaths and ReadWritePaths directives, respectively.
The PrivateDevices setting turns off access to physical devices (/dev/sda, /dev/mem, …) for the executed process.
Additionally, ProtectKernelTunables makes kernel settings exposed through procfs and sysfs (/proc/timer_stats, …) read-only to all processes of the unit.
Furthermore, ProtectControlGroups makes the Linux Control Groups hierarchies accessible through /sys/fs/cgroup read-only as well.
Finally, ProtectHome can be set to either “true”, “read-only” or “tmpfs” which subsequently makes the directories /home, /root and /run/user either inaccessible, read-only or mounted through a loopback tmpfs for the process.
| |
The PrivateTmp directive sets up a new file system namespace for the processes and mounts empty /tmp and /var/tmp directories inside it.
As a corollary, this makes sharing between processes via /tmp or /var/tmp impossible. If this is enabled, all temporary files created by a service in these directories will be removed after the service is stopped.
There is one last thing I’d like to do: limit the number of the processes the unit can spawn:
| |
Depending on the kind of application you’re running this setting may or may not be useful, but since Traefik uses Go’s internal concurrency and is therefore running in a single process anyway, it makes sense to ensure it can never spawn another process.
Assembling all of the snippets above together, we obtain the final service file:
| |
I will contribute these enhancements to Traefik’s sample systemd service file.
I have to admit that locking down a service so rigorously is actually quite enjoyable. I also learned a lot during the process of reading through various man pages, investigating process capabilities and application requirements. Also, systemd makes it super easy to apply these arcane and convoluted security measures by exposing them through simple configuration directives and documenting them well. Systemd’s manual pages are very comprehensive and contain lots of useful hints as well as references to other documentation. But you really need to make sure to actually read it and not just skim it!
Don’t forget to systemctl daemon-reload after changing your service files!