Restic Backups

In this post I describe the workflow I currently use to create offsite backups of my servers.

I use restic to create backups since it supports snapshots really well and saves the data in a [content-addressable storage]() which makes it very bandwidth- and space-efficient. Thus it is also quite fast. Another nice feature of restic is also that it supports multiple backends for storing data. I’m using the Backblaze B2 backend.

So let’s just dive right in. I’m not going to explain every single command in the script, because mostly their meaning and actions are rather obvious.

/usr/local/sbin/restic-backup.sh:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
#!/bin/bash

set -e

# export environment keys
export RESTIC_REPOSITORY="b2:b2-bucket-name:/"
export RESTIC_PASSWORD="restic-repository-key"
# ...

# check if repository needs to be initialized first
if ! restic snapshots 2>&1 >/dev/null; then
    restic init
fi

# check if repository is ok
restic check

# create new backup
restic backup \
       --one-file-system \
       --exclude-caches \
       '/etc' \
       '/root' \
       '/mnt/data' \
       '/var/backups'
       # ...

# expire old snapshots
# keep one snapshot per month for the last 12 month
# keep all snaphots within the last 30 days
restic forget \
       --keep-monthly 12 \
       --keep-within 30d

# remove unreferenced data from repo
restic prune

# collect log (since last time)
lastrun_timestamp="$(systemctl show -p InactiveEnterTimestamp restic-backup.service | awk '{print $2 $3}')"
log="$(journalctl -u restic-backup.service --since $lastrun_timestamp)"

# send email notification
echo "$log" | mail -s "Restic Backup $(hostname -f) successful" admin@example.com

exit 0

The interesting bit comes at the end: I’m using a systemd service and timer to trigger the backup job at predefined times.

Since systemd is collecting logs for each of the services its running anyway, I use this feature to fetch the logs of the current run. This is implemented by first getting the timestamp of when that last service activation ended (InactiveEnterTimestamp) and the getting the logs since that timestamp from journalctl.

Then I simply mail these logs to myself through the system mail.

TODO: always send e-mail, send mail when unit fails (systemd?) systemd.unit OnFailure

/etc/systemd/system/restic-backup.service:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
[Unit]
Description=Run backup job
Documentation=man:restic(1)
Documentation=https://restic.readthedocs.io/en/stable/
Requires=local-fs.target
Requires=network.target

[Service]
Type=oneshot
Environment="XDG_CACHE_HOME=/var/cache"
ExecStartPre=/usr/local/bin/cleanup-backups.sh
ExecStart=/usr/local/sbin/restic-backup.sh

[Install]
WantedBy=multi-user.target

/etc/systemd/system/restic-backup.timer:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
[Unit]
Description=Activates Backup Job

[Timer]
# see man 7 systemd.time for possible formats
OnCalendar=*-*-* 01:30:00
RandomizedDelaySec=120

[Install]
WantedBy=timers.target

The nice thing about systemd timers is that they are a lot more flexible that traditional cronjobs, i.e. you can specify multiple run times for them (e.g. at noon, at midnight, on reboot). It also makes it really simply to random delay the execution of units. This is helpful when you have multiple servers backing up the same backend, so not all servers run it at the same time and the backend does not get overloaded.