Backups

Backups are important, so I've put a lot of effort into designing a simple yet functional system for backup up all necessary services in my homelab.

Restic

At the center of my backup architecture lies Restic, a modern, fast and space-efficient backup tool. It's installed on all systems in my homelab, providing the main tool for backing up my data.

Restic also provides Rest Server, an HTTP server that implements the Restic backup API. This server runs on the NAS portion of my homelab, and it is where all backups reside.

Scripts

My backup solution consists of a couple of scripts wrapped around Restic. On each host, a cron job runs the main backup script at 2AM every night. This main script is responsible for running the individual backup scripts for each service. It collects the results of each script and bundles them into a Ntfy notification, allowing me to know in the morning whether all my backups have succeeded. This script also handles evicting older backups.

The main script executes all files that match the glob pattern /etc/backups/*.backup.sh. This method allows me to define backup scripts in the role for a specific service without having to reconfigure the backup system. New backup scripts get picked up automatically the next time the main script is executed.

Btrfs

My backup solution leverages Btrfs atomic snapshots to create consistent backups. Using Ansible, Btrfs subvolumes are created for a service which can then be used to create consistent backups. A Btrfs backup script looks like this:

# Snapshot data directory
data_dir='/mnt/data1/monica/monica'
snapshot_dir="${data_dir}.snapshot"

# Read-only snapshot for atomic backup
btrfs subvolume snapshot -r "$data_dir" "$snapshot_dir" || exit $?

/usr/local/bin/restic backup "$snapshot_dir"

# Always remove snapshot subvolume, even if restic fails
btrfs subvolume delete "$snapshot_dir"

As you can see, restic can be used without any repository or password configuration. These values are provided as environment variables by the main backup script. This configuration tells restic to upload the backups to the Rest Server running on my NAS.

Miscellaneous

Not everything can/should be backed up using a Btrfs snapshot. For databases, I use the respective database dump tool to create a Gzip-compressed database dump. For example, this is a Postgres backup script:

cd /etc/atuin

/usr/bin/docker compose exec -T db pg_dump -U atuin atuin |
    /usr/bin/gzip --rsyncable |
    /usr/local/bin/restic backup --stdin --stdin-filename atuin-server-postgres.sql.gz

Of note here is the --rsyncable flag used with gzip. This flag allows Restic to properly deduplicate the compressed file, reducing the space needed on disk.