add backups description

_Sidebar.md

@@ -1,2 +1,3 @@
 - [Overview services](overview-services)
 - [Integrate a new system](integrate-new-system)
+- [Backups](backups)

backups.md

@@ -0,0 +1,74 @@
+# Backups
+
+Backups are important, so I've put a lot of effort into designing a simple yet
+functional system for backup up all necessary services in my homelab.
+
+## Restic
+
+At the center of my backup architecture lies [Restic](https://restic.net/), a
+modern, fast and space-efficient backup tool. It's installed on all systems in
+my homelab, providing the main tool for backing up my data.
+
+Restic also provides [Rest Server](https://github.com/restic/rest-server), an
+HTTP server that implements the Restic backup API. This server runs on the NAS
+portion of my homelab, and it is where all backups reside.
+
+## Scripts
+
+My backup solution consists of a couple of scripts wrapped around Restic. On
+each host, a cron job runs the main backup script at 2AM every night. This main
+script is responsible for running the individual backup scripts for each
+service. It collects the results of each script and bundles them into a
+[Ntfy](https://ntfy.sh/) notification, allowing me to know in the morning
+whether all my backups have succeeded. This script also handles evicting older
+backups.
+
+The main script executes all files that match the glob pattern
+`/etc/backups/*.backup.sh`. This method allows me to define backup scripts in
+the role for a specific service without having to reconfigure the backup
+system. New backup scripts get picked up automatically the next time the main
+script is executed.
+
+## Btrfs
+
+My backup solution leverages Btrfs atomic snapshots to create consistent
+backups. Using Ansible, Btrfs subvolumes are created for a service which can
+then be used to create consistent backups. A Btrfs backup script looks like
+this:
+
+```bash
+# Snapshot data directory
+data_dir='/mnt/data1/monica/monica'
+snapshot_dir="${data_dir}.snapshot"
+
+# Read-only snapshot for atomic backup
+btrfs subvolume snapshot -r "$data_dir" "$snapshot_dir" || exit $?
+
+/usr/local/bin/restic backup "$snapshot_dir"
+
+# Always remove snapshot subvolume, even if restic fails
+btrfs subvolume delete "$snapshot_dir"
+```
+
+As you can see, `restic` can be used without any repository or password
+configuration. These values are provided as environment variables by the main
+backup script. This configuration tells `restic` to upload the backups to the
+Rest Server running on my NAS.
+
+## Miscellaneous
+
+Not everything can/should be backed up using a Btrfs snapshot. For databases, I
+use the respective database dump tool to create a Gzip-compressed database
+dump. For example, this is a Postgres backup script:
+
+```bash
+cd /etc/atuin
+
+/usr/bin/docker compose exec -T db pg_dump -U atuin atuin |
+    /usr/bin/gzip --rsyncable |
+    /usr/local/bin/restic backup --stdin --stdin-filename atuin-server-postgres.sql.gz
+```
+
+Of note here is the `--rsyncable` flag used with `gzip`. This flag allows
+Restic to properly deduplicate the compressed file, reducing the space needed
+on disk.