How to back up a server

Which tool

Only restic writes straight to S3 storage, which is why the steps below use it.

Steps

1. List what to save

   Save what you cannot rebuild: configs in /etc, your project folders, uploads and a dump of each database. Skip caches, node_modules and anything a package manager can download again. Dump a running database instead of copying its files.

   $ sudo mkdir -p /var/backups/db
   $ sudo -u postgres pg_dump mydb | gzip > /var/backups/db/mydb.sql.gz

2. Make a bucket and a key

   Create a bucket for backups only, and a key that can reach that bucket and nothing else. If the server is ever broken into, the thief gets that one bucket, not your account. Any S3 provider works: Amazon S3, Cloudflare R2, Backblaze B2 and others. Uno cloud storage does not give S3 access keys yet, so restic cannot write to it; on Uno, the daily disk copies below are the layer Uno keeps for you.

3. Install restic

   $ sudo apt-get install restic
   $ restic version

4. Keep the keys and the password in one root-only file

   restic encrypts the repository with a password. Make a long one and keep a second copy in a password manager, outside the server: without it the snapshots cannot be read. The names below are the ones restic reads from the environment.

   $ sudo mkdir -m 700 /etc/restic
   $ openssl rand -base64 32 | sudo tee /etc/restic/password
   $ sudo nano /etc/restic/env
   
   RESTIC_REPOSITORY=s3:https://s3.example.com/my-backups
   RESTIC_PASSWORD_FILE=/etc/restic/password
   AWS_ACCESS_KEY_ID=<your access key>
   AWS_SECRET_ACCESS_KEY=<your secret key>
   
   $ sudo chmod 600 /etc/restic/env /etc/restic/password

   Replace s3.example.com with your provider's S3 address and my-backups with your bucket. Write it as above: the bucket comes after the host, not inside the host name.

5. Create the repository and run the first backup

   The first run uploads everything. Later runs send only what changed.

   $ sudo bash -c 'set -a; . /etc/restic/env; set +a; restic init'
   $ sudo bash -c 'set -a; . /etc/restic/env; set +a; restic backup /etc /home/dev/app /var/backups/db'

6. Put it in a script and on a timer

   The script dumps the database, backs up, then keeps seven daily, four weekly and six monthly snapshots and removes the rest. Name your own numbers with --keep-daily, --keep-weekly and --keep-monthly. While restic prunes, the repository is locked and no backup can finish, so run it at night.

   $ sudo nano /usr/local/bin/backup.sh
   
   #!/usr/bin/env bash
   set -euo pipefail
   set -a; . /etc/restic/env; set +a
   sudo -u postgres pg_dump mydb | gzip > /var/backups/db/mydb.sql.gz
   restic backup /etc /home/dev/app /var/backups/db
   restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune
   
   $ sudo chmod 700 /usr/local/bin/backup.sh

   A systemd timer runs the script every night. Persistent=true makes systemd run a missed backup after the machine was off.

   $ sudo nano /etc/systemd/system/backup.service
   
   [Unit]
   Description=Back up to S3
   
   [Service]
   Type=oneshot
   ExecStart=/usr/local/bin/backup.sh
   
   $ sudo nano /etc/systemd/system/backup.timer
   
   [Unit]
   Description=Nightly backup
   
   [Timer]
   OnCalendar=*-*-* 03:00:00
   RandomizedDelaySec=15min
   Persistent=true
   
   [Install]
   WantedBy=timers.target
   
   $ sudo systemctl daemon-reload
   $ sudo systemctl enable --now backup.timer

7. Restore something, today

   Restore the latest snapshot into an empty folder and open a file from it. This is the only step that proves the backup. Then ask restic to check the repository.

   $ sudo bash -c 'set -a; . /etc/restic/env; set +a; restic restore latest --target /tmp/restore-test'
   $ ls /tmp/restore-test/etc
   $ sudo bash -c 'set -a; . /etc/restic/env; set +a; restic check'

Check that it works

Next morning, check when the timer ran and list the snapshots. You should see one new snapshot a day.

$ systemctl list-timers backup.timer
$ journalctl -u backup.service -n 20
$ sudo bash -c 'set -a; . /etc/restic/env; set +a; restic snapshots'

Why keep both layers

• The daily copy is a copy of the disk, not a dump of one database: it was made while the computer ran. Keep the database dump from step 1 as well.
• There are no hourly copies, so you can lose up to a day of changes. Run your own timer more often if that is too much.

Common problems