Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 52 additions & 6 deletions docs/content/self-hosting/operations-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -410,23 +410,69 @@ The restore test restores the latest dump into a scratch database, prints its us

When `coder-backup.service` fails, `coder-backup-failed.service` posts to Discord through the relay from [Step 11 of the server setup](server-setup.md#step-11-set-up-discord-notifications). A backup that stops running entirely does not fail, so also monitor the age of `/var/lib/coder-backup/last-success`, which holds the time of the last successful backup. In Icinga Director, add a service on the host that runs `check_file_age` with `-f /var/lib/coder-backup/last-success -w 93600 -c 180000`, which warns after 26 hours and goes critical after 50.

#### Practice a recovery

A backup that has never been recovered isn't known to work. Once a quarter, and after any change to the backup setup, recover it on a machine other than the server, with only the secrets from 1Password. This is the situation after a server is lost.

`scripts/coder-backup-verify.sh` needs only Docker. It downloads the latest backup with duplicity in an Ubuntu container, and loads the database dump into a scratch PostgreSQL 16 container. Then it lists the users, templates and workspaces in the dump, and the time of its newest audit log entry.

1. Write the backup secrets to a private env file. This example reads them with the 1Password CLI:

```bash
ITEM="op://YOUR_VAULT/YOUR_BACKUP_ITEM"
umask 077
cat > ~/tmp/coder-backup.env <<EOF
BACKUP_TARGET=s3://$(op read "$ITEM/bucket")/coder.example.com
PASSPHRASE=$(op read "$ITEM/duplicity passphrase")
AWS_ACCESS_KEY_ID=$(op read "$ITEM/AWS_ACCESS_KEY_ID")
AWS_SECRET_ACCESS_KEY=$(op read "$ITEM/AWS_SECRET_ACCESS_KEY")
AWS_DEFAULT_REGION=$(op read "$ITEM/region")
EOF
```

2. Run the script from a checkout of this repository:

```bash
scripts/coder-backup-verify.sh ~/tmp/coder-backup.env
```

3. Check the output:
- "Chain end time" is from the last nightly backup.
- `/var/backups/coder/coder.dump`, `/root/coder-pg-password` and `/etc/coder.d/coder.env` are present.
- The users, templates and workspaces match `coder users list` and `coder list -a` on the live server, apart from changes made since the backup.

4. Delete the env file, because it contains the secrets. The script has already deleted the restored files:

```bash
rm ~/tmp/coder-backup.env
```

`/etc/coder-backup.env` itself is not in the backup, because its secrets come from 1Password.

#### Restore after losing the server

1. Rebuild the host by following the [server setup](server-setup.md) through PostgreSQL (Step 5). Install `duplicity` and `python3-boto3`, and write `/etc/coder-backup.env` from 1Password.
2. Restore the backup to a staging directory, then copy back what you need:
1. Rebuild the host by following the [server setup](server-setup.md) through Step 5, but skip "Create the Coder database and user". Install `duplicity` and `python3-boto3`, and write `/etc/coder-backup.env` from 1Password as shown in [Install](#install).
2. Restore the backup to a staging directory:

```bash
sudo bash -c 'set -a; source /etc/coder-backup.env; set +a; duplicity restore "$BACKUP_TARGET" /root/restore'
```

3. Restore the database before starting Coder. The dump owns its objects as `coder`, so create that user first with the password from `/root/restore/root/coder-pg-password`:
3. Restore the database before starting Coder. The dump assigns its objects to the `coder` user, so create that user first with the old password:

```bash
sudo -u postgres createdb -O coder coder
sudo -u postgres pg_restore --dbname coder /root/restore/var/backups/coder/coder.dump
sudo -u postgres psql -v pw="$(sudo cat /root/restore/root/coder-pg-password)" <<'EOF'
CREATE USER coder WITH PASSWORD :'pw';
CREATE DATABASE coder OWNER coder;
EOF
sudo cat /root/restore/var/backups/coder/coder.dump | sudo -u postgres pg_restore --dbname coder
sudo install -m 600 /root/restore/root/coder-pg-password /root/coder-pg-password
```

4. Copy `/etc/coder.d`, `/etc/letsencrypt` and the other files from `/root/restore` into place, then continue with the remaining setup steps.
The `postgres` user can't read `/root`, so `pg_restore` reads the dump from standard input.

4. Copy `/etc/coder.d`, `/etc/letsencrypt` and the other files you need from `/root/restore` into place with `sudo cp -a`.
5. Continue the server setup from Step 6. Where a step creates a file that the backup holds, use the restored file instead. The database already holds the users, templates and API tokens, so skip the steps that create them.

### Workspace Data Backup

Expand Down
53 changes: 53 additions & 0 deletions scripts/coder-backup-verify.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
#!/usr/bin/env bash
# Download the latest Coder backup from S3 and load its database dump into a
# scratch PostgreSQL, on any machine with Docker. Use it to practice recovery
# without the server: it needs only the backup secrets, not the server itself.
#
# Usage: coder-backup-verify.sh <env-file>
# The env file sets BACKUP_TARGET, PASSPHRASE, AWS_ACCESS_KEY_ID,
# AWS_SECRET_ACCESS_KEY and AWS_DEFAULT_REGION, like /etc/coder-backup.env.
# The restored files contain secrets, so they are deleted on exit.
set -euo pipefail

ENVFILE=${1:?usage: coder-backup-verify.sh <env-file>}
WORK=$(mktemp -d "${TMPDIR:-/tmp}/coder-backup-verify.XXXX")
PG=coder-backup-verify-pg

cleanup() {
docker rm -f "$PG" >/dev/null 2>&1 || true
# Files restored in the container are owned by root, so delete them there.
docker run --rm -v "$WORK:/restore" ubuntu:24.04 rm -rf /restore/files >/dev/null 2>&1 || true
rmdir "$WORK" 2>/dev/null || true
}
trap cleanup EXIT

echo "== Downloading the latest backup"
docker run --rm --env-file "$ENVFILE" -v "$WORK:/restore" ubuntu:24.04 bash -c '
set -e
apt-get update -qq >/dev/null
DEBIAN_FRONTEND=noninteractive apt-get install -y -qq duplicity python3-boto3 >/dev/null 2>&1
duplicity collection-status "$BACKUP_TARGET" | grep -E "Last full backup date|Chain end time|Number of contained backup sets"
duplicity restore "$BACKUP_TARGET" /restore/files >/dev/null
chmod -R a+rX /restore/files
'
for f in var/backups/coder/coder.dump root/coder-pg-password etc/coder.d/coder.env; do
[ -s "$WORK/files/$f" ] && echo "present /$f" || echo "MISSING /$f"
done

echo "== Loading the dump into a scratch PostgreSQL 16"
docker run -d --name "$PG" -e POSTGRES_PASSWORD=verify \
-v "$WORK/files/var/backups/coder:/dump:ro" postgres:16 >/dev/null
until docker exec "$PG" pg_isready -U postgres -q; do sleep 1; done
sleep 2
docker exec "$PG" createdb -U postgres coder
docker exec "$PG" pg_restore -U postgres --no-owner --dbname coder /dump/coder.dump

q() { docker exec "$PG" psql -U postgres -d coder -tA -F ' ' -c "$1"; }
echo "== Users"
q "select username, login_type, created_at::date from users where not deleted order by created_at"
echo "== Templates and version counts"
q "select name, (select count(*) from template_versions v where v.template_id = t.id) from templates t where not deleted order by name"
echo "== Workspaces"
q "select u.username || '/' || w.name from workspaces w join users u on u.id = w.owner_id where not w.deleted order by 1"
echo "== Newest audit log entry"
q "select max(time) from audit_logs"
Loading