From a2080c711160e80afeeebc5434088ce98b8a8303 Mon Sep 17 00:00:00 2001 From: Randy Fay Date: Tue, 17 Nov 2026 09:22:38 -0700 Subject: [PATCH] docs: add a backup recovery drill and fix the database restore steps MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Short Summary (TL;DR) The operations guide now explains how to recover the Coder database backup on another machine, using only Docker and the secrets in 1Password, with a new `scripts/coder-backup-verify.sh`. It also fixes the restore steps for a lost server, which could not work as written. ## The Issue - Refs #259, which lists a practice recovery as a follow-up to the 2026-11-16 outage. The only documented check of the backup, `coder-restore-test`, runs on the server itself. After the server is lost, nothing showed that the backup could be recovered with only the secrets stored elsewhere. The "Restore after losing the server" steps had three errors: - Step 1 followed the setup through Step 5, which creates the `coder` database and user, but step 3 creates them again. - Step 3 said to create the `coder` user but gave no command for it. - `sudo -u postgres pg_restore /root/restore/...` fails, because the `postgres` user can't read `/root`. ## How This PR Solves The Issue - New "Practice a recovery" section. It writes the backup secrets from 1Password to a private env file and runs `scripts/coder-backup-verify.sh`. It also says what to check in the output. It recommends a drill once a quarter and after any change to the backup setup. - `scripts/coder-backup-verify.sh` downloads the latest backup with duplicity in an `ubuntu:24.04` container. It loads the dump into a `postgres:16` container and lists users, templates, workspaces and the newest audit log entry. The restored files contain secrets, so it deletes them on exit. - The restore steps now skip "Create the Coder database and user" in the setup. They create the user with the password from the backup, pipe the dump to `pg_restore`, and install `/root/coder-pg-password`. ## Manual Testing Instructions On any machine with Docker and the 1Password CLI, follow [Practice a recovery](https://github.com/ddev/coder-ddev/blob/20261117_rfay_backup_restore_drill/docs/content/self-hosting/operations-guide.md#practice-a-recovery) with your vault and item names. Expect: ```text Chain end time: present /var/backups/coder/coder.dump present /root/coder-pg-password present /etc/coder.d/coder.env == Users ... ``` The users, templates and workspaces should match `coder users list` and `coder list -a`. Afterwards, `ls -d "${TMPDIR:-/tmp}"/coder-backup-verify.*` should find nothing. ## Automated Testing Overview - Claude Code ran the documented drill on macOS with OrbStack against the coder.ddev.com backup from 2026-11-17 02:38 UTC. It restored 3 users, 4 templates and 2 workspaces, which match the live server, and it left no files or containers behind. - Claude Code ran restore step 3 verbatim in an `ubuntu:24.04` container with Ubuntu's PostgreSQL 16 and a test dump. The `coder` user logged in with a restored password containing quotes, `$`, a backtick and a space, and the restored table kept `coder` as owner. - Claude Code ran `shellcheck` on the script and `make docs-check`, which passed. - Nobody has tested a full server restore with steps 4 and 5. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 5.5 --- docs/content/self-hosting/operations-guide.md | 58 +++++++++++++++++-- scripts/coder-backup-verify.sh | 53 +++++++++++++++++ 2 files changed, 105 insertions(+), 6 deletions(-) create mode 100755 scripts/coder-backup-verify.sh diff --git a/docs/content/self-hosting/operations-guide.md b/docs/content/self-hosting/operations-guide.md index aa530e7..9ba3c85 100644 --- a/docs/content/self-hosting/operations-guide.md +++ b/docs/content/self-hosting/operations-guide.md @@ -410,23 +410,69 @@ The restore test restores the latest dump into a scratch database, prints its us When `coder-backup.service` fails, `coder-backup-failed.service` posts to Discord through the relay from [Step 11 of the server setup](server-setup.md#step-11-set-up-discord-notifications). A backup that stops running entirely does not fail, so also monitor the age of `/var/lib/coder-backup/last-success`, which holds the time of the last successful backup. In Icinga Director, add a service on the host that runs `check_file_age` with `-f /var/lib/coder-backup/last-success -w 93600 -c 180000`, which warns after 26 hours and goes critical after 50. +#### Practice a recovery + +A backup that has never been recovered isn't known to work. Once a quarter, and after any change to the backup setup, recover it on a machine other than the server, with only the secrets from 1Password. This is the situation after a server is lost. + +`scripts/coder-backup-verify.sh` needs only Docker. It downloads the latest backup with duplicity in an Ubuntu container, and loads the database dump into a scratch PostgreSQL 16 container. Then it lists the users, templates and workspaces in the dump, and the time of its newest audit log entry. + +1. Write the backup secrets to a private env file. This example reads them with the 1Password CLI: + + ```bash + ITEM="op://YOUR_VAULT/YOUR_BACKUP_ITEM" + umask 077 + cat > ~/tmp/coder-backup.env < +# The env file sets BACKUP_TARGET, PASSPHRASE, AWS_ACCESS_KEY_ID, +# AWS_SECRET_ACCESS_KEY and AWS_DEFAULT_REGION, like /etc/coder-backup.env. +# The restored files contain secrets, so they are deleted on exit. +set -euo pipefail + +ENVFILE=${1:?usage: coder-backup-verify.sh } +WORK=$(mktemp -d "${TMPDIR:-/tmp}/coder-backup-verify.XXXX") +PG=coder-backup-verify-pg + +cleanup() { + docker rm -f "$PG" >/dev/null 2>&1 || true + # Files restored in the container are owned by root, so delete them there. + docker run --rm -v "$WORK:/restore" ubuntu:24.04 rm -rf /restore/files >/dev/null 2>&1 || true + rmdir "$WORK" 2>/dev/null || true +} +trap cleanup EXIT + +echo "== Downloading the latest backup" +docker run --rm --env-file "$ENVFILE" -v "$WORK:/restore" ubuntu:24.04 bash -c ' + set -e + apt-get update -qq >/dev/null + DEBIAN_FRONTEND=noninteractive apt-get install -y -qq duplicity python3-boto3 >/dev/null 2>&1 + duplicity collection-status "$BACKUP_TARGET" | grep -E "Last full backup date|Chain end time|Number of contained backup sets" + duplicity restore "$BACKUP_TARGET" /restore/files >/dev/null + chmod -R a+rX /restore/files +' +for f in var/backups/coder/coder.dump root/coder-pg-password etc/coder.d/coder.env; do + [ -s "$WORK/files/$f" ] && echo "present /$f" || echo "MISSING /$f" +done + +echo "== Loading the dump into a scratch PostgreSQL 16" +docker run -d --name "$PG" -e POSTGRES_PASSWORD=verify \ + -v "$WORK/files/var/backups/coder:/dump:ro" postgres:16 >/dev/null +until docker exec "$PG" pg_isready -U postgres -q; do sleep 1; done +sleep 2 +docker exec "$PG" createdb -U postgres coder +docker exec "$PG" pg_restore -U postgres --no-owner --dbname coder /dump/coder.dump + +q() { docker exec "$PG" psql -U postgres -d coder -tA -F ' ' -c "$1"; } +echo "== Users" +q "select username, login_type, created_at::date from users where not deleted order by created_at" +echo "== Templates and version counts" +q "select name, (select count(*) from template_versions v where v.template_id = t.id) from templates t where not deleted order by name" +echo "== Workspaces" +q "select u.username || '/' || w.name from workspaces w join users u on u.id = w.owner_id where not w.deleted order by 1" +echo "== Newest audit log entry" +q "select max(time) from audit_logs"