Skip to content

Commit 292bda8

Browse files
benpsnyderbfops
andauthored
docs: self-hosted prod/test/dev Azure VM guide with key rotation, Azure Key Vault workflows, and rsync data migration pattern (#4545)
## Summary This PR creates a comprehensive self-hosted JWT key rotation how-to for SpacetimeDB with reproducible defaults, identity-preserving rotation guidance, and host-scoped data migration runbooks. It is written for both operators and automation workflows, with explicit guardrails for production-like VM environments. ## Key Changes - Adds a top-level assumptions/risk section for a multi-host topology (`prod`, `test`, `dev`, optional `local`) including backup prerequisites before rotation or sync. - Defines an opinionated, end-to-end path contract for keys: - host source: `./.generated/spacetimedb-keys` - runtime mount: `/etc/spacetimedb` - startup args pinned to `/etc/spacetimedb/id_ecdsa` and `/etc/spacetimedb/id_ecdsa.pub` - Includes reproducible quickstart scaffolding: - OpenSSL commands for compatible ES256/P-256 key generation - non-container and Docker startup examples - rotation/verify command sequence and marker checks - Documents one unified tooling surface around `spacetimedb-tooling.ts` for: - rotate (`--dry-run`, `--yes`) - verify (`--verify-only`) - token continuity (`--resign-token-only`) - explicit token/key path overrides (`--publisher-cli-toml-path`, `--private-key-path`) - Clarifies 401 vs 403 behavior and why identity drift causes publish failures after rotation. - Covers all rotation strategies in one place: - clean-slate rotation (stateless/dev) - identity-preserving rotation (stateful) - OIDC-backed identity model for production - Adds host-scoped migration runbooks for staged sync (`prod -> test -> dev`) with destination-side token re-signing semantics (run on destination host context, including SSH examples). - Explicitly separates conceptual topology guidance from currently implemented sync primitives in reference tooling to avoid over-claiming. - Adds operational guardrails for sync promotions: - `rsync --delete` is destructive - stop/start ordering around sync - required acceptance gates: key parity, destination re-sign, restart/redeploy, publish marker validation - Expands verification and automation content: - PEM/parity/fingerprint/newline checks - AI/automation contract with inputs, outputs, required command order, and success/failure markers - troubleshooting flow for `InvalidSignature` and ownership mismatch errors ## Why This Matters The most common failure mode in self-hosted fleets is identity drift: signatures validate, but `spacetime publish` still fails with `403 Forbidden` because destination ownership no longer matches the publishing identity after key rotation or data movement. The updated how-to makes these operator requirements explicit: 1. preserve identity continuity across rotations and host promotions 2. re-sign destination tokens after `rsync` in destination host context 3. treat restart/redeploy and publish markers as promotion gates Following this runbook prevents regressions where key material is correct but publish still fails due to stale signatures or owner mismatch state. ## Why JWT Key Rotation Is Essential for Self-Hosters In self-hosted deployments, operators are the signing authority. Rotation is both a security control and an operational correctness requirement. ### Security value - reduces exposure window for compromised private keys - invalidates old/stolen tokens after rotation - mitigates risk from snapshots, backups, clones, and long-lived VM access ### Operational value - keeps key material consistent across multi-VM environments - reduces publish outages after promotion/sync operations - prevents avoidable `401`/`403` failures in routine release workflows ## Scope - docs-only change - public-safe examples only (no internal org names, private vault IDs, or private secret prefixes) - includes non-container and Docker workflows, self-publish/versioning guidance, and marker-based validation (`PUBLISH_SUCCESS`, `PUBLISH_FAILED`) - includes data migration workflow guidance (`rsync` + destination re-sign + restart/redeploy + publish checks) --------- Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
1 parent 476d157 commit 292bda8

1 file changed

Lines changed: 1297 additions & 0 deletions

File tree

0 commit comments

Comments
 (0)