Commit 292bda8
docs: self-hosted prod/test/dev Azure VM guide with key rotation, Azure Key Vault workflows, and rsync data migration pattern (#4545)
## Summary
This PR creates a comprehensive self-hosted JWT key rotation how-to for
SpacetimeDB with reproducible defaults, identity-preserving rotation
guidance, and host-scoped data migration runbooks. It is written for
both operators and automation workflows, with explicit guardrails for
production-like VM environments.
## Key Changes
- Adds a top-level assumptions/risk section for a multi-host topology
(`prod`, `test`, `dev`, optional `local`) including backup prerequisites
before rotation or sync.
- Defines an opinionated, end-to-end path contract for keys:
- host source: `./.generated/spacetimedb-keys`
- runtime mount: `/etc/spacetimedb`
- startup args pinned to `/etc/spacetimedb/id_ecdsa` and
`/etc/spacetimedb/id_ecdsa.pub`
- Includes reproducible quickstart scaffolding:
- OpenSSL commands for compatible ES256/P-256 key generation
- non-container and Docker startup examples
- rotation/verify command sequence and marker checks
- Documents one unified tooling surface around `spacetimedb-tooling.ts`
for:
- rotate (`--dry-run`, `--yes`)
- verify (`--verify-only`)
- token continuity (`--resign-token-only`)
- explicit token/key path overrides (`--publisher-cli-toml-path`,
`--private-key-path`)
- Clarifies 401 vs 403 behavior and why identity drift causes publish
failures after rotation.
- Covers all rotation strategies in one place:
- clean-slate rotation (stateless/dev)
- identity-preserving rotation (stateful)
- OIDC-backed identity model for production
- Adds host-scoped migration runbooks for staged sync (`prod -> test ->
dev`) with destination-side token re-signing semantics (run on
destination host context, including SSH examples).
- Explicitly separates conceptual topology guidance from currently
implemented sync primitives in reference tooling to avoid over-claiming.
- Adds operational guardrails for sync promotions:
- `rsync --delete` is destructive
- stop/start ordering around sync
- required acceptance gates: key parity, destination re-sign,
restart/redeploy, publish marker validation
- Expands verification and automation content:
- PEM/parity/fingerprint/newline checks
- AI/automation contract with inputs, outputs, required command order,
and success/failure markers
- troubleshooting flow for `InvalidSignature` and ownership mismatch
errors
## Why This Matters
The most common failure mode in self-hosted fleets is identity drift:
signatures validate, but `spacetime publish` still fails with `403
Forbidden` because destination ownership no longer matches the
publishing identity after key rotation or data movement.
The updated how-to makes these operator requirements explicit:
1. preserve identity continuity across rotations and host promotions
2. re-sign destination tokens after `rsync` in destination host context
3. treat restart/redeploy and publish markers as promotion gates
Following this runbook prevents regressions where key material is
correct but publish still fails due to stale signatures or owner
mismatch state.
## Why JWT Key Rotation Is Essential for Self-Hosters
In self-hosted deployments, operators are the signing authority.
Rotation is both a security control and an operational correctness
requirement.
### Security value
- reduces exposure window for compromised private keys
- invalidates old/stolen tokens after rotation
- mitigates risk from snapshots, backups, clones, and long-lived VM
access
### Operational value
- keeps key material consistent across multi-VM environments
- reduces publish outages after promotion/sync operations
- prevents avoidable `401`/`403` failures in routine release workflows
## Scope
- docs-only change
- public-safe examples only (no internal org names, private vault IDs,
or private secret prefixes)
- includes non-container and Docker workflows, self-publish/versioning
guidance, and marker-based validation (`PUBLISH_SUCCESS`,
`PUBLISH_FAILED`)
- includes data migration workflow guidance (`rsync` + destination
re-sign + restart/redeploy + publish checks)
---------
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>1 parent 476d157 commit 292bda8
1 file changed
Lines changed: 1297 additions & 0 deletions
0 commit comments