Keynote Spacetime simulation comparisons (#4072)

# Description of Changes

Adds a comprehensive benchmarking suite for comparing SpacetimeDB
performance against other databases and backends.

**Key features:**
- Load testing framework that runs repeatable benchmarks across multiple
connectors
- Supports SpacetimeDB, PostgreSQL, CockroachDB, SQLite, Supabase,
Convex, and Bun
- Configurable test parameters: duration, concurrency, Zipf distribution
(α) for hot/cold account skew
- Outputs JSON reports with TPS and latency statistics to `./runs/`
- Docker Compose support for containerized benchmarking
- Includes a test SpacetimeDB Rust module (`modules/test-1/server`)

**Connectors included:**
- Direct SQL connectors (Postgres, CockroachDB, SQLite)
- Drizzle ORM wrappers
- RPC-based connectors for network-isolated benchmarking
- SpacetimeDB WebSocket connector
- Supabase and Convex cloud connectors

# API and ABI breaking changes

None. This is a new standalone template with no changes to existing
code.

# Expected complexity level and risk

**1** - This is an additive change that introduces a new isolated
template folder. It has no interactions with existing SpacetimeDB crates
or modules.

# Testing

- [ ] Run `pnpm install` in `templates/keynote-2/`
- [ ] Verify SpacetimeDB module compiles: `cd
templates/keynote-2/modules/test-1/server && cargo build`
- [ ] Run a basic benchmark against SpacetimeDB (requires local
SpacetimeDB instance)

---------

Signed-off-by: bradleyshep <148254416+bradleyshep@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Co-authored-by: = <cloutiertyler@gmail.com>

Author: 3 people

templates/keynote-2/.dockerignore

@@ -0,0 +1,6 @@
+node_modules
+**/node_modules
+.git
+dist
+build
+.tmp

templates/keynote-2/.env.example

@@ -0,0 +1,63 @@
+# ===== Runtime toggles =====
+USE_DOCKER=1          # 1 = run docker compose up for pg/crdb; 0 = skip
+#SKIP_PG=1             # 1 = don't init Postgres in prep
+#SKIP_CRDB=1           # 1 = don't init Cockroach in prep
+#SKIP_SQLITE=1         # 1 = don't init SQLite in prep
+#SKIP_SUPABASE=1       # 1 = don't init Supabase in prep
+SKIP_CONVEX=1          # 1 = don't init Convex in prep
+USE_SPACETIME_METRICS_ENDPOINT=0
+
+# ===== PostgreSQL =====
+POSTGRES_USER=postgres
+POSTGRES_PASSWORD=postgres
+POSTGRES_DB=postgres
+PG_URL=postgres://postgres:postgres@127.0.0.1:5432/postgres
+
+# ===== CockroachDB =====
+CRDB_URL=postgresql://root@127.0.0.1:26257/defaultdb?sslmode=disable
+# Example managed CockroachDB URL (fill in your own values):
+#CRDB_URL=postgresql://username:password@your-cluster.region.cockroachlabs.cloud:26257/defaultdb?sslmode=verify-full
+
+# ===== SQLite =====
+SQLITE_FILE=./.data/accounts.sqlite
+SQLITE_MODE=default
+
+# ===== SpacetimeDB =====
+STDB_URL=ws://127.0.0.1:3000
+STDB_MODULE=test-1
+STDB_MODULE_PATH=./spacetimedb
+STDB_METRICS_URL=http://127.0.0.1:3000/v1/metrics
+
+# ===== Supabase  =====
+# Replace with your actual Supabase project ref
+SUPABASE_URL=http://127.0.0.1:54321
+SUPABASE_ANON_KEY=your_supabase_anon_key_here
+SUPABASE_DB_URL=postgresql://postgres:postgres@127.0.0.1:54322/postgres
+
+# ===== Convex =====
+#CONVEX_URL=https://your-deployment.convex.cloud
+CONVEX_URL=http://127.0.0.1:3210
+CONVEX_SITE_URL=http://127.0.0.1:3210
+CLEAR_CONVEX_ON_PREP=0
+CONVEX_USE_SHARDED_COUNTER=1
+
+# ===== Bun =====
+BUN_URL=http://127.0.0.1:4001
+BUN_PG_URL=postgres://postgres:postgres@pg_bun:5432/bun_bench
+
+# ===== RPC Servers =====
+PG_RPC_URL=http://127.0.0.1:4101
+CRDB_RPC_URL=http://127.0.0.1:4102
+SQLITE_RPC_URL=http://127.0.0.1:4103
+SUPABASE_RPC_URL=http://127.0.0.1:4106
+
+# ===== PlanetScale (optional) =====
+#PLANETSCALE_PG_URL=postgresql://user:password@your-planetscale-host:5432/database
+#PLANETSCALE_RPC_URL=http://127.0.0.1:4104
+
+# ===== Seeding knobs  =====
+SEED_ACCOUNTS=100000
+SEED_INITIAL_BALANCE=10000000
+
+VERIFY=0
+ENABLE_RPC_SERVERS=0

templates/keynote-2/.gitignore

@@ -0,0 +1,39 @@
+# Environment files
+.env
+.env.*
+!.env.example
+
+# Data files
+.data/
+
+# Auto-generated files
+convex-app/convex/_generated/
+modules/test-1/module_bindings/
+
+# SpacetimeDB
+.spacetime/
+
+# Logs
+logs
+*.log
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+pnpm-debug.log*
+lerna-debug.log*
+
+node_modules
+dist
+dist-ssr
+*.local
+
+# Editor directories and files
+.vscode/
+.idea
+.DS_Store
+*.suo
+*.ntvs*
+*.njsproj
+*.sln
+*.sw?
+/runs

templates/keynote-2/.prettierrc

@@ -0,0 +1,5 @@
+{
+  "singleQuote": true,
+  "printWidth": 80,
+  "tabWidth": 2
+}

templates/keynote-2/DEVELOP.md

@@ -0,0 +1,241 @@
+## Overview
+
+This app runs repeatable load tests against multiple connectors (Bun+Postgres, CockroachDB, SQLite, Supabase, Convex, SpacetimeDB, etc.).
+
+Each run:
+
+- Loads a scenario from `src/tests/<test-name>/`
+- Runs it against one or more connectors
+- Writes a JSON report into `./runs/` with TPS and latency stats
+
+---
+
+## Demo Mode
+
+Run a quick performance comparison:
+
+```bash
+npm run demo
+```
+
+The script will:
+
+- Check that required services are running (prompts you to start them if not)
+- Seed databases with test data
+- Run benchmarks at high contention
+- Display animated results comparing your chosen systems (SpacetimeDB and Convex by default, as both are simple to run locally)
+
+**Options:**
+
+- `--seconds N` - Benchmark duration (default: 10)
+- `--concurrency N` - Concurrent connections (default: 50)
+- `--alpha N` - Contention level (default: 1.5)
+- `--systems a,b,c` - Systems to compare (default: convex,spacetimedb)
+- `--skip-prep` - Skip database seeding
+- `--no-animation` - Disable animated output
+
+---
+
+## Prerequisites
+
+- **Node.js** ≥ 20.x
+- **pnpm** installed globally
+- **Docker** for local Postgres / Cockroach / Supabase
+- Local/Cloud Convex
+
+From a fresh clone:
+
+```bash
+pnpm install
+```
+
+---
+
+## Configuration (`.env`)
+
+Copy `.env.example` to `.env` and adjust.
+
+**Seeding / verification:**
+
+- `SEED_ACCOUNTS` – number of accounts to seed (if unset, code defaults to `100_000`)
+- `SEED_INITIAL_BALANCE` – starting balance per account
+- `VERIFY` – enable extra verification passes when non-zero
+- `ENABLE_RPC_SERVERS` – flag used by scripts that start the RPC benchmark servers
+
+**Runtime toggles:**
+
+- `USE_DOCKER` – `1` = run Docker Compose for Postgres / CockroachDB; `0` = skip
+- `SKIP_PG` – `1` = don't init Postgres in prep
+- `SKIP_CRDB` – `1` = don't init CockroachDB in prep
+- `SKIP_SQLITE` – `1` = don't init SQLite in prep
+- `SKIP_SUPABASE` – `1` = don't init Supabase in prep
+- `SKIP_CONVEX` – `1` = don't init Convex in prep
+- `USE_SPACETIME_METRICS_ENDPOINT` – `1` = read committed transfer counts from the SpacetimeDB metrics endpoint; otherwise only local counters are used
+
+**PostgreSQL / CockroachDB:**
+
+- `PG_URL` – Postgres connection string
+- `CRDB_URL` – CockroachDB connection string
+
+**SQLite:**
+
+- `SQLITE_FILE` – path to the SQLite file
+- `SQLITE_MODE` – tuning preset for the SQLite connector
+
+**SpacetimeDB:**
+
+- `STDB_URL` – WebSocket URL for SpacetimeDB
+- `STDB_MODULE` – module name to load (e.g. `test-1`)
+- `STDB_MODULE_PATH` – filesystem path to the module source (for local dev)
+- `STDB_METRICS_URL` – HTTP URL for the SpacetimeDB metrics endpoint
+
+**Supabase:**
+
+- `SUPABASE_URL` – Supabase project URL
+- `SUPABASE_ANON_KEY` – Supabase anon/public key
+- `SUPABASE_DB_URL` – Postgres connection string for the Supabase database
+
+**Convex:**
+
+- `CONVEX_URL` – Convex deployment URL
+- `CONVEX_SITE_URL` – Convex site URL
+- `CLEAR_CONVEX_ON_PREP` – Convex prep flag (clears data when enabled)
+- `CONVEX_USE_SHARDED_COUNTER` – flag for using the sharded-counter implementation
+
+**Bun / RPC helpers:**
+
+- `BUN_URL` – Bun HTTP benchmark server URL
+- `BUN_PG_URL` – Postgres connection string for the Bun benchmark service
+
+**RPC benchmark servers:**
+
+- `PG_RPC_URL` – HTTP URL for the Postgres RPC server
+- `CRDB_RPC_URL` – HTTP URL for the CockroachDB RPC server
+- `SQLITE_RPC_URL` – HTTP URL for the SQLite RPC server
+
+---
+
+## Setup
+
+### Generate bindings (first time after clone)
+
+**SpacetimeDB module bindings:**
+
+```bash
+cd spacetimedb
+spacetimedb generate --lang typescript --out-dir ../module_bindings
+cd ..
+```
+
+**Convex generated files:**
+
+```bash
+cd convex-app
+npx convex dev
+# Wait for it to generate files, then Ctrl+C
+cd ..
+```
+
+### Start services
+
+1. Start SpacetimeDB (`spacetimedb start`)
+2. Start Convex (inside convex-app run `npx convex dev`)
+3. Init Supabase (run `supabase init`) inside project root.
+4. `npm run prep` to seed the databases.
+5. `npm run bench` to run the test against all connectors.
+
+## Commands & Examples
+
+### 1. Run a test
+
+```bash
+npm run bench [test-name] [--seconds N] [--concurrency N] [--alpha A] [--connectors list]
+```
+
+Examples:
+
+```bash
+# Default test (test-1), default args (note: only 1 test right now, and it's embedded)
+npm run bench
+
+# Explicit test name
+npm run bench test-1
+
+# Short run, 100 concurrent workers
+npm run bench test-1 --seconds 10 --concurrency 100
+
+# Heavier skew on hot accounts
+npm run bench test-1 --alpha 2.0
+
+# Only run selected connectors
+npm run bench test-1 --connectors spacetimedb,sqlite
+```
+
+---
+
+## CLI Arguments
+
+From `src/cli.ts`:
+
+- **`test-name`** (positional)
+  - Name of the test folder under `src/tests/`
+  - Default: `test-1`
+
+- **`--seconds N`**
+  - Duration of the benchmark in seconds
+  - Default: `1`
+
+- **`--concurrency N`**
+  - Number of workers / in-flight operations
+  - Default: `10`
+
+- **`--alpha A`**
+  - Zipf α parameter for account selection (hot vs cold distribution)
+  - Default: `0.5`
+
+- **`--connectors list`**
+  - Optional, comma-separated list of connector `system` names
+  - Example:
+
+    ```bash
+    --connectors spacetimedb,sqlite,postgres
+    ```
+
+  - If omitted, all connectors for that test are run
+  - The valid names come from `tc.system` in the test modules and the keys in `CONNECTORS`
+
+- **`--contention-tests startAlpha endAlpha step concurrency`**
+  - Runs a sweep over Zipf α values for a single connector
+  - Uses `startAlpha`, `endAlpha`, and `step` to choose the α values
+  - Uses the provided `concurrency` for all runs
+
+- **`--concurrency-tests startConc endConc step alpha`**
+  - Runs a sweep over concurrency levels for a single connector
+  - Uses `startConc`, `endConc`, and `step` to choose the concurrency values
+  - Uses the provided `alpha` for all runs
+
+---
+
+### Running in Docker
+
+You can also run the benchmark via Docker instead of Node directly:
+
+```bash
+docker compose run --rm bench \
+  --seconds 5 \
+  --concurrency 50 \
+  --alpha 1 \
+  --connectors convex
+```
+
+If using Docker, make sure to set `USE_DOCKER=1` in `.env`, verify docker-compose env variables, verify you've run supabase init, and run `npm prep` before running bench.
+
+## Output
+
+Every run writes a JSON file into `./runs/`:
+
+- Directory: `./runs/`
+- Filename: `<test-name>-<timestamp>.json`
+  - Example: `test-1-2025-11-17T16-45-12-345Z.json`
+
+Point your visualizations / CSV exports at `./runs/` and you’re good.

templates/keynote-2/Dockerfile.bench

@@ -0,0 +1,17 @@
+# Dockerfile.bench
+FROM node:20.18.1-bookworm-slim
+
+WORKDIR /app
+RUN npm install -g pnpm@9.15.3
+
+# Install build tools for better-sqlite3
+RUN apt-get update && apt-get install -y python3 make g++ && rm -rf /var/lib/apt/lists/*
+
+COPY package.json pnpm-lock.yaml* tsconfig.json* ./
+RUN pnpm install --frozen-lockfile || pnpm install
+
+COPY src ./src
+COPY modules ./modules
+
+# Correct: use bench script with passed args
+ENTRYPOINT ["pnpm", "tsx", "src/cli.ts"]

templates/keynote-2/Dockerfile.bun

@@ -0,0 +1,14 @@
+FROM oven/bun:latest
+
+WORKDIR /app
+
+COPY package.json bun.lockb* tsconfig.json ./
+
+RUN bun install --ignore-scripts
+
+# Copy source
+COPY bun ./bun
+
+ENV PORT=4001
+
+CMD ["bun", "bun/bun-server.ts"]