Skip to content

Commit 7143ed9

Browse files
authored
Clean up keynote-2 template README & DEVELOP (#4624)
# Description of Changes Refresh this stale PR against current `master`. Several original items were already applied upstream or conflict with recent keynote-2 work (#4616, #4647, #4678, #4682, #4698, #4703, #4743, #4745, #4753, #4757), so those are dropped. What remains is the still-relevant subset, rebased onto the current file structures. **README.md:** - Use `pnpm run demo` in Quick Demo (consistency with pnpm workspace) - Add `--concurrency` and `--alpha` to demo options - Add `--` separator to `docker compose run` bench example - Fix hardware config punctuation (add comma before "OS:") - Remove redundant Quick Start section; replace with link to DEVELOP.md for prerequisites and CLI reference - Add symlink for license **DEVELOP.md:** - Use `pnpm run` throughout (demo, prep, bench) instead of `npm run` - Drop the `-- ` pass-through after `pnpm run bench` (not needed with pnpm; matches the `#4703` testing examples) - Add Rust to Prerequisites - Add explicit list of valid connector names (`convex`, `spacetimedb`, `bun`, `postgres_rpc`, `cockroach_rpc`, `sqlite_rpc`, `supabase_rpc`, `planetscale_pg_rpc`) - Update CLI reference defaults to match methodology (seconds: 1→10, concurrency: 10→50) - Condense `docker compose run` bench example to a single line with `--` separator; fix `npm prep` → `pnpm run prep` **src/opts.ts:** (CLI parsing moved here in `#4703`; original PR targeted the now-gone inline parsers in `cli.ts`/`demo.ts`) - `parseBenchOptions`: bench `--seconds` default `1` → `10` - `parseDemoOptions`: demo `--concurrency` default `10` → `50` **.env.example:** - Comment out `USE_DOCKER=1` and `SKIP_CONVEX=1` so demo defaults (convex, spacetimedb) work out of the box - Comment out `CONVEX_USE_SHARDED_COUNTER=1` (still a supported knob, just off by default) # Dropped as superseded by master - Rust Client README section tweaks (heading capitalization, `bottlnecked`/`then` typo fixes) — section was removed by `#4753` - Rename `SPACETIME_METRICS_ENDPOINT` → `USE_SPACETIME_METRICS_ENDPOINT` — master's `src/config.ts` still reads the original name - Connector-name fixes in examples (`sqlite` → `sqlite_rpc`, `postgres` → `postgres_rpc`) — already corrected on master # API and ABI breaking changes None. # Expected complexity level and risk **1** – Documentation and default-value changes. No functional changes to core logic. # Testing - [x] `pnpm install` in `templates/keynote-2/` succeeds - [x] `pnpm run bench --help` / `pnpm run demo --help` render with valid-connec
1 parent 6d7b3b5 commit 7143ed9

File tree

5 files changed

+28
-48
lines changed

5 files changed

+28
-48
lines changed

templates/keynote-2/.env.example

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
11
# ===== Runtime toggles =====
2-
USE_DOCKER=1 # 1 = run docker compose up for pg/crdb; 0 = skip
2+
#USE_DOCKER=1 # 1 = run docker compose up for pg/crdb; 0 = skip
33
#SKIP_PG=1 # 1 = don't init Postgres in prep
44
#SKIP_CRDB=1 # 1 = don't init Cockroach in prep
55
#SKIP_SQLITE=1 # 1 = don't init SQLite in prep
66
#SKIP_SUPABASE=1 # 1 = don't init Supabase in prep
7-
SKIP_CONVEX=1 # 1 = don't init Convex in prep
7+
#SKIP_CONVEX=1 # 1 = don't init Convex in prep
88
SPACETIME_METRICS_ENDPOINT=0
99

1010
# ===== PostgreSQL =====
@@ -40,7 +40,7 @@ SUPABASE_DB_URL=postgresql://postgres:postgres@127.0.0.1:54322/postgres
4040
CONVEX_URL=http://127.0.0.1:3210
4141
CONVEX_SITE_URL=http://127.0.0.1:3210
4242
CLEAR_CONVEX_ON_PREP=0
43-
CONVEX_USE_SHARDED_COUNTER=1
43+
#CONVEX_USE_SHARDED_COUNTER=1
4444

4545
# ===== Bun =====
4646
BUN_URL=http://127.0.0.1:4001

templates/keynote-2/DEVELOP.md

Lines changed: 17 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ Each run:
1515
Run a quick performance comparison:
1616

1717
```bash
18-
npm run demo
18+
pnpm run demo
1919
```
2020

2121
The script will:
@@ -44,6 +44,7 @@ The script will:
4444

4545
- **Node.js** ≥ 22.x
4646
- **pnpm** installed globally
47+
- **Rust** (required for SpacetimeDB benchmarks) -- [install](https://rust-lang.org/tools/install/)
4748
- **Docker** for local Postgres / Cockroach / Supabase
4849
- Local/Cloud Convex
4950

@@ -160,42 +161,42 @@ cd ..
160161
1. Start SpacetimeDB (`cargo run -p spacetimedb-cli -- start` or `spacetime start`)
161162
2. Start Convex (inside convex-app run `npx convex dev`)
162163
3. Init Supabase (run `supabase init`) inside project root.
163-
4. `npm run prep` to seed the databases.
164-
5. `npm run bench` to run the test against all connectors.
164+
4. `pnpm run prep` to seed the databases.
165+
5. `pnpm run bench` to run the test against all connectors.
165166

166167
## Commands & Examples
167168

168169
### 1. Run a test
169170

170171
```bash
171-
npm run bench -- [test-name] [--seconds N] [--concurrency N] [--alpha A] [--connectors list] [--stdb-compression none|gzip]
172+
pnpm run bench [test-name] [--seconds N] [--concurrency N] [--alpha A] [--connectors list] [--stdb-compression none|gzip]
172173
```
173174

174175
Examples:
175176

176177
```bash
177178
# Default test (test-1), default args
178-
npm run bench
179+
pnpm run bench
179180

180181
# Explicit test name
181-
npm run bench -- test-1
182+
pnpm run bench test-1
182183

183184
# Short run, 100 concurrent workers
184-
npm run bench -- test-1 --seconds 10 --concurrency 100
185+
pnpm run bench test-1 --seconds 10 --concurrency 100
185186

186187
# Heavier skew on hot accounts
187-
npm run bench -- test-1 --alpha 2.0
188+
pnpm run bench test-1 --alpha 2.0
188189

189190
# Enable gzip for the SpacetimeDB benchmark client
190-
npm run bench -- test-1 --connectors spacetimedb --stdb-compression gzip
191+
pnpm run bench test-1 --connectors spacetimedb --stdb-compression gzip
191192

192193
# Only run selected connectors
193-
npm run bench -- test-1 --connectors spacetimedb,sqlite_rpc
194+
pnpm run bench test-1 --connectors spacetimedb,sqlite_rpc
194195
```
195196

196197
### 2. Run the distributed TypeScript SpacetimeDB benchmark
197198

198-
Use this mode when you want to spread explicit TypeScript client connections across multiple machines. The existing `npm run bench` flow is still the single-process benchmark; the distributed flow is a separate coordinator + generator setup.
199+
Use this mode when you want to spread explicit TypeScript client connections across multiple machines. The existing `pnpm run bench` flow is still the single-process benchmark; the distributed flow is a separate coordinator + generator setup.
199200

200201
The commands below are written so they run unchanged on a single machine. For a true multi-machine run, replace `127.0.0.1` with the actual coordinator and server hostnames or IP addresses reachable from each generator machine.
201202

@@ -387,11 +388,11 @@ From `src/cli.ts`:
387388

388389
- **`--seconds N`**
389390
- Duration of the benchmark in seconds
390-
- Default: `1`
391+
- Default: `10`
391392

392393
- **`--concurrency N`**
393394
- Number of workers / in-flight operations
394-
- Default: `10`
395+
- Default: `50`
395396

396397
- **`--alpha A`**
397398
- Zipf α parameter for account selection (hot vs cold distribution)
@@ -407,6 +408,7 @@ From `src/cli.ts`:
407408

408409
- If omitted, all connectors for that test are run
409410
- The valid names come from `tc.system` in the test modules and the keys in `CONNECTORS`
411+
- Valid names: `convex`, `spacetimedb`, `bun`, `postgres_rpc`, `cockroach_rpc`, `sqlite_rpc`, `supabase_rpc`, `planetscale_pg_rpc`
410412

411413
- **`--contention-tests startAlpha endAlpha step concurrency`**
412414
- Runs a sweep over Zipf α values for a single connector
@@ -425,14 +427,10 @@ From `src/cli.ts`:
425427
You can also run the benchmark via Docker instead of Node directly:
426428

427429
```bash
428-
docker compose run --rm bench \
429-
--seconds 5 \
430-
--concurrency 50 \
431-
--alpha 1 \
432-
--connectors convex
430+
docker compose run --rm bench -- --seconds 5 --concurrency 50 --alpha 1 --connectors convex
433431
```
434432

435-
If using Docker, make sure to set `USE_DOCKER=1` in `.env`, verify docker-compose env variables, verify you've run supabase init, and run `npm run prep` before running bench.
433+
If using Docker, make sure to set `USE_DOCKER=1` in `.env`, verify docker-compose env variables, verify you've run supabase init, and run `pnpm run prep` before running bench.
436434
437435
## Output
438436

templates/keynote-2/LICENSE

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
../../licenses/apache2.txt

templates/keynote-2/README.md

Lines changed: 5 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -8,12 +8,12 @@ See SpacetimeDB's performance advantage with one command:
88

99
```bash
1010
pnpm install
11-
npm run demo
11+
pnpm run demo
1212
```
1313

1414
The demo compares SpacetimeDB and Convex by default, since both are easy for anyone to set up and run locally without additional infrastructure. Other systems (Postgres, CockroachDB, SQLite, etc.) are also supported but require more setup. The demo checks that required services are running (prompts you to start them if not), seeds databases, and displays animated results.
1515

16-
**Options:** `--systems a,b,c` | `--seconds N` | `--skip-prep` | `--no-animation`
16+
**Options:** `--systems a,b,c` | `--seconds N` | `--concurrency N` | `--alpha N` | `--skip-prep` | `--no-animation`
1717

1818
**Note:** `demo` always runs the built-in `test-1` scenario. Use `bench` if you need to specify a test name directly.
1919
**Note:** `demo` selects targets with `--systems`; `bench` filters test connectors with `--connectors`.
@@ -84,7 +84,7 @@ This is a classic read-modify-write workload that tests transactional integrity
8484
### Test Command
8585

8686
```bash
87-
docker compose run --rm bench --seconds 10 --concurrency 50 --alpha XX --connectors YY
87+
docker compose run --rm bench -- --seconds 10 --concurrency 50 --alpha XX --connectors YY
8888
```
8989

9090
- `--seconds 10`: Duration of benchmark run
@@ -97,7 +97,7 @@ docker compose run --rm bench --seconds 10 --concurrency 50 --alpha XX --connect
9797

9898
**Server Machine (Variant A - PhoenixNAP):**
9999

100-
- s3.c3.medium bare metal instance - Intel i9-14900k 24 cores (32 threads), 128GB DDR5 Memory OS: Ubuntu 24.04
100+
- s3.c3.medium bare metal instance - Intel i9-14900k 24 cores (32 threads), 128GB DDR5 Memory, OS: Ubuntu 24.04
101101

102102
**Server Machine (Variant B - Google Cloud):**
103103

@@ -174,28 +174,9 @@ PlanetScale results (~477 TPS) demonstrate the **significant impact of cloud dat
174174

175175
## Running the Benchmarks
176176

177-
See [DEVELOP.md](./DEVELOP.md) for detailed setup and execution instructions.
177+
See [DEVELOP.md](./DEVELOP.md) for prerequisites, configuration, and full CLI reference.
178178
The distributed TypeScript SpacetimeDB workflow is documented there as `Run the distributed TypeScript SpacetimeDB benchmark`.
179179

180-
### Quick Start
181-
182-
```bash
183-
# Install dependencies
184-
pnpm install
185-
186-
# Copy environment config
187-
cp .env.example .env
188-
189-
# Start services (PostgreSQL, CockroachDB, etc.)
190-
docker compose up -d pg crdb
191-
192-
# Seed databases
193-
npm run prep
194-
195-
# Run benchmark
196-
npm run bench -- test-1 --seconds 10 --concurrency 50 --alpha 1.5 --connectors spacetimedb,postgres_rpc,sqlite_rpc
197-
```
198-
199180
## Output
200181

201182
Benchmark results are written to `./runs/` as JSON files with TPS and latency statistics.

templates/keynote-2/src/opts.ts

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -306,7 +306,7 @@ export function parseDemoOptions(argv: string[] = process.argv): DemoOptions {
306306
return {
307307
...runtimeOptions,
308308
seconds: options.seconds ?? 10,
309-
concurrency: options.concurrency ?? 10,
309+
concurrency: options.concurrency ?? 50,
310310
alpha: options.alpha ?? 1.5,
311311
systems:
312312
options.systems ?? options.connectors ?? [...defaultDemoSystems],
@@ -378,7 +378,7 @@ export function parseBenchOptions(argv: string[] = process.argv): BenchOptions {
378378
return {
379379
...runtimeOptions,
380380
testName: args[0] ?? defaultBenchTestName,
381-
seconds: options.seconds ?? 1,
381+
seconds: options.seconds ?? 10,
382382
concurrency:
383383
contentionTests?.concurrency ?? options.concurrency ?? 50,
384384
alpha: concurrencyTests?.alpha ?? options.alpha ?? 1.5,

0 commit comments

Comments
 (0)