Add distributed typescript benchmark harness (#4698)

# Description of Changes

The current keynote-2 benchmarks pipelines operations via
`MAX_INFLIGHT_PER_WORKER` in order to simulate a large number of client
connections while running the benchmark locally or on a single machine.

This patch adds a distributed benchmark mode for `templates/keynote-2`
so explicit SpacetimeDB client connections can be spread across multiple
machines without changing the existing single-process benchmark flow.

This is a pure extension. `npm run bench` and the current
`src/core/runner.ts` path remain intact. The new distributed path adds a
small coordinator/generator/control-plane harness specifically for
multi-machine ts client runs.

- New CLI entry points `bench-dist-coordinator`, `bench-dist-generator`,
and `bench-dist-control` were added
- The coordinator defines the benchmark window
- Generators begin submitting requests during warmup, but warmup
transactions are excluded from TPS
- Throughput is measured from the server-side committed transfer
counter, not client-local TPS
- Each connection runs closed-loop with one request at a time in this
distributed mode
- Connection startup is bounded-parallel (`--open-parallelism`) to avoid
a connection storm
- Verification is run by the coordinator after the epoch
- Late generators can be registered after a run to increase load on the
server incrementally
- If a participating generator dies and never sends `/stopped`, the
epoch result is flagged with an error so the run can be retried cleanly

See `DEVELOP.md` for instructions on how to run.

# API and ABI breaking changes

N/A

# Expected complexity level and risk

3

# Testing

Manual

Author: joshua-spacetime

templates/keynote-2/DEVELOP.md

@@ -186,6 +186,185 @@ npm run bench test-1 --alpha 2.0
 npm run bench test-1 --connectors spacetimedb,sqlite
 ```
 
+### 2. Run the distributed TypeScript SpacetimeDB benchmark
+
+Use this mode when you want to spread explicit TypeScript client connections across multiple machines. The existing `npm run bench` flow is still the single-process benchmark; the distributed flow is a separate coordinator + generator setup.
+
+The commands below are written so they run unchanged on a single machine. For a true multi-machine run, replace `127.0.0.1` with the actual coordinator and server hostnames or IP addresses reachable from each generator machine.
+
+#### Machine roles
+
+- **Server machine**: runs SpacetimeDB and hosts the benchmarked module.
+- **Coordinator machine**: runs `bench-dist-coordinator` and `bench-dist-control`. It may also run one or more generators if you want.
+- **Generator machines**: run `bench-dist-generator`. You can run multiple generator processes on the same machine as long as each one has a unique `--id`.
+
+#### Distributed setup
+
+All coordinator and generator machines should use the same `templates/keynote-2` checkout and have dependencies installed:
+
+```bash
+cd templates/keynote-2
+pnpm install
+cp .env.example .env
+```
+
+Generate TypeScript bindings in that checkout on each machine that will run the coordinator or a generator:
+
+```bash
+spacetime generate --lang typescript --out-dir module_bindings --module-path ./spacetimedb
+```
+
+#### Step 1: Start the server
+
+On the **server machine**:
+
+```bash
+spacetime start
+```
+
+#### Step 2: Publish the module and seed the accounts
+
+On any machine with the repo checkout and CLI access to the server, publish the module and seed the database once before starting the distributed run:
+
+```bash
+cd templates/keynote-2
+
+export STDB_URL=ws://127.0.0.1:3000
+export STDB_MODULE=test-1
+export STDB_MODULE_PATH=./spacetimedb
+
+spacetime publish -c -y --server local --module-path "$STDB_MODULE_PATH" "$STDB_MODULE"
+spacetime call --server local "$STDB_MODULE" seed 100000 10000000
+```
+
+If you are using a named server instead of `local`, replace `--server local` with the correct server name.
+
+#### Step 3: Start the coordinator
+
+On the **coordinator machine**:
+
+```bash
+cd templates/keynote-2
+
+pnpm run bench-dist-coordinator -- \
+  --test test-1 \
+  --connector spacetimedb \
+  --warmup-seconds 15 \
+  --window-seconds 30 \
+  --verify 1 \
+  --stdb-url ws://127.0.0.1:3000 \
+  --stdb-module test-1 \
+  --bind 127.0.0.1 \
+  --port 8080
+```
+
+Notes:
+
+- `--warmup-seconds` is the unmeasured warmup period. Generators submit requests during warmup, but those transactions are excluded from TPS.
+- `--window-seconds` is the measured interval.
+- `--verify 1` preserves the existing benchmark semantics by running one verification pass centrally after the epoch completes.
+- The coordinator derives the HTTP metrics endpoint from `--stdb-url` by switching to `http://` or `https://` and appending `/v1/metrics`.
+- For a real multi-machine run, change `--bind 127.0.0.1` to `--bind 0.0.0.0` so remote generators can reach the coordinator.
+- For a real multi-machine run, set `--stdb-url` to the server machine's reachable address.
+
+#### Step 4: Start generators on one or more client machines
+
+On **generator machine 1**:
+
+```bash
+cd templates/keynote-2
+
+pnpm run bench-dist-generator -- \
+  --id gen-a \
+  --coordinator-url http://127.0.0.1:8080 \
+  --test test-1 \
+  --connector spacetimedb \
+  --concurrency 2500 \
+  --accounts 100000 \
+  --alpha 1.5 \
+  --open-parallelism 128 \
+  --control-retries 3 \
+  --stdb-url ws://127.0.0.1:3000 \
+  --stdb-module test-1
+```
+
+On **generator machine 2**:
+
+```bash
+cd templates/keynote-2
+
+pnpm run bench-dist-generator -- \
+  --id gen-b \
+  --coordinator-url http://127.0.0.1:8080 \
+  --test test-1 \
+  --connector spacetimedb \
+  --concurrency 2500 \
+  --accounts 100000 \
+  --alpha 1.5 \
+  --open-parallelism 128 \
+  --control-retries 3 \
+  --stdb-url ws://127.0.0.1:3000 \
+  --stdb-module test-1
+```
+
+Repeat that on as many generator machines as needed, adjusting `--id` and `--concurrency` for each process.
+For a real multi-machine run, replace `127.0.0.1` with the coordinator host in `--coordinator-url` and the SpacetimeDB server host in `--stdb-url`.
+
+`--open-parallelism` controls connection ramp-up only. It deliberately avoids a connection storm by opening connections in bounded parallel batches.
+`--control-retries` sets the retry cap for `register`, `ready`, `/state`, and `/stopped`. The default is `3`.
+
+#### Step 5: Confirm generators are ready
+
+On the **coordinator machine**:
+
+```bash
+cd templates/keynote-2
+
+pnpm run bench-dist-control -- status --coordinator-url http://127.0.0.1:8080
+```
+
+Wait until each generator shows `state=ready` and `opened=N/N`.
+
+#### Step 6: Start an epoch
+
+On the **coordinator machine**:
+
+```bash
+cd templates/keynote-2
+
+pnpm run bench-dist-control -- start-epoch --coordinator-url http://127.0.0.1:8080 --label run-1
+```
+
+`start-epoch` waits for the epoch to finish, then prints the final result.
+
+#### Step 7: Check results
+
+Each completed epoch writes one JSON result file on the coordinator machine under:
+
+```text
+templates/keynote-2/runs/distributed/
+```
+
+The result contains:
+
+- participating generator IDs
+- total participating connections
+- committed transaction delta from the server metrics endpoint
+- measured window duration
+- computed TPS
+- verification result
+
+#### Operational notes
+
+- Start the coordinator before the generators.
+- Generators begin submitting requests when the coordinator enters `warmup`, not when the measured window begins.
+- Throughput is measured only from the committed transaction counter delta recorded after warmup, so warmup transactions are excluded.
+- For this distributed TypeScript mode, each connection runs closed-loop with one request at a time. There is no pipelining in this flow.
+- Late generators are allowed to register and become ready while an epoch is already running, but they only participate in the next epoch.
+- The coordinator does not use heartbeats. It includes generators that most recently reported `ready`.
+- If a participating generator dies and never sends `/stopped`, the epoch result is written with an `error`, and that generator remains `running` in coordinator status until you restart it and let it register again.
+- You can run multiple generator processes on the same machine if you want to test the harness locally. Just make sure each process uses a unique `--id`.
+
 ---
 
 ## CLI Arguments

templates/keynote-2/README.md

@@ -180,6 +180,7 @@ for the other backends/databases as they maxed out before the client.
 ## Running the Benchmarks
 
 See [DEVELOP.md](./DEVELOP.md) for detailed setup and execution instructions.
+The distributed TypeScript SpacetimeDB workflow is documented there as `Run the distributed TypeScript SpacetimeDB benchmark`.
 
 ### Quick Start
 

templates/keynote-2/package.json

@@ -11,7 +11,10 @@
     "prep": "tsx src/init/init-all.ts",
     "down": "docker compose down || exit 0",
     "test-1": "tsx src/cli.ts test-1",
-    "bench": "tsx src/cli.ts"
+    "bench": "tsx src/cli.ts",
+    "bench-dist-coordinator": "tsx src/distributed/coordinator.ts",
+    "bench-dist-generator": "tsx src/distributed/generator.ts",
+    "bench-dist-control": "tsx src/distributed/control.ts"
   },
   "devDependencies": {
     "@types/better-sqlite3": "^7.6.13",

templates/keynote-2/src/connectors/index.ts

@@ -7,6 +7,7 @@ import sqlite_rpc from './rpc/sqlite_rpc.ts';
 import supabase_rpc from './rpc/supabase_rpc.ts';
 import planetscale_pg_rpc from './rpc/planetscale_pg_rpc.ts';
 import { ConnectorKey } from '../opts.ts';
+export type { ConnectorKey } from '../opts.ts';
 
 export const CONNECTORS = {
   convex,

templates/keynote-2/src/connectors/spacetimedb.ts

@@ -6,6 +6,7 @@ import {
   stdbModule,
   stdbUrl,
 } from '../opts';
+import { deriveWebsocketUrl } from '../core/stdbUrl';
 
 export function spacetimedb(
   url = stdbUrl,
@@ -31,7 +32,7 @@ export function spacetimedb(
     if (subscriptions.length === 0) subscribed.resolve();
 
     const builder = Db.builder()
-      .withUri('ws://' + url)
+      .withUri(deriveWebsocketUrl(url))
       .withDatabaseName(moduleName)
       .withConfirmedReads(stdbConfirmedReads)
       .onConnect((ctx) => {

templates/keynote-2/src/core/spacetimeMetrics.ts

@@ -1,9 +1,26 @@
 import { stdbUrl } from '../opts';
+import { deriveMetricsUrl } from './stdbUrl';
 
 type LabelFilter = Record<string, string>;
 
+function formatErrorWithCause(err: unknown): string {
+  if (!(err instanceof Error)) {
+    return String(err);
+  }
+
+  const cause =
+    'cause' in err && err.cause != null ? `; cause: ${String(err.cause)}` : '';
+  return `${err.message}${cause}`;
+}
+
 export async function fetchMetrics(url: string): Promise<string> {
-  const res = await fetch(url);
+  let res: Response;
+  try {
+    res = await fetch(url);
+  } catch (err) {
+    throw new Error(`metrics GET ${url} failed: ${formatErrorWithCause(err)}`);
+  }
+
   if (!res.ok) {
     throw new Error(
       `metrics GET ${url} failed: ${res.status} ${res.statusText}`,
@@ -63,8 +80,10 @@ export function parseMetricCounter(
   return null;
 }
 
-export async function getSpacetimeCommittedTransfers(): Promise<bigint | null> {
-  const url = `http://${stdbUrl}/v1/metrics`;
+export async function getSpacetimeCommittedTransfers(
+  rawStdbUrl = stdbUrl,
+): Promise<bigint | null> {
+  const url = deriveMetricsUrl(rawStdbUrl);
 
   const labels: LabelFilter = {
     committed: 'true',

templates/keynote-2/src/core/stdbUrl.ts

@@ -0,0 +1,33 @@
+function parseStdbUrl(rawUrl: string): URL {
+  const trimmed = rawUrl.trim();
+  if (!trimmed) {
+    throw new Error('STDB_URL not set');
+  }
+
+  const withScheme = /^[a-z]+:\/\//i.test(trimmed)
+    ? trimmed
+    : `ws://${trimmed}`;
+  return new URL(withScheme);
+}
+
+export function normalizeStdbUrl(rawUrl: string): string {
+  return parseStdbUrl(rawUrl).host;
+}
+
+export function deriveMetricsUrl(rawUrl: string): string {
+  const url = parseStdbUrl(rawUrl);
+  url.protocol = 'http:';
+  url.pathname = '/v1/metrics';
+  url.search = '';
+  url.hash = '';
+  return url.toString();
+}
+
+export function deriveWebsocketUrl(rawUrl: string): string {
+  const url = parseStdbUrl(rawUrl);
+  url.protocol = 'ws:';
+  url.pathname = '';
+  url.search = '';
+  url.hash = '';
+  return url.toString();
+}