From 7ce0b991c6cbea7145727326a77e6d00d1efa6bc Mon Sep 17 00:00:00 2001 From: Drew Stone Date: Wed, 10 Jun 2026 19:24:55 -0600 Subject: [PATCH] docs(bench): HARNESS.md catches up to the suite's current surface The map was a day behind the code (the anti-staleness law): non-inferiority gate mode + funnel-alignment law + endurance envs (CHECKPOINT/GYM_RECREATE) + waterfall/anytime observability in the header; ablation-grid, steering-modes, and the model-matrix runner join the entry-point table; the powered-tournament queue item marked RAN with a ledger pointer. --- bench/HARNESS.md | 30 ++++++++++++++++++++---------- bench/run-model-matrix.sh | 29 +++++++++++++++++++++++++++++ 2 files changed, 49 insertions(+), 10 deletions(-) create mode 100755 bench/run-model-matrix.sh diff --git a/bench/HARNESS.md b/bench/HARNESS.md index 49e754c5..5f43d7c1 100644 --- a/bench/HARNESS.md +++ b/bench/HARNESS.md @@ -69,15 +69,22 @@ and offline (zero new rollouts, zero judge calls). ## THE CANONICAL SUITE (2026-06-10) — the published path; start here -The optimization layer now ships from the package (`PR #209–#211`); bench scripts compose it. -A domain = an `Environment` (5 hooks); a strategy = how budget is spent to beat its check; -`runBenchmark` returns per-strategy means + the per-task LOSSES table + the (score,$) Pareto -frontier + paired-bootstrap refine-vs-sample. Promotion is the package gate -(`promotionGate` — seeded paired bootstrap via agent-eval `heldoutSignificance`, minimum -6 paired tasks, CI lower bound must clear zero); the authoring is the package primitive -(`authorStrategy`, with a named `fallbackModel` retry). Models policy: cheap router models -only (worker + author default `deepseek-v4-pro`; `moonshotai/kimi-k2.6` is selectable but -needs `maxTokens` and times out on long authoring prompts) — never CC models. +The optimization layer ships from the package; bench scripts compose it. A domain = an +`Environment` (5 hooks); a strategy = how budget is spent to beat its check; `runBenchmark` +returns per-strategy means + the per-task LOSSES table + the (score,$) Pareto frontier. +Promotion is the package gate (`promotionGate` — seeded paired bootstrap, evidence floor, +two modes: **superiority** and **non-inferiority** = score CI low > −tolerance AND cost +savings CI low > 0, the "same quality, cheaper" gate; verdicts carry paired Δlatency). +Authoring is `authorStrategy` (named `fallbackModel` retry). **Funnel-alignment law**: the +search-side champion tie-band must be no stricter than the gate's tolerance (under +`OBJECTIVE=cost` it defaults to it). Endurance envs on the evolve runner: `CHECKPOINT=path` +(phase ledger + resume — a killed run re-pays ONE phase), `GYM_RECREATE='docker …'` +(recreate the container at phase boundaries — the wedge killer). Observability: +`createWaterfallCollector` (every spawn billed+timed) + `anytimeReport` (TTT / shots-to- +target / COCO ERT / hill-climb AUC per satisficing target). Models policy: cheap router +models only (defaults `deepseek-v4-pro`/`deepseek-v4-flash`; compressor = flash with +`gpt-4o-mini` fallback) — never CC models; every verdict banner + artifact is +self-describing (models + config). | entry point | what it answers | one-liner | |---|---|---| @@ -89,6 +96,9 @@ needs `maxTokens` and times out on long authoring prompts) — never CC models. | `src/eops-corpus-ab.mts` | the across-run flywheel A/B (naive priming = **−11.6pp NEGATIVE**; `PRIME_MODE=relevance` = the unrun surviving design) | `N=16 HOLDOUT=4 PRIME_MODE=relevance tsx src/eops-corpus-ab.mts` | | `src/commit0-env-run.mts` | the HARD domain (implement whole libraries vs their test suites) through `runBenchmark` | `IDS=commit-0/wcwidth BUDGET=3 INNER_TURNS=10 tsx src/commit0-env-run.mts` | | `src/prompt-compression-gate.mts` | **THE COMPRESSION GATE** — does prompt compression preserve quality while cutting real cost? (nth-char/word-drop baselines vs LLM compression, each gated non-inferiority vs the original) | `N=12 BUDGET=2 tsx src/prompt-compression-gate.mts` | +| `src/ablation-grid.mts` | **THE FACTOR GRID** — σ steering × α evolve × γ prompt-artifact × κ compression as named cells on identical tasks, each gated vs `base` (κ cells non-inferiority); machinery bill + break-even printed | `ENV=aime\|math\|eops CELLS=base,steer,compress,steer+compress N=20 HOLDOUT=16 tsx src/ablation-grid.mts` | +| `src/steering-modes.mts` | **THE STEERING HYPERCUBE** — sample / refine / structural (deterministic floor) / contrastive / belief arms, paired vs refine, with waterfalls + anytime tables | `N=16 OFFSET=40 BUDGET=3 tsx src/steering-modes.mts` | +| `run-model-matrix.sh` | **THE MODEL MATRIX** — the 4-cell grid per worker model (deepseek ×2, kimi, glm, gpt-5.4-mini/5.5, haiku, sonnet), sequential, one artifact per model | `bash run-model-matrix.sh` | | `src/examples/strategy-demo.mts` | the 3-layer API demo (gym-free) | `WORKER_MODEL=gpt-4o-mini tsx src/examples/strategy-demo.mts` | | `src/examples/math-demo.mts` | any-domain proof: math via `createVerifierEnvironment` (the tax/legal/gtm answer-shape) | `BUDGET=3 tsx src/examples/math-demo.mts` | @@ -102,7 +112,7 @@ is firewalled (trace-only), costs are real (router usage → `{usd, ms, tokens}` ### The QUEUED runs for the test fleet (wired, one command each, unrun) 1. **Relevance-primed corpus A/B** — `PRIME_MODE=relevance K_FACTS=2 N=16 HOLDOUT=4` (the read-side design that survived the naive-priming negative). -2. **Strategy tournament at power** — flywheel-run with `N=24 HOLDOUT=12` (or runBenchmark with all four built-ins), the within-run topology verdict. +2. ~~Strategy tournament at power~~ — RAN (n=24, budget 4, ×3 configs): HOLD verdicts; the cost-frontier finding ×3 + the funnel-alignment law came out of these. Live ledger: `.evolve/current.json` + the findings gist. 3. **Commit0 at real budget** — `BUDGET=3 INNER_TURNS=12 N=3` sample-vs-refine on the hard domain. 4. **Cross-domain replication** — blocked on sourcing the csm/hr gym containers (`EOPS_SPLIT` is wired). diff --git a/bench/run-model-matrix.sh b/bench/run-model-matrix.sh new file mode 100755 index 00000000..275558e2 --- /dev/null +++ b/bench/run-model-matrix.sh @@ -0,0 +1,29 @@ +#!/usr/bin/env bash +# MODEL × CELL matrix: the 4-cell AIME grid per worker model, sequential (one model at +# a time — keeps router load comparable across models for the latency columns). +# κ treatment: compressor fixed to deepseek-v4-flash (gpt-4o-mini fallback) for every +# model, so the compression operator is constant; each run re-compresses (temp 0.2) — +# near-identical, noted as a caveat for cross-model κ deltas. +set -uo pipefail +cd "$(dirname "$0")" +MODELS=( + deepseek-v4-flash + deepseek-v4-pro + moonshotai/kimi-k2.6 + zai/glm-4.7 + gpt-5.4-mini + gpt-5.5 + claude-haiku-4-5 + claude-sonnet-4-6 +) +for MODEL in "${MODELS[@]}"; do + SAFE=$(echo "$MODEL" | tr '/.' '--') + echo "=== matrix: $MODEL ===" + ENV=aime CELLS=base,steer,compress,steer+compress \ + N=20 HOLDOUT=12 HOLDOUT_OFFSET=4 BUDGET=3 INNER_TURNS=2 CONCURRENCY=3 \ + KAPPA=llm-50 WORKER_MODEL="$MODEL" \ + OUT="/tmp/matrix-$SAFE.json" \ + npx tsx src/ablation-grid.mts 2>&1 | tail -12 + echo "=== done: $MODEL ===" +done +echo "MATRIX COMPLETE"