Spike harness for baseline-example anchoring (T0.5, part 1 of #28) by GustavoSena · Pull Request #40 · subvisual/sluice

GustavoSena · 2026-07-26T02:12:43Z

Part of #28 — its first acceptance criterion (the script + fixed request set). The other two (the comparison table, the go/no-go) need a funded Galileo key, so #28 stays open; this makes producing the numbers one command instead of an afternoon.

What it does

Two arms over 24 requests (6 promptings × single/two-token budgets × maxStrategies 1 and 3 — the mix #28 specified):

control — the production prompt, exactly as buildComposeMessages emits it
baseline — the same prompt with a worked, valid baseline object appended

The baseline is generated from the live request via templateFallback, never a static example: a fixed blob would hand the model stale constants to echo, which is precisely the chainId/deadline fabrication compose.ts already had to fix once.

The arms belong to the experiment, not to the shipped prompt, so the spike appends the baseline itself and re-implements the reject-and-re-infer loop in miniature (~20 lines). No flag, no env sniffing, nothing spike-shaped left in compose.ts.

One metric beyond what #28 asked for

Echo fidelity — does the model actually use the pairing amounts and band tiers we precompute, or quietly re-derive them? That is the assumption every Tier 0 change rests on (#24, #25, #26) and it has never been measured against a live 7B. Arguably worth more than the baseline question this issue was opened for. Measured generously on purpose: a model echoing two of three legs is a different animal from one ignoring the block.

Verification, given the real thing can't run yet

--dry-run answers from a script keyed per run — malformed-then-good, verbatim parrot, over-budget-then-good, clean first try — so every path the report measures is exercised with no key and no network:

| metric | control | baseline |
| --- | --- | --- |
| accepted | 100% (24/24) | 100% (24/24) |
| firstTry | 50% | 50% |
| malformed | 17% of attempts | 17% of attempts |
| parrotRate | 0% (0/24) | 25% (6/24) |
| echoAmounts | 50% (6/12) | 100% (12/12) |
| avgPromptTokensEst | 1781 | 1996 |

Parrots are detected in the baseline arm and correctly absent from control (nothing to parrot). These are canned numbers — they say the instrument works, nothing about the model.

Writing the dry run earned its keep by catching two harness bugs that would otherwise have surfaced as findings: the parrot response was sliced from "schema" rather than the opening brace, so it arrived as a fragment and scored as malformed; and a global call counter made a first-attempt success impossible, pinning firstTry and parrotRate to zero forever.

The gate (≥30% relative drop in malformed+fallback and parrot rate <1/3) is evaluated by the script rather than eyeballed by whoever hoped the idea would work.

Running it

npx tsx spike/baseline-anchoring.ts --dry-run

Live, once a funded key exists — up to 96 inferences, so --limit first:

ZG_PRIVATE_KEY=0x... npx tsx spike/baseline-anchoring.ts --limit 4

Fund the ledger out-of-band (npm run fund); this script never funds.

Notes

Runs against stubContext() deliberately — the experiment varies the prompt, so the market and book must not move underneath the arms.
Token counts are chars/4 estimates, labelled as such in the output: comparable between arms, not exact. inferChat drops the provider's usage field; surfacing it would be a production change this spike doesn't need.
SDK 169/169, typecheck clean. No production files touched.

🤖 Generated with Claude Code

Acceptance criterion 1 of #28: the script and its fixed request set. The numbers still need a funded Galileo key — this makes producing them one command rather than an afternoon. Two arms over 24 requests (6 promptings x single/two-token budgets x maxStrategies 1 and 3): the production prompt, and the same prompt with a worked baseline object appended. The baseline is built from the live request via templateFallback, never a static example — a fixed blob would hand the model stale constants to echo, which is the fabrication bug compose.ts already had to fix once. The arms belong to the experiment, so the spike appends the baseline itself and re-implements the reject-and-re-infer loop in miniature rather than adding a flag to the shipped prompt builder. Beyond the metrics #28 asked for, it measures ECHO FIDELITY: whether the model actually uses the amounts and band tiers we precompute for it. That is the assumption all of Tier 0 rests on and it has never been measured against a live 7B. --dry-run answers from a script keyed per run (malformed-then-good, verbatim parrot, over-budget-then-good, clean first try) so every path the report measures is exercised without a key. Writing it caught two harness bugs: the parrot response was sliced from "schema" rather than the opening brace, so it arrived as a fragment and read as malformed; and a global call counter made a first-attempt success impossible, leaving firstTry and parrotRate permanently zero. The gate is evaluated by the script, not eyeballed by whoever hoped the idea would work. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Two fixes found by the first live run against Galileo. dotenv: PR #30 moved `import "dotenv/config"` out of config.ts into each CLI entrypoint. The spike is an entrypoint and never imported it, so loadConfig could not see ZG_PRIVATE_KEY. The dry run could not catch this — it never calls loadConfig. Failure diagnostics: the report said 75% fell back and gave nobody a lead. It now prints the failure kind per run and a violation-code histogram per arm, plus one sample rejection text. That is what turned an opaque "100% malformed" into a diagnosis: sub-second empty responses, which is a drained compute ledger rejecting requests, not a model failing to comply. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot

Pull request overview

Adds a reproducible spike harness to measure whether appending a request-specific “baseline” (from templateFallback) improves model compliance versus increasing parroting, per Issue #28. This lives under spike/ and re-implements a minimal reject-and-re-infer loop so the experiment can vary prompt/loop behavior without touching production compose code.

Changes:

Introduces spike/baseline-anchoring.ts to run a fixed 24-request suite across two arms (control vs baseline) with metrics and a decision gate.
Adds a --dry-run mode with canned responses to exercise parsing/validation/parrot detection without a funded key or network.
Reports acceptance/fallback, malformed rate, parroting, echo-fidelity checks, latency, and estimated token costs.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+	while (attempts < MAX_COMPOSE_ATTEMPTS) {
+		const messages = messagesFor(arm, item.req, feedback);
+		promptChars = messages.reduce((n, m) => n + m.content.length, 0);
+		const raw = await infer(messages);
+		attempts += 1;
+		latencyMs += raw.latencyMs;
+		completionChars = raw.resultText.length;
+


+	// The gate, evaluated rather than eyeballed — the numbers decide, not the
+	// person who hoped the idea would work.
+	const badness = (rows_: RunResult[]) => {
+		const rs = rows_.filter((r) => r.arm !== undefined);
+		const attempts = rs.reduce((s, r) => s + r.attempts, 0);
+		const bad =
+			rs.reduce((s, r) => s + r.malformedAttempts, 0) +
+			rs.filter((r) => !r.accepted).length;
+		return attempts === 0 ? 0 : bad / attempts;
+	};
+	const c = badness(rows.filter((r) => r.arm === "control"));
+	const b = badness(rows.filter((r) => r.arm === "baseline"));
+	const drop = c === 0 ? 0 : (c - b) / c;
+	const acceptedBaseline = rows.filter((r) => r.arm === "baseline" && r.accepted);
+	const parrotRate =
+		acceptedBaseline.length === 0
+			? 0
+			: acceptedBaseline.filter((r) => r.parrot).length / acceptedBaseline.length;
+
+	console.log(
+		`\nGATE: malformed+fallback dropped ${Math.round(drop * 100)}% relative (bar: >= 30%), ` +
+			`parrot rate ${Math.round(parrotRate * 100)}% (bar: < 33%)`,
+	);


+	const dry = process.argv.includes("--dry-run");
+	const limitArg = process.argv.indexOf("--limit");
+	const limit = limitArg === -1 ? Infinity : Number(process.argv[limitArg + 1]);
+
+	const items = requestSet().slice(0, limit);


This was referenced Jul 26, 2026

Prompt T0.5 (spike) — baseline-example anchoring: measure compliance gain vs parroting #28

Open

Prompt building Tier 0 — echo-don't-compute improvements (tracking) #29

Open

josevazf requested a review from Copilot July 26, 2026 02:34

Copilot started reviewing on behalf of josevazf July 26, 2026 02:34 View session

Copilot AI reviewed Jul 26, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Spike harness for baseline-example anchoring (T0.5, part 1 of #28) - #40

Spike harness for baseline-example anchoring (T0.5, part 1 of #28)#40
GustavoSena wants to merge 2 commits into
mainfrom
gs/spike-baseline-anchoring-harness

GustavoSena commented Jul 26, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

GustavoSena commented Jul 26, 2026

What it does

One metric beyond what #28 asked for

Verification, given the real thing can't run yet

Running it

Notes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants