feat(bench): renderer-pinned samples, topic-coverage gate, error-recovery quizzes; trim skillgym to agentic checks

[thymikee]

Summary

Follow-up to #1404: makes the help conformance bench the single command-planning benchmark and trims SkillGym to the checks only an agentic runner can provide.

• Renderer-pinned samples. All CLI output quoted in benchmark prompts moves to scripts/help-conformance-sample-outputs.mjs, and scripts/__tests__/help-conformance-sample-outputs.test.ts rebuilds every sample through the real production code: the settle output formatters (including the ADR 0014 pinned tail refs the old samples were missing), printHumanError, formatSnapshotText with the real private-ax quality warning, and the actual refMutationAdmissionResponse rejection. A rendering or message change now fails a deterministic unit test instead of leaving the bench grading models against output the CLI no longer prints.
• Fabricated sample retired. The sample-output-recoverable-failure-retries-in-session case quoted an error COMMAND_FAILED: / retriable: / details: envelope the CLI never renders — a production settle timeout is a success with not settled after Nms. It is replaced by a real DEVICE_IN_USE recovery quiz whose message and recovery hint come from session-open.ts and buildSessionRecoveryHint.
• Topic-coverage gate. Cases move to scripts/help-conformance-cases.mjs; helpTopicIds() is exported from cli-help.ts, and scripts/__tests__/help-conformance-topic-coverage.test.ts fails when a help topic has neither a bench case nor an explicit, reasoned waiver (waived: cdp, macos, maestro, physical-device, react-devtools, remote). Stale or already-covered waivers also fail.
• New case families (18 cases total): error-recovery quizzes with production-true text (device-in-use → reuse the owning session; stale pinned ref → fresh snapshot -i; ambiguous find → re-observe rather than guess unseen refs; app-not-installed → apps discovery, no invented installs), topic-keyed planning cases for tv, web, react-native, debugging, and workflow, and a metamorphic twin of the settled-diff quiz to resist overfitting the help text to the benchmark.
• SkillGym trimmed 119 → 5 cases (suite: 2,778 → ~450 lines). Kept: the routing smoke plus the four output-interpretation cases that require a proven local CLI help probe (local-cli-help-policy). Their embedded samples import the same pinned constants through a small .d.mts, replacing hand-transcribed output that had already drifted from the renderer. README and docs/agents/testing.md now route command-planning knowledge checks to the bench and live fixture behavior to the iOS simulator e2e suite (test(ios): expand simulator e2e coverage #1408).

Validation

• pnpm format:check, pnpm typecheck, pnpm lint, pnpm check:layering, pnpm check:fallow, pnpm check:production-exports all green.
• Deterministic gates: 56/56 across scripts/__tests__, test/skillgym/suites/local-cli-help-policy.test.ts, and src/cli/parser/__tests__/cli-help-topics.test.ts.
• Full unit-core run: 499 files passed; 4 timeout-shaped failures in the documented contention-flaky files (request-router-open, runner-client, plus the subprocess-spawning bench test under full-suite load) — all 92 tests in those files pass in isolation.
• Bench dry run builds prompts for all 18 cases × 2 default runners; the parser-backed validator accepts the new cases' grammar.
• Known boundary: the new cases' model-side pass rates are not yet sampled — run pnpm bench:help-conformance --repeat 3 (paid LLM calls, local) before treating them as regression signal; the tv/web matchers are the most likely to need loosening.

[github-actions]

PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-07-27 12:24 UTC

[github-actions]

Size Report

Metric	Base	Current	Diff
JS raw	1.85 MB	1.85 MB	+120 B
JS gzip	592.7 kB	592.8 kB	+63 B
npm tarball	706.9 kB	707.1 kB	+112 B
npm unpacked	2.47 MB	2.47 MB	+510 B

Startup median (7 runs, lower is better):

Scenario	Base	Current	Diff
CLI --version	28.6 ms	26.2 ms	-2.4 ms
CLI --help	60.2 ms	55.5 ms	-4.7 ms

Top changed chunks:

Chunk	Raw diff	Gzip diff
dist/src/session.js	+32 B	+20 B

[thymikee]

Reviewed exact head f1bd3733. The SkillGym reduction is intentional and the topic registry/waiver checks are coherent, but two P2 gaps remain:

• Three supposedly renderer-pinned error samples still hand-author producer messages for DEVICE_IN_USE, AMBIGUOUS_MATCH, and APP_NOT_INSTALLED before calling printHumanError. Changes in the real session-open/find/app-resolution producers can therefore drift while these tests stay green. Exercise the actual producer path or extract/use a shared error factory.
• The retained open-and-snapshot routing smoke is documented as proving “skill plus local help,” but it sets neither requireLocalCliHelp nor allowOnlyLocalCliHelpCommands; skill telemetry is also conditional. It can pass entirely from the supplied contract/model prior. Add the observed local-help requirements or narrow the claim.

Keep ready-for-human withheld until addressed.

[thymikee]

Both P2s addressed at ec31963:

• Hand-authored producer messages: the three error samples now render through the actual producers — buildDeviceInUseBySessionError (extracted in session-open.ts, called by the handler), buildAmbiguousMatchError (exported from find.ts), buildAppNotInstalledError (extracted in app-resolution.ts, thrown by the resolver). The wiring is itself gate-enforced: each factory is exported from its producer file and called by the production path, so a handler that stops calling its factory leaves a test-only export and fails check:production-exports — the factory can't quietly stop being the real producer.
• open-and-snapshot: now sets requireLocalCliHelp + allowOnlyLocalCliHelpCommands, so the "skill plus local help" claim is observed (local-cli-help-policy over command events), not assumed.

Gates: format/typecheck/lint green; scripts/tests + local-cli-help-policy 37/37; find/session-open/app-resolution suites 53/53; fallow audit + production-exports clean.

🤖 Addressed by Claude Code

[thymikee]

Re-reviewed exact head ec31963a. Clean: each error sample now drives the actual production producer before CLI rendering; the exported factories remain production-used and therefore gate-visible; and the retained routing smoke now requires a real local CLI-help probe while rejecting wrapper-only or mixed runtime-command bypasses. Regressions are non-vacuous, scope stays within the six-file benchmark/routing follow-up, and all completed checks are green. Pending iOS smoke is not a code-review blocker.