Skip to content

feat(setup-pstack): pin the trail reviewer and show its step - #295

Merged
mdsmithaustin merged 5 commits into
mainfrom
feat/trail-reviewer-pin
Oct 21, 2026
Merged

mdsmithaustin merged 5 commits into
mainfrom
feat/trail-reviewer-pin

Conversation

@mdsmithaustin

@mdsmithaustin mdsmithaustin commented Oct 21, 2026 •

Copy link
Copy Markdown
Owner

Why

Closes #294. On Codex, trail reviewer: gpt-6.1-sol@xhigh ran as gpt-6-astra@high whenever the work ran on sol, because one other role allows astra. Setup never said so, and the user could not turn the step off.

What changed

  • trail reviewer step: off pins the reviewer. A pinned reviewer that matches the work model keeps its written model and effort and reports "step": "same-model". A pin never labels an inherit-parent reviewer same-model, and it keeps the arm's earlier notes.
  • The step note names the roles that allow the target when one to three do. Setup steps 3c and 4 print the stepped reviewer next to the written one.
  • Rule 7 now says a stepped reviewer's effort comes from the work effort and the written effort is ignored.
  • Prose that said the reviewer always moves off the work model now says "unless pinned": skills/swarm/SKILL.md, skills/poteto-mode/playbooks/orchestrate.md, skills/pstack-harness/references/roles.md, the models-config Grammar bullet, and the PORTING.md keep-list. The matching promise-ledger claims and quote (evals/promises/ledger.json) and the oracle evidence string (evals/promises/oracles.py) change in the same commit.

Scope

Covers the resolver, the lint, setup prose, the prose sites above with their promise-ledger entries, and tests. Unpinned stepping and its effort rule stay as PR 150 set them. docs/guide is unchanged, because the guide promise ledger needs a checked promise for every new guide sentence. The setup preview for inherit-parent or auto work roles is filed as #296 and not touched here.

Tradeoffs

A setting line beats a marker on the reviewer entry. It fits the role: entry grammar and the layer order, and it leaves the (model, effort) tuples that the role contract and tests read alone. The note lists roles only when one to three allow the target, because a long list does not say which line to change.

Blast Radius

A step note now carries a role suffix, so a consumer that matches the exact note text changes (one orch recovery test updated). Configs without the new line resolve as before.

Verification

  • The new resolver and lint tests fail on main and pass here: test_check_models_config runs 203 tests, all OK. Round 2 added tests for the pinned inherit-parent reviewer, the kept notes, and the lint accepting the setting; each fails against a mutant of the resolver or lint.
  • On a copy of the real Codex config with work on gpt-6.1-sol@high, the reviewer was gpt-6-astra@high before. It is the same arm with (allowed by arena cross-judge pool) now. With trail reviewer: gpt-6.1-sol@high and trail reviewer step: off it resolves to gpt-6.1-sol@high, same-model.
  • At 44ef4b8, tools/gate-check.sh lint passes 9/9 and bun test passes 542. The test tier fails only on host-bound tests this diff does not touch (an rg path from the ChatGPT app). The no-mistakes run passed review, test, document, lint, and CI.

✅ Low: The resolver change is small and does what the intent asks: the pin follows layer order, the lint validates the setting, the roles that allow the stepped tier are counted by tier, and setting lines are excluded from ROLES and from _written_tiers. Tests run the CLI and the parser and check what they output; the only open item is a minor wording gap in the setup preview.

Updates from git push no-mistakes

The trail reviewer steps off the work model by rule 7, and a Codex user
who wrote sol got astra with no sign of it at setup. Setup now prints the
stepped reviewer beside the written one, the step note names the roles
that allow the target, and `trail reviewer step: off` keeps the written
model and effort on a match, labeled same-model. The rules now say a
stepped reviewer's effort follows the work effort.
@mdsmithaustin mdsmithaustin changed the title feat(setup-pstack): allow pinning the trail reviewer model and effort feat(setup-pstack): pin the trail reviewer and show its step Oct 21, 2026
@mdsmithaustin

Copy link
Copy Markdown
Owner Author

Verdict: FAIL
Head: fee15bd
Patch-id: 1125e3485169a89f1f53f0c7f175c58406aaa404
Docs: needs changes

Round 1, full swarm at the code-ready head: gates, live, regression against trunk (swarm workers, sonnet@high) and two audit lanes (trail reviewer, opus@xhigh, resolved with --work-model sonnet@high; three pipeline commits have an unknown author model, so those may be same-model).

Lane Result
Gates PASS. Lint 9/9; resolver suite 200; bun 541/0 fail; CI green. Local test-tier failures match trunk (host-only).
Live PASS. Issue config steps to astra with (allowed by arena cross-judge pool); trail reviewer step: off keeps sol at the written effort, labeled same-model; invalid values fail closed.
Regression vs trunk PASS. 3150 runs per tree; every arm's model, effort, source and step match; only the note suffix differs. Trunk has no pin (unknown role).
Audit: consumers ISSUES (low)
Audit: config safety ISSUES (low). 23k random configs match trunk once the suffix is stripped.

Sent back to the owner:

  1. Stale prose the pin makes false: skills/swarm/SKILL.md:26, skills/poteto-mode/playbooks/orchestrate.md:19 and :92, skills/setup-pstack/references/models-config.md:18, and the matching claims in evals/promises/ledger.json (around 3345 and 3459) still say the reviewer always moves off the work model when the config allows one.
  2. Tests: no test kills a mutant where the pin labels an inherit-parent reviewer same-model, or one where the pin drops the arm's earlier notes; test_check_models_config.py:1298 asserts parser storage, not output.

Follow-up: #296 (setup preview with inherit-parent/auto work roles; reached only by an invented config).

…its edges

The swarm, orchestrate, models-config, roles, and promise-ledger text said
the trail reviewer always moves off the work model when the config allows
another. Each now says so only while `trail reviewer step` is not `off`, and
the ledger claims and quote change with the prose.

New tests cover a pinned inherit-parent reviewer staying unlabeled and the
pin keeping the arm's earlier notes. The section test now asserts lint
output and the resolved step instead of parser storage.
@mdsmithaustin

Copy link
Copy Markdown
Owner Author

Verdict: PASS+NOTES
Head: 44ef4b8
Patch-id: 46962f1b41c687d9919fda0597539e8a23056683
Docs: pass

Round 2, single independent verifier (trail reviewer, opus@xhigh, cross-model to the sonnet@high owner). The diff since round 1 is a fix: 7 files, +54/-13, no resolver code.

  • Gates: lint incl. promise-ledger audit pass; setup-pstack suite 203/203; bun 0 failures; local test-tier failures match trunk e649e23.
  • Live: issue config still steps sol to astra with (allowed by arena cross-judge pool); trail reviewer step: off keeps gpt-6.1-sol@xhigh, labeled same-model; a pinned inherit-parent reviewer stays unlabeled on all four harnesses; the pin keeps earlier notes.
  • Round 1 finding 1 (stale prose): fixed; a repo-wide grep finds no unqualified claim left; ledger claims, quote and oracle evidence string match.
  • Round 1 finding 2 (tests): both new tests fail on the described mutants; the storage assertion now checks CLI output.

Note: the PR body still cites 174 tests and fee15bd and its Scope line omits the swarm, orchestrate, roles and ledger prose; the owner will refresh it (no head change).
Follow-up: #296 (setup preview with inheriting work roles, plus the by-model dedupe noted on the issue).

@mdsmithaustin
mdsmithaustin merged commit 6283892 into main Oct 21, 2026
7 checks passed
@mdsmithaustin
mdsmithaustin deleted the feat/trail-reviewer-pin branch October 21, 2026 05:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

models-config rule 7: trail reviewer step is invisible at setup and cannot be pinned

1 participant