test(review): add fanout-diversity fixtures that separate the arms

Parent: #7584

Problem

In #12151 both lens and model named every planted defect in every run (24/24 each), so the suite is at ceiling and cannot tell the arms apart: M minus L +0.01, 95% interval -0.03 to +0.05. The ground-truth axis also disagrees with where fanout files two defects: csv-handle-leak and search-sqli are labeled standards but land under ### Spec in most runs.

Done when

At least four new defect fixtures where a pilot shows at least one arm misses the planted defect in some runs (subtler defects, larger diffs, or defects spread across files).
The csv-handle-leak and search-sqli ground-truth axis values are checked against the commit-message claim rule in plugins/review/evals/fanout-diversity.md and corrected where wrong.
At least two fixtures plant two or more defects under one heading, so the per-heading positions can set feat(review): add a per-heading act-on cap to fanout's report #12503's act-on cap.
Decoy runs report IMPORTANT+ findings by label (false, already on main, preference, about the changed lines). Baseline from test(review): run the fanout reviewer-diversity eval and post its probe result #12151: L 9/12 runs and M 7/12 had one; 0 of 31 were false, 19 were preferences.
The L and M arms are rerun over the new fixtures and the result is posted on review: spec-first fanout and ratchet move to toolchain #7584.

Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.

Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none

I understand this as a test-fixture task under #7584. The #12151 suite is at ceiling (lens and model both 24/24), so it can't separate the arms. The work is in four parts:

Add at least four harder defect fixtures.
Re-check the axis labels for csv-handle-leak and search-sqli against the commit-message claim rule.
Add at least two fixtures with two or more defects under one heading.
Report decoy IMPORTANT+ findings by label, then rerun the L and M arms and post the result on review: spec-first fanout and ratchet move to toolchain #7584.

The done-when items are specific, so I'm asking nothing further. The pilot and rerun need model runs. #12503 is related, since the multi-defect fixtures feed its act-on cap. I found no duplicate.

_{Automated triage, applied by the lanes App.}

The triage sweep dispatched intake triage for this issue (attempt 1 of 2).

Claim. Backlog HITL host-run worker is taking this now.

docs(review): say the two-defect fixtures share an axis, not a heading

Claim. Backlog HITL host-run worker is taking this now.

Host run. A 3-run L and M pass of the fanout-diversity suite over main's fixtures: four decoys and the two two-defect fixtures from #12664. Three of the five done-when boxes are met: (b), (c) and (e). Boxes (a) and (d) are not.

What ran

claude plugin eval ran from an out-of-repo staged copy of review and multi-agent at origin/main, with these settings:
- Opus 5.5 as both subject and judge, --runs 3 --ablation none -j 2.
- --scaffold --allow-tools Bash Write Workflow, with the real node first on PATH.
- An isolated HOME and a short TMPDIR under ~/worktrees.
Each arm ran two suites. Decoys: typed-user, sort-column, page-links, return-promise. Two-defect: request-options, month-export.
Claude Code 2.1.317, plus 2.1.318 for one M decoy rerun.
Cost in USD (decoys + two-defect):
- L: 23.32 + 16.42.
- M: 13.94 + 8.82, plus 13.35 for the rerun.
Validity: run-validity.py --runs 3 --scaffolded on each result file, plus a count of tool calls in every trace.
- Bash ran in every run: 10 to 17 allowed Bash calls and one Workflow call per run.
- No trace shows a sandbox failure.
- arm-check passed in every run.

Suite	L	M
Two-defect	VALID (warnings: 2 denials, each in a run that scored the same as its case's denial-free runs; skill-fired not checked)	VALID (warning: skill-fired not checked)
Decoys	VALID (same two warnings)	INVALID in both passes

M decoys, first pass: INVALID.
- One return-promise-m run was denied a grep of its own session's workflow script, which is outside the plugin.
- The gate counts grep's pattern operand as a possible plugin path (run-validity.py:1333-1340), so the denial is a FAIL.
M decoys, rerun: INVALID.
- Two runs (sort-column-m r1, typed-user-m r1) were denied a Glob aimed at the staged review plugin directory.
- A third run was denied an ls of its own workflow directory.
- It was not retried.
So no M decoy number, and no L-against-M decoy comparison, is reported.

Against the done-when

(a) At least four new fixtures where an arm misses: NOT MET (0 of 4).
- The two new defect fixtures hit the ceiling. Both recall graders passed in all 6 runs of each arm (L 1.00, M 1.00), so these fixtures do not separate the arms.
- The decoys plant no defect, so they are not miss fixtures.
- test(review): add two-defect fanout-diversity fixtures and fix the search-sqli axis #12664's pilot dropped three single-defect candidates that also hit the ceiling.
(b) csv-handle-leak and search-sqli axis checked: MET.
- The rule: axis is spec when the change fails the commit message's own claim (plugins/review/evals/fanout-diversity.md:117).
- search-sqli is now spec. Its commit claims a "Read-only lookup over the catalogue", and the injection reads the users table.
- csv-handle-leak stays standards. Its claim, that a file with other columns reports an error, still holds.
(c) At least two fixtures with two or more defects under one heading: MET.
- Both fixtures put their two defects on one ground-truth axis: request-options on standards, month-export on spec.
- Positions within the ## By axis headings, from the VALID suites:
  - L kept both defects together in 6/6 runs, at positions 1 and 2 (month-export under Spec, request-options under Standards). This is also the first L measurement of the shipped request-options, which test(review): add two-defect fanout-diversity fixtures and fix the search-sqli axis #12664 left unmeasured.
  - M kept them together in 3/6 runs (month-export r2, request-options r2 and r3). The other three runs put each defect at chore: initialize marketplace scaffold #1 of a different heading.
  - An act-on cap of 2 keeps both defects in every run of both arms.
- Side finding: request-options carries an unplanted real regression, found in all 3 L runs and all 3 M runs of that fixture.
  - On the branch, { ...DEFAULTS, ...rest } lets an explicit timeoutMs: undefined replace the default; main's ?? kept it.
  - No caller passes undefined today.
  - Reviewers rated it CRITICAL in 2 L runs and 1 M run, and it always ranked after both planted defects.
(d) Decoy IMPORTANT+ findings by label: NOT MET.
- The decoy suite is VALID for L and INVALID for M in both passes.
- Decoy numbers are read only from runs VALID on both arms, so no labeled count is posted for either arm.
- The L reports have been labeled. That count is held back until M has a VALID decoy run to pair it with.
(e) L and M rerun over the new fixtures, posted on review: spec-first fanout and ratchet move to toolchain #7584: MET.
- The two-defect result is VALID on both arms.
- It is posted at review: spec-first fanout and ratchet move to toolchain #7584 (comment), with M's decoy INVALID lines in place of its numbers.

Missing, and the smallest next step

(a): four single-defect fixtures with much larger diffs (several files, the defect deep inside one hunk). Pilot them with --runs 1 on both arms and keep only those where an arm misses.
(d): one VALID M decoy run, then post both arms' labeled counts. Either stop the M arm's Glob of the plugin directory in the fanout sweep, or narrow the gate rule that counts a grep pattern as a plugin path (run-validity.py:1333-1340). Then rerun the M decoys only.

Not closable yet: (a) and (d) are unmet.

Target labels: add none; remove none.

Claim. Backlog HITL host-run worker is taking this now (round 2: harder fixtures, clean two-defect and decoy reruns).

Host run. Round 2 of #12505: a 1-run pilot of six new single-defect candidates on the L and M arms, and the clean 3-run L and M reruns of the two-defect fixtures and the four decoys that the #7584 correction asked for. No box changes state: (a) and (d) stay unmet, and (c) and (e) are not re-confirmed because the L arm's rerun is INVALID.

What ran

claude plugin eval from out-of-repo staged copies of review and multi-agent (reruns: origin/main f4b1e40; pilot: the round-2 branch head 46b8a6515d), Opus 5.5 subject and judge, --ablation none -j 2 --scaffold --allow-tools Bash Write Workflow, the real node first on PATH, isolated HOME, short TMPDIR under ~/worktrees, each suite in its own capped scope. Claude Code 2.1.318.
Reruns, --runs 3: two-defect (request-options, month-export) and decoys (typed-user, sort-column, page-links, return-promise) as separate suites per arm. Pilot, --runs 1: list-limit-null, sku-regex-state, bulk-ids, cache-key, shared-sort, rename-overdue.
Cost in USD, L / M: two-defect 15.69 / 9.16; decoys 23.36 / 14.00; pilot 16.21 / 9.28.
Wall time: two-defect L 20 min, M 14 min; decoys L 29 min, M 16 min; pilot L 19 min, M 10 min. L takes 1.4 to 1.9 times as long as M on the same cases.
Validity: the fixed run-validity.py from #13024.

Suite	L	M
Two-defect	INVALID (3 denials in 2 runs)	VALID (warning: skill-fired not checked)
Decoys	INVALID (1 denial)	INVALID (1 denial; a second only warned)
Pilot	INVALID (3 denials in 3 runs)	INVALID (3 denials in 3 runs)

Denials (the finding). M's Glob of the staged plugin directory for fanout-sweep* did not recur. Two other classes did, across the six suites:

A Bash read of the review plugin's own reference files (context/severity.md, skills/quality-gate/context/code-design.md, reference/findings-file-shape.md) by absolute staged path: L 4 runs, M 1. This is an in-plugin reach, and it FAILs.
An ls, grep or Grep of the session's own workflow script, workflow directory or journal under the run's config dir, outside the plugin: L 3 runs, M 4 (one only warned). These FAIL when the run scored differently from its case's denial-free runs, or had none to compare (every pilot run). That is the gate as designed.
rewrite-a-plugin.md lets a reader leave denied cases out only for a negative control, not for an arm comparison, so no run was dropped to make a suite VALID.

Against the done-when

(a) Four fixtures where an arm misses: NOT MET, and it looks unmeetable as written.
- Six candidates, each proven with node on the branch against main: a null a destructuring default does not replace (12 handlers), a g-flag regex .test keeping lastIndex, map(Number.parseInt), a cache key with no separator between user id and page, an in-place sort of an array an unchanged module returns by reference, and a < turned <= inside a 196-line rename.
- Recall 12/12: both arms named every planted defect at IMPORTANT or CRITICAL.
- With test(review): add two-defect fanout-diversity fixtures and fix the search-sqli axis #12664's pilot (3 single-defect candidates), that is two fixture iterations, 9 candidates, 18 arm-runs and 0 misses.
- Why: in 11 of the 12 pilot reports the reviewers say they reproduced or verified a finding by running the code. With Bash, a subtle defect gets executed, not just read.
- Criterion (a) was filed by the lane-3 agent session under the kyle-sexton login on 2026-11-02 at 20:43:38Z, as a test(review): run the fanout reviewer-diversity eval and post its probe result #12151 follow-up. The lane-3 hub outbox records filing "harder-fixture and cap sub-issues" at 20:4xZ and test(review): add fanout-diversity fixtures that separate the arms #12505 at 20:5xZ. It was not written by the owner in person.
- Fixture lesson: shared-sort's not_defects entry "categoryPage crashes on an empty category" is wrong. Both arms reproduced a real crash on an unknown category, so its false-positive grader failed in both runs. Fix the scaffold before reusing it.
- The candidates are commit 46b8a6515d on local branch host-run/12505-round2 (not pushed). No PR was opened, because none passes the keep rule.
(c) Two-defect positions: not re-confirmed. Recall was 1.00 in all 12 runs (L 6, M 6), but L is INVALID.
- Within-heading positions, all runs: L kept both defects under one heading in 4 of 6 runs (request-options 3/3, month-export 1/3), M in 2 of 6 (one run of each).
- Every planted defect sat at position 1 or 2 of its heading in every run, so a cap of 2 keeps both.
(d) Decoy IMPORTANT+ by label: NOT MET. Both arms are INVALID. The labels are done (0 false in either arm, no CRITICAL in any of 24 runs) and are held back until both arms have a VALID decoy run.
(e) Rerun posted on review: spec-first fanout and ratchet move to toolchain #7584: partly. The result is posted on review: spec-first fanout and ratchet move to toolchain #7584 with these INVALID verdicts, and no number from an INVALID suite is counted.

Remaining, smallest next step

Owner decision on (a):
- drop it;
- rewrite it for a setup where reviewers cannot run the fixture;
- or keep iterating fixtures.
- Recommendation: rewrite it. Basis: 0 misses in 18 arm-runs, and 11 of 12 pilot reports say they executed the code.
Clean L two-defect, L decoy and M decoy suites: the same flags, after a decision on whether the skill's reads of its own reference files by staged path should be allowed or avoided. That reach is in-plugin, so it is a skill or sandbox question, not a gate bug.
The pilot started while another session's eval suite was running: three suites on the host, at load 1.2 with 27 GB free.

Target labels: add none; remove none.

Decision recorded.
(agent; not an owner decision)

Criterion (a), "at least four new defect fixtures where a pilot shows at least one arm misses the planted defect", is retired.

Who set it: the lane-3 agent session filed test(review): add fanout-diversity fixtures that separate the arms #12505 on 2026-11-02 as a test(review): run the fanout reviewer-diversity eval and post its probe result #12151 follow-up. The owner did not write it, so an agent may amend it under the owner's 2026-11-02 direction.
Why retire it: two fixture iterations produced 9 single-defect candidates and 18 arm-runs, with 0 misses on either arm (test(review): add two-defect fanout-diversity fixtures and fix the search-sqli axis #12664's pilot plus round 2). In 11 of the 12 round-2 pilot reports, the reviewers say they ran the code to reproduce or verify the defect. With Bash available, subtle defects get executed rather than only read, so single-defect recall sits at ceiling in the setup fanout really ships in.
Why not rewrite it for a no-Bash setup: a no-Bash setup would measure a configuration fanout does not run in. Arm separation stays measured by (c), the per-heading positions, and (d), the decoy labels.
Basis: the round-2 host-run record on this issue (comment of 2026-11-04T03:14Z) and test(review): run the fanout reviewer-diversity eval and post its probe result #12151.

(a) is closed by this record. Still open: (d) and (e), which need VALID L and M decoy runs and a VALID L two-defect run. Before those reruns, decide whether fanout's Bash reads of its own reference files at the staged plugin path should be allowed or avoided.

Decision recorded.
(agent; not an owner decision)

Decision: the review plugin's reads of its own reference files are avoided in the plugin, not allowed in the eval setup. The review:code-reviewer agent now reads every ${CLAUDE_PLUGIN_ROOT} file it names with the Read tool, never through Bash. The change is one line in plugins/review/agents/code-reviewer.md plus a patch changelog fragment, in #13092.

Problem. In round 2, the L two-defect suite and both pilot suites were INVALID. Bash reads of context/severity.md and skills/quality-gate/context/code-design.md, by the staged plugin's absolute path, were denied, and the run-validity gate counts a denial aimed inside the plugin as a FAIL.

Basis

Who issued the reads. The kept round-2 L traces show the denied calls came from review:code-reviewer subagents dispatched inside the fanout-sweep workflow, not from the fanout skill itself.
- The denial record names decision_reason_type: subcommandResults.
- Both denied L two-defect commands were compound commands carrying a sed -n '/Bundled criteria/,…/p' of code-design.md.
Read worked and Bash did not. I tallied code-reviewer calls naming these files across the round-2 L and L-pilot sandboxes. My tally did not find the M-pilot denial that the round-2 record names, so the M side is not counted here.

Tool Calls Outcome

Read 43 no denial

Grep 6 no denial

Bash 13 6 ok, 3 errors, 4 denials

No main-session read of these files was denied in the 40 round-2 L and M sandboxes.
The eval setup has no lever that allows these reads.
- The kept run settings already list the plugin root under the sandbox's allowRead.
- The suites already pass --allow-tools Bash, and the reads were still denied.
- The plugin-evals page makes a grant the only way to widen what a run may do (Grant tools).
- Anything wider means changing permission config, which is the owner's call.
Real users. Read is the file-read tool, and it is the path that never hit a denial in the eval. Whether a real session prompts on these shell reads is UNVERIFIED.
The cause of the denial is UNVERIFIED. I did not trace why the permission layer denied these compound commands, while one main-session grep … severity.md | head passed. The fix does not depend on the cause.

Options

Option	Assessment
1. Allow the reads in the eval setup	No flag does this short of a permission change.
2. Use a plugin-root-relative path	A relative path in Bash resolves against the reviewed repository's working directory, not the plugin.
3. Use the `Read` tool	Chosen. One file changes, and it is the path that held in every eval run.

Left as designed: the second denial class is unchanged. It covers an ls, grep or Grep of the session's own workflow script or journal under the run's config dir, and that is the gate working as designed.

Host run. Round 3 of #12505: the 3-run L and M reruns of the two-defect fixtures and the four decoys, on the plugin head that merged as #13092 (review:code-reviewer reads its bundled files with Read, not Bash). Two of the four suites are VALID (L two-defect, M decoys), but neither fixture set has both arms VALID, so no arm comparison is counted and no box changes state. The #13092 instruction held: code-reviewer made no Bash read of a bundled file, and no Bash read of any bundled file was denied.

What ran

claude plugin eval from an out-of-repo staged copy of review and multi-agent built from fix(review): have code-reviewer read its bundled files with the Read tool #13092's head, content-identical over those plugins to the merged commit a0700ce. Opus 5.5 subject and judge, --ablation none -j 2 --runs 3 --scaffold --allow-tools Bash Write Workflow, the real node first on PATH, isolated HOME, short TMPDIR under ~/worktrees, each suite in its own capped scope. Claude Code 2.1.318.
Two chains in parallel from 06:05Z: L two-defect then L decoys (to 06:57Z); M decoys then M two-defect (to 06:36Z). Load at start 2.2.
Cost in USD, L / M: two-defect 16.21 / 9.02; decoys 23.68 / 14.46. Total 63.37.
Wall time: two-defect L 24 min, M 11 min; decoys L 28 min, M 20 min.
Validity: run-validity.py --runs 3 --scaffolded, origin/main's version (it includes #13024).

Suite	L	M
Two-defect	VALID (no denial; warning: skill-fired not checked)	INVALID (1 counted denial)
Decoys	INVALID (1 counted denial; 2 more only warned)	VALID (warnings: 2 denials in runs that scored like their case's denial-free runs; skill-fired not checked)

Denials

M two-defect, the FAIL: month-export-m run 2, a main-session Glob of the staged review plugin directory for the sweep workflow. This is an in-plugin reach. The same class made round 1's M decoy rerun INVALID. A second denial (request-options-m run 3, an ls of its own workflow directory) only warned.
L decoys, the FAIL: sort-column-l run 1, an ls of the session's own workflow directory under the config dir, outside the plugin. It counts because that run's false-positive grader failed, unlike its case's denial-free runs. Two return-promise-l denials only warned: a Grep of its own workflow script, and a main-session Read of a fixture file joined onto the plugin directory, a path that does not exist.
M decoys: two Greps of the session's own workflow script, both warnings.
No run was dropped and no verdict was recomputed on a subset.

Bundled-file reads (the #13092 check)

review:code-reviewer ran in all 18 L runs: 56 Read calls on bundled files, 0 Bash calls on them, none denied. M runs do not dispatch it.
Bash reads of bundled files still come from other callers, all allowed this round: review:security-reviewer (a Bash command naming the bundled context/ directory or a context/security-*.md file, 3 L runs) and the main session (grep of the agents' frontmatter, cat of context/severity.md, head of an agent file; 5 L runs, 2 M runs). Round 2 had Bash reads of bundled files denied in L 4 runs and M 1; M never dispatches code-reviewer, so a caller other than code-reviewer has already hit this. The denial did not recur, but it is not ruled out for the callers fix(review): have code-reviewer read its bundled files with the Read tool #13092 left alone.

Numbers from the VALID suites (single-arm; no comparison)

L two-defect: recall 1.00, both recall graders passed in 6 of 6 runs. Positions within ## By axis, read from the finding summaries (the position script's line window matches both request-options defects on lines 14 to 16, so its raw output was not used): L kept both defects under one heading at positions 1 and 2 in 6 of 6 runs (month-export under Spec, request-options under Standards). A cap of 2 keeps both.
M decoys, IMPORTANT+ by label (Findings table only, as in round 2): 6 of 12 runs had one, 13 findings: false 0, already on main 4, preference 5, about the changed lines 4.
- page-links 0 in 3 runs; typed-user 1 (main) in 1 run; sort-column 2 per run in 3 runs (no tests: preference; no ORDER BY tiebreaker: changed); return-promise 6 in 2 runs.
- One CRITICAL (round 2 reported none in 24 decoy runs): return-promise-m run 2 rated a missing pool error listener in the unchanged src/db.js CRITICAL, and its false-positive grader failed. The claim is true per the node-postgres Pool docs (an unhandled pool error can crash the process), and src/db.js is in the base commit, so it is labeled already on main, not false.
Not counted: M two-defect and L decoys (INVALID).
No cross-round pairing. The #7584 correction set round 1's two-defect runs aside, round 2's L suites are INVALID, and the staged plugins differ in 66 files between round 2's base (f4b1e40) and fix(review): have code-reviewer read its bundled files with the Read tool #13092, including the fanout skill.

Against the done-when

(a) Four fixtures where an arm misses: retired by the agent decision record (issuecomment-6622736575). The box is not ticked.
(b) csv-handle-leak and search-sqli axis: MET in round 1. The #7584 correction keeps it, since it is a fixture check, not a run.
(c) Two-defect positions: NOT MET. L is VALID on this head (6 of 6 runs at positions 1 and 2 under one heading); M is INVALID, so the clean two-arm rerun the correction asks for is still missing.
(d) Decoy IMPORTANT+ by label: NOT MET. M is VALID and labeled (above); L is INVALID, so there is no pair.
(e) Rerun posted on review: spec-first fanout and ratchet move to toolchain #7584: partly. The clean two-arm result still does not exist.

Remaining, smallest next step

Rerun L decoys and M two-defect only, from the same staged copy. That pairs each with this round's VALID suite on one plugin head, which settles (c), (d) and (e) if both are VALID. About 33 USD and 28 minutes with the two suites in parallel. The pair also needs Claude Code 2.1.318; if the CLI has updated, the pair is cross-version and all four suites rerun. Rerunning on current main would need all four suites again, since plugins/review has changed since fix(review): have code-reviewer read its bundled files with the Read tool #13092.
Recommended, not made: the main-session Glob of the plugin directory for the sweep workflow has now made an M suite INVALID in rounds 1 and 3. A one-line instruction in the fanout skill, like fix(review): have code-reviewer read its bundled files with the Read tool #13092's, naming the workflow by its exact plugin-root path would stop it. The workflow-directory ls and grep class remains the gate as designed, and it can still make a rerun INVALID.

Not closable yet: (c) and (d) are unmet, and (e) is partly met.

Target labels: add none; remove none.

Host run. Round 3b of #12505: the M two-defect rerun on round 3's staged copy (content of #13092's merge), which pairs with round 3's VALID L two-defect suite on one plugin copy and settles box (c).

What ran

claude plugin eval of the two M two-defect cases (month-export-m, request-options-m), same staged copy, flags and isolation as round 3: Opus 5.5 subject and judge, --ablation none -j 2 --runs 3 --scaffold --allow-tools Bash Write Workflow, capped scope, short TMPDIR. Claude Code 2.1.318, the same version as round 3's L suite. Cost 9.46 USD, 12 minutes.
Validity, origin/main's run-validity.py --runs 3 --scaffolded:

Suite	Verdict
L two-defect (round 3)	VALID, 0 denials in 6 traces (warning: skill-fired not checked)
M two-defect (this rerun)	VALID, 0 denials in 6 traces (warning: skill-fired not checked)
L decoys (rerun)	INVALID: `return-promise-l` run 1, a `review:doc-drift-detector` workflow agent's `Grep` for `fanout_diversity` at the staged `review` plugin root

Reports pulled from the kept traces' report Write (never sealed/), positions read from each finding's summary under ## By axis. A line window does not work here: L request-options run 1 and M run 2 each have a second src/http/options.js:16 row that is a different finding, and L run 3 cites the header leak at line 10.

Box (c): two-defect positions, both arms VALID on one copy

Fixture	L (3 runs)	M (3 runs)
`month-export`	both defects under Spec at positions 1 and 2, 3 of 3	both under Spec at positions 1 and 2, 3 of 3
`request-options`	both under Standards at positions 1 and 2, 3 of 3	both at positions 1 and 2 under one heading, 3 of 3 (Spec in run 1, Standards in runs 2 and 3)

Recall 1.00 in both arms: both recall graders passed in 6 of 6 runs per arm. Every other grader also passed in M.
L reports are longer (19 to 29 ## By axis rows per report, against 7 to 12 in M), but the planted defects' positions within their heading do not differ between arms.
A per-heading act-on cap of 2 keeps both planted defects in 12 of 12 runs; a cap of 1 drops the second in 12 of 12.
Axis note: request-options M run 1 filed both defects under Spec; the ground truth says standards. The box asks for two defects under one heading, which still holds.
Verdict: (c) MET. Two fixtures plant two defects each, and both arms put both under one heading in every run, so the positions can set feat(review): add a per-heading act-on cap to fanout's report #12503's act-on cap.

Against the done-when

(a) Four fixtures where an arm misses: retired by the agent decision record (issuecomment-6622736575); not ticked.
(b) csv-handle-leak and search-sqli axis: MET in round 1.
(c) Two-defect positions: MET (above).
(d) Decoy IMPORTANT+ by label: NOT MET. M decoys VALID (round 3); L decoys INVALID twice on this copy. Next: rerun both decoy arms on one new copy that carries the plugin-root search fix.
(e) Rerun posted on review: spec-first fanout and ratchet move to toolchain #7584: the two-defect half now exists; the decoy half does not.

Target labels: add none; remove none.

Host run. Round 4 of #12505: the L and M decoy suites on one staged copy of #13159's head, which names the plugin files the sweep and the doc-drift-detector agent had been searching the plugin directory for. M is VALID and L is INVALID, so (d) still has no pair. The fix held: no search aimed at the plugin root in 24 runs.

What ran

#13159 (head 1e2659fa37): a **Bundled files.** bullet in skills/fanout/context/run-everything-mode.md naming <plugin-root>/workflows/fanout-sweep.js, and one in agents/doc-drift-detector.md naming ${CLAUDE_PLUGIN_ROOT}/reference/config.md for the review.yaml keys. These were grounded in the two plugin-root denials of round 3 and its reruns: a main-session Glob **/*sweep* (M two-defect) and a doc-drift-detector Grep fanout_diversity (L decoy rerun). scripts/check-affected.sh --tests: 15 passed, 2 skipped.
claude plugin eval from a staged copy of review and multi-agent at that head. The eval directories were identical to round 3's stage. Flags and isolation were the same as round 3: Opus 5.5 subject and judge, --ablation none -j 2 --runs 3 --scaffold --allow-tools Bash Write Workflow, capped scope, short TMPDIR, under the heavy-run lock. Claude Code 2.1.318.
Both suites started 08:01Z at load 1.5. M ended 08:17Z (14.33 USD) and L ended 08:28Z (24.39 USD), 38.72 USD in total.
Validity: origin/main's run-validity.py --runs 3 --scaffolded.

Suite	Verdict
L decoys	INVALID (1 counted denial, 1 warning; skill-fired not checked)
M decoys	VALID (0 denials in 12 traces; skill-fired not checked)

Denials (reported, not rerun)

L, the FAIL: in return-promise-l run 2, the main session ran a Bash ls of its own workflow directory under the session's config directory, which is outside the plugin. The call came after the background Workflow launch and before its completion notification, so the session was still waiting on the sweep. The gate counted it. The case's denial-free runs disagree on the recall grader (run 1 fails it, run 3 passes it). This is the same class as round 3's sort-column-l FAIL.
L, warning only: in page-links-l run 2, the main session ran a Bash grep of its own workflow's journal.jsonl for one leaf's result text. This came after the workflow's completion notification, so it was recovering a leaf's output, not waiting.
M: none.

Plugin-directory reach (the #13159 check)

Main session: no Glob, Grep or shell search aimed at the plugin root in any of the 24 traces. In 2 L runs it named the workflow script and searched only within that file.
Workflow agents: no plugin-root denial, against 2 such denials in round 3 and its reruns. Their progress summaries show no plugin-root search either, but those summaries show only each agent's latest tool.
In 6 L runs, doc-drift-detector read a file under reference/ by name. The progress summary truncates the path, so which file it read is UNVERIFIED.
No reach into the plugin was denied.

Against the done-when

(a): retired by the agent decision record (issuecomment-6622736575); not ticked.
(b): MET in round 1.
(c): MET in round 3b (both arms VALID on fix(review): have code-reviewer read its bundled files with the Read tool #13092's copy; both defects at positions 1 and 2 under one heading in 12 of 12 runs).
(d) Decoy IMPORTANT+ by label: NOT MET. M is VALID on fix(review): name the plugin files the sweep and doc-drift agent look for #13159's copy, but L is INVALID, so there is no pair. No third rerun was made, and no labels were computed from an unpaired arm.
(e): the two-defect half exists; the decoy half does not.

Remaining, smallest next step

The remaining cause of the L decoy failures (round 3, round 4) is the main session reading its own workflow's run directory. It does this in two shapes: an ls while it waits on the background sweep (the counted denial), and a grep of journal.jsonl after the sweep returns, to recover one leaf's output (a warning). The fix is two lines in run-everything-mode.md:
- Step 5: wait for the sweep's completion notification, without reading its run directory.
- Step 6: take each leaf's text from the returned raw array, or from its findingsPaths file.
Whether raw held that leaf's full text in the warned run is UNVERIFIED. After the fix, rerun both decoy arms on one copy.

Target labels: add none; remove none.

Host run. Round 5 of #12505: the L and M decoy suites on one staged copy of #13167's head. That head adds step 5 and 6 lines so the sweep waits for its workflow's completion notification and takes leaf text from the returned raw array or the findings file, instead of reading the workflow's run directory. Both suites are VALID with no denials, so (d) is labeled on one plugin copy.

What ran

#13167 (head 86378a50e9), on top of #13159, which had merged in the meantime.
- Grounding: in round 4's warned run, the result file named by the completion notification already held the leaf's full text that the session then went looking for in journal.jsonl. The raw entries carry each leaf's whole output (workflows/fanout-sweep.js:265).
- Both reads are named: the raw entry's output and the findingsPaths file.
- scripts/check-affected.sh --tests: 15 passed, 2 skipped.
claude plugin eval from a staged copy of review and multi-agent at that head. The eval directories are identical to rounds 3 and 4.
- Opus 5.5 subject and judge, --ablation none -j 2 --runs 3 --scaffold --allow-tools Bash Write Workflow, capped scope, short TMPDIR, under the heavy-run lock. Claude Code 2.1.318.
- Both suites started at 08:35Z, at load 2.1. M ran 15 minutes (14.00 USD) and L ran 26 minutes (23.98 USD), 37.98 USD in total.
Validity: origin/main's run-validity.py --runs 3 --scaffolded.

Suite	Verdict
L decoys	VALID, 0 denials in 12 traces (warning: skill-fired not checked)
M decoys	VALID, 0 denials in 12 traces (warning: skill-fired not checked)

Main session: no Glob, Grep or shell search at the plugin root, and no tool call naming the workflow run directory, in any of the 24 traces.

Box (d): decoy IMPORTANT+ by label, both arms VALID on one copy

Labels (round 2's scheme, Findings table only):

false: the claim is wrong.
main: true, but already on main, outside the diff.
pref: a preference (tests, naming, a hypothetical future edit).
changed: true and about the changed lines.

Run	L findings	M findings
page-links r1-r3	none	none
return-promise r1	main (no auth on the user route)	none
return-promise r2	changed (commit message calls the await redundant; the async stack frame is lost), main (no auth)	none
return-promise r3	none	none
sort-column r1	pref (no tests), pref (map name vs. its comment)	pref (no tests), changed (no ORDER BY tiebreaker)
sort-column r2	pref (no tests)	pref (no tests), changed (no tiebreaker), main (LIKE wildcards not escaped)
sort-column r3	changed (no tiebreaker), main (case-sensitive name sort), pref (no tests), pref (map name vs. comment), main (text timestamps sort only in one format)	pref (no tests), changed (no tiebreaker), main (LIKE wildcards and case-sensitive sort)
typed-user r1	none	none
typed-user r2	none	main (empty full name renders "Hi, ")
typed-user r3	changed (commit message misstates why the guest branch was unreachable)	none

L: 6 of 12 runs had an IMPORTANT+ finding, 12 findings in total: false 0, main 4, pref 5, changed 3.
M: 4 of 12 runs had one, 9 findings in total: false 0, main 3, pref 3, changed 3.
No CRITICAL in any of the 24 runs; the false-positive grader passed 24 of 24.
Against the test(review): run the fanout reviewer-diversity eval and post its probe result #12151 baseline (L 9/12, M 7/12, 0 of 31 false, 19 preferences): both arms flag fewer runs, and no finding is false in either arm.
Each arm's sort-column recall grader failed in 3 of 3 runs. Recall graders also failed in L return-promise r1 and r2 and typed-user r3, and in M return-promise r2 and typed-user r2. These graders sit on the decoy cases and are not part of (d).
Verdict: (d) MET. Decoy runs report IMPORTANT+ findings by label for both arms on one VALID pair.

Against the done-when

(a): retired by the agent decision record (issuecomment-6622736575); not ticked.
(b): MET in round 1.
(c): MET in round 3b (two-defect pair on fix(review): have code-reviewer read its bundled files with the Read tool #13092's copy).
(d): MET (above, on fix(review): have the sweep wait for the workflow result, not read its run directory #13167's copy).
(e): both halves now exist. The two-defect pair (round 3b) and the decoy pair (round 5) each have both arms VALID on one copy, and they are on different copies. The box is met when they are posted on review: spec-first fanout and ratchet move to toolchain #7584.

Target labels: add none; remove none.

Done.

Every done-when box, checked 2026-11-04:

(a) Four fixtures where an arm misses: retired by agent decision record. With Bash, single-defect recall sat at ceiling (0 misses in 18 arm-runs), and a no-Bash setup is not one fanout ships in.
(b) Ground-truth axis values checked and corrected: met in round 1, kept by the review: spec-first fanout and ratchet move to toolchain #7584 correction.
(c) Two-defect positions: met. Both arms VALID on one copy; both defects at positions 1-2 under one heading in 12 of 12 runs (round 3b).
(d) Decoy IMPORTANT+ by label: met. Both arms VALID on one copy; L 6/12 runs, M 4/12, 0 false (round 5).
(e) Rerun posted on review: spec-first fanout and ratchet move to toolchain #7584: met (comment).

The three skill fixes that made both arms VALID merged: #13092, #13159, #13167.

Navigation Menu

Repository navigation

test(review): add fanout-diversity fixtures that separate the arms #12505

Problem

Done when

6 remaining items

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Tool	Calls	Outcome
`Read`	43	no denial
`Grep`	6	no denial
Bash	13	6 ok, 3 errors, 4 denials

test(review): add fanout-diversity fixtures that separate the arms #12505

Description

Problem

Done when

Activity

github-actions commented on Nov 2, 2026

github-actions commented on Nov 2, 2026

6 remaining items

kyle-sexton commented on Nov 3, 2026

kyle-sexton commented on Nov 3, 2026

kyle-sexton commented on Nov 4, 2026

kyle-sexton commented on Nov 4, 2026

kyle-sexton commented on Nov 4, 2026

kyle-sexton commented on Nov 4, 2026

kyle-sexton commented on Nov 4, 2026

kyle-sexton commented on Nov 4, 2026

kyle-sexton commented on Nov 4, 2026

kyle-sexton commented on Nov 4, 2026

kyle-sexton commented on Nov 4, 2026

kyle-sexton commented on Nov 4, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions