[kyle-sexton]

Closes #12661

Summary

The /evals:plugin-eval hub loaded its iteration loop on every invocation, though no action runs it. This PR moves that section word for word into reference/iterating.md, and the hub says to answer tuning and confirm questions from it. It is the third progressive-disclosure cut under #12243 and follows the #12387 and #12704 recipe. The cut is small because two existing contracts pin the rest of the body to the hub (see Fix).

Fix

• "Iterating" moves to reference/iterating.md. Only its relative link changes, to ../../methodology/....
• Hub body (SKILL.md with frontmatter stripped): 41,783 bytes on main, 40,550 bytes on this branch.
• docs/upstream/addyosmani-agent-skills.md cited two Iterating steps by line. Those citations now point at reference/iterating.md:5 and :17.
• Two other sections were tried and kept in the hub:
  • "Calibrating a judge" is Run step 2, and the hub says it "holds every rule the actions use". The first meaning-diff labeler called its move a SEMANTIC LOSS.
  • "Boundary, the built-in plugin eval command" is pinned by the native-overlap registry. Its plugin eval row sets baked.boundary_section, and validate-plugins fails when the hub has no ## Boundary section.
• Further cuts would need one of two changes, which is perf: cut the bodies of the most-invoked skills through progressive disclosure - part 1 of 2 #12243's call: drop the hub's every-rule contract, or update the registry.

Verification

• awk frontmatter strip piped to wc -c printed 41783 on main's copy and 40550 on this branch.

• /docs-hygiene:compress compare ran with two independent compare-labeler agents per version (rewrite-a-plugin.md step 6). On the final head both returned VERDICT: PASS with no SEMANTIC LOSS and no AMBIGUITY rows. Earlier versions drew one SEMANTIC LOSS (the Calibrating move) and three AMBIGUITY rows, each fixed before the counted round.

• scripts/validate-plugins.sh --only evals passed. markdownlint passed on every changed file.

• Paired eval on the 7 cases whose case.yaml names this skill: pe-ci-pin-full-ids, pe-estimate-under-ceiling, pe-invalid-no-number, pe-rules-target, pe-windows-sandbox-refusal, pe-worktree-guard-stop and va-unknown-timeout-key.

  • Command: claude plugin eval <copy> --runs 2 --ablation none --scaffold -j 2 --keep-temp --model claude-opus-5-5, Claude Code 2.1.317.
  • Each arm ran from an out-of-repo copy holding only those cases, under a capped scope. The arms of each round were launched together.
  • Before is this branch's copy with main's skills/plugin-eval swapped in. Every case fired the skill on main, so no prompt needed the skill named.

• Negative control, preregistered on perf(evals): cut the plugin-eval skill body through progressive disclosure #12661 before the counted control round: main's copy with the hub body cut to its H1 and ## Next.

  Round	Arms	Scores	Change, 95% interval	noise-report	run-validity.py --runs 2 --scaffolded
  control (r3)	before vs degraded	0.98 vs 0.72	-0.26 (-0.39 to -0.13)	3 case drops (pe-rules-target, pe-windows-sandbox-refusal, pe-worktree-guard-stop)	VALID both (degraded with --own-copy-search-warns)
  cut (r6)	before vs after	0.92 vs 0.93	+0.01 (-0.01 to +0.04)	non-inferior at margin 0.05, no case drop	VALID both

  | iterate-then-confirm (r7) | before vs after | 1.00 vs 1.00 (3 of 3 runs each) | n too small to call (1 case) | judges unanimous, 9 of 9 votes per grader | VALID both |
  | iterate-then-confirm control (r8) | before vs main minus ## Iterating | 1.00 vs 1.00 | n too small to call | no fall | VALID both |

  • The control falls by more than 0.15 and calls case drops, so this suite detects a loss of the body.
  • iterate-then-confirm has no case.yaml, so the first selection missed it. claude-review flagged it as the one case that exercises the moved text. r7 and r8 ran with --runs 3 --judge-model opus, and the r8 control was posted on the review thread before either round ran.
    • The after arm read reference/iterating.md in all 3 runs, so the hub's pointer is followed.
    • The degraded arm read no file and still passed, because the hub's Cost section (SKILL.md:215) already says "--runs 1 for iteration with a confirm at 3". This case therefore cannot isolate the Iterating section.
    • Decision (agent, not an owner decision): ship the cut. rewrite-a-plugin.md:61-63 defines the body-cut control as deleting the skill's procedure steps, which is r3, and r3 detects a loss.
  • Not counted:
    • r1 is an earlier control round that ran before the preregistration comment was posted. It showed the same drop (-0.21).
    • r2 and r5 are cut rounds on intermediate versions that also moved Boundary and, in r2, Calibrating a judge. Both were VALID on both arms and non-inferior (+0.01 and +0.04).
  • The only warning on every arm is that the judge model is not recorded. Every grader is deterministic, so no judge ran.

Related

• perf: cut the bodies of the most-invoked skills through progressive disclosure - part 1 of 2 #12243 (parent), perf(source-control): move the worktree nesting measurement out of the skill hub #12387 (cut 1 recipe), perf(source-control): move the commit skill's rationale out of its hub #12704 (commit cut).

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-03T17:02:40.054600Z	e14054b	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[github-actions]

claude-security-review reviewed e14054b052154a16ac0e43d075ba78691e93697d: read 4 of 4 changed file(s); 0 finding(s), 0 as inline comments.

Security review: no security issues found

I read every hunk of all 4 changed files in .claude-lane/pr.diff:

• .changes/evals/perf-12661-plugin-eval-sections-to-reference-7d3e91a4.md: a new changelog fragment, prose only.
• docs/upstream/addyosmani-agent-skills.md: one table cell's line citations now point at the new reference file.
• plugins/evals/skills/plugin-eval/SKILL.md: the ## Iterating section is removed, and a sentence is added that sends tuning and confirm questions to reference/iterating.md.
• plugins/evals/skills/plugin-eval/reference/iterating.md: new file holding the removed section.

Instruction-surface lens. The diff removes text from a skill body, so I read it under this lens before applying any skip. I compared the removed steps 1–5 with the new file line by line. They are identical except for one relative link, which changes from ../methodology/... to ../../methodology/... because the file now sits one directory deeper. No rule is deleted, narrowed or softened. The hub now links to the file and says those questions are answered "from iterating.md, never from this page alone." The moved steps cover how to tune and score eval cases. They hold no guardrail on secrets, permissions, destructive actions, trust or posting. No protected class applies, so I did not need to consult the register.

Other checks.

• No changed file matches any TRUSTED PATHS glob.
• No trust control, merge hold, workflow, action, token request or kill switch is touched.
• No plugin manifest (plugin.json / marketplace.json) changes, so there are no manifest lines.
• No executable code changes.
• The PR context contains no embedded instructions aimed at this lane.

Prior findings: none. Round 1 of a cap of 3, not past the cap.

The diff itself spends nothing. The PR description reports paired claude plugin eval runs already made on claude-opus-5-5, which used prepaid model usage. That is not a cost hold.

critical-findings: 0

[github-actions]

claude-review: 1 inline finding(s) on e14054b052154a16ac0e43d075ba78691e93697d; the summary is in this lane's pull request comment.

claude-lane: claude-review

[github-actions]

claude-review reviewed e14054b052154a16ac0e43d075ba78691e93697d: read 4 of 4 changed file(s); 1 finding(s), 1 as inline comments.

Code review: PR #12875, moving plugin-eval's "Iterating" section to reference/iterating.md

Scope: I read all 4 changed files in full. This is a complete review.

What I checked:

• The move is verbatim. I compared each moved line with the lines removed from SKILL.md. Steps 1–5 are word for word the same. The only change is the relative link, ../methodology/… → ../../methodology/…. That is correct for the new depth: it resolves to plugins/evals/skills/methodology/reference/local-decisions.md from both locations.
• The repointed citations are right. In docs/upstream/addyosmani-agent-skills.md, the old SKILL.md:378 / :390 now point at iterating.md:5 / :17. Those land on the same steps (step 2 and step 5).
• Nothing else links to the removed section. I found no #iterating anchor, test, hook or script that references the old ## Iterating heading.
• Path-scoped rules. skill-bodies-state-current-rules.md and cost-claims.md match the changed files. The new hub sentence and the moved text break neither rule.
• Not counted as a finding. Two older dated records cite SKILL.md line numbers above the cut: docs/upstream/opus-5-5-task-cost.md:283 (:139-146) and docs/upstream/cursor-pstack.md:97 (:318). Both already pointed at the wrong lines before this PR (line 139 at head is the ### Tested model and judge area, not USD anchors). This PR's two inserted lines shift them by +2, so the drift is not new.

Finding (1, inline on SKILL.md:33-35): The paired eval leaves out plugins/evals/evals/iterate-then-confirm. That is the only case that tests the moved text, and its skill-fired grader names this skill. Because it has no case.yaml, it fell outside the 7-case suite. The non-inferiority result therefore says nothing about whether the model still follows the moved rules when they live behind a pointer.

Closing issue #12661, "Done when":

• Body bytes fall; text moved verbatim; citations repointed. Met. I checked that the text is verbatim and the citations are repointed. The byte counts (41,783 → 40,550) are the author's numbers; I did not run them.
• /docs-hygiene:compress compare returns PASS. The author says so; I did not verify it.
• Paired eval, VALID on both arms, non-inferior. The author reports this for the 7 cases that have a case.yaml; I did not verify it. That meets the criterion's wording, but see the inline finding: the one case that targets the moved content was not in the suite.

I found no embedded instructions in the PR context or prior-findings files. This is round 1 of a cap of 3.