Repository navigation
perf(evals): cut the plugin-eval skill body through progressive disclosure #12661
Description
Refs #12243
Outcome
The /evals:plugin-eval body is cut through progressive disclosure: action-specific detail moves to reference files read on demand, and the hub keeps routing and gates. This is one of five slices under #12243; the first cut (/source-control:worktree) landed in #12387, whose recipe this follows.
Size at origin/main 3bd42cabfe: body 41,783 bytes (SKILL.md with frontmatter stripped, the method #12387 used); the skill directory holds 570,818 tracked bytes. The binding limit is the paired eval run each cut needs, so each skill is its own PR.
Done when
- The hub's body bytes fall, measured the same way. Moved text sits verbatim in
reference/, and every citation of it (tests, hooks, scripts, fixtures, README) is repointed. -
/docs-hygiene:compress comparereturns VERDICT PASS with no SEMANTIC LOSS. - The cases in
plugins/evals/evals/that target this skill (thepe-*cases and any other whosecase.yamlnames it) run on the before and after copies with/evals:plugin-eval --baseline <before.json> --margin <m>, on a roundrun-validity.pymarks VALID on both arms, and the after arm is non-inferior (plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md, "Cutting text").
Related
- perf: cut the bodies of the most-invoked skills through progressive disclosure - part 1 of 2 #12243 (parent), perf(source-control): move the worktree nesting measurement out of the skill hub #12387 (cut 1 recipe), perf: cut the bodies of the most-invoked skills through progressive disclosure - part 2 of 2 #12244 (after figure).
Activity
github-actions commentedon Nov 3, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Medium
Complexity: Medium
Uncertainty: Known
Flags: none
Blocked by: none
#12661 (comment)
That decision settles the open questions, so I'm asking nothing further. I read this as a bounded change to the plugin-eval skill: move action-specific text into
reference/, repoint its citations, and confirm with a compress check and a paired eval run. The paired eval run needs a host, so admission can route it. I found no duplicate.Automated triage, applied by the lanes App.
melodic-automation-lanes commentedon Nov 3, 2026
Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 1 (measure-over-time: "The hub's body bytes fall, measured the same way. Moved text sits verbatim in..."), item 3 (a model eval run: "The cases in plugins/evals/evals/ that target this skill (the pe-* cases and ..."); recommended split: keep the other items here as the admissible issue and move these to a child issue for the host queue (
host-run); it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).melodic-automation-lanes commentedon Nov 3, 2026
The latest triage verdict holds nothing: admission moved the issue to
status: confirmed; its comment says why.kyle-sexton commentedon Nov 3, 2026
Claim. Backlog HITL host-run worker is taking this now.
kyle-sexton commentedon Nov 3, 2026
Negative control, preregistered before the first paid run (
rewrite-a-plugin.mdstep 4).case.yaml(pe-ci-pin-full-ids,pe-estimate-under-ceiling,pe-invalid-no-number,pe-rules-target,pe-windows-sandbox-refusal,pe-worktree-guard-stop,va-unknown-timeout-key). All graders are deterministic (regex,tool_used).SKILL.mdbody's procedure sections deleted (everything after the H1 except## Next), frontmatter,reference/andscripts/kept.claude-opus-5-5,--runs 2 --ablation none --scaffold --keep-temp, launched together with a before run.--baseline before.json) calls acase drop, or the mean case score falls by 0.15 or more. Neither means the body is unmeasurable by this suite and the before text ships.run-validity.py --own-copy-search-warnson the degraded side only.noise-report.py --baseline before.json --margin 0.05, VALID on both arms.kyle-sexton commentedon Nov 3, 2026
Decision recorded.
(agent; not an owner decision)
The Iterating cut in #12875 ships. The one case that reads the moved text,
iterate-then-confirm, cannot detect its deletion. Its control, main's skill with only## Iteratingdeleted, still scores 1.00 on 3 of 3 runs, because the hub's Cost section already says "--runs 1for iteration with a confirm at 3".Basis:
plugins/playbooks/skills/skill-authoring/reference/rewrite-a-plugin.md:61-63defines the control for a body cut as deleting the skill's procedure steps.iterate-then-confirm, the after arm readreference/iterating.mdin all 3 runs.github-actions commentedon Nov 3, 2026
The triage sweep dispatched intake triage for this issue (dispatch 1; 2 runs that reach intake, or 6 dispatches, hand it to a maintainer).