Repository navigation
perf(source-control): cut the pull-request skill body through progressive disclosure #12655
Description
Refs #12243
Outcome
The /source-control:pull-request body is cut through progressive disclosure: action-specific detail moves to reference files read on demand, and the hub keeps routing and gates. This is one of five slices under #12243; the first cut (/source-control:worktree) landed in #12387, whose recipe this follows.
Size at origin/main 3bd42cabfe: body 50,149 bytes (SKILL.md with frontmatter stripped, the method #12387 used); the skill directory holds 799,595 tracked bytes. A cut that moves about half the body is a diff well under the 262,144-byte cap. The binding limit is the paired eval run each cut needs, so each skill is its own PR.
Done when
- The hub's body bytes fall, measured the same way. Moved text sits verbatim in
reference/(or an action file), and every citation of it (tests, hooks, scripts, fixtures, README) is repointed. -
/docs-hygiene:compress comparereturns VERDICT PASS with no SEMANTIC LOSS. - The
pull-request-*cases inplugins/source-control/evals/run on the before and after copies with/evals:plugin-eval --baseline <before.json> --margin <m>, on a roundrun-validity.pymarks VALID on both arms, and the after arm is non-inferior (plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md, "Cutting text").
Related
- perf: cut the bodies of the most-invoked skills through progressive disclosure - part 1 of 2 #12243 (parent), perf(source-control): move the worktree nesting measurement out of the skill hub #12387 (cut 1 recipe), perf: cut the bodies of the most-invoked skills through progressive disclosure - part 2 of 2 #12244 (after figure).
Activity
github-actions commentedon Nov 3, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
Understood: cut the
/source-control:pull-requestskill body through progressive disclosure. Move action detail verbatim to reference files, repoint every citation, and pass the compress compare and the paired eval non-inferiority check. The eval and compare steps may need a host run, which admission routes.#12655 (comment)
This decision settles the open question about the measured-fire case, so nothing more is asked.
Automated triage, applied by the lanes App.
melodic-automation-lanes commentedon Nov 3, 2026
Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 1 (measure-over-time: "The hub's body bytes fall, measured the same way. Moved text sits verbatim in..."), item 3 (a model eval run: "The pull-request-* cases in plugins/source-control/evals/ run on the before a..."); recommended split: keep the other items here as the admissible issue and move these to a child issue for the host queue (
host-run); it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).melodic-automation-lanes commentedon Nov 3, 2026
The latest triage verdict holds nothing: admission moved the issue to
status: confirmed; its comment says why.kyle-sexton commentedon Nov 3, 2026
Claim. Backlog HITL host-run worker is taking this now, with #12468: its pull-request cut is this slice, so one PR serves both.
kyle-sexton commentedon Nov 3, 2026
Claim. Backlog HITL host-run worker is taking done-when 3 (the paired pull-request-* eval for #12739) now, resuming the earlier run that stopped in a host lockup.
kyle-sexton commentedon Nov 3, 2026
Decision recorded.
(agent; not an owner decision)
run-validity runs on both arms of the done-when 3 round with
--measured-fire pull-request-ready-hands-pr-to-explain-change, and that case keeps its plain prompt.Basis: in the firing check (
--runs 1on the before arm), the case loaded the skill under test: its first tool call isSkill source-control:pull-request ready 512 .... run-validity still reports "the skill did not fire" because the case grades the hand-off toexplain-changewith atool: Skill,min: 1grader (graders/digest-at-ready-event.md). That hand-off is the behavior under test, which run-validity --help describes as "a should-trigger case whose firing is the quantity a comparison measures". Naming explain-change in the prompt would hand the model the graded answer. Dropping the case would leave the ready flow, which #12739 touched, untested. Every other check is unchanged. The missed hand-off is reported as a behavior finding with per-arm counts.kyle-sexton commentedon Nov 3, 2026
Negative control, preregistered before the first paid run.
#12739 cut the pull-request skill's body and merged without the negative control that
plugins/playbooks/skills/skill-authoring/reference/rewrite-a-plugin.md:60-66and:79require. This runs it after the fact.pull-request-*cases inplugins/source-control/evals/.claude-opus-5-5), same flags (--runs 2 -j 2 --scaffold --ablation none --keep-temp):plugins/source-control/skills/pull-request/.SKILL.mdbody deleted (frontmatter kept).run-validity.py --runs 2 --scaffoldedon both arms, plus--own-copy-search-warnson degraded only.noise-report.py <degraded> --baseline <before> --margin 0.05.case drop). Cases whose degraded runs carry a guard denial are left out of that comparison. If neither holds, the body cut is unmeasurable by this suite.kyle-sexton commentedon Nov 3, 2026
Host run. Done-when 3: a paired eval of every
pull-request-*case inplugins/source-control/evals/, run on a before arm with #12739's diff reverted and on an after arm at main.What ran (Claude Code 2.1.317, subscription login, isolated HOME, the claude.slice scope):
git archivecopies of the case-fix branch head f400737, which is origin/main 1e321c6 plus the case fix. The before copy has perf(source-control): move the pull-request monitor checklists to monitor.md #12739's diff (5f63e3d) reverted underplugins/source-control/skills/pull-request/only, anddiff -rqshows the copies differ only in that directory.claude plugin eval <before copy> --case 'pull-request-*' --runs 1 -j 2 --scaffold --ablation none --keep-temp --model claude-opus-5-5. Every non-slash case loaded the pull-request skill (11 of 11). The 10ci-modecases open with the slash command.--runs 2, on each arm. Before 18:39 to 18:52Z, after 18:39 to 18:55Z, both under the host's heavy-run lock at load < 2.run-validity.py --runs 2 --scaffolded --measured-fire pull-request-ready-hands-pr-to-explain-changeon each arm. The reason for the flag is in the Decision recorded comment on this issue.noise-report.py <after> --baseline <before> --margin 0.05.Verdicts:
verdict: VALID (warnings: measured cases fired in 0 of 2 runs, models)verdict: VALID (warnings: measured cases fired in 0 of 2 runs, models)with-arm mean: 0.91, 95% interval 0.82 to 0.99 (normal, 21 cases)with-arm mean: 0.90, 95% interval 0.81 to 0.98 (normal, 21 cases)with-arm change (this minus baseline), paired over 21 cases: -0.01, 95% interval -0.05 to +0.03compare verdict: non-inferior at margin 0.05 (the interval's lower end -0.05 is at or above -0.05)Done-when 3: met. Both arms are VALID, and the after arm is non-inferior at margin 0.05. The lower end sits exactly on the margin.
Findings:
pull-request-dirty-worktree-stashes-before-checkoutwith its plain prompt on main fired the skill in 0 of 4 runs. Each time the model stashed withgit stash push --include-untrackedand rangit switch mainitself. The case-fix PR names the skill in that prompt, so the case now tests behavior, not triggering. The trigger gap belongs to the skill's description and trigger probes (/skill-quality:check measure-invocation).pull-request-ready-hands-pr-to-explain-changeloaded the pull-request skill but never handed off toexplain-change: 0 of 2 runs per arm, the same on both arms, so perf(source-control): move the pull-request monitor checklists to monitor.md #12739 did not cause it.reference/ready-for-review.md2.5.5 carries the instruction in both arms.modelsWARN). The round passed no--judge-model, the same on both arms, and neither the result nor the traces record which judge model ran.PR: the case fix is in #12915.
Target labels: add none; remove host-run.
kyle-sexton commentedon Nov 3, 2026
Host run. The negative control for #12739's pull-request body cut, run after the fact and preregistered in #12655 (comment).
What ran (Claude Code 2.1.317, subscription login, isolated HOME, the claude.slice scope, under the host's heavy-run lock at load 0.97):
git archivecopies of origin/main e3bf854: before has perf(source-control): move the pull-request monitor checklists to monitor.md #12739's diff (5f63e3d) reverted underplugins/source-control/skills/pull-request/; degraded is the before copy with the pull-requestSKILL.mdbody deleted (frontmatter kept).diff -rqshows they differ only in that file. The earlier round's before copy was not reused: it was built on the unmerged case-fix branch and differs from main in two files.claude plugin eval <copy> --case 'pull-request-*' --runs 2 -j 2 --scaffold --ablation none --keep-temp --model claude-opus-5-5on both arms, launched together 19:38:58Z; degraded ended 19:51:42Z, before 19:54:04Z.run-validity.py --runs 2 --scaffoldedon before; the same plus--own-copy-search-warnson degraded.noise-report.py <degraded> --baseline <before> --margin 0.05.Verdicts:
verdict: INVALID (the skill did not fire in 3 with-arm runs of a should-trigger case)verdict: INVALID (4 permission denials in the traces; the skill did not fire in 4 with-arm runs of a should-trigger case)with-arm mean: 0.91, 95% interval 0.84 to 0.98 (normal, 21 cases)with-arm mean: 0.85, 95% interval 0.76 to 0.94 (normal, 21 cases)with-arm change (this minus baseline), paired over 21 cases: -0.06, 95% interval -0.14 to +0.02case drop: case pull-request-create-default-open-state-ready with-arm fell from 1.00 to 0.50, a third or more; name its cause and prove it before shippingcase drop: case pull-request-create-verification-ends-left-out with-arm fell from 1.00 to 0.50, a third or more; name its cause and prove it before shippingcompare verdict: not shown non-inferior at margin 0.05 (the interval's lower end -0.14 is below -0.05), and 2 case(s) dropped by a third or moreValidity:
pull-request-dirty-worktree-stashes-before-checkoutfired 0 of 2 per arm. Its plain prompt on main never fires the skill (the earlier host run's finding; the prompt fix is in the unmerged test(source-control): name the pull-request skill in the dirty-worktree eval case #12915).pull-request-ready-hands-pr-to-explain-changealso missed once on before and twice on degraded.cdinto the copy's skill directory, orsedon its reference file), inci-mode-escalates-instead-of-cutting-criterion-line,ci-mode-fixes-markdown-lint,ci-mode-makes-room-without-cutting-criterion-lineandready-hands-pr-to-explain-change.--own-copy-search-warnsdid not downgrade them.Means with those cases left out (per-case mean over both runs):
Result: two readings, for the coordinator to pick.
case drop, as inrewrite-a-plugin.md:63-64): detect, through the 2case drops. Neither falls on a denial case, so both hold with those cases left out. Each drop is consistent across runs: both degraded runs of each case scored 0.50, against 1.00 and 1.00 on before. One judge vote increate-default-open-state-readyrun 2 was split (FAIL FAIL PASS).The degraded copy keeps
reference/, and the traces show it searching its own copy for the lost text; reads that were not denied may have recovered much of the procedure, which would explain the small mean fall.Traces are kept under the run's TMPDIR for the coordinator.
Target labels: add none; remove none.
kyle-sexton commentedon Nov 3, 2026
Decision recorded.
(agent; not an owner decision)
The negative controls for #12739 (#12655) and #12704 (#12666) both detect a loss, so both cuts stand.
Basis:
plugins/playbooks/skills/skill-authoring/reference/rewrite-a-plugin.md:60-75. A suite detects a loss on acase dropor a 0.15 fall. A case whose degraded runs carry a denial cannot decide the control, so the drop or fall must hold with those cases left out.#12655 (pull-request): detect.
case drops,create-default-open-state-readyandcreate-verification-ends-left-out, 1.00 to 0.50 each in both degraded runs. Neither case had a denial.dirty-worktree, already reported as a triggering finding;ready-hands-pr-to-explain-change) sit in other cases.#12666 (commit): detect.
no-push-no-pr) left out, it falls 0.213.case drops,conventional-heredoc-trailer(1.00 to 0.60) andexec-bit-backstop(1.00 to 0.33).Target labels: add none; remove none.