Repository navigation
perf(source-control): cut the commit skill body through progressive disclosure #12666
Description
Refs #12243
Outcome
The /source-control:commit body is cut through progressive disclosure: action-specific detail moves to reference files read on demand, and the hub keeps routing and gates. This is one of five slices under #12243; the first cut (/source-control:worktree) landed in #12387, whose recipe this follows.
Size at origin/main 3bd42cabfe: body 28,967 bytes (SKILL.md with frontmatter stripped, the method #12387 used); the skill directory holds 132,506 tracked bytes. The binding limit is the paired eval run each cut needs, so each skill is its own PR.
Done when
- The hub's body bytes fall, measured the same way. Moved text sits verbatim in
reference/, and every citation of it (tests, hooks, scripts, fixtures, README) is repointed. -
/docs-hygiene:compress comparereturns VERDICT PASS with no SEMANTIC LOSS. - The
commit-*cases inplugins/source-control/evals/run on the before and after copies with/evals:plugin-eval --baseline <before.json> --margin <m>, on a roundrun-validity.pymarks VALID on both arms, and the after arm is non-inferior (plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md, "Cutting text").
Related
- perf: cut the bodies of the most-invoked skills through progressive disclosure - part 1 of 2 #12243 (parent), perf(source-control): move the worktree nesting measurement out of the skill hub #12387 (cut 1 recipe), perf: cut the bodies of the most-invoked skills through progressive disclosure - part 2 of 2 #12244 (after figure).
Activity
github-actions commentedon Nov 3, 2026
Proposed work class: work-class: structural
Basis: automated intake triage; the lanes App applied this class.
Size: Medium
Complexity: Medium
Uncertainty: Known
Flags: none
Blocked by: none
I read this as a bounded cut of the
/source-control:commitskill body. Detail moves verbatim toreference/, citations are repointed, and the compress and paired-eval checks follow.A recorded decision is on the issue:
#12666 (comment)
That decision settles the open questions. The unfired eval cases it names are a separate fix.
Automated triage, applied by the lanes App.
melodic-automation-lanes commentedon Nov 3, 2026
Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 1 (measure-over-time: "The hub's body bytes fall, measured the same way. Moved text sits verbatim in..."), item 3 (a model eval run: "The commit-* cases in plugins/source-control/evals/ run on the before and aft..."); recommended split: keep the other items here as the admissible issue and move these to a child issue for the host queue (
host-run); it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).melodic-automation-lanes commentedon Nov 3, 2026
The latest triage verdict holds nothing: admission moved the issue to
status: confirmed; its comment says why.kyle-sexton commentedon Nov 3, 2026
Claim. Backlog HITL host-run worker is taking this now.
kyle-sexton commentedon Nov 3, 2026
Decision recorded.
(agent; not an owner decision)
PR #12704 ships on its round-2 eval, and done-when 3 counts as met on the cases that can measure the change.
Basis:
--case 'commit-*' --runs 2 --scaffold, Opus, Claude Code 2.1.317, noise-report margin 0.05): before 0.743, after 0.792, change +0.05, 95% interval -0.01 to +0.11, so non-inferior. Round 1 also moved up (+0.08) but its interval was too wide to call.run-validity.pymarks both arms INVALID on one check only: the skill-fired check. Every other validity check passes.commit-hook-rejection-no-bypassandcommit-never-stage-everything. They fired the skill in 0 of 8 runs per case on main's unchanged copy, on both arms and in both rounds. In those traces the model runsgit addand a heredocgit commit -F -itself and never calls the Skill tool. Because those cases never load the skill body, they cannot measure a change to it; their scores are the same task without the skill on both arms.skill-criteria.mdstep 6 asks for no lost trigger, case or directive and the 5-point margin. Every case that loads the skill meets it, and/docs-hygiene:compress comparefound no semantic loss.Finding, eval cases: the two cases above do not fire the commit skill on main, so the commit suite cannot produce a VALID round until they are fixed. That fix belongs to those cases (their prompts or
case.yaml), not to this PR.Target labels: add none; remove none.
1 remaining item
github-actions commentedon Nov 3, 2026
The triage sweep dispatched intake triage for this issue (attempt 1 of 2).
kyle-sexton commentedon Nov 3, 2026
Correction: the decision above is withdrawn.
PR #12704 does not ship on round 2. Done-when 3 is not met.
skill-criteria.mdmeasurement step 4 says to read a number only from a run/evals:plugin-evaljudges valid. Both rounds are INVALID on both arms. So neither round's +0.05 or +0.08 counts as evidence. Recomputing the score on a subset of cases does not make the run valid.The finding stands.
commit-hook-rejection-no-bypassandcommit-never-stage-everythingfire the commit skill in 0 of 8 runs per case on main, so the commit suite cannot produce a VALID round.Next step: fix those two cases on #12704's branch so the skill fires, then run one before-and-after round. #12704 keeps
do-not-mergeuntil that round is VALID and non-inferior.kyle-sexton commentedon Nov 3, 2026
Negative control, preregistered before the first paid run.
#12704 cut the commit skill's body and merged without the negative control that
plugins/playbooks/skills/skill-authoring/reference/rewrite-a-plugin.md:60-66and:79require. This runs it after the fact.commit-*cases inplugins/source-control/evals/, with the case prompts as fixed in perf(source-control): move the commit skill's rationale out of its hub #12704.claude-opus-5-5), same flags (--runs 2 -j 2 --scaffold --ablation none --keep-temp):plugins/source-control/skills/commit/only.SKILL.mdbody deleted (frontmatter kept).run-validity.py --runs 2 --scaffoldedon both arms, plus--own-copy-search-warnson degraded only.noise-report.py <degraded> --baseline <before> --margin 0.05.case drop). Cases whose degraded runs carry a guard denial are left out of that comparison. If neither holds, the body cut is unmeasurable by this suite.kyle-sexton commentedon Nov 3, 2026
Host run. The negative control for #12704's commit body cut, run after the fact and preregistered in #12666 (comment).
What ran (Claude Code 2.1.317, subscription login, isolated HOME, the claude.slice scope, under the host's heavy-run lock at load 3.63):
git archivecopies of origin/main e3bf854, so both carry perf(source-control): move the commit skill's rationale out of its hub #12704's fixed case prompts: before has perf(source-control): move the commit skill's rationale out of its hub #12704's change (db9344f) reverted underplugins/source-control/skills/commit/only; degraded is the before copy with the commitSKILL.mdbody deleted (frontmatter kept).diff -rqshows they differ only in that file.claude plugin eval <copy> --case 'commit-*' --runs 2 -j 2 --scaffold --ablation none --keep-temp --model claude-opus-5-5on both arms, launched together 19:57:16Z; degraded ended 19:59:21Z, before 20:00:38Z.run-validity.py --runs 2 --scaffoldedon before; the same plus--own-copy-search-warnson degraded.noise-report.py <degraded> --baseline <before> --margin 0.05.Verdicts:
verdict: INVALID (the skill did not fire in 1 with-arm run of a should-trigger case)verdict: VALID (warnings: models)with-arm mean: 0.94, 95% interval 0.82 to 1.00 (normal, 6 cases)with-arm mean: 0.76, 95% interval 0.53 to 0.99 (normal, 6 cases)with-arm change (this minus baseline), paired over 6 cases: -0.18, 95% interval -0.41 to +0.05case drop: case commit-conventional-heredoc-trailer with-arm fell from 1.00 to 0.60, a third or more; name its cause and prove it before shippingcase drop: case commit-exec-bit-backstop with-arm fell from 1.00 to 0.33, a third or more; name its cause and prove it before shippingcompare verdict: not shown non-inferior at margin 0.05 (the interval's lower end -0.41 is below -0.05), and 2 case(s) dropped by a third or moreValidity:
commit-no-push-no-prrun 2 did not call the Skill tool (3 turns, score 0.50). Degraded fired the skill in 12 of 12 runs and had no denials, so no case is left out for denials.commit-no-push-no-prleft out as well (per-case mean over both runs): 5 cases, before 1.000, degraded 0.787, fall 0.213.Result: detect. The degraded mean falls 0.18 below before (0.21 with the unfired case left out), above the preregistered 0.15, and the noise report calls 2
case drops. Both drops fall on both runs of each case (0.60 and 0.33 in each run), so they are not one bad run. The commit suite can detect a body loss.Traces are kept under the run's TMPDIR for the coordinator.
Target labels: add none; remove none.
kyle-sexton commentedon Nov 3, 2026
Decision recorded.
(agent; not an owner decision)
The negative controls for #12739 (#12655) and #12704 (#12666) both detect a loss, so both cuts stand.
Basis:
plugins/playbooks/skills/skill-authoring/reference/rewrite-a-plugin.md:60-75. A suite detects a loss on acase dropor a 0.15 fall. A case whose degraded runs carry a denial cannot decide the control, so the drop or fall must hold with those cases left out.#12655 (pull-request): detect.
case drops,create-default-open-state-readyandcreate-verification-ends-left-out, 1.00 to 0.50 each in both degraded runs. Neither case had a denial.dirty-worktree, already reported as a triggering finding;ready-hands-pr-to-explain-change) sit in other cases.#12666 (commit): detect.
no-push-no-pr) left out, it falls 0.213.case drops,conventional-heredoc-trailer(1.00 to 0.60) andexec-bit-backstop(1.00 to 0.33).Target labels: add none; remove none.
kyle-sexton commentedon Nov 4, 2026
Correction: the "both controls detect" decision above is withdrawn until a clean re-run.
In both controls the before arm read INVALID on run-validity, because some with-arm runs never fired the skill. The decision leaned on cases unaffected by those runs. That is a looser reading than this lane's own rule, which reads numbers only from runs that pass validity.
rewrite-a-plugin.md:60-75permits leaving out degraded-side cases that have denials. It does not permit an INVALID before arm.Next: re-run both controls on current main. #12915 has merged and fixed the dirty-worktree case. A control counts only when the before arm reads VALID; the degraded arm may leave out cases with denials, per the doc. If a clean control does not detect a loss (a
case dropor a 0.15 fall), that cut is unmeasurable and its skill body reverts to the before text by PR. Both cuts stay merged until the re-run reports.kyle-sexton commentedon Nov 4, 2026
Host run. The clean re-run of the negative control for #12704's commit body cut, preregistered in #12666 (comment). It replaces the earlier control, whose before arm read INVALID.
What ran (Claude Code 2.1.318, subscription login, isolated HOME, the claude.slice scope, under the host's heavy-run lock):
git archivecopies of origin/main 97ba98f. No commit since the earlier control's base (e3bf854) touchesskills/commit/. The before arm has perf(source-control): move the commit skill's rationale out of its hub #12704's change (db9344f) reverted underplugins/source-control/skills/commit/only, and the patch applied with no fuzz. The degraded arm is the before copy with the commitSKILL.mdbody deleted and its frontmatter kept.diff -rqshows the two differ only in that file. Following the preregistration, "before" is the pre-cut text, not the current hub.claude plugin eval <copy> --case 'commit-*' --runs 2 -j 2 --scaffold --ablation none --keep-temp --model claude-opus-5-5on both arms, launched together at 00:17:19Z. Degraded ended at 00:19:13Z, before at 00:20:52Z.run-validity.py --runs 2 --scaffoldedon before; the same plus--own-copy-search-warnson degraded.noise-report.py <degraded> --baseline <before> --margin 0.05.Verdicts:
verdict: VALID (warnings: models). The skill fired in 12 of 12 with-arm runs, with no denials, and every session started.verdict: VALID (warnings: models). The skill fired in 12 of 12 runs, with no denials, so no case is left out.with-arm mean: 1.00, 95% interval 1.00 to 1.00 (normal, 6 cases)with-arm mean: 0.76, 95% interval 0.53 to 0.99 (normal, 6 cases)with-arm change (this minus baseline), paired over 6 cases: -0.24, 95% interval -0.47 to -0.01case drop: case commit-conventional-heredoc-trailer with-arm fell from 1.00 to 0.60case drop: case commit-exec-bit-backstop with-arm fell from 1.00 to 0.33case drop: case commit-no-push-no-pr with-arm fell from 1.00 to 0.62Result: detect. Both arms read VALID, the mean falls 0.24 (the threshold is 0.15), and the noise report calls 3
case drops. The figures cover all 6 cases, with none left out. Underrewrite-a-plugin.md:60-75the commit suite can detect a body loss, so #12704's cut stands.Traces are kept under the run's TMPDIR for the coordinator.
Target labels: add none; remove none.