Skip to content

perf(source-control): move the commit skill's rationale out of its hub - #12704

Merged
kyle-sexton merged 5 commits into
mainfrom
host-run/12666-commit-skill-cut
Nov 3, 2026
Merged

kyle-sexton merged 5 commits into
mainfrom
host-run/12666-commit-skill-cut

Conversation

@kyle-sexton

@kyle-sexton kyle-sexton commented Nov 3, 2026 •

Copy link
Copy Markdown
Member

Closes #12666

Summary

The /source-control:commit hub loaded explanatory prose on every invocation that a commit rarely needs. This PR moves that prose word for word into reference files read on demand, and keeps every command, table and gate in the hub. It is the second progressive-disclosure cut under #12243, following the #12387 recipe.

Fix

  • New reference/config-probes.md, reference/changelog-fragments.md and reference/design-rationale.md hold the moved explanations. The three native-surface descriptions move into the existing reference/native-commit.md.
  • The hub's reference index has a row for each file, so the agent still knows where each topic lives.
  • Hub body (SKILL.md with frontmatter stripped): 28,967 bytes on main, 25,098 bytes on this branch.
  • Four commit-* eval cases never fired the skill on main, so no round could be VALID. Each prompt now names the skill: commit-hook-rejection-no-bypass, commit-never-stage-everything, commit-no-push-no-pr and commit-trailer-model-from-session. Only the prompts changed; graders and case.yaml are untouched.

Verification

  • awk frontmatter strip piped to wc -c on both copies printed 28967 and 25098.

  • /docs-hygiene:compress compare (fresh compare-labeler, before against after) returned VERDICT: PASS with no SEMANTIC LOSS and no AMBIGUITY rows.

  • A repo-wide grep found no external citation of the moved text to repoint.

  • markdownlint passed on every changed file; the pre-push check-affected hook passed.

  • Paired commit-* eval. Command: claude plugin eval --case 'commit-*' --runs 2 --scaffold --ablation none --keep-temp --model claude-opus-5-5, Claude Code 2.1.317, under a capped scope.

    • Before is this branch's copy with main's skills/commit swapped in. The two arms differ only in the commit skill.
    • Result: both arms VALID under run-validity.py --runs 2 --scaffolded, with 12 of 12 traces resolving per arm. The only warning is that the judge model was not recorded.
    Round Before After Change, 95% interval noise-report at margin 0.05 run-validity.py
    3 1.00 1.00 +0.00 (+0.00 to +0.00) non-inferior VALID both arms
    • Rounds 1 and 2 ran before the case fixes. They were INVALID on the skill-fired check, so they do not count.

Related

🤖 Generated with Claude Code

kyle-sexton and others added 2 commits November 2, 2026 23:33
The /source-control:commit SKILL.md body drops from 28,967 to 25,098
bytes (frontmatter stripped). Explanatory prose moves verbatim to
reference files read on demand: the repository-context probe reasons
to reference/config-probes.md, the changelog-fragment detail to
reference/changelog-fragments.md, the type-set sources, mod note, key
spelling and recorded deviation to reference/design-rationale.md, and
the three native-surface descriptions to reference/native-commit.md.
Every command, table and gate stays in the hub.

Refs: #12666

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VcuponMacsFYBfNzLKb3ws
@kyle-sexton
kyle-sexton marked this pull request as ready for review November 3, 2026 05:55
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-03T05:56:49.928495Z 5d80bd5 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5d80bd5dcb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

claude-security-review reviewed 2ea70e7cdb622e2971e100a0e3d8129615dd4441: read 1 of 1 changed file(s); 0 finding(s), 0 as inline comments.

Security review: PR #12704, incremental since 6e8daa73

No security issues found.

I read the one hunk in the one file this push changed (.claude-lane/incremental.diff): plugins/source-control/evals/commit-trailer-model-from-session/prompt.md. The file is not on a trusted path. It is not a trust control, a merge hold or a plugin manifest.

  • The only change is to the user prompt of an eval case. It now ends "Commit the staged fix with the source-control:commit skill." instead of "Commit the staged fix.", so the skill fires in that case. This matches the other eval cases the previous push fixed and the PR's "Remaining" note.
  • The case's adversarial setup is unchanged: the prompt still carries the mismatched Co-Authored-By: Claude Fable 5.1 trailer. expected_outcome still says the trailer must name the actual model, and allowed_tools ([Read, Glob, Grep, Skill, Bash]) is the same.
  • This is an eval fixture, not a standing instruction. The change removes no guardrail and grants no tool, so the instruction-surface lens finds nothing.

The PR context file contains no text aimed at the review. The one settled thread, on design-rationale.md, is outside this push's files and stays settled. The only prior post is the last round's clean summary, so there is nothing to repeat. The round cap was not reached (0 of 3 rounds).

critical-findings: 0

@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

claude-review reviewed 2ea70e7cdb622e2971e100a0e3d8129615dd4441: read 1 of 1 changed file(s); 0 finding(s), 0 as inline comments.

Code review (incremental): PR #12704, changes since 6e8daa7

No new findings. I read the whole of the one file changed since the last review.

  • plugins/source-control/evals/commit-trailer-model-from-session/prompt.md

What changed. The prompt now ends "Commit the staged fix with the source-control:commit skill." The last push did the same for commit-hook-rejection-no-bypass and commit-never-stage-everything. The point is to make the skill fire, so the paired eval round isn't INVALID on the skill-fired check.

Does the case still test what it says? Yes:

  • The prompt still sets the trap. It carries the mismatched Co-Authored-By: Claude Fable 5.1 instruction and states the real model (Opus 5.5).
  • The expected_outcome is unchanged and still matches the prompt.
  • Naming the skill changes what the case tests: whether the skill's trailer rule overrides the stale string, rather than whether the model picks the skill. That fits a before/after comparison of the hub content.

Carried-over observation (not a finding). commit-no-push-no-pr still doesn't name the skill. The previous review raised this, and it still holds: on the before arm it could still miss the skill-fired check. Done-when 3 of #12666 needs a round that is VALID on both arms, so this case may need the same edit.

Other checks

  • Path-scoped rule (.claude/rules/eval-case-transcripts.md): the prompt is a short synthetic line with no transcript text or identifying details, so the rule is met.
  • Plugin manifests: none changed in this increment.
  • Closing issue: none. The PR only refs perf(source-control): cut the commit skill body through progressive disclosure #12666, so there were no acceptance criteria to check.
  • Settled threads / prior findings: the one settled thread (design-rationale.md, no lines changed since it was resolved) is outside this increment. The only prior post is the last round's summary, which had no findings, so nothing is repeated.
  • Embedded instructions: none found in the PR context, the settled-threads file or the prior-findings file.
  • Round cap: 0 rounds with findings so far against a cap of 3, so the cap did not apply.
  • Verification: I had no shell, so I ran nothing. Whether the skill now fires in this case stays unverified until an eval round runs.

Spend (not a finding): only the normal lane runs and the planned paired eval rounds, which use prepaid API credit.

Keep in the hub the step-7 rule that a plugin the fragment list does not
name still needs its hand version bump and CHANGELOG entry. Replace the
restated mod-hook and GitHub trailer behavior in design-rationale.md with
pointer, as-of and recheck records.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VcuponMacsFYBfNzLKb3ws
@kyle-sexton

Copy link
Copy Markdown
Member Author

Reply to the claude-review finding on plugins/source-control/skills/commit/SKILL.md:195:

# Finding Classification Basis
1 Step 7's release-record obligation left the hub VALID, fixed verified: the hub's step 7 named only exit 2 and the reference file. The rule that a plugin the list does not name still needs the hand version bump and CHANGELOG entry was only in reference/changelog-fragments.md

Fixed in 9a365b4. Step 7 now states that a fragment replaces the hand release record, and that for a plugin the list does not name the script writes nothing and the hand record still applies.

@kyle-sexton kyle-sexton added the do-not-merge Hard merge gate: do not merge while applied. label Nov 3, 2026
commit-hook-rejection-no-bypass and commit-never-stage-everything never
fired the commit skill (0 of 8 runs each), and commit-no-push-no-pr
missed on main's copy: the prompts led with a raw git instruction, so
the model ran git itself or routed to the pull-request skill. Each
prompt now opens with the commit request; the first two name the
source-control:commit skill, since a plain request still ran git
directly. The --no-verify, git add -A, and push-and-PR asks the graders
judge are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VcuponMacsFYBfNzLKb3ws
commit-trailer-model-from-session run 2 on main's skill body committed
with a heredoc and never called Skill, so the skill-fired grader missed
and the before arm read as invalid. The prompt now asks for the commit
with the source-control:commit skill, the same way the other three
commit cases do. The Fable 5.1 trailer string the graders judge is
unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton kyle-sexton removed the do-not-merge Hard merge gate: do not merge while applied. label Nov 3, 2026
@kyle-sexton
kyle-sexton added this pull request to the merge queue Nov 3, 2026
Merged via the queue into main with commit db9344f Nov 3, 2026
78 of 80 checks passed
@kyle-sexton
kyle-sexton deleted the host-run/12666-commit-skill-cut branch November 3, 2026 16:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf(source-control): cut the commit skill body through progressive disclosure

1 participant