You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
refactor(code-metrics): give the six audit-* skills distinct, trimmed descriptions #12491
The three code-metrics:audit-* leaves (coverage, duplication, suppressions) and the audit router stay as they are on main. Each leaf description opens with a first clause no sibling shares and is trimmed to the repository's description standard, with no loss in triggering.
Scope is option 3, chosen by the owner on 2026-11-03 (decision: #12491 (comment)), amended on 2026-11-04 to three leaves plus the router, since PR 12803 had already folded size, complexity and type-debt into the router. Owner, 2026-11-04: "Go with recomemndations".
Done when
No two plugins/code-metrics/skills/audit-*/SKILL.md descriptions share a first clause, and /skill-quality:check passes on each.
Each changed description passes a two-labeler /docs-hygiene:compress compare meaning diff before any paid run.
plugins/code-metrics/probes exists (the plugin has none today), written per plugins/skill-quality/probes/template.md by an author who never sees the rewrite.
Triggering is re-measured on those probes before and after (plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md, "Cutting text", steps 3 and 4): no lost positive and no new false fire. A description that fails keeps its main text.
ls -d plugins/code-metrics/skills/audit-* | wc -l is 3, plugins/code-metrics/skills/audit exists, and scripts/validate-plugins.sh passes.
Proposed work class: work-class: scoped
Basis: automated intake triage; a maintainer confirms or changes it.
Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
Understood: keep the six code-metrics:audit-* skills as separate leaves. Give each description a distinct first clause, trimmed to the repo standard, with no triggering loss. This includes creating plugins/code-metrics/probes and measuring triggering before and after.
The owner's decision comment (#issuecomment-6618125708) chose this option-3 scope. The rewritten body matches it, so no open questions remain.
Some done-when items, such as paid probe runs and the two-labeler compare, may need a host or an attended run. Admission handles routing for those.
No duplicates found. #12490 covers the same work for architecture:map-* and is separate.
Refinement. A read-only run explored this repository, searched its issues and pull requests, and read the allowed docs to draft what follows. On an issue a listed human filed, it counts as the issue's spec once posted, while unedited; it is still model text, so check it.
Intended outcome: Consolidate the six code-metrics audit-* skills into one code-metrics:audit router skill with actions per D107, delivered in at least two waves each under 262144 bytes.
Dependencies
#12486: Blocked by it; the *:check consolidation goes first per #7582 order.
Which consolidation target: one code-metrics:audit router skill with six actions, two skills split by data source, or keep six leaves with distinct first clauses?
RECOMMENDED: Option 1: one code-metrics:audit router skill with actions/ per D107, confirmed against the #11954 visibility measurement and each skill's eval baseline in the plan. Basis: judgment
Is the first deliverable only the sized wave plan comment, with the skill moves filed as later issues, or must this issue also complete the moves?
RECOMMENDED: The plan comment is this issue's first step; moves land in follow-up wave issues filed from it, so the remaining done-when items belong to those issues. Basis: judgment
Should the old audit-* skill names remain as aliases or redirects during the transition, or be removed outright?
RECOMMENDED: Remove them at the end of the final wave and repoint all references in the same wave; the done-when count then equals 1 for the router. Basis: judgment
Sources read
/home/runner/work/claude-code-plugins/claude-code-plugins/.base/plugins/code-metrics/skills: six audit-* directories exist beside principles and setup; I did not measure bytes or read their contents.
To answer, post a comment that starts with this line:
**Decision recorded.** <answer>
Automated refinement by the lanes App. Answer with a comment whose first line is **Decision recorded.**.
Body and title rewritten to the option-3 scope recorded above. Dropped: the wave plan, the patch-cap sizing and the "Blocked by #12486" line (#12486 is closed, and keeping the leaves no longer depends on it). Added: write plugins/code-metrics/probes first, since the plugin has none.
Setup fact, and a done-when this cannot meet. At base cb26b84af99 only three audit-* leaves exist: audit-coverage, audit-duplication and audit-suppressions. audit-size, audit-complexity and audit-type-debt became actions of the code-metrics:audit router in #12803, merged before the option 3 decision. So ls -d plugins/code-metrics/skills/audit-* | wc -l reads 3, not 6. Restoring the leaves is outside this run's scope. This run measures the three leaves and leaves the router's text unchanged; the count comes back to the owner as a question in the hand-back.
Probes. Frozen at commit 259db6b1712 on refactor/12491-code-metrics-audit-descriptions, before any rewrite existed. A fresh agent with no tools wrote them from before descriptions and plugins/skill-quality/probes/template.md alone: 18 per skill, 9 should-trigger (at least 6 contesting a named rival, none using the skill's own name words) and 9 near-misses. Competitors: code-metrics:audit in all three, plus verification:measure, testing:plan, testing:audit, mutation-testing:audit (coverage), docs-hygiene:extract-ssot, verification:measure (duplication), toolchain:ratchet, toolchain:lint (suppressions). No probe changes after this comment. No second probe round.
Rewrites. A second fresh agent with no tools wrote them, without seeing any probe. Each is under 419 characters (coverage 400, duplication 380, suppressions 410), with no first clause shared. /skill-quality:check: 6 passed, 0 failed. /docs-hygiene:compress compare, two labelers, intended-cuts file: both VERDICT: PASS in round 1, no SEMANTIC LOSS or AMBIGUITY rows. Text is fixed from here.
Arms. All are copies under ~/.ev/c12491/arms from git archive 259db6b1712 of the six plugins the cases load, outside any git repository, each with its own emitted cases.
before: as committed.
after: all three rewrites.
ccov, cdup, csup: before with one skill's description: line replaced by "Helper for repository maintenance tasks.".
Run. Per skill, its before, after and control launch together on that skill's 18 cases, one skill at a time. claude plugin eval <arm> --eval-dir probe-evals/code-metrics --case 'code-metrics-<skill>-*' --runs 3 -j 2 --ablation none --keep-temp --model opus --disallow-tools Read Glob Grep Agent --trust-plugin --no-publish --threshold 0 --max-cost-usd 25, through docker-links.py run, with TMPDIR under ~/.ev of at most 24 characters and one directory per arm. Claude Code 2.1.318. Launch waits while load is 24 or more, memory is under 16G, or more than 5 evals run. That is 54 runs per arm, 486 in total, and at most 6 runs of this work live at once.
Ceiling. 25 USD per arm, 225 USD across the 9 arm runs. Basis: the highest measured figure to hand, about 0.38 USD per run on a worktree suite, times 54 is 20.5 USD, plus headroom. #13011's measured figure for this case shape (about 99 USD over 822 runs, 0.12 per run) gives an expected total near 58 USD. These are list-price estimates against plan usage, and nothing is billed.
Smoke, which does not count. One should-trigger case, one run, before arm, to prove loading and read one run's cost. Its result enters no decision. A setup defect it finds is fixed before the first counted run, and the fix is posted here first.
Validity, per arm result. All of these:
run-validity.py <json> --runs 3 prints verdict: VALID (its exit code alone is not read, since a total failure can exit 0);
partial is false;
54 rows;
no non-null run error, except a max-turns error after the graded Skill call, which run-validity.py already reads as a warning;
each arm's kept init lines list every plugin the case names, and the served model is Opus.
run-validity.py reports a should-trigger miss as a measured rate, in every arm, so a miss never invalidates a run.
Control fire miss. It is the detection signal and never a validity failure, whether no skill fired or a rival fired. A miss in a run carrying a non-null error is not a miss: it is that run's error, under validity above.
Denials, in every arm alike, before arm included. A denied call other than Skill that run-validity.py reads as a warning does not invalidate the run, and the run's fire bit counts. A denial it reads as a FAIL makes that arm INVALID, which is a named setup defect under the re-run rule. No case is dropped from a comparison, on any side.
Detect, per skill, control against before. On the 9 should-trigger cases, either:
the mean case score falls by 0.15 or more; or
noise-report.py <control.json> --baseline <before.json> reports a case drop on one of them, and a fresh before and control pair, launched together, 3 runs each on each dropped case, reports a drop on the same case.
Near-miss cases do not decide detection. The rule comes from plugins/skill-quality/reference/invocation-probes.md, "Description controls".
Decide, per skill, after against before. Every clause is checked and reported before a verdict.
(a) Trigger rate on the should-trigger cases: after ≥ before − 0.05.
(b) False-trigger rate on the near-misses: after ≤ before + 0.05.
(c) No lost positive and no new false fire: no case drop from noise-report.py <after.json> --baseline <before.json>, where a drop is re-run once on a fresh before and after pair, launched together, 3 runs each, and counts only if it recurs on the same case.
(d) That noise report's compare verdict reads non-inferior at margin 0.05.
(e) The control detects.
Outcomes.
Valid, control detects, (a) to (d) hold: the new text ships.
Valid, control detects, any of (a) to (d) fails: the before text ships.
Valid, control does not detect: unmeasurable, the before text ships, probes stay committed. This is a likely result: #13050 saw no control drop on four skills, and the listing keeps each skill's name under a generic description.
Any of the skill's arms still invalid after the one re-run: that skill's run does not count, and the before text ships.
Re-run. One per skill, of all three of its arms launched together, and only for a named setup defect: a usage-limit, auth, crash or sandbox-start error, a ceiling hit, a validity FAIL that names a denial or a missing plugin. The defect is posted here before the re-run. The reproduction pairs in Detect and (c) are part of those rules and are not this re-run.
Disclosed now. The after arm holds all three rewrites together. If only some ship, a shipped skill was measured beside a sibling's rewrite that did not ship; no further run follows from that.
Named setup defect, before the one re-run for audit-coverage. Under the preregistration's re-run rule: its before and after arms read verdict: INVALID, with 30 and 32 of 54 runs ending Failed to authenticate: OAuth token revoked. The ccov arm launched with them finished VALID, and so did all six arms of the two skills launched afterward, so this looks like a credential change partway through those arms; the cause is otherwise UNVERIFIED. All three audit-coverage arms now re-run together with the same flags. The first attempt's results do not count, and no rule changes. This is that skill's one re-run.
Result: no control detected a loss, so all three leaves keep their before text. Decided by the preregistration, outcome "control does not detect". The probes stay committed.
skill
before chars
shipped
should-trigger fired, before / after / control
false fires, before / after / control
mean fall, control
control case drops
after vs before
audit-coverage (re-run)
474
before text
27/27, 27/27, 27/27
0/27, 0/27, 0/27
0.000
0
non-inferior at 0.05, no drop
audit-duplication
434
before text
27/27, 27/27, 27/27
1/27, 0/27, 0/27
0.000
0
non-inferior at 0.05, no drop
audit-suppressions
898
before text
27/27, 27/27, 27/27
0/27, 0/27, 0/27
0.000
0
non-inferior at 0.05, no drop
The after text held on all of (a) to (d) for every skill; (e) is what fails. Every Skill call on a should-trigger case named the target, under the generic description too, so no rival took a probe.
Validity. All 9 counted arm results read verdict: VALID, with 54 rows, partial false and no run error. The warnings are models (no judge is recorded, and these cases have none) and skill fired (near-miss cases carry no fire grader). Every init line listed each plugin the case names, and every trace shows claude-opus-5-5. Claude Code 2.1.318.
The one re-run.audit-coverage's first attempt lost its before and after arms to OAuth token revoked, posted here before the re-run. The re-run is what the table reports. In that first attempt the control arm alone was valid, at 27/27 fired.
Cost. About 53 USD list-price estimate over 649 runs, smoke and the failed attempt included. This is plan usage, and nothing is billed.
Done-when.
First clauses: PASS on the before text, none shared. /skill-quality:check: PASS on each.
Meaning diff: PASS, two labelers, on the rewrites that did not ship.
plugins/code-metrics/probes: PASS, written blind.
Triggering: no lost positive and no new false fire. The cuts are unmeasurable on these probes, so none ships and all three stay over the 419-character warn line.
ls -d plugins/code-metrics/skills/audit-* | wc -l: FAIL, reads 3, not 6, since #12803 folded size, complexity and type-debt into the router before the option 3 decision.
Not done here: the router's text and the six-leaf count both wait on the owner's call between restoring the leaves and amending this issue. The rewrites are kept out of the branch.
Preregistration, name-and-description control, before any paid run.
Question. The generic-description control fired 81 of 81 here and 72 of 72 in #13050, so it may be unable to fail while the skill's name stays in the listing. This run tests a control that also changes the name, then reports whether that control can detect a loss.
This run changes no shipped text. Any change to the certification rule in plugins/skill-quality/reference/invocation-probes.md, or to a shipped description, is a separate step.
Base.origin/main at ce2bc9d09c2, Claude Code 2.1.318, --model opus. Scoring scripts are that commit's run-validity.py and noise-report.py, extracted under ~/.ev/c12491n/tools.
Arms. Copies under ~/.ev/c12491n/arms, outside any git repository, from git archive of the six plugins the cases load.
before: as committed, cases emitted by measure-invocation.sh emit-plugin-eval.
ncov, ndup, nsup: before with that one skill's folder renamed to helper-a, which shares no word with the original, and its description: line replaced by the earlier control's "Helper for repository maintenance tasks.".
Setup fact. These skills have no name: line, so the folder rename is the whole rename. The skills page gives the name field as "Defaults to the directory name", and a plugin skill's command as "Frontmatter name or the directory name, namespaced by plugin" (skills). No name: line is added, because that would be a second change.
How a fire is scored under the rename. Cases are not re-emitted, because re-emitting would rename them and noise-report.py pairs by case name. Each control holds a copy of before's cases, and only the skill's 9 graders/skill-fired.md change, from (?:code-metrics:)?<skill> to (?:code-metrics:)?helper-a. prompt.md keeps its text. Its description and expected_outcome frontmatter still name the old skill. No prompt body names any of the three leaves (checked).
Checked with diff -r before this comment. Each control differs from before by the folder rename, one description: line and the 9 grader lines, and nothing else.
Scope. The 9 should-trigger cases only, since near-misses did not decide detection. Probes are the frozen ones, unchanged since 259db6b1712.
Run. Per skill, its before and control launch together, one skill at a time, so at most 4 runs of this work are live.
<dir> is ~/.ev/c12491n/b or ~/.ev/c12491n/c, 24 characters expanded. A launch waits while load is 24 or more, memory is under 16G, or more than 5 evals run. That is 27 runs per arm, 162 in total.
Ceiling. 5 USD per arm, 30 USD across the 6 arm runs. Basis: the earlier counted results measured 0.052 to 0.073 USD per should-trigger run, 0.082 at most, so 2.2 USD per arm at that maximum. Expected total is about 11 USD. A control miss's cost has not been measured, which is what the rest of the ceiling covers. These are list-price estimates against plan usage, and nothing is billed.
Smoke, which does not count. One run of code-metrics-audit-coverage-pos-train-01 in ncov. It shows that the init line lists code-metrics:helper-a and not code-metrics:audit-coverage. Its score enters no decision. A setup defect it finds is fixed before the first counted run, and the fix is posted here first.
Validity, per arm result. All of these:
run-validity.py <json> --runs 3 prints verdict: VALID (its exit code alone is not read);
partial is false;
27 rows;
no non-null run error, except a max-turns error after the graded Skill call, which run-validity.py reads as a warning;
each arm's kept init lines list every plugin the case names, and the served model is Opus;
each control's kept init lines list code-metrics:helper-a and not the old name.
Fire miss, in either arm. It is the measured quantity and never a validity failure, whether no skill fired or a rival fired. A miss in a run carrying a non-null error is not a miss: it is that run's error, under validity above. run-validity.py gives a max-turns error with no graded Skill call no exemption, so such a run makes that arm INVALID. before misses stay in its mean, and no case is dropped on either side.
Denials, in either arm. A denied call other than Skill that run-validity.py reads as a warning does not invalidate the run, and the run's fire bit counts. A denial it reads as a FAIL makes that arm INVALID, which is a named setup defect under the re-run rule. Its rules give a should-fire miss with a denial no probe exemption, so a miss with a denial fails unless it scored the same as every denial-free run of its case. No flag is added to change that reading.
Detect, per skill, control against before, the earlier rule unchanged, and only on a VALID pair. On the 9 should-trigger cases, either:
the mean case score falls by 0.15 or more; or
noise-report.py <control.json> --baseline <before.json> --margin 0.05 --threshold 0 reports a case drop, and a fresh before and control pair, launched together, 3 runs each on each dropped case, shows a drop on the same case.
noise-report.py prints no case-drop line under 3 paired cases, so the reproduction applies its rule directly to the pair's case scores: a fall of a third or more.
Description-only control: cited, not re-run. It fired 81 of 81 with mean fall 0.000, under the same version and model and with all 3 arms read VALID by the script of that run. git diff 259db6b1712 ce2bc9d09c2 and diff -r show no skill or probe change in these six plugins since then: five READMEs and one hook test differ, and the emitted cases are identical. Its sandboxes are gone, so today's run-validity.py cannot re-read its traces. The recorded verdict stands as recorded.
Outcomes.
(a) The control detects on a skill. That shows the name carries the trigger, since the same probes fired 27 of 27 with the name kept and the description generic, and it tells us the description-only control cannot certify a cut on that skill. Reported per skill.
(b) All three pairs VALID and the control detects on no skill. That shows the probes themselves cannot fail.
Any pair still INVALID after the re-run: that skill's pair does not count. If no other skill detects, neither (a) nor (b) holds, and the result says so.
Limits, stated now.
The namespace stays: each control lists code-metrics:helper-a, so the plugin's name is still beside the renamed skill.
The code-metrics:audit router's listing description ends "Coverage and CRAP: /code-metrics:audit-coverage.", and it stays, because everything else is identical. In ncov a run may call that old name, which no longer exists. That scores as a miss under the copied grader, and the result reports its count separately. No sibling's listing description names audit-duplication or audit-suppressions; code-metrics:principles has only the wildcard "/code-metrics:audit-* skills".
The result also reports which skill each should-trigger run called. That is descriptive and decides nothing.
Re-run. One per skill, of its pair launched together, and only for a named setup defect: a usage-limit, auth, crash or sandbox-start error, a ceiling hit, a validity FAIL that names a denial or a missing plugin. The defect is posted here before the re-run. The reproduction pair in Detect is part of that rule and is not this re-run.
I stopped the first counted launch about a minute in, during its audit-coverage pair. My own shell wrapper had a 590-second limit, and three pairs at about 180 seconds each could have reached it and killed the third pair. This is not one of the named defects; the stop was mine.
The stopped CLIs wrote result files, which I moved aside unread. They enter no decision. So that no discretion enters, I count this as audit-coverage's one re-run. No rule changes. The pairs now launch one at a time in separate commands, with the same flags. Smoke result: code-metrics:helper-a is on the init line, the old name is not, and the run took no denial.
Result: the name-and-description control detected a loss on all three skills, so outcome (a) holds on each. Decided by the preregistration. This run changes no shipped text. Any change to the certification rule or a shipped description is a separate step.
What this shows. On these probes the name carries the trigger: with the name kept and the description generic, every run fired; with both changed, none did. So on these three skills the description-only control cannot certify a cut: it passed with a generic description, so on these probes it cannot fail. The probes themselves can fail, so outcome (b) does not hold.
Compare line, identical for all three controls.with-arm change (this minus baseline), paired over 9 cases: -1.00, 95% interval -1.00 to -1.00; compare verdict: not shown non-inferior at margin 0.05 (the interval's lower end -1.00 is below -0.05), and 9 case(s) dropped by a third or more. The mean fall alone meets the detect rule, so no reproduction pair ran.
Validity. All 6 arm results read verdict: VALID, with 27 rows, partial false, no run error and no denial. The warnings are models in all 6 (no judge is recorded, and these cases have none) and, in the 3 controls, skill fired as the measured rate. Every init line lists each plugin the case names, and every trace shows claude-opus-5-5. Claude Code 2.1.318. Every control init line lists code-metrics:helper-a and not the old name; every before init line lists the old name.
What the control runs called (descriptive, decides nothing). In helper-a's place:
control
Skill calls
runs with no Skill call
calls to helper-a
calls to the old name
audit-coverage
code-metrics:audit 6
21
0
0
audit-duplication
code-review 4, code-metrics:audit 3
21
0
0
audit-suppressions
toolchain:ratchet 5, code-metrics:audit 3
19
0
0
Spot-read of 9 miss runs, three per control, from the kept traces: each attempted the request and reported that it could not do it, because the session has no read or shell tool and no listed skill fits. All 27 runs of each before called only the target skill.
Limits on reading this.
The control is close to removing the skill from view, and the fall is all or nothing. It shows these probes can fail. It says nothing about whether they could detect a smaller loss, such as a weaker description under the real name.
No arm kept the description and changed the name, so this does not show whether the description could carry the trigger alone. The name is sufficient here, which is why description cuts are unmeasurable, but it is not shown to be the only carrier.
In the coverage control, the router's description still names /code-metrics:audit-coverage. No run called it, and 25 of 27 answers name it, the ones sampled as unavailable, so that skill's misses were reached with the old name in view. The duplication and suppressions controls have no such pointer in the listing and fell the same: no answer in either names the old name. In the suppressions control, the 5 runs that loaded toolchain:ratchet saw the old name in its body, and none called it.
helper-a was visible and was not chosen: 11 of 27 duplication answers and 2 of 27 coverage answers name it, 0 of 27 suppressions answers, and no run called it.
This covers three skills on Opus 5.5. The architecture:map-* skills of #13050 were not run under this control, and carrying the finding to them is UNVERIFIED.
Deviations.
These skills carry no name: line, so the folder rename was the rename, as the preregistration stated.
The first counted launch was stopped by me, as the restart note says. Its files stayed unread until the counted runs ended, then only their costUsd and partial were read: both partial true, interrupted. It counts as audit-coverage's one re-run. No named setup defect came up, and no other re-run happened.
Cost. About 11.8 USD list-price estimate: 11.2 over the 162 counted runs, 0.1 for the smoke run, 0.5 for the stopped launch. Each arm cost 1.47 to 2.13 USD, under its 5 USD ceiling. This is plan usage, and nothing is billed.
Activity
github-actions commentedon Nov 2, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; a maintainer confirms or changes it.
Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
Understood: keep the six
code-metrics:audit-*skills as separate leaves. Give each description a distinct first clause, trimmed to the repo standard, with no triggering loss. This includes creatingplugins/code-metrics/probesand measuring triggering before and after.The owner's decision comment (#issuecomment-6618125708) chose this option-3 scope. The rewritten body matches it, so no open questions remain.
Some done-when items, such as paid probe runs and the two-labeler compare, may need a host or an attended run. Admission handles routing for those.
No duplicates found. #12490 covers the same work for
architecture:map-*and is separate.Automated triage. A maintainer reviews it.
melodic-automation-lanes commentedon Nov 2, 2026
The latest triage verdict holds nothing: the issue is at
status: ready.melodic-automation-lanes commentedon Nov 2, 2026
Refinement. A read-only run explored this repository, searched its issues and pull requests, and read the allowed docs to draft what follows. On an issue a listed human filed, it counts as the issue's spec once posted, while unedited; it is still model text, so check it.
Intended outcome: Consolidate the six code-metrics audit-* skills into one code-metrics:audit router skill with actions per D107, delivered in at least two waves each under 262144 bytes.
Dependencies
Related work
Open questions
RECOMMENDED: Option 1: one code-metrics:audit router skill with actions/ per D107, confirmed against the #11954 visibility measurement and each skill's eval baseline in the plan. Basis: judgment
RECOMMENDED: The plan comment is this issue's first step; moves land in follow-up wave issues filed from it, so the remaining done-when items belong to those issues. Basis: judgment
RECOMMENDED: Remove them at the end of the final wave and repoint all references in the same wave; the done-when count then equals 1 for the router. Basis: judgment
Sources read
To answer, post a comment that starts with this line:
Automated refinement by the lanes App. Answer with a comment whose first line is
**Decision recorded.**.39 remaining items
kyle-sexton commentedon Nov 3, 2026
Body and title rewritten to the option-3 scope recorded above. Dropped: the wave plan, the patch-cap sizing and the "Blocked by #12486" line (#12486 is closed, and keeping the leaves no longer depends on it). Added: write
plugins/code-metrics/probesfirst, since the plugin has none.kyle-sexton commentedon Nov 4, 2026
kyle-sexton commentedon Nov 4, 2026
Preregistration, before any paid run.
Setup fact, and a done-when this cannot meet. At base
cb26b84af99only threeaudit-*leaves exist:audit-coverage,audit-duplicationandaudit-suppressions.audit-size,audit-complexityandaudit-type-debtbecame actions of thecode-metrics:auditrouter in #12803, merged before the option 3 decision. Sols -d plugins/code-metrics/skills/audit-* | wc -lreads 3, not 6. Restoring the leaves is outside this run's scope. This run measures the three leaves and leaves the router's text unchanged; the count comes back to the owner as a question in the hand-back.Probes. Frozen at commit
259db6b1712onrefactor/12491-code-metrics-audit-descriptions, before any rewrite existed. A fresh agent with no tools wrote them from before descriptions andplugins/skill-quality/probes/template.mdalone: 18 per skill, 9 should-trigger (at least 6 contesting a named rival, none using the skill's own name words) and 9 near-misses. Competitors:code-metrics:auditin all three, plusverification:measure,testing:plan,testing:audit,mutation-testing:audit(coverage),docs-hygiene:extract-ssot,verification:measure(duplication),toolchain:ratchet,toolchain:lint(suppressions). No probe changes after this comment. No second probe round.Rewrites. A second fresh agent with no tools wrote them, without seeing any probe. Each is under 419 characters (coverage 400, duplication 380, suppressions 410), with no first clause shared.
/skill-quality:check: 6 passed, 0 failed./docs-hygiene:compress compare, two labelers, intended-cuts file: bothVERDICT: PASSin round 1, no SEMANTIC LOSS or AMBIGUITY rows. Text is fixed from here.Arms. All are copies under
~/.ev/c12491/armsfromgit archive 259db6b1712of the six plugins the cases load, outside any git repository, each with its own emitted cases.before: as committed.after: all three rewrites.ccov,cdup,csup:beforewith one skill'sdescription:line replaced by"Helper for repository maintenance tasks.".Run. Per skill, its before, after and control launch together on that skill's 18 cases, one skill at a time.
claude plugin eval <arm> --eval-dir probe-evals/code-metrics --case 'code-metrics-<skill>-*' --runs 3 -j 2 --ablation none --keep-temp --model opus --disallow-tools Read Glob Grep Agent --trust-plugin --no-publish --threshold 0 --max-cost-usd 25, throughdocker-links.py run, withTMPDIRunder~/.evof at most 24 characters and one directory per arm. Claude Code 2.1.318. Launch waits while load is 24 or more, memory is under 16G, or more than 5 evals run. That is 54 runs per arm, 486 in total, and at most 6 runs of this work live at once.Ceiling. 25 USD per arm, 225 USD across the 9 arm runs. Basis: the highest measured figure to hand, about 0.38 USD per run on a worktree suite, times 54 is 20.5 USD, plus headroom. #13011's measured figure for this case shape (about 99 USD over 822 runs, 0.12 per run) gives an expected total near 58 USD. These are list-price estimates against plan usage, and nothing is billed.
Smoke, which does not count. One should-trigger case, one run,
beforearm, to prove loading and read one run's cost. Its result enters no decision. A setup defect it finds is fixed before the first counted run, and the fix is posted here first.Validity, per arm result. All of these:
run-validity.py <json> --runs 3printsverdict: VALID(its exit code alone is not read, since a total failure can exit 0);partialis false;run-validity.pyalready reads as a warning;run-validity.pyreports a should-trigger miss as a measured rate, in every arm, so a miss never invalidates a run.Control fire miss. It is the detection signal and never a validity failure, whether no skill fired or a rival fired. A miss in a run carrying a non-null error is not a miss: it is that run's error, under validity above.
Denials, in every arm alike, before arm included. A denied call other than Skill that
run-validity.pyreads as a warning does not invalidate the run, and the run's fire bit counts. A denial it reads as a FAIL makes that arm INVALID, which is a named setup defect under the re-run rule. No case is dropped from a comparison, on any side.Detect, per skill, control against before. On the 9 should-trigger cases, either:
noise-report.py <control.json> --baseline <before.json>reports acase dropon one of them, and a fresh before and control pair, launched together, 3 runs each on each dropped case, reports a drop on the same case.Near-miss cases do not decide detection. The rule comes from
plugins/skill-quality/reference/invocation-probes.md, "Description controls".Decide, per skill, after against before. Every clause is checked and reported before a verdict.
case dropfromnoise-report.py <after.json> --baseline <before.json>, where a drop is re-run once on a fresh before and after pair, launched together, 3 runs each, and counts only if it recurs on the same case.compare verdictreadsnon-inferior at margin 0.05.Outcomes.
Re-run. One per skill, of all three of its arms launched together, and only for a named setup defect: a usage-limit, auth, crash or sandbox-start error, a ceiling hit, a validity FAIL that names a denial or a missing plugin. The defect is posted here before the re-run. The reproduction pairs in Detect and (c) are part of those rules and are not this re-run.
Disclosed now. The
afterarm holds all three rewrites together. If only some ship, a shipped skill was measured beside a sibling's rewrite that did not ship; no further run follows from that.kyle-sexton commentedon Nov 4, 2026
Named setup defect, before the one re-run for
audit-coverage. Under the preregistration's re-run rule: itsbeforeandafterarms readverdict: INVALID, with 30 and 32 of 54 runs endingFailed to authenticate: OAuth token revoked. Theccovarm launched with them finished VALID, and so did all six arms of the two skills launched afterward, so this looks like a credential change partway through those arms; the cause is otherwise UNVERIFIED. All threeaudit-coveragearms now re-run together with the same flags. The first attempt's results do not count, and no rule changes. This is that skill's one re-run.kyle-sexton commentedon Nov 4, 2026
Result: no control detected a loss, so all three leaves keep their before text. Decided by the preregistration, outcome "control does not detect". The probes stay committed.
The after text held on all of (a) to (d) for every skill; (e) is what fails. Every Skill call on a should-trigger case named the target, under the generic description too, so no rival took a probe.
Validity. All 9 counted arm results read
verdict: VALID, with 54 rows,partialfalse and no run error. The warnings aremodels(no judge is recorded, and these cases have none) andskill fired(near-miss cases carry no fire grader). Every init line listed each plugin the case names, and every trace showsclaude-opus-5-5. Claude Code 2.1.318.The one re-run.
audit-coverage's first attempt lost its before and after arms toOAuth token revoked, posted here before the re-run. The re-run is what the table reports. In that first attempt the control arm alone was valid, at 27/27 fired.Cost. About 53 USD list-price estimate over 649 runs, smoke and the failed attempt included. This is plan usage, and nothing is billed.
Done-when.
/skill-quality:check: PASS on each.plugins/code-metrics/probes: PASS, written blind.ls -d plugins/code-metrics/skills/audit-* | wc -l: FAIL, reads 3, not 6, since #12803 folded size, complexity and type-debt into the router before the option 3 decision.Not done here: the router's text and the six-leaf count both wait on the owner's call between restoring the leaves and amending this issue. The rewrites are kept out of the branch.
chore(code-metrics): add the changelog fragment for the audit probes
chore(code-metrics): mark the probes fragment as no release
kyle-sexton commentedon Nov 4, 2026
Preregistration, name-and-description control, before any paid run.
Question. The generic-description control fired 81 of 81 here and 72 of 72 in #13050, so it may be unable to fail while the skill's name stays in the listing. This run tests a control that also changes the name, then reports whether that control can detect a loss.
This run changes no shipped text. Any change to the certification rule in
plugins/skill-quality/reference/invocation-probes.md, or to a shipped description, is a separate step.Base.
origin/mainatce2bc9d09c2, Claude Code 2.1.318,--model opus. Scoring scripts are that commit'srun-validity.pyandnoise-report.py, extracted under~/.ev/c12491n/tools.Arms. Copies under
~/.ev/c12491n/arms, outside any git repository, fromgit archiveof the six plugins the cases load.before: as committed, cases emitted bymeasure-invocation.sh emit-plugin-eval.ncov,ndup,nsup:beforewith that one skill's folder renamed tohelper-a, which shares no word with the original, and itsdescription:line replaced by the earlier control's"Helper for repository maintenance tasks.".Setup fact. These skills have no
name:line, so the folder rename is the whole rename. The skills page gives thenamefield as "Defaults to the directory name", and a plugin skill's command as "Frontmatternameor the directory name, namespaced by plugin" (skills). Noname:line is added, because that would be a second change.How a fire is scored under the rename. Cases are not re-emitted, because re-emitting would rename them and
noise-report.pypairs by case name. Each control holds a copy ofbefore's cases, and only the skill's 9graders/skill-fired.mdchange, from(?:code-metrics:)?<skill>to(?:code-metrics:)?helper-a.prompt.mdkeeps its text. Itsdescriptionandexpected_outcomefrontmatter still name the old skill. No prompt body names any of the three leaves (checked).Checked with
diff -rbefore this comment. Each control differs frombeforeby the folder rename, onedescription:line and the 9 grader lines, and nothing else.Scope. The 9 should-trigger cases only, since near-misses did not decide detection. Probes are the frozen ones, unchanged since
259db6b1712.Run. Per skill, its
beforeand control launch together, one skill at a time, so at most 4 runs of this work are live.python3 -I <repo>/plugins/evals/skills/plugin-eval/scripts/docker-links.py run -- env TMPDIR=<dir> timeout 3000 claude plugin eval <arm> --eval-dir probe-evals/code-metrics --case 'code-metrics-<skill>-pos-*' --runs 3 -j 2 --ablation none --keep-temp --model opus --disallow-tools Read Glob Grep Agent --trust-plugin --no-publish --threshold 0 --max-cost-usd 5 --json <result><dir>is~/.ev/c12491n/bor~/.ev/c12491n/c, 24 characters expanded. A launch waits while load is 24 or more, memory is under 16G, or more than 5 evals run. That is 27 runs per arm, 162 in total.Ceiling. 5 USD per arm, 30 USD across the 6 arm runs. Basis: the earlier counted results measured 0.052 to 0.073 USD per should-trigger run, 0.082 at most, so 2.2 USD per arm at that maximum. Expected total is about 11 USD. A control miss's cost has not been measured, which is what the rest of the ceiling covers. These are list-price estimates against plan usage, and nothing is billed.
Smoke, which does not count. One run of
code-metrics-audit-coverage-pos-train-01inncov. It shows that the init line listscode-metrics:helper-aand notcode-metrics:audit-coverage. Its score enters no decision. A setup defect it finds is fixed before the first counted run, and the fix is posted here first.Validity, per arm result. All of these:
run-validity.py <json> --runs 3printsverdict: VALID(its exit code alone is not read);partialis false;run-validity.pyreads as a warning;code-metrics:helper-aand not the old name.Fire miss, in either arm. It is the measured quantity and never a validity failure, whether no skill fired or a rival fired. A miss in a run carrying a non-null error is not a miss: it is that run's error, under validity above.
run-validity.pygives a max-turns error with no graded Skill call no exemption, so such a run makes that arm INVALID.beforemisses stay in its mean, and no case is dropped on either side.Denials, in either arm. A denied call other than Skill that
run-validity.pyreads as a warning does not invalidate the run, and the run's fire bit counts. A denial it reads as a FAIL makes that arm INVALID, which is a named setup defect under the re-run rule. Its rules give a should-fire miss with a denial no probe exemption, so a miss with a denial fails unless it scored the same as every denial-free run of its case. No flag is added to change that reading.Detect, per skill, control against
before, the earlier rule unchanged, and only on a VALID pair. On the 9 should-trigger cases, either:noise-report.py <control.json> --baseline <before.json> --margin 0.05 --threshold 0reports acase drop, and a fresh before and control pair, launched together, 3 runs each on each dropped case, shows a drop on the same case.noise-report.pyprints no case-drop line under 3 paired cases, so the reproduction applies its rule directly to the pair's case scores: a fall of a third or more.Description-only control: cited, not re-run. It fired 81 of 81 with mean fall 0.000, under the same version and model and with all 3 arms read
VALIDby the script of that run.git diff 259db6b1712 ce2bc9d09c2anddiff -rshow no skill or probe change in these six plugins since then: five READMEs and one hook test differ, and the emitted cases are identical. Its sandboxes are gone, so today'srun-validity.pycannot re-read its traces. The recorded verdict stands as recorded.Outcomes.
Limits, stated now.
code-metrics:helper-a, so the plugin's name is still beside the renamed skill.code-metrics:auditrouter's listing description ends "Coverage and CRAP: /code-metrics:audit-coverage.", and it stays, because everything else is identical. Inncova run may call that old name, which no longer exists. That scores as a miss under the copied grader, and the result reports its count separately. No sibling's listing description namesaudit-duplicationoraudit-suppressions;code-metrics:principleshas only the wildcard "/code-metrics:audit-* skills".Re-run. One per skill, of its pair launched together, and only for a named setup defect: a usage-limit, auth, crash or sandbox-start error, a ceiling hit, a validity FAIL that names a denial or a missing plugin. The defect is posted here before the re-run. The reproduction pair in Detect is part of that rule and is not this re-run.
kyle-sexton commentedon Nov 4, 2026
Restart note, before any counted result was read.
I stopped the first counted launch about a minute in, during its
audit-coveragepair. My own shell wrapper had a 590-second limit, and three pairs at about 180 seconds each could have reached it and killed the third pair. This is not one of the named defects; the stop was mine.The stopped CLIs wrote result files, which I moved aside unread. They enter no decision. So that no discretion enters, I count this as
audit-coverage's one re-run. No rule changes. The pairs now launch one at a time in separate commands, with the same flags. Smoke result:code-metrics:helper-ais on the init line, the old name is not, and the run took no denial.kyle-sexton commentedon Nov 4, 2026
Result: the name-and-description control detected a loss on all three skills, so outcome (a) holds on each. Decided by the preregistration. This run changes no shipped text. Any change to the certification rule or a shipped description is a separate step.
What this shows. On these probes the name carries the trigger: with the name kept and the description generic, every run fired; with both changed, none did. So on these three skills the description-only control cannot certify a cut: it passed with a generic description, so on these probes it cannot fail. The probes themselves can fail, so outcome (b) does not hold.
Compare line, identical for all three controls.
with-arm change (this minus baseline), paired over 9 cases: -1.00, 95% interval -1.00 to -1.00;compare verdict: not shown non-inferior at margin 0.05 (the interval's lower end -1.00 is below -0.05), and 9 case(s) dropped by a third or more. The mean fall alone meets the detect rule, so no reproduction pair ran.Validity. All 6 arm results read
verdict: VALID, with 27 rows,partialfalse, no run error and no denial. The warnings aremodelsin all 6 (no judge is recorded, and these cases have none) and, in the 3 controls,skill firedas the measured rate. Every init line lists each plugin the case names, and every trace showsclaude-opus-5-5. Claude Code 2.1.318. Every control init line listscode-metrics:helper-aand not the old name; everybeforeinit line lists the old name.What the control runs called (descriptive, decides nothing). In
helper-a's place:helper-acode-metrics:audit6code-review4,code-metrics:audit3toolchain:ratchet5,code-metrics:audit3Spot-read of 9 miss runs, three per control, from the kept traces: each attempted the request and reported that it could not do it, because the session has no read or shell tool and no listed skill fits. All 27 runs of each
beforecalled only the target skill.Limits on reading this.
/code-metrics:audit-coverage. No run called it, and 25 of 27 answers name it, the ones sampled as unavailable, so that skill's misses were reached with the old name in view. The duplication and suppressions controls have no such pointer in the listing and fell the same: no answer in either names the old name. In the suppressions control, the 5 runs that loadedtoolchain:ratchetsaw the old name in its body, and none called it.helper-awas visible and was not chosen: 11 of 27 duplication answers and 2 of 27 coverage answers name it, 0 of 27 suppressions answers, and no run called it.architecture:map-*skills of #13050 were not run under this control, and carrying the finding to them is UNVERIFIED.Deviations.
name:line, so the folder rename was the rename, as the preregistration stated.costUsdandpartialwere read: bothpartialtrue,interrupted. It counts asaudit-coverage's one re-run. No named setup defect came up, and no other re-run happened.Cost. About 11.8 USD list-price estimate: 11.2 over the 162 counted runs, 0.1 for the smoke run, 0.5 for the stopped launch. Each arm cost 1.47 to 2.13 USD, under its 5 USD ceiling. This is plan usage, and nothing is billed.