[kyle-sexton]

Closes #12940

Summary

The host run #12979 left for the three change-scope /code-metrics:audit eval cases (size, complexity, type-debt), plus one grader fix it exposed. Each case now reaches the report it grades.

Fix

• audit-complexity-reports-the-change-with-provenance-and-no-verdict/case.yaml: expectation-2 accepted only a Coverage table naming a collector and version per lane and measure. In the sandbox no complexity collector is on PATH, so every measure reads unavailable and the old wording could not be met even by a correct report (one of three judge votes failed it). It now also accepts a measure reported unavailable with the reason the run gave, as long as the run installs no collector and states no number it did not collect. test(code-metrics): scaffold the change-scope audit eval cases #12979 reworded the type-debt case's expectation-2 the same way.
• The same case's description and expected_outcome now say a measure is taken only on a host with its collector on PATH, and otherwise reads unavailable with its reason, matching the type-debt case.
• .changes/code-metrics/: a bump: none fragment (eval suite only).
• Scaffold mode: the three scaffold.sh files are already 100755 on main (git ls-files -s), so nothing changed there.

Departure from #12940: the issue asked for the complexity scaffold to be "a function with branches, in a lane that has a collector". The scaffold has the function, but the sandbox has no collector, so the case checks the report's shape (the unavailable reason, the ISO 5055 reference, no verdict) and leaves the numbers measured for discount() unchecked there. Basis for not making a collector available: judgment on cost, since each eval run costs about 1.2 USD, and the type-debt case took the same path; why collectors are not on the sandbox's PATH is UNVERIFIED.

Verification

• validate-cases.py over plugins/code-metrics/evals: exit 0, 7 WARN lines, all about the Bash grant the runs pass with --allow-tools Bash. Re-run on the current head.
• One claude plugin eval --case <name> --scaffold --runs 1 --ablation none --judge-model opus run per action. All three reached their report with no dispatch.sh exit 2, and each run's validity gate was VALID:
  • size: score 1.0, 5/5 graders. The first audit-size.sh call exited 3 (a broken scc mise shim); the run then used the bundled counter and exited 0.
  • complexity: score 1.0, exit 0. expectation-2 votes FAIL, PASS, PASS on the old wording, which is the grader fixed above.
  • type-debt: score 1.0. Both lanes reported unavailable with install hints.
• The run total was 3.56 USD. Traces are kept on the host, outside the repository.
• Not run: the complexity case again on the new expectation-2 and reworded fields. The rerun was refused before its first turn, at no cost, because symbolic links in ~/.docker failed the eval runner's credential-store preflight. CI runs no plugin evals, so these edits' only check is the reading above.

Related

• test(code-metrics): scaffold the change-scope audit eval cases #12979 added the scaffolds; refactor(code-metrics)!: fold audit-size, audit-complexity and audit-type-debt into the audit router #12803 is the compare that found the empty-sandbox failure.
• Found during the runs and left as observations, not filed: a present-but-broken collector shim gives a lane "collect failed (exit 3)" with no fallback to the bundled counter; each run cost about 1.2 USD.

🤖 Generated with Claude Code

https://claude.ai/code/session_015te8CrNfM6v83JDbLRFDA9

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-04T02:06:57.770133Z	e2b4865	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[github-actions]

claude-security-review reviewed 0c9fb81ce0fa091de7666e04be521a005e76ff90: read 2 of 2 changed file(s); 0 finding(s), 0 as inline comments.

Security review: PR #13039 (incremental since e2b4865)

This incremental review is complete: I read the change to both files listed in scope. Neither file matches a trusted-path glob.

• .changes/code-metrics/test-12940-code-metrics-scaffold-runs-7c41e2a9.md: this push renamed the earlier changelog fragment, and the API returned no patch for it. I read the file at head. It is a bump: none fragment with one line of prose and no security surface.
• plugins/code-metrics/evals/audit-complexity-reports-the-change-with-provenance-and-no-verdict/case.yaml: this push rewords only the description and expected_outcome text of the eval case. They now say a measure is taken only on a host where its collector is on PATH, and otherwise reads unavailable with its reason. context, execution and the grader settings are unchanged.

Instruction-surface lens: this is eval case text, not a standing instruction (it is not a CLAUDE.md, rules file, skill or agent body, or hook list). No guardrail is removed or softened.

There are no trust-control, workflow, token, manifest or billing changes in this push. Plugin eval runs spend only prepaid API usage; the PR describes about 1.2 USD per run, made outside CI.

The PR context file contains no embedded instructions aimed at the review lane. The earlier summary on this lane (#13039 (comment)) reported no findings, so nothing carries over.

I found no security issues.

critical-findings: 0

[github-actions]

claude-review: 1 inline finding(s) on e2b48651329135db65257786fea9e39af18991af; the summary is in this lane's pull request comment.

claude-lane: claude-review

[github-actions]

claude-review reviewed 0c9fb81ce0fa091de7666e04be521a005e76ff90: read 2 of 2 changed file(s); 0 finding(s), 0 as inline comments.

Code review: PR #13039 (incremental, e2b48651 → 0c9fb81c)

Scope: I read both files on the incremental list.

• .changes/code-metrics/test-12940-code-metrics-scaffold-runs-7c41e2a9.md is a pure rename, so the diff has no patch for it. I read the file at head. It is still a bump: none fragment whose text describes only the eval-suite change, which fits a PR that touches only evals.
• In plugins/code-metrics/evals/audit-complexity-reports-the-change-with-provenance-and-no-verdict/case.yaml, I read the description and expected_outcome hunk and the head file.

Neither file matches a trusted path. No plugin manifest changed. The path-scoped rule .claude/rules/eval-case-transcripts.md matches the eval case, but this push adds no transcript, so the rule isn't broken.

Prior finding: last round's advisory said the case still claimed it measures the change. That is now fixed. description (line 3) now says:

• the scaffold adds no collector;
• a measure is taken only where its collector is on PATH;
• anywhere else, the eval sandbox included, the measure reads unavailable with its reason, and the measured numbers go unchecked.

expected_outcome (line 5) now accepts a Coverage table that names either the collector or the reason a measure is unavailable. Both fields now agree with expectation-2 (line 24) and with the type-debt case's wording. The owner's reply settled the thread, and I'm not raising it again.

Closing issue #12940: the departure from "a function with branches, in a lane that has a collector" is now stated in three places: the PR body, a comment on the issue, and the case's own description. The "Done when" items still rest on the author's report: validate-cases.py exit 0 and the kept traces. I did not run either. The PR says the complexity case was not rerun on the new expectation-2 and the reworded fields, so this push's edits have never been run by an eval.

Findings: none in this push.

Spend: none in this change.

I found no embedded instructions in the PR context, the settled threads or the prior findings.