Repository navigation
test(code-metrics): scaffold the change-scope audit eval cases #12940
Description
Why
The change-scope eval cases for the code-metrics audit actions (size, complexity, type-debt) were converted from evals.json, with no context.scaffold_script.
Each run therefore starts in an empty sandbox: no git history and no source files. On main ab409f0a and on #12803's head, every size run in the #12803 compare (2026-11-03) ended in the same error: dispatch.sh: cannot find a default branch for the merge-base; pass --base <ref> (exit 2). The score then depended on whether the model widened the scope to --all or ., which measures home dotfiles. It did not depend on the report the case grades.
This weakens any future no-loss evidence for a cut to these actions. docs/review/skills-and-agents.md:6-8 requires that evidence.
Change
Add a scaffold.sh to each change-scope case. It builds a small git repo, commits it on a default branch, then makes an uncommitted change to measure, so the case reaches the report it grades:
- size: a few files, one long;
- complexity: a function with branches, in a lane that has a collector;
- type-debt: TypeScript and Python with some
any.
audit-suppressions-count-quoted (case.yaml with scaffold_script: scaffold.sh) shows the shape.
Done when
- Every converted change-scope case under
plugins/code-metricshas a scaffold, andvalidate-cases.pypasses. - One
--caserun per action reaches its report (no exit 2), shown by a kept trace.
Refs: #12803
Metadata
Metadata
Assignees
Labels
Type
Fields
Work status
Activity
github-actions commentedon Nov 3, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Small
Complexity: Low
Uncertainty: Known
Flags: none
Blocked by: none
Understood: the code-metrics change-scope audit eval cases (size, complexity, type-debt) have no scaffold_script. Each run starts in an empty sandbox, so dispatch.sh exits 2 with no default branch and the graded report is never reached. The fix is to add a scaffold.sh to each case that builds a small git repo with an uncommitted change, following
audit-suppressions-count-quoted. Scope and done-when are clear. The kept-trace run per action may need a run outside this lane. No duplicate found among open issues.Automated triage, applied by the lanes App.
github-actions commentedon Nov 3, 2026
melodic-automation-lanes commentedon Nov 3, 2026
The work-resolve-issue run that
agent-readystarted waited in its slot past a change to its lane workflow files on main, so it stopped before any model turn. The run's report-outcome removed and addedagent-readyback to start a fresh run on main's workflow files. If this run ends the same way, it re-fires the issue again, waiting longer each time.github-actions commentedon Nov 3, 2026
github-actions commentedon Nov 3, 2026
2 remaining items
github-actions commentedon Nov 4, 2026
The triage sweep dispatched intake triage for this issue (dispatch 1; 2 runs that reach intake, or 6 dispatches, hand it to a maintainer).
melodic-automation-lanes commentedon Nov 4, 2026
Not admitted to the issue lane: merged pull request #12979 names it with
Refs:after its newest decision or refinement comment, so its work may already be done: close the issue if it is, or post a comment whose first line is**Decision recorded.**saying what is left, which sends it back through intake. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).melodic-automation-lanes commentedon Nov 4, 2026
The latest triage verdict holds nothing: admission moved the issue to
status: confirmed; its comment says why.kyle-sexton commentedon Nov 4, 2026
kyle-sexton commentedon Nov 4, 2026
PR 13039 departs from this issue's "a function with branches, in a lane that has a collector" for the complexity case: the scaffold has the function, but the eval sandbox has no collector on PATH, so the case checks the report's shape and leaves the measured numbers unchecked there. Basis: the type-debt case took the same path (its case.yaml description); judgment on cost, since each eval run costs about 1.2 USD. Why collectors are not on the sandbox's PATH is UNVERIFIED.
kyle-sexton commentedon Nov 4, 2026
Correction to comment 6622972119: that comment was an agent choice, not an owner answer. A session wrote it under the owner's login, and the owner has not seen it. PR 13039's pr-verify verdict read it as acceptance by "the issue author, a trusted human"; that reading is wrong. The owner reviews the departure in the morning.