You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
test(evals): add eval cases for pointer-swept skills that have none - part 9 of 9 #13311
I read this as part 9 of 9 of #13300. It asks for eval cases for pointer-swept skills that have none, then one run per plugin with a valid both-arm verdict, recorded here. Any regression found goes in its own bug issue.
The body says "every skill above", but no skill list appears in it. The list should be in #13300 or the split plan, so the issue lane needs to read the parent to find which skills belong to part 9.
#13303 to #13310 are the sibling parts and #13300 is the parent. They cover other slices, so I did not mark any as a duplicate.
The both-arm verdict run may need an eval environment. The cases themselves can be written without one.
Part 9 (#13311): VALID both-arm runs, playwright and playbooks on Claude Code 2.1.318, opus judge, 3 runs per arm, -j 1, each plugin run from a copy outside git:
Plugin
Case
With
Without
playwright
dv-plans-pipeline-in-order
1.00
0.00
playwright
dv-qc-fail-holds-post
0.78
0.67
playbooks
boris-parallel-sessions-from-reference
1.00
0.83
playbooks
repo-sweep-next-stops-on-conflicting-pr
1.00
0.83
playbooks
update-check-runs-pack-scripts
1.00
0.50
Validity:run-validity.py reads VALID for both plugins. playwright: verdict: VALID, no warnings. playbooks: verdict: VALID (warnings: permission denials in case update-check-runs-pack-scripts, skill fired): one without-arm Glob denied on the plugin copy's directory, and no tool_used: Skill grader in that case, since update sets disable-model-invocation: true and the case is slash-invoked.
Gains: playwright's noise report reads "n too small to call" (2 cases). playbooks: delta +0.28, 95% interval +0.06 to +0.50 over 3 cases, so the difference is larger than the noise.
Findings:
In one of three with-arm dv-qc-fail-holds-post runs the model followed "just post it" and gave the gh pr comment --attach command before explaining the QC failure; holds-post correctly failed it.
holds-post passes a without-arm reply that posts through gh release create plus a link with no --attach, which inflates that arm.
In update-check-runs-pack-scripts both --check scripts ran in order but exited 2 because the sandbox blocks howborisusesclaudecode.com, so the case measures the procedure, not a drift result.
The stops-before-step llm judge has no calibration samples but was unanimous in 6 of 6 runs.
No regression found. Remaining in part 9: instruction-placement and work-items (runs in progress), and disk-hygiene's setup case (INVALID, fix and re-run pending).
Part 9 (#13311): instruction-placement and work-items on Claude Code 2.1.318, deterministic graders only, 3 runs per arm, -j 1, each plugin run from a copy outside git:
Plugin
Case
With
Without
instruction-placement
au-safety-rail-held-back (retry)
0.87
0.87
instruction-placement
dl-no-baseline-routes-to-audit
1.00
0.75
instruction-placement
mg-plan-stops-before-write
1.00
1.00
work-items
su-check-flags-duplicate-schedule-id
0.75
0.75
Validity:
instruction-placement, full 3-case pass: verdict: INVALID (2 permission denials in the traces (warnings only, not counted: case au-safety-rail-held-back)). Both counted denials are in au-safety-rail-held-back: a with-arm sed -n read of the plugin's context/findings-artifact.md blocked by a deny rule, and a without-arm mkdir -p .claude/rules denied in dontAsk mode. One retry of that case alone reads verdict: VALID. The dl and mg rows are from the INVALID pass. This plugin still owes one clean full pass; it re-runs after the grader fix below.
work-items: verdict: VALID (warnings: skill fired). The setup skill sets disable-model-invocation: true, so the case has no Skill grader; the with-only ran-home-resolver passed in 3 of 3 with-arm runs.
Gains: "n too small to call" throughout. mg-plan-stops-before-write is at ceiling; the plugin shows only through its with-only graders (3 of 3).
Grader defects, being fixed:
holds-back-git-safety (au-safety-rail-held-back) fails correct replies in both arms ("Not moved: "Git safety"", "Moving the git section: held back", "Git safety: I'd keep this in CLAUDE.md").
names-duplicate-id (su-check-flags-duplicate-schedule-id) fails every with-arm run although each FAIL row names dependency-review and says "both rows use the id". Its 0.75 / 0.75 hides a with-arm that did the task while the baseline did nothing.
Part 9 (#13311): re-runs after PR 13433, from origin/main at 7b4a144, Claude Code 2.1.318, tested model claude-opus-5-5, opus judge (no llm grader selected), 3 runs per arm, -j 1, no-plugin baseline arm, each plugin run from a copy outside git.
verdict: VALID (warnings: skill fired). 0 permission denials in 6 traces (previous run: 7, from the clean engine gate on compound Bash lines). The warning is expected: the setup skill sets disable-model-invocation: true.
With 1.00, without 0.50, delta +0.50. Noise: "n too small to call" (1 case).
With-arm runs read the engine with Read and Grep (^MIN_PYTHON), as the setup skill now directs.
instruction-placement, all three cases in one pass, --tag migrate --tag audit --tag delta --scaffold --allow-tools Bash Write Edit:
verdict: VALID (warnings: ceiling). 0 permission denials in 18 traces (previous full pass: 2); the skill fired in 9 of 9 with-arm runs.
Case
With
Without
mg-plan-stops-before-write
1.00
1.00 (ceiling, excluded)
au-safety-rail-held-back
0.87
0.80
dl-no-baseline-routes-to-audit
1.00
0.83
Paired delta +0.08, 95% interval -0.02 to +0.17: within noise.
holds-back-git-safety again failed 2 with-arm replies that correctly hold Git safety back ("has to stay in CLAUDE.md"; "Not moved: "Git safety" … keep it in CLAUDE.md"). PR 13433 left this grader at main's pattern on purpose: two verifier rounds found every regex widening also accepted wrong answers. It under-scores the with-arm; it does not make the run valid when it should not be.
Activity
github-actions commentedon Nov 4, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Medium
Complexity: Low
Uncertainty: Knowable
Flags: none
Blocked by: none
I read this as part 9 of 9 of #13300. It asks for eval cases for pointer-swept skills that have none, then one run per plugin with a valid both-arm verdict, recorded here. Any regression found goes in its own bug issue.
The body says "every skill above", but no skill list appears in it. The list should be in #13300 or the split plan, so the issue lane needs to read the parent to find which skills belong to part 9.
#13303 to #13310 are the sibling parts and #13300 is the parent. They cover other slices, so I did not mark any as a duplicate.
The both-arm verdict run may need an eval environment. The cases themselves can be written without one.
Automated triage, applied by the lanes App.
14 remaining items
kyle-sexton commentedon Nov 5, 2026
Part 9 (#13311): VALID both-arm runs, playwright and playbooks on Claude Code 2.1.318, opus judge, 3 runs per arm,
-j 1, each plugin run from a copy outside git:dv-plans-pipeline-in-orderdv-qc-fail-holds-postboris-parallel-sessions-from-referencerepo-sweep-next-stops-on-conflicting-prupdate-check-runs-pack-scriptsrun-validity.pyreads VALID for both plugins. playwright:verdict: VALID, no warnings. playbooks:verdict: VALID (warnings: permission denials in case update-check-runs-pack-scripts, skill fired): one without-armGlobdenied on the plugin copy's directory, and notool_used: Skillgrader in that case, sinceupdatesetsdisable-model-invocation: trueand the case is slash-invoked.dv-qc-fail-holds-postruns the model followed "just post it" and gave thegh pr comment --attachcommand before explaining the QC failure;holds-postcorrectly failed it.holds-postpasses a without-arm reply that posts throughgh release createplus a link with no--attach, which inflates that arm.update-check-runs-pack-scriptsboth--checkscripts ran in order but exited 2 because the sandbox blockshowborisusesclaudecode.com, so the case measures the procedure, not a drift result.stops-before-stepllm judge has no calibration samples but was unanimous in 6 of 6 runs.kyle-sexton commentedon Nov 5, 2026
Part 9 (#13311): instruction-placement and work-items on Claude Code 2.1.318, deterministic graders only, 3 runs per arm,
-j 1, each plugin run from a copy outside git:au-safety-rail-held-back(retry)dl-no-baseline-routes-to-auditmg-plan-stops-before-writesu-check-flags-duplicate-schedule-idverdict: INVALID (2 permission denials in the traces (warnings only, not counted: case au-safety-rail-held-back)). Both counted denials are inau-safety-rail-held-back: a with-armsed -nread of the plugin'scontext/findings-artifact.mdblocked by a deny rule, and a without-armmkdir -p .claude/rulesdenied in dontAsk mode. One retry of that case alone readsverdict: VALID. Thedlandmgrows are from the INVALID pass. This plugin still owes one clean full pass; it re-runs after the grader fix below.verdict: VALID (warnings: skill fired). The setup skill setsdisable-model-invocation: true, so the case has noSkillgrader; the with-onlyran-home-resolverpassed in 3 of 3 with-arm runs.mg-plan-stops-before-writeis at ceiling; the plugin shows only through its with-only graders (3 of 3).holds-back-git-safety(au-safety-rail-held-back) fails correct replies in both arms ("Not moved: "Git safety"", "Moving the git section: held back", "Git safety: I'd keep this inCLAUDE.md").names-duplicate-id(su-check-flags-duplicate-schedule-id) fails every with-arm run although each FAIL row namesdependency-reviewand says "both rows use the id". Its 0.75 / 0.75 hides a with-arm that did the task while the baseline did nothing.test(instruction-placement): accept held-back phrasings of the git sa…
test(work-items): accept both-rows and same-id duplicate wording
test(playwright): tighten holds-post and posts-before-recheck against…
test(firecrawl): tighten states-drift-verdict against over-broad matches
test(event-storming): tighten offers-markdown against over-broad matches
test(disk-hygiene): tighten unwalked-child-is-coverage-gap against ov…
test(instruction-placement): tighten holds-back-git-safety against ov…
test(work-items): tighten names-duplicate-id against over-broad matches
kyle-sexton commentedon Nov 5, 2026
Part 9 (#13311): re-runs after PR 13433, from origin/main at 7b4a144, Claude Code 2.1.318, tested model claude-opus-5-5, opus judge (no llm grader selected), 3 runs per arm,
-j 1, no-plugin baseline arm, each plugin run from a copy outside git.disk-hygiene,
su-check-reports-floor-and-kill-switch,--allow-tools Bash Write Edit:verdict: VALID (warnings: skill fired). 0 permission denials in 6 traces (previous run: 7, from the clean engine gate on compound Bash lines). The warning is expected: the setup skill setsdisable-model-invocation: true.^MIN_PYTHON), as the setup skill now directs.instruction-placement, all three cases in one pass,
--tag migrate --tag audit --tag delta --scaffold --allow-tools Bash Write Edit:verdict: VALID (warnings: ceiling). 0 permission denials in 18 traces (previous full pass: 2); the skill fired in 9 of 9 with-arm runs.mg-plan-stops-before-writeau-safety-rail-held-backdl-no-baseline-routes-to-auditholds-back-git-safetyagain failed 2 with-arm replies that correctly hold Git safety back ("has to stay inCLAUDE.md"; "Not moved: "Git safety" … keep it inCLAUDE.md"). PR 13433 left this grader at main's pattern on purpose: two verifier rounds found every regex widening also accepted wrong answers. It under-scores the with-arm; it does not make the run valid when it should not be.Part 9 done-when, item by item:
kyle-sexton commentedon Nov 5, 2026
Closing: both done-when items hold, item by item, in issuecomment-6648828965.