Repository navigation
test(evals): add eval cases for pointer-swept skills that have none #13300
Description
Slice of #6405, carried over from #7296 (eval runs owed after the upstream-pointer sweep, PRs #7061-#7274).
Problem
The sweep replaced restated Claude Code docs text with pointers in these skills. None of them has a plugin eval case that targets it, by a Skill input_match or by tag, on origin/main at 90d3c6962e. So nothing checks that they still do their job:
- instruction-placement
migrate,audit,delta - work-items
setup - autonomy
setup - disk-hygiene
audit,check,setup - skill-quality
setup - bugs
setup - harness-memory
audit,stateless - docs-naming
generate-file-name-gate - event-storming
methodology,simulation - firecrawl
update - github
advise - code-metrics
principles - mutation-testing
audit - playwright
demo-video - playbooks
boris,repo-sweep,update
Change
For each skill, add at least one case that tests its main behavior. Ship one plugin per PR.
- Each case has a
skill-firedgrader plus deterministic graders where the answer allows them. - Each case has pass and fail samples, and
validate-cases.pyreports 0 FAIL.
Done when
- Every skill above has a case on main.
- Each plugin's new cases have one run with a VALID both-arm verdict, recorded on this issue. Any regression found goes in its own bug issue.
Activity
kyle-sexton commentedon Nov 4, 2026
Scope amendment.
disk-hygienecheckcomes off the list. main has no such skill:origin/mainhas onlyplugins/disk-hygiene/skills/{audit,clean,setup}, because the*:checkconsolidation waves moved each plugin's check into itssetupskill. Its check action is covered by thedisk-hygienesetupcase this issue already requires.Done-when row 1 now covers the remaining 22 skills.
github-actions commentedon Nov 4, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Large
Complexity: Medium
Uncertainty: Known
Flags: none
Blocked by: none
I understood this as: add at least one plugin eval case (a
skill-firedgrader plus deterministic graders, with pass and fail samples, and 0 FAIL fromvalidate-cases.py) for each of the 22 pointer-swept skills that have none. The work ships one plugin per PR. The scope amendment dropsdisk-hygienecheck, becausesetupnow covers it. The done-when also asks for one VALID both-arm eval run per plugin, recorded here. That part needs eval runs, so a maintainer or a host run will have to do it.Because it spans about 15 plugins, the work should be split into one issue or PR per plugin. Related, but not duplicates: #7296 (running the deferred eval suites), #6405 (the parent slice) and #13298 (plain prompts missing two skills).
Automated triage, applied by the lanes App.
melodic-automation-lanes commentedon Nov 4, 2026
Not admitted to the issue lane yet: sub-issue #13311 is open.
melodic-automation-lanes commentedon Nov 4, 2026
Split at intake (owner Q93). This issue is Size Large, so its done-when is split into 9 sub-issues, each holding its task-list lines as written.
Automated split by the lanes App. Admission waits on this issue while a sub-issue is open; work-unblock closes it as completed once every sub-issue is closed as completed by a merge or a listed human and its done-when is covered.
kyle-sexton commentedon Nov 4, 2026
Part 4 (#13306): VALID both-arm runs on Claude Code 2.1.318, opus judge, 3 runs per arm, each plugin run from a copy outside git:
su-check-default-rootsu-check-missing-root-routes-to-configuresu-check-untracked-team-filesu-check-zero-config-is-inforun-validity.pyreads VALID for both plugins. Its one warning is that notool_used: Skillgrader exists. Both setup skills setdisable-model-invocation: true, so a case invokes them by slash command and noSkillcall happens. Each case uses awith-onlygrader on skill-only content instead.Skill-grader limit.kyle-sexton commentedon Nov 4, 2026
Part 7 (#13309): VALID both-arm runs on Claude Code 2.1.318, opus judge, 3 runs per arm, each plugin run from a copy outside git:
advise-ruleset-walkthroughadvise-set-it-up-stays-proposedprinciples-cyclomatic-22-answered-with-provenanceprinciples-coverage-no-universal-targetau-dirty-target-stopsau-no-test-command-asksrun-validity.pyreads VALID for every run.advise-ruleset-walkthroughcarries a warning for one with-armGlobdenied on the user's.claudedirectory, outside the plugin.principles-coverage-no-universal-target(tightened) andau-no-test-command-asks(widened, for a false negative on a correct reply; its 0.89 is before that fix). Both were regraded offline over the recorded replies: only that false negative changes verdict. Neither final pattern has had its own paid run.kyle-sexton commentedon Nov 4, 2026
Eval record for #13307 (part 5 of 9): harness-memory and docs-naming. Each plugin's new cases have one run with a VALID both-arm verdict.
Setup for both: Claude Code 2.1.318,
claude-opus-5-5, judgeopus,--runs 3 -j 4 --scaffold --allow-tools Bash Write Edit, default with/without ablation, deterministic graders only. Each plugin was copied outside git.run-validity.pynoise-reportgenerate-file-name-gate)audit,stateless)stateless-env-zero-beats-settingscores 1.00 without the plugin and is excluded, so the delta over the other 2 cases is +0.36).claude/, which the sandbox protects, and one with-arm run made a scratch copy outside the repository. Each prompt now rules that out.ccf3966a,3c1cb00e1aae4810,c18c84abmelodic-automation-lanes commentedon Nov 5, 2026
Every sub-issue of this split is closed as completed by a listed human or a merge (#13303, #13304, #13305, #13306, #13307, #13308, #13309, #13310, #13311), so the lanes App closes it as completed.