Repository navigation
feat(evals): convert other plugins' evals.json files to claude plugin eval suites #8635
Description
Refs: #5879
Summary
Convert other plugins' skill-creator evals.json files to claude plugin eval suites (case.yaml). This is follow-up 4 of #5879, split out by the owner's decision recorded on 2026-10-12 (#5879 (comment)). #5879 keeps follow-ups 1-3, the judge-calibration fixes in the evals plugin's own suite.
Current state
As of the 2026-10-04 triage (#5879 (comment)), six plugins have case.yaml suites (code-tidying, discovery, education, improvement, planning, source-control), and the repository holds 341 evals.json files. Recount both before planning.
Desired behavior
Plugins with skill-creator evals.json cases gain claude plugin eval suites, converted per plugin, each passing the suite's validity gate.
Next step
This is a cross-plugin migration, so it needs decomposing per plugin (/work-items:decompose) before any slice goes to the lane. Each converted suite needs paid claude plugin eval runs to validate it, and eval spend is the owner's call. So this stays needs-human until the decomposition and the spend are decided.
Out of scope
- Follow-ups 1-3 of feat(evals): close the eval suite's calibration gaps and convert other plugins' evals #5879 (the evals plugin's own judge calibration).
- Changing rubric intent while converting.
Activity
kyle-sexton commentedon Oct 25, 2026
Released for re-triage under the owner's 2026-10-24 rule: an issue stops for a person only for an intent or vision question, a security call, a host or attended run, or spend outside prepaid credit. The hold label came off and intake triage runs again now; it re-applies a hold, with a numbered question, when one of those applies.
To hold this issue again, add
needs-human(orstatus: needs-decision) and say why in a comment.kyle-sexton commentedon Oct 25, 2026
Hold restored: the 2026-10-25 release took
needs-humanoff, but this issue does need a person under the owner's stop rule. Reason (attended): its Done-when needs eval runs, and the issue lane cannot run node, gh or the claude CLI (lane run 40213739706 on #9786).kyle-sexton commentedon Oct 31, 2026
Decision recorded. Replace, staged:
case.yaml(claude plugin eval) becomes the one eval format for skills, and the skill-creatorevals/evals.jsonfiles are converted plugin by plugin, then removed.The owner chose this in host lane 4's session on 2026-10-31, answering "go w/ recommendations" to the question "For #8635, what happens to the 377
evals.jsonfiles now thatclaude plugin evalsuites exist?" The options were replace (staged, recommended), keep both formats, or convert and keep both.State on main (2026-10-31)
plugins/*/skills/*/evals/evals.jsonfiles across 89 plugins.case.yamlsuites, and 45 plugins withevals.jsonhave none./evals:designstill writes newevals.jsonfiles (plugins/evals/skills/design/SKILL.md:177).evals.jsoncase has no no-plugin baseline arm (design/SKILL.md:190).Slices
evals.json:/evals:designemitscase.yamlcases for a skill, and the references toevals.jsonmove to the suite format. This includesskill-criteria.md,/skill-quality:check validate-evalsand the evals skills. Filed below.evals.jsoncases intocase.yamlcases (prompt, expected outcome and graders) and removes the originals, checked with/evals:validate. Filed below.Posted by AI from the owner's answer in that session.
kyle-sexton commentedon Nov 1, 2026
Finding for slice 3, from the intake triage session, not the owner.
The converter has landed (#11720), so slice 3's per-plugin conversions are next. Each conversion removes the plugin's
evals/evals.jsonfiles (plugins/evals/skills/plugin-eval/scripts/convert-evals-json.py:44,:434). Check 14 still fails a new or changed skill that ships noevals/evals.json. It fails with "skill ships no evals/evals.json", required when the skill is new or itsSKILL.mdchanged (plugins/skill-quality/scripts/check-skill.sh:1553-1559).scripts/check-changed-skills.sh:173-177passes--require-evalsfor every skill whoseSKILL.mdchanged against the base and is not listed inscripts/evals-warrant-exemptions.txt. So after a plugin is converted, the next pull request that edits one of itsSKILL.mdfiles fails that check.Check 14 has to accept a
case.yamlsuite case that names the skill before, or in, the first conversion pull request. #11711 noted this needs a rule for how a case names its skill. No slice in the decision lists it, so it goes with slice 3. #11650 does not cover it.kyle-sexton commentedon Nov 1, 2026
Claimed by host-run lane 4 (melo-lap-001, session 827b528f-6a95-44ac-a548-726f5e85fda3), 2026-11-01T04:30Z: the check-14 fix first, so a skill counts as having evals when a plugin
evals/*/case.yamltags its name (the tag the converter writes,convert-evals-json.py:279).fix(skill-quality): read the skill tag from a converted case's prompt…
fix(skill-quality): read prompt.md front matter through a process sub…
26 remaining items
kyle-sexton commentedon Nov 4, 2026
Claimed by host-run lane 4 (session f674da34-2c6a-489c-94cf-8a8887b5023d), 2026-11-04T01:5xZ: group 2, next plugin actionlint (
skills/setup/evals/evals.json, 6 cases, beside the existingsu-*suites; duplicates dropped under checklist item 7). Its PR opens after lane 4's open PR merges.Posted by the lane 4 session. This was generated by AI.
kyle-sexton commentedon Nov 4, 2026
Group 2, actionlint converted and merged in #13051 (a2b2298, 2026-11-04T02:57:07Z): 4 cases converted, 1 merged into su-check-default, 1 dropped as stale; smoke VALID, with 0.93 vs without 0.53. Disclosed and unchanged: the headless-toggle project/local-scope gap (same as eol-normalizer). Legacy
skills/*/evals/evals.jsonleft on main: 330. Next group 2 plugin follows lane 4's #12973 PR.Posted by the lane 4 session. This was generated by AI.
kyle-sexton commentedon Nov 4, 2026
Claimed by host-run lane 4 (session 9d340af9-34ef-4b86-9579-7ea86e71e77e), 2026-11-04T03:0xZ: group 2, next plugin ruff-format (
skills/setup/evals/evals.json, 6 cases, beside the existingck-notice-runs-bundled-checkersuite; duplicates dropped under checklist item 7). Its PR opens after lane 4's open PR (#13060) merges.Posted by the lane 4 session. This was generated by AI.
test(ruff-format): allow a final period in the venv grader, ask about…
kyle-sexton commentedon Nov 4, 2026
Claimed by host-run lane 4 (session 839ddb58-b732-4a71-bb69-79a384292e2e), 2026-11-04T05:1xZ: group 2, next plugin powershell-format (
skills/setup/evals/evals.json, 6 cases, beside the existingck-notice-runs-bundled-checker). Its PR follows the ruff-format PR #13081.kyle-sexton commentedon Nov 4, 2026
Group 2, ruff-format converted and merged in #13081 (e7dbf97, 2026-11-04T05:08:54Z): 6 legacy cases became 7
su-*suites. Smoke VALID, 0.91 with the plugin against a delta of +0.30; the changed Poetry case re-ran VALID at 1.00 against 0.80. Known gaps are listed in the PR body.test(powershell-format): narrow the --help exemption and grade the no…
kyle-sexton commentedon Nov 4, 2026
Group 2, powershell-format converted and merged in #13116 (98c3a04, 2026-11-04T06:48:29Z): 6 legacy cases became 6
su-*suites. Smoke VALID, with the plugin 0.97, delta +0.43, run with--disallow-tools Grep Glob. Follow-ups for this issue: theno-plugin-cli-writepattern in eol-normalizer, actionlint and ruff-format still fails a read-onlyclaude plugin configure --help; #13116 exempts a--helpright after the verb, and later plugins copy that form. A run's Grep of the plugin copy root is denied, and--disallow-tools Grep Globis now the lane's smoke flag where Bash is granted.kyle-sexton commentedon Nov 4, 2026
Claimed by host-run lane 4 (session 839ddb58-b732-4a71-bb69-79a384292e2e), 2026-11-04T07:3xZ: group 2, next plugin bash-format (
skills/setup/evals/evals.json). Its PR follows lane 4's #12908 PR.kyle-sexton commentedon Nov 4, 2026
Group 2, bash-format converted and merged in #13144 (8f5489f, 2026-11-04T07:52:22Z): 7 legacy cases became 7
su-*suites. Smoke VALID, with the plugin 0.97, delta +0.43. Next plugin claimed: go-format, whose PR follows #7573's. Claimed by host-run lane 4 (session 839ddb58-b732-4a71-bb69-79a384292e2e).kyle-sexton commentedon Nov 4, 2026
Group 2, next plugin claimed: typos-format, whose PR follows go-format's #13171. Claimed by host-run lane 4 (session 839ddb58-b732-4a71-bb69-79a384292e2e). Finding for this issue: the shared
no-installmatcher does not catchgo get -toolorgo mod edit -tool=, thego.modroute go-format's setup refuses.test(go-format): catch a single-quoted target in no-config-shell-write
kyle-sexton commentedon Nov 4, 2026
Group 2, go-format converted and merged in #13171 (38de7da, 2026-11-04T09:04:59Z): 7 legacy cases became 7
su-*suites. Smoke VALID, with the plugin 1.00, delta +0.31.kyle-sexton commentedon Nov 4, 2026
Group 2, next plugin claimed: biome-format, whose PR follows typos-format's #13175. Claimed by host-run lane 4 (session 839ddb58-b732-4a71-bb69-79a384292e2e).
kyle-sexton commentedon Nov 4, 2026
Group 2, typos-format converted and merged in #13175 (ead773c, 2026-11-04T09:18:52Z): 7 legacy cases became 7
su-*suites. Smoke VALID, with the plugin 0.95, delta +0.26.