You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
test(evals): add eval cases for pointer-swept skills that have none - part 8 of 9 #13310
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Medium
Complexity: Low
Uncertainty: Known
Flags: none
Blocked by: none
Decision recorded. The playwright demo-video row is delivered by #13336; the playbooks boris, repo-sweep and update row is still open: none of the three skills has a plugin eval case on main.
Agent choice, not an owner answer.
[quote truncated; the full decision is at the link]
The decision settles the open questions. The playwright demo-video row is done via #13336. What remains is adding eval cases under plugins/playbooks/evals/ for playbooks:boris, playbooks:repo-sweep and playbooks:update. Each needs a skill-fired grader, deterministic graders where the answer allows, pass and fail samples, and 0 FAIL from validate-cases.py. That is bounded, test-only work with a clear approach.
Decision recorded. The update item of this issue's 21:55Z record is narrowed: its case needs no skill-fired grader, because playbooks:update is not model-invocable on its own, so a slash-invoked case makes no Skill call.
Agent choice, not an owner answer.
Why a skill-fired grader is not the right test for this skill:
plugins/playbooks/skills/update/SKILL.md:5 sets disable-model-invocation: true. The Claude Code skills docs say "Claude can't invoke the skill on its own" with that setting (https://code.claude.com/docs/en/skills#control-who-invokes-a-skill), so the case invokes it by slash command.
plugins/evals/skills/plugin-eval/reference/case-authoring.md:100-106: a prompt that starts with the skill's slash command loads the skill without a Skill tool call, so a tool_used: Skill grader fails and the validity gate marks the run INVALID (seen at Claude Code 2.1.289, 2026-10-04, with a recheck trigger in that file).
The skill-fired wording came from this lane's own record of 2026-11-04T21:55Z, not from the issue body, whose row reads only "playbooks boris, repo-sweep, update".
Replaces the record's update item:
playbooks update: at least one plugin eval case under plugins/playbooks/evals/ that invokes /playbooks:update by slash command, with deterministic graders on what the skill does (no skill-fired grader), pass and fail samples per deterministic grader, and 0 FAIL from validate-cases.py.
The boris and repo-sweep items stand unchanged, skill-fired grader included. The VALID both-arm verdict stays with #13311.
Checked on #13388's head 24c1d69d3c: python3 -I plugins/evals/skills/validate/scripts/validate-cases.py plugins/playbooks/evals exits 0 with 0 FAIL; none of its WARNs is on the three new cases (boris-parallel-sessions-from-reference, repo-sweep-next-stops-on-conflicting-pr, update-check-runs-pack-scripts). PR CI does not run this validator on eval-case changes (it runs only in the eval dispatch workflows), so this run is the evidence for that clause.
Activity
github-actions commentedon Nov 4, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Medium
Complexity: Low
Uncertainty: Known
Flags: none
Blocked by: none
#13310 (comment)
The decision settles the open questions. The playwright
demo-videorow is done via #13336. What remains is adding eval cases underplugins/playbooks/evals/forplaybooks:boris,playbooks:repo-sweepandplaybooks:update. Each needs askill-firedgrader, deterministic graders where the answer allows, pass and fail samples, and 0 FAIL fromvalidate-cases.py. That is bounded, test-only work with a clear approach.Automated triage, applied by the lanes App.
18 remaining items
github-actions commentedon Nov 4, 2026
github-actions commentedon Nov 4, 2026
kyle-sexton commentedon Nov 5, 2026
Decision recorded. The
updateitem of this issue's 21:55Z record is narrowed: its case needs noskill-firedgrader, becauseplaybooks:updateis not model-invocable on its own, so a slash-invoked case makes noSkillcall.Agent choice, not an owner answer.
Why a
skill-firedgrader is not the right test for this skill:plugins/playbooks/skills/update/SKILL.md:5setsdisable-model-invocation: true. The Claude Code skills docs say "Claude can't invoke the skill on its own" with that setting (https://code.claude.com/docs/en/skills#control-who-invokes-a-skill), so the case invokes it by slash command.plugins/evals/skills/plugin-eval/reference/case-authoring.md:100-106: a prompt that starts with the skill's slash command loads the skill without aSkilltool call, so atool_used: Skillgrader fails and the validity gate marks the run INVALID (seen at Claude Code 2.1.289, 2026-10-04, with a recheck trigger in that file).skill-firedwording came from this lane's own record of 2026-11-04T21:55Z, not from the issue body, whose row reads only "playbooksboris,repo-sweep,update".Replaces the record's
updateitem:update: at least one plugin eval case underplugins/playbooks/evals/that invokes/playbooks:updateby slash command, with deterministic graders on what the skill does (noskill-firedgrader), pass and fail samples per deterministic grader, and 0 FAIL fromvalidate-cases.py.The
borisandrepo-sweepitems stand unchanged,skill-firedgrader included. The VALID both-arm verdict stays with #13311.Checked on #13388's head
24c1d69d3c:python3 -I plugins/evals/skills/validate/scripts/validate-cases.py plugins/playbooks/evalsexits 0 with 0 FAIL; none of its WARNs is on the three new cases (boris-parallel-sessions-from-reference,repo-sweep-next-stops-on-conflicting-pr,update-check-runs-pack-scripts). PR CI does not run this validator on eval-case changes (it runs only in the eval dispatch workflows), so this run is the evidence for that clause.