Repository navigation
skill-quality: skill authoring, quality and catalog #7582
Description
Parent
Refs #7555. Source: PLAN Phase 14, topic poteto-review. Thread: Skill authoring, quality and
catalog.
Depends on
- config-cascade: settings resolution and convention home #7571 Settings resolution and convention home (description target and warn settings)
Outcome
Skills follow Anthropic's authoring guidance as measured rules: a SKILL.md router with actions,
evals on all skills, a description standard with probe-proven triggers, consolidated *:check
skills, and catalog files generated in the release PR.
Decisions
- Skills follow Anthropic's authoring guidance: SKILL.md a router under 5k tokens,
actions/per
action, evals on all skills, split only by exception, rollout across every skill (D107, Q203).
Origin: ours (owner reworked the size options). - Description standard: one what-sentence, "Use when" with one trigger per branch, at most three
probe-proven hook terms, target 250, warn above 419, fail at 1,024; check 12 drops the single-quote
rule (D82, Q183, Q184). Settings: description target 250; warn 419 (1,024 fixed). Check 3's
unquoted-trigger tracking lands first, in the same PR, with a red-then-green test; two attempts
ranked on the session-flow trim-pilot diff (B2). - Consolidate the
*:checkskills (recount first: 29), thenarchitecture:map-*and
code-metrics:audit-*; fix the Cursor/checkcollision by consolidation plus distinct first
clauses (D82, Q179, Q186; owner observed the collision). - Stack-specific tools: detect, research live, interview, write setup into the project; use an
existing tool without an interview (D66, Q96). - Bare invocation: infer the action; read-only proceeds and names what it inferred (setting
proceed); state changes propose and wait (fixed floor); unattended state changes stop and report
(D79, Q93). - Invocation convention: any automation starter means model-invocable, safety in body gates and
scripts; user-invoked only for starved skills no automation starts (D53, Q138). - Skill criteria gain "choose leading words on purpose" with
--baseline; write-for-agents gains a
leading-words section, a sentence-level no-op test, the context-boundary line (D4, Q525). - Don't wrap skill-creator; add a tested route for cutting text (D7).
- Files, not registry entries; catalog, cheat-sheet and marketplace manifest generated in the release
PR; problem-first section above the generated cheat-sheet block (Q143, Q144, Q146, Q546;
Lauren's files-not-registry rule). - Shared copies stay copies; one shared-copies record (D83, Q180). Plugin READMEs use one install form
or point to the root README (Q526). Re-audit shipped skills on each model release (D6; measured gap:
median 1,477 words versus Matt 506 and pstack 364). - Probes on this thread's slices: four invocation probes, routine first (D53); picker label
re-observation before leaf-name registry removal (Q179); skill-visibility measurement gates
Q183-Q187 (Q137, Q182); trim pilot on session-flow, non-inferiority with a 5-point margin (Q185);
each consolidation merge probed and the recount (Q186); hook terms probe-proven (Q183); hand-off
phrasing eval, 12 cases, 3 arms, 3 runs (D85, Q125, Q188); steering words with--baseline(D4);
each cut tested with and without removed text (D6, D107);skillListingBudgetFractionstays 0.07
until the pilot ships (Q187). - Sub-topic promotion: this thread gets its own plan at phase start (B7).
Key interfaces
From the design's module layout: "skill-quality: check 12 drops the single-quote rule; description
standard (target 250, warn 419, fail 1,024). *:check skills consolidate after a recount, then
architecture:map-* and code-metrics:audit-*." Settings: "Description target 250, warn 419 (1,024
fixed). ... Bare run read-only proceed."
Done when
- The check-skill test suite exits 0 and includes a case asserting check 3 warns on a dropped
unquoted trigger (no such case on main). -
rg -c "not single-quoted" plugins/skill-quality/scripts/check-skill.shreturns 0 (nonzero on
main), andbash plugins/skill-quality/scripts/check-skill.sh plugins/session-flow/skillsexits 0. - Every slice sub-issue is closed by a merged PR, each with its probe or eval result.
Out of scope
- Setting the fleet listing budget (housekeeping thread, after the pilot).
Related
- Concise skills rollout: the 15 check-and-setup plugins #6619 concise skills rollout, check-and-setup plugins (stays under Concise skills: one skill-creation standard, evals, and a fleet-wide rollout #6476; linked, not moved).
- Concise skills rollout: playwright (3 skills, 2 model-invoked) #6587 concise skills rollout, playwright (stays under Concise skills: one skill-creation standard, evals, and a fleet-wide rollout #6476; linked, not moved).
Activity
github-actions commentedon Oct 16, 2026
Proposed work class: work-class: structural
Basis: automated intake triage; the lanes App applied this class.
Size: Large
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
Understood: this is the parent tracker for skill authoring, quality and catalog work. The slices are mostly delivered. #11957 and #11960 are still open, and the
*:check,architecture:map-*andcode-metrics:audit-*consolidation slices are not filed yet.#7582 (comment)
That decision settles the open questions. The two exit-code checks and the host-run probes are for a host run, not a person decision.
Automated triage, applied by the lanes App.
kyle-sexton commentedon Oct 24, 2026
Released for re-triage under the owner's 2026-10-24 rule: an issue stops for a person only for an intent or vision question, a security call, a host or attended run, or spend outside prepaid credit. The hold label came off and intake triage runs again now; it re-applies a hold, with a numbered question, when one of those applies.
To hold this issue again, add
needs-human(orstatus: needs-decision) and say why in a comment.34 remaining items
kyle-sexton commentedon Nov 2, 2026
Lane 1 (hub assignment 2026-11-02 17:2xZ) took #7582. Criteria 1 and 2 of the acceptance record (issuecomment-6582916179) are met on main b000bb5. No file under plugins/skill-quality, plugins/session-flow or plugins/playbooks/skills/skill-authoring differs from origin/main 57477a9.
bash plugins/skill-quality/scripts/check-skill.test.sh: rc=0, 302 ok, 0 not ok. Case 9a (check-skill.test.sh:442-472) passes.bash plugins/skill-quality/scripts/check-skill.sh plugins/session-flow/skills: rc=0, 19 passed, 0 failed.Criteria 3 and 4 are next. Before the consolidation waves can be filed, the target they consolidate into (D82/Q179/Q186) has to be found: no durable copy is in this repo, in architecture owner-decisions.md, or in Epic: absorb Lauren Tan's and Matt Pocock's agent workflows (poteto-review) #7555 or docs(invocation-mode): record the invocation probes and narrow the scheduled-task rule (#11960) #12056. Lane 1 has asked the owner for it.
kyle-sexton commentedon Nov 2, 2026
Slices filed for the
*:checkconsolidation, the Cursor/checkcollision fix, and thearchitecture:map-*andcode-metrics:audit-*consolidations. All are sub-issues of this issue, labeledhost-run, with native blocked-by links.lib/prerequisites.mjs, not a/<plugin>:checkskill*:checkwave 1: 14 hook-runtime checks*:checkwave 2: 8 formatter and linter checks*:checkwave 3: 6 external-tool checks and the 4check-prerequisitesskillsinstruction-placement:check,skill-quality:checkandtoolchain:check, so nocheckleaf remains (Q179 picker re-observation)architecture:map-*consolidation; first item is a sized wave plan, and the target is an open questioncode-metrics:audit-*consolidation; first item is a sized wave plan, and the target is an open questionWhy the contract is its own slice:
docs/contracts/prerequisites/prerequisites.schema.json:36,63requires each entry'scheckand restricts it to a slash command.lib/prerequisites.mjs:207-208rejects any other value, and:520and:719print it in the notice.All 35 plugins with a
checkorcheck-prerequisitesskill already ship aprerequisites.jsonandlib/prerequisites.mjs(origin/main b000bb5). Each wave still adds one, per the schema, for any plugin that lacks one when the wave starts.Posted by a lane 1 worker session. This was generated by AI.
kyle-sexton commentedon Nov 4, 2026
Status. One item is left before this issue closes: the D4 steering-words probe.
Agent choice, not an owner answer.
The other Done-when rows were checked against origin/main c19e5ed:
bash plugins/skill-quality/scripts/check-skill.test.shexited 0 with 309 ok and 0 not ok, including the check-3 dropped unquoted Use-when case.not single-quotedno longer appears incheck-skill.sh.check-skill.sh plugins/session-flow/skillsexited 0.The one gap is #8645. It closed through #9597, and that PR's "Not verified" list says: "Eval case 10 was not executed against a model, only linted". Case 10 is
plugins/docs-hygiene/skills/write-for-agents/evals/evals.json(leading word proposed, no-op sentence deleted whole). No--baselinerun of it is recorded on this issue or its slices, so row 3 ("each with its probe or eval result") is not met for #8645./evals:plugin-eval --baseline) and post the result here.The body's consolidation wording was re-scoped by owner answers, and those decisions govern:
/checkrenames were dropped (#12489, not planned).architecture:map-*andcode-metrics:audit-*keep their leaf skills instead of merging into one router (#12490).kyle-sexton commentedon Nov 4, 2026
Status amendment. This amends the open item in the status above: case 10 has to be converted before
/evals:plugin-evalcan run it.Agent choice, not an owner answer.
On origin/main the case exists only in the legacy format:
plugins/docs-hygiene/skills/write-for-agents/evals/evals.json, with fixturesdraft-rule.mdanddraft-rule-leading-words.md. No converted case for it is underplugins/docs-hygiene/evals/; a grep for "leading word" and "no-op sentence" there finds nothing.claude plugin evalreadscase.yamlsuites, and the evals plugin's own description says it is not for the skill-creator JSON eval format. Converting these files is #8635's scope (laptop Lane 4).The lane that runs the item should first convert case 10 into a
plugins/docs-hygiene/evals/<case>/suite case (prompt.md,case.yaml, graders), coordinating with Lane 4 so #8635 does not convert it twice. It then runs that case with and without the D4 text and posts the result here.kyle-sexton commentedon Nov 4, 2026
Eval result for the D4 steering-words text (#8645, from #9597): VALID. The leading-word behavior shows only with the text. This was the one Done-when gap left in the status record.
What ran
evals.jsoncase 10 (behavior-leading-word-and-no-op-sentence), converted withconvert-evals-json.py. Its fixture was seeded into the workspace by a scaffold script as.claude/rules/testing.md, with the same text asdraft-rule-leading-words.md, because the converter leaves fixtures unrouted.## Leading wordsand## The no-op testsections and the "Hiding later steps" bullet.claude plugin eval <copy> --case write-for-agents-behavior-leading-word-and-no-op-sentence --ablation none --runs 3 -j 1 --scaffold --trust-plugin --keep-temp --judge-model opus --threshold 0.8, Claude Code 2.1.318, model under testclaude-opus-5-5, read-only tools, 2026-11-04 16:51Z to 16:54Z.run-validity.py --runs 3 --scaffolded:verdict: VALIDfor both runs, with no denials, skill fired in 6 of 6, and no split judge votes.Result
Cost (list-price estimates, judge included): about 1.41 USD for the after run and about 1.35 USD before judge calls for the before run. That is about 0.47 USD per run, over the 0.2 anchor.
Read:
noise-report.py --baselinereadscompare verdict: n too small to call, since one case cannot give an interval. So this records that the probe ran and what it showed, not a measured gain.Spot-read: I checked the judge votes against the outputs, and they hold up.
The judge graders are uncalibrated: their samples are the converter's single pass and fail answers.
Done-when
*:checkand description item has its probe or eval result. For docs(playbooks): add leading words, the no-op test and the context-boundary line to the authoring guidance #8645, this comment is the result.All 17 sub-issues are closed, and rows 1 and 2 were verified at c19e5ed in the status record, so this closes the issue.