Repository navigation
Concise skills rollout: domain-driven-design (1 skills, 1 model-invoked) #6604
Description
Part of #6476. Rewrite domain-driven-design's skills to the skill criteria in one draft PR, following the per-plugin runbook (/playbooks:skill-authoring, reference/rewrite-a-plugin.md). The runbook and the rewrite protocol it cites decide what ships; this issue holds only what is specific to the plugin.
Path: description-only, measured by trigger probes (owner decision on #6476, 2026-10-18: the 1-skill tail takes the description-only path). The body is not rewritten.
The plugin today
As of origin/main on 2026-10-24:
- Skills:
curate-language(model-invoked). - Outcome eval case directories under the plugin: 0. Probe files under
probes/: 0.
Done when
-
git diff origin/main -- plugins/domain-driven-design/skillschanges no line outside thedescriptionkey ofplugins/domain-driven-design/skills/curate-language/SKILL.md. -
plugins/domain-driven-design/probes/curate-language.jsonexists in the PR. - The PR body shows
/skill-quality:check measure-invocationtrigger scores for the before and the after description on the same probes. - The after description scores at least the before score, or the PR ships the before description and says so.
-
/skill-quality:check plugins/domain-driven-designon the PR head reports no FAIL that the run on main does not. - The plugin's version or changelog entry is in the PR, per the plugin's release mode.
Eval spend and review
- Eval passes within prepaid API credit are pre-approved (owner decision Q46, 2026-10-24), at the
--max-cost-usd 5per-pass ceiling set on #6476 on 2026-10-18. - No person gates the PR: the owner's 2026-10-24 decision (C12) leaves only an intent mismatch during the build or a security issue at review as stops.
Generated by Claude Code
Metadata
Metadata
Assignees
Labels
Type
Fields
No fields configured for Task.
Activity
github-actions commentedon Oct 7, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Small
Complexity: Low
Uncertainty: Known
Flags: none
Blocked by: none
Understood: a description-only rewrite of the
curate-languageskill indomain-driven-design. It also adds a probe file, a before/after trigger-score comparison in the PR body, and a version or changelog entry. The scope and done-when list are clear, and the eval spend is already pre-approved in the issue. No duplicates found among the open issues. It is part of #6476.Automated triage, applied by the lanes App.
kyle-sexton commentedon Oct 24, 2026
Released by the owner's HITL session (2026-10-24, C12): it waited only on a removed gate. Intake triage re-runs on it to set its fields, then admission takes it.
github-actions commentedon Oct 25, 2026
The triage sweep dispatched intake triage for this issue (attempt 1 of 2).
6 remaining items
kyle-sexton commentedon Oct 31, 2026
Host run. Description-only trigger screen for
domain-driven-design:curate-language: before and after copies launched together, frozen probes.What ran
a781adf89a, modelsclaude-opus-5-5andclaude-sonnet-5-5,--ablation none,-j 2, cases frommeasure-invocation.sh emit-plugin-eval, 20 probes (10 positive, 10 negative), digest5977464f6fe4.--runs 1screen, then--runs 3, where after must score at least before on every model (the done-when), or the before description ships.claude plugin eval <copy> --eval-dir trigger --model <model> --ablation none --runs N -j 2 --keep-temp --max-cost-usd 5, each in a capped scope./home/kyle/worktrees/host-run-tail-eval/out/<label>-before.jsonand-after.json, labelss1-domain-driven-design-opus,s1-domain-driven-design-sonnet. Rule:PREREG.mdbeside them. Frozen probes:/home/kyle/worktrees/host-run-tail-eval/probes/domain-driven-design/curate-language.json.Done-when
--runs 3gate ran./skill-quality:check, no new FAIL: not covered: no PR head.Scores (hits on both polarities, over probes x runs)
Result
Passed the screen. No
--runs 3gate ran inside the time box, so the description is unchanged on main and nothing ships yet. Remaining: the gate, then the PR with the probe file.Caveats
run-validity.pyhas no trigger mode: a missed positive prints INVALID onskill fired, and an ungranted-toolNo such tool availableerror prints a denial. Both were read per an addendum written before any--runs 3pass, the same for both versions. That addendum fails a pass on a denial of a granted tool, and every other row had to pass.Target labels: add none; remove none.
kyle-sexton commentedon Nov 1, 2026
kyle-sexton commentedon Nov 1, 2026
This was generated by AI.
Progress: #11828 merged (dd26f63). It adds five outcome eval cases (27 deterministic graders) and 18 frozen trigger probes for the
domain-driven-designcurate-languageskill, and tightens one grader that passed a merged Claimant definition. No skill text changed. Its smoke pass was INVALID (one sandbox denial of a plugin-data write), so its scores are a signal, not a measurement.This issue stays open. The concise rewrite and its measured passes wait on the eval validity fixes in #11618 and #11628. The rewrite should also reword
cl-existing-conventionso "never the insured organization" reads unambiguously as a rejected name (see the PR body). The frozen suite digest for those runs is0ecb152f77dd79791ff9d3abb5e4c6371a637cb5(plugins/domain-driven-design/evalsat e44363c). The claim on this issue is released.kyle-sexton commentedon Nov 1, 2026
kyle-sexton commentedon Nov 2, 2026
Reopened (lane 3): the #12114 description cut failed a retroactive negative control, so it is being reverted.
Owner rule (2026-11-02): a cut ships only if its suite scores a deliberately broken copy clearly below the before version. #12114 merged before the rule existed.
Setup: main's
curate-languagedescription against"Helper for repository maintenance tasks."(522 → 56 characters), on Sonnet 5.5. Both versions were launched together on the round's trigger sets with the round's flags, and every pass was VALID.The frozen set's one case drop was a single run on
neg-train-05. It did not reproduce in a confirmation pair launched together, which was quiet 3/3 on both versions.FAIL. A description that says nothing about glossaries fired on every should-trigger probe, so the suite cannot tell the shipped text from a generic one, and the round's no-loss result is no evidence the cut is safe.
Next: a revert PR restores the earlier description. This issue stays open for a routing suite with competing skills loaded; with one plugin installed, nothing competes for the queries. Cost of the control: 10.40 USD (list-price estimate, the sum of
costUsd).kyle-sexton commentedon Nov 2, 2026
This was generated by AI.
Negative control with competitor plugins loaded (#12249): FAIL
Part: domain-driven-design
curate-languagedescription (#6604).Copies.
main:git archiveof origin/mainplugins/at cf60764 (#12300 merged).broken:the same, with only the skill's
description:line replaced by"Helper for repository maintenance tasks.".diff -r main brokenshows that one line only, beforeand after emission.
Suite. The 18 frozen probes in
probes/curate-language.json(9 should-trigger, 9 should-not),emitted per copy with the merged
measure-invocation.sh emit-plugin-eval. Every case'splugins:line names domain-driven-design and its competitors' plugins: planning, naming, docs-hygiene,
architecture. The emitted cases are byte-identical in both copies. Main and broken were launched
together on claude-sonnet-5-5 (Claude Code 2.1.316), each with
--runs 3 --ablation none --disallow-tools Read Glob Grep Agent -j 2. The rule was preregisteredbefore any paid run.
Loading. All 54 with-arm init lines on each side list exactly domain-driven-design, planning,
naming, docs-hygiene and architecture, with no
plugin_errors. They also list two Claude Codebuilt-ins (
cc-plugin-agents-md,cc-plugin-plugin-authoring), the same on both sides.run-validity's
plugins loadedcheck passes on every pass.Noise report, broken against main (18 cases, margin 0.05):
with-arm change (this minus baseline), paired over 18 cases: -0.02, 95% interval -0.05 to +0.02case drop: case domain-driven-design-curate-language-neg-train-01 with-arm fell from 1.00 to 0.67, a third or more; name its cause and prove it before shippingcompare verdict: not shown non-inferior at margin 0.05 (the interval's lower end -0.05 is below -0.05), and 1 case(s) dropped by a third or moreThe case drop did not reproduce. It rests on one run. In run 3 of should-not probe
neg-train-01("read me the glossary entry"), the broken copy searched for Read, Grep and Glob andfound none. It then called the skill as a way to read the file, and told the user it had no
file-reading tool. The trace shows no denial or run error. The preregistered confirmation pair (that
case alone, launched together, 3 runs) had both sides quiet in 3 of 3 runs, both VALID.
Result: FAIL. With its competitors loaded, the generic description still fired on every
should-trigger probe in every run: 27 of 27, the same as the shipped description. The mean fell by
0.02, short of the 0.15 the rule needs, and the only case drop did not reproduce. The suite still
cannot tell the shipped description from a generic one, so the #6604 cut remains unmeasurable.
Cost: 7.27 USD (list-price estimate, sum of
costUsdover 4 passes, confirmation included).Next for this issue: loading the probe file's competitors was not enough: every should-trigger probe still routes to the generic description. The probe set needs queries that a neighbouring skill (planning, naming, docs-hygiene, architecture) could plausibly take, frozen before any description cut.
kyle-sexton commentedon Nov 2, 2026
This was generated by AI.
Negative control rerun with knowledge loaded (#12249): FAIL
Part: domain-driven-design
curate-languagedescription (#6604), rerun because the probe file nowlists
knowledge:book-distillas a competitor, so every emitted case loads the knowledge plugin too.Copies.
main: theplugins/tree of branchfix/12249-probe-evals-dir(8e47b21 plus theprobe-file edit that adds the knowledge competitor), copied as is.
broken: the same, with only theskill's
description:line replaced by"Helper for repository maintenance tasks.".diff -r main brokenshows that one line only, before and after emission.Suite. The 18 frozen probes in
probes/curate-language.json(9 should-trigger, 9 should-not),emitted per copy with that copy's
measure-invocation.sh emit-plugin-eval. Every case'splugins:line names domain-driven-design, architecture, docs-hygiene, knowledge, naming and planning. Main
and broken were launched together on claude-sonnet-5-5 (Claude Code 2.1.316), each with
--runs 3 --ablation none --disallow-tools Read Glob Grep Agent -j 2. The rule (unchanged from thefirst run) was amended into the preregistration before this run.
Loading. All 54 with-arm init lines on each side list exactly architecture, docs-hygiene,
domain-driven-design, knowledge, naming and planning, with no
plugin_errors, plus the same twoClaude Code built-ins on both sides (
cc-plugin-agents-md,cc-plugin-plugin-authoring).run-validity (origin/main bcd36ec) passes
plugins loadedandpermission denialson bothpasses.
Noise report, broken against main (18 cases, margin 0.05):
with-arm change (this minus baseline), paired over 18 cases: +0.00, 95% interval +0.00 to +0.00compare verdict: non-inferior at margin 0.05 (the interval's lower end +0.00 is at or above -0.05)No case dropped, so no confirmation pair was needed.
Competitors. No should-trigger run on either side sent the probe to a competitor: in all 54,
curate-language was the first skill called. In one main run of
pos-val-04the model also called thebuilt-in
update-configafter curate-language. Competitors took near-miss should-not probes on bothsides:
naming:name-it-betteron the helper-naming probe (main 2 of 3 runs, broken 1 of 3),docs-hygiene:rename-referenceson the leftover-references probe (3 of 3 each), andplanning:designon the type-design probe (3 of 3 each).knowledge:book-distillwas not called inany run.
Result: FAIL. With knowledge loaded beside the other competitors, the generic description still
fired on every should-trigger probe in every run (27 of 27), the same as the shipped description, and
both stayed quiet on every should-not probe. The mean did not move (rule needs -0.15) and no case
dropped. The suite still cannot tell the shipped description from a generic one, so the #6604 cut
remains unmeasurable.
Cost: 7.51 USD (list-price estimate, sum of
costUsdover 2 passes: main 3.96, broken 3.55).kyle-sexton commentedon Nov 2, 2026
Closed under the owner's Q2 decision, which stops the concise-rewrite series: #6476 (comment). The curate-language description cut was reverted in #12247; the description stayed unmeasurable even with competitor plugins loaded.