Repository navigation
test(architecture): give four map-* skills probes a description cut can fail, then trim them #13050
Description
Refs: #12490
Problem
#13011 shortened five architecture:map-* descriptions under the option-3 scope of #12490. It kept the before text for four, because their negative controls detected no loss: map-data, map-events, map-flow and map-deployment (the map-deployment drop did not reproduce). On the held-out set, a generic description lost no request for these four, so the probes cannot show whether a cut is safe. map-data (473 characters) and map-flow (468) stay over the 419-character warn line. The PR body's results table and disclosures hold the numbers: #13011
plugins/skill-quality/reference/invocation-probes.md ("Description controls") names the cause for a similar case: no competitor took the skill's should-trigger probes even against the generic description, so the skill won them by elimination.
Done when
- Each of the four skills has should-trigger probes that a sibling could plausibly take, written blind from the before description and frozen before any cut. A generic-description control on them shows a reproduced
case dropor a mean fall of 0.15 or more. - Each description is then rewritten under 419 characters and measured as refactor(architecture): give the nine map-* skills distinct, trimmed descriptions #13011 was: two-labeler meaning diff before any paid run, held-out probes on Opus, and the pre-registered no-loss rule. A part whose control still detects nothing keeps its before text.
Related
- refactor(architecture): give the nine map-* skills distinct, trimmed descriptions #12490 (closed by refactor(architecture): give the nine map-* skills distinct, trimmed descriptions #13011), refactor(code-metrics): give the six audit-* skills distinct, trimmed descriptions #12491 (the code-metrics counterpart).
Filed by host-run Lane 2. This was generated by AI.
Activity
github-actions commentedon Nov 4, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
Understood: the map-data, map-events, map-flow and map-deployment descriptions kept their before text in #13011 because their negative controls detected no loss. The request is to write new should-trigger probes that a sibling skill could plausibly take, and freeze them before any cut. After that, trim each description under 419 characters and measure it the way #13011 did. A skill whose control still detects nothing keeps its before text.
Some done-when steps are paid Opus runs and labeler passes. Admission will route those to a host run. No open duplicate found.
Automated triage, applied by the lanes App.
github-actions commentedon Nov 4, 2026
github-actions commentedon Nov 4, 2026
melodic-automation-lanes commentedon Nov 4, 2026
The work-resolve-issue lane stopped on this issue (
host-only): the run reports the work needs a host run the lane cannot make.It added
host-runand removedagent-ready: an agent run on a host builds it, and it waits on no decision from a person.In the run's words:
What the run says it did not do:
Run: https://github.com/melodic-software/claude-code-plugins/actions/runs/41762336755
To hand it back to the lane, remove
host-runand addagent-readyagain.kyle-sexton commentedon Nov 4, 2026
Claim. Host-run lane 1 (melo-lap-001) is taking this now, on the hub's H19 order.
kyle-sexton commentedon Nov 4, 2026
Preregistration, before any paid run.
Why #13011's controls could not fail. Every probe file's
competitors[]named onlyarchitecture:*skills, so each emitted case loaded only the architecture plugin. This set adds rivals from other plugins to the listing, so the skills no longer win by elimination.Probes. Frozen at commit e25fa86 on
test-13050-map-probes. A fresh agent wrote them blind from the before descriptions and the rival listings, with no tools and no repository access: 6 should-trigger (pos-contested-*) and 3 near-miss (neg-contested-*) per skill. It was told not to use the skill's own name words. One map-deployment probe had "AKS" replaced by "Kubernetes" before any run, because the typos check flags it. No probe changes after this comment. A second round, if any, is one new blind author under a dated addendum written before its run.Rivals added.
Step 1, detection (144 model runs). Arms: before (
origin/main284b530) and one control per skill, identical but for that skill's description, replaced by"Helper for repository maintenance tasks.". Per skill, before and control launch together on the 6 should-trigger cases.--model opus --runs 3 --ablation none --keep-temp --disallow-tools Read Glob Grep Agent, Claude Code 2.1.318.A skill's control detects when, on those 6 cases, either:
noise-report.pyreports acase dropthat reproduces: a fresh before and control pair, launched together, 3 runs each on each dropped case, reports a drop on the same case.The margin 0.15 and the drop rule are the issue's and
plugins/skill-quality/reference/invocation-probes.md's, fixed before any data. A skill whose control does not detect keeps its before text, and that is a result: its probes stay and no rewrite runs.Step 2, only for skills that detect.
probes/or any emitted case./docs-hygiene:compress compareruns with two labelers until both returnVERDICT: PASS, before any paid run.Decision per skill, as #13011.
case dropwithout a measured cause.Validity. Each pass is VALID per
run-validity.pywith--runsequal to the actual rows. The skill-fired check is not applied, because it is the measured quantity. A denial or turn cap after the graded Skill call does not invalidate a run. A usage-limit, auth or crash error is rerun as a fresh pair. No other exclusions.Host limits. At most 4 of this lane's runs at once, under the host's cap of 8 across lanes.
kyle-sexton commentedon Nov 4, 2026
Result: the controls detect nothing, so all four skills keep their before text. Decided by the preregistration, step 1.
144 runs, Opus 5.5 in every trace, Claude Code 2.1.318. Every result file reads VALID, with the
modelswarning only because no judge model is recorded, and these cases use no judge.Checked before accepting 144 of 144:
description:line.argument-hintandmetadata.summaryremain, and neither is in the listing: the listing holds "skill names and descriptions" (Skill descriptions are cut short),argument-hintis "shown during autocomplete", and formetadata"Claude Code doesn't act on its contents" (frontmatter reference). The five skills whose controls did detect in refactor(architecture): give the nine map-* skills distinct, trimmed descriptions #13011 carry equally specific summaries.Done-when status.
No second probe round. The preregistration allowed one. With 72 of 72 control runs firing against real rivals, more rewording would only fit probes to the outcome.
Cause: UNVERIFIED. The listing always keeps a skill's name, and the generic description leaves that in place, but nothing here tested whether the name does the work.
Decisions (agent, not the owner): rivals and probes stay committed;
plugins/skill-quality/reference/invocation-probes.mdrecords these runs, so its advice to write contested probes is not read as sufficient.docs(skill-quality): record that contested probes did not make four c…
kyle-sexton commentedon Nov 4, 2026
Closing as not planned, not completed. #13088 merged the frozen probes and the record of these runs. Done-when 1 is unmet: the probes exist, but no control showed a reproduced drop or a fall of 0.15. Done-when 2: no rewrite, under its clause that a part whose control still detects nothing keeps its before text. The preregistration allowed one more probe round; I skipped it (judgment), so this does not prove the bar is unreachable. The owner reviews this closure in the morning and can reopen it.
kyle-sexton commentedon Nov 4, 2026
Owner decision, 2026-11-04: this issue stays closed as not planned, as recommended ("go w/ recommendations", with the standing direction). Done-when 1 stays unmet, as the closing comment records.