Skip to content

test(architecture): add contested trigger probes for four map skills - #13088

Merged
kyle-sexton merged 4 commits into
mainfrom
test-13050-map-probes
Nov 4, 2026
Merged

kyle-sexton merged 4 commits into
mainfrom
test-13050-map-probes

Conversation

@kyle-sexton

Copy link
Copy Markdown
Member

Refs: #13050

Summary

Adds contested trigger probes for four architecture:map-* skills and records what they measured. No description changes: the generic-description controls detected no loss on any of the four, so map-data, map-events, map-flow and map-deployment keep their before text under the issue's own rule.

Fix

  • plugins/architecture/probes/: 6 should-trigger and 3 near-miss probes per skill, written blind from the before descriptions and frozen before any run. Each probe file's competitors[] gains cross-plugin rivals, so a live run loads them. refactor(architecture): give the nine map-* skills distinct, trimmed descriptions #13011's lists named only architecture:* skills.
  • plugins/skill-quality/reference/invocation-probes.md: records these runs, because the page's advice to write contested probes reads as sufficient and these runs show it was not.

Verification

Preregistration, posted before any paid run, and result.

skill should-trigger fired, before control mean fall case drops
map-data 18/18 18/18 0.000 0
map-events 18/18 18/18 0.000 0
map-flow 18/18 18/18 0.000 0
map-deployment 18/18 18/18 0.000 0

144 runs, Opus 5.5 in every trace, Claude Code 2.1.318, every result file VALID.

Disclosures.

  • One blind probe had "AKS" replaced by "Kubernetes" before any run, because the typos check flags it.
  • The near-miss probes were not run. They belong to the preregistration's step 2, which runs only for a skill whose control detects.
  • measure-invocation.sh validate plugins/architecture/probes: PASS. It warns on 27 queries against a target of roughly 20 and on the cross-plugin rivals, which is the point of this change.

The issue's Done-when.

  • Item 1: the probes exist and were frozen first. The control bar was not reached: no reproduced drop and no fall of 0.15.
  • Item 2: no rewrite, under its clause "A part whose control still detects nothing keeps its before text". map-data and map-flow keep their length warnings.

Remaining

After merge, #13050 closes as kept with reason, on the result above.

Related

🤖 Generated with Claude Code

kyle-sexton and others added 4 commits November 3, 2026 23:39
…vals

Freezes the should-trigger and near-miss probes for map-data, map-events,
map-flow and map-deployment before any description cut. A blind author wrote
them from the before descriptions. Each probe's rivals now include skills from
other plugins, so a generic description has something to lose to.

Refs: #13050

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replaces a word the typos check flags in a map-deployment probe, before any
run of that probe.

Refs: #13050

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ontrols drop

Refs: #13050

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton
kyle-sexton marked this pull request as ready for review November 4, 2026 05:08
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-04T05:10:27.198256Z 4b240ce Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

claude-security-review reviewed 4b240ce86e3707cda7ba5cebd64f22f34b519343: read 7 of 7 changed file(s); 0 finding(s), 0 as inline comments.

Security review: PR #13088 (head 4b240ce)

I found no security issues. I read every hunk of all 7 changed files, so this review is complete.

What changed

  • Two .changes/ changelog fragments: bump: none for architecture and bump: patch for skill-quality.
  • Four probe data files under plugins/architecture/probes/ (map-data, map-deployment, map-events, map-flow):
    • Each gains 9 new probe queries: 6 that should trigger the skill and 3 near-misses.
    • Each competitors[] list gains cross-plugin rivals: visualization:visualize, discovery:explore, event-storming:methodology and debugging:debug.
  • One prose addition to plugins/skill-quality/reference/invocation-probes.md that records the measurement results.

What I checked

  • Trusted paths and trust controls: none of the changed files matches a listed trusted path. The diff touches no trust control: no trusted-actor list, token request, kill switch, merge hold, action or workflow. It also changes no plugin manifest (plugin.json / marketplace.json).
  • How the competitor names are used: plugins/skill-quality/scripts/measure-invocation.sh reads them with jq. It uses each plugin name in a plugins: line of an eval that the operator starts and runs locally (for example L563). Every name added here is a plugin in this marketplace, written as a fixed string. Nothing outside the repository and no input a contributor controls reaches a shell or a credentialed lane through them.
  • Instruction surface: the doc change only adds text. It removes or softens no standing instruction or guardrail.
  • Embedded instructions: I found no instructions aimed at the review lane in the PR context or the diff.

Spend: the PR describes 144 paid probe runs made before the PR was opened. Merging adds no billing setting, paid resource or recurring spend.

The prior-findings file lists no earlier findings: 0 rounds with findings, cap 3, not past the cap.

critical-findings: 0

@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

claude-review reviewed 4b240ce86e3707cda7ba5cebd64f22f34b519343: read 7 of 7 changed file(s); 0 finding(s), 0 as inline comments.

Code review: no findings

I read every hunk in all 7 changed files. This is a complete review, not a partial one.

What changed

  • Four architecture probe files (map-data, map-events, map-flow, map-deployment) each get 6 contested should-trigger probes and 3 near-miss probes. Each file's competitors[] also gains cross-plugin rivals.
  • plugins/skill-quality/reference/invocation-probes.md records the 2026-11-04 control runs.
  • Two changelog fragments: architecture at bump: none and skill-quality at bump: patch.

What I checked

  • Rival skills exist. Each added competitor has a SKILL.md in the repo: visualization:visualize, discovery:explore, event-storming:methodology and debugging:debug.
  • Probe ids are unique. Each file has 9 new pos-/neg-contested-* ids, and none repeats an existing one. The JSON stays well-formed: each new block follows the last existing entry with a comma and closes the array.
  • Near-miss probes point at the right neighbour. Each one describes a sibling skill's job. For example, map-flow neg-contested-01 is a publisher/consumer question, which belongs to map-events, and map-deployment neg-contested-01 is a deployables-and-bindings question, which belongs to map-containers. That is consistent with what they are meant to test.
  • Reference doc matches the PR. The new table agrees with the PR's results table: 6 probes × 3 runs = 18 per arm. It also applies the page's own rule correctly: no reproduced case drop and no 0.15 mean fall means unmeasurable, so the before text stays.
  • Cost-claims rule. .claude/rules/cost-claims.md matches plugins/*/reference/**. The added text states no prices or per-task costs.
  • Bump levels. none fits the probe-only architecture change, and patch fits the documentation-only skill-quality change.

Context

  • Closing issue: the PR says Refs: #13050 and closes no issue, so there were no acceptance criteria to check.
  • Trusted paths: no changed file matches a trusted-path glob.
  • Prior findings: none, and the round cap was not reached.
  • Embedded instructions: the PR context contains none.
  • Verification: the run results (144 runs, all VALID) and the validate PASS come from the PR author. I could not check them because I have no shell.

Spend: none. The change adds probe data and documentation only and starts no runs.

@kyle-sexton
kyle-sexton added this pull request to the merge queue Nov 4, 2026
Merged via the queue into main with commit d130ba4 Nov 4, 2026
66 of 68 checks passed
@kyle-sexton
kyle-sexton deleted the test-13050-map-probes branch November 4, 2026 05:20
@kyle-sexton

Copy link
Copy Markdown
Member Author

Why this PR says Refs and not Closes. Done-when 1 of #13050 is unmet: the controls never dropped. I used Refs for that reason, and also to avoid pr-verify failing this PR on the missed bar, which the body does not say. After merge I will close #13050 as not planned, not completed, with item 1 marked unmet, and list the closure for the owner to review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant