Skip to content

perf(discovery): move the research practitioner test to a reference file - #12876

Merged
kyle-sexton merged 4 commits into
mainfrom
host-run/12659-research-cut
Nov 3, 2026
Merged

kyle-sexton merged 4 commits into
mainfrom
host-run/12659-research-cut

Conversation

@kyle-sexton

@kyle-sexton kyle-sexton commented Nov 3, 2026 •

Copy link
Copy Markdown
Member

Closes #12659

Summary

The /discovery:research hub carried the practitioner-source test on every load, though a run needs it only when it weighs a person as a recognized author. This PR moves that text word for word into a reference file read at that point. It is the third progressive-disclosure cut under #12243, following the #12387 recipe.

Fix

  • New skills/research/reference/practitioner-test.md holds the two moved paragraphs: the proof-of-shipped-work test and the untrusted-profile paragraph. "Reconciling sources at the gate" keeps its heading, first paragraph and scoped-exception quote, and points at the file before a person's post, talk, profile or repository is weighed. "See also" lists the file.
  • Hub body (SKILL.md with frontmatter stripped, awk strip piped to wc -c): 45,165 bytes on main 221bfbabf1, 43,646 on this branch. The issue's 44,641 predates fix(discovery): give the research Effort-ceiling sentence headroom in the re-attach slice #12647.
  • The two external citations of the moved text, docs/upstream/aiforui-course.md rows L8 and L9, now name the reference file (L9 was fixed in review). No test, contract window, hook or script cites it; the 20,000-byte window pins in contract.test.sh are untouched, since the cut sits after byte 20,000.
  • Two new cases read the moved text: rs-practitioner-needs-shipped-work and rs-profile-imperative-is-red-flag. Neither grants Bash, and both end before any web research.

Verification

  • /docs-hygiene:compress compare, two independent compare-labeler agents, before against after: both returned VERDICT: PASS. Union of rows: 2 RELOCATED, 2 FALSE POSITIVE, no SEMANTIC LOSS, no AMBIGUITY.
  • plugins/discovery/scripts/contract.test.sh: all contract assertions passed. validate-cases.py on plugins/discovery/evals: no finding on the rs-* cases.
  • Evals: claude plugin eval --case 'rs-*' --ablation none --runs 2 -j 2 --keep-temp --model claude-opus-5-5, Claude Code 2.1.317, capped scopes, arms launched together. Before is this branch's copy with main's skills/research swapped in. Firing check on main (--runs 1): all 3 cases scored 1.00, VALID. The suite was frozen after it, with one later edit: the spelling check flagged the optional recognised alternative in the not-cited-as-authority grader, so it was dropped. Regrading all 9 observed replies of that case with the old and new pattern gives the same verdict on each.
  • The negative control was preregistered on perf(discovery): cut the research skill body through progressive disclosure #12659 before the first paid run.
Round Arm Mean noise-report run-validity.py --runs 2
B before 1.00 VALID
B after 1.00 +0.00 (+0.00 to +0.00), non-inferior at margin 0.05 VALID
C before 1.00 VALID
C degraded (body deleted) 0.50 over 2 cases n too small to call (2 cases comparable) INVALID: rs-practitioner-needs-shipped-work hit max turns and one own-copy Grep was denied
  • After-arm traces show the reference file read in all 4 runs of the new cases; no denied call in either B arm.
  • Control decision: rewrite-a-plugin.md step 4 says a case whose degraded runs carry a denial cannot decide the control, so the fall must hold without it. Without rs-practitioner-needs-shipped-work, the mean falls from 1.00 to 0.50, past the 0.15 threshold. The suite detects a loss, so the cut is measurable and ships.

Related

🤖 Generated with Claude Code

kyle-sexton and others added 3 commits November 3, 2026 11:40
The /discovery:research hub body (SKILL.md, frontmatter stripped) drops
from 45,165 to 43,646 bytes. The proof-of-shipped-work test and the
untrusted-profile paragraph move word for word to
skills/research/reference/practitioner-test.md; the hub keeps the
heading's other paragraphs and points at the file before a person is
weighed as a recognized author, and its See also lists it.

Two rs-* cases read the moved text: rs-practitioner-needs-shipped-work
and rs-profile-imperative-is-red-flag.

Refs: #12659

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pelling

The changelog fragment takes the <branch-slug>-<8 hex> name the
fragment check requires. The rs-practitioner-needs-shipped-work grader
drops its optional "recognised" alternative, which the spelling check
flags; regrading all 9 observed replies gives the same verdict with and
without it.

Refs: #12659

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton
kyle-sexton marked this pull request as ready for review November 3, 2026 17:02
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-03T17:04:05.549446Z f64988d Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

claude-security-review reviewed 7fe6f09b363f9ada63f1179bfc3c496c9177584d: read 1 of 1 changed file(s); 0 finding(s), 0 as inline comments.

Security review (incremental since f64988d): no security issues found

critical-findings: 0

I read the one file changed since the last review, docs/upstream/aiforui-course.md, and its single hunk. The only change is in the L9 row of the lesson table. The "Be selective" citation used to point at plugins/discovery/skills/research/SKILL.md and now points at plugins/discovery/skills/research/reference/practitioner-test.md. That is where this PR moved the practitioner-source rule, and the L8 row on the line above already cites the same file.

  • Security surface: none. This is a provenance record in documentation. It doesn't change any standing instruction, gate, workflow, token request or manifest.
  • Trusted paths: the file doesn't match any of the TRUSTED PATHS globs.
  • Manifests: no plugin.json or marketplace.json changed.
  • Prior findings: the last review left only a summary comment with no findings. No prior finding applies to this change.
  • Embedded text: none. The change contains no instructions aimed at the review lane.
  • Spend: nothing new.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

claude-review: 1 inline finding(s) on f64988da9a7d31a4f7cdf504489ee195949a0102; the summary is in this lane's pull request comment.

claude-lane: claude-review

Comment thread docs/upstream/aiforui-course.md Resolved
@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

claude-review reviewed 7fe6f09b363f9ada63f1179bfc3c496c9177584d: read 1 of 1 changed file(s); 0 finding(s), 0 as inline comments.

Code review (incremental): PR #12876, f64988d → 7fe6f09

I reviewed 1 of 1 changed files: docs/upstream/aiforui-course.md. I found nothing new.

Last round's finding is fixed. Row L9 (line 312) cited plugins/discovery/skills/research/SKILL.md. It now cites plugins/discovery/skills/research/reference/practitioner-test.md, the same file row L8 names. The edit changes only that path. With it, both outside citations of the moved text point at the reference file. That meets the citation part of #12659's criterion, "every citation of it (tests, contract windows, hooks, scripts, fixtures, README) is repointed". The PR body now names rows L8 and L9 as those two citations, which matches the diff.

Not verified: the owner says a sweep of docs/ and plugins/ found no other citations of the moved text. I didn't run that sweep, so this is the owner's claim. The byte-count, /docs-hygiene:compress compare and eval non-inferiority criteria are also still author-claimed. I can't run those tools from this lane.

Spend: the two new eval cases from earlier rounds still use prepaid model credits on each rs-* run. This isn't a billing change.

I found no embedded instructions in the PR context, the settled threads or the prior findings.

…file

Row L9's "Be selective" why-line cites the practitioner-source rule,
which this branch moved from the research SKILL.md to
reference/practitioner-test.md. L8 was repointed; L9 now is too.

Refs: #12659

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton
kyle-sexton added this pull request to the merge queue Nov 3, 2026
Merged via the queue into main with commit 22baf23 Nov 3, 2026
69 of 71 checks passed
@kyle-sexton
kyle-sexton deleted the host-run/12659-research-cut branch November 3, 2026 18:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf(discovery): cut the research skill body through progressive disclosure

1 participant