From 88fdeaca522b05a1274ce636309a8979de5c400e Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Tue, 3 Nov 2026 11:26:21 -0500 Subject: [PATCH 1/3] perf(evals): move three plugin-eval sections to reference files The plugin-eval hub loaded the judge-calibration procedure, the iteration loop and the CLI boundary notes on every invocation, though only a calibration or a tuning loop needs them. They move verbatim to reference/calibrating-a-judge.md, reference/iterating.md and reference/boundary.md; the hub keeps the "Calibrating a judge" heading with a pointer and gains a read-when index row per reference file. Citations of the moved text in case-authoring.md and the addyosmani upstream record are repointed. Body bytes (SKILL.md with frontmatter stripped): 41,783 to 38,238. Refs: #12661 Co-Authored-By: Claude Opus 5.5 --- ...gin-eval-sections-to-reference-7d3e91a4.md | 12 ++++ docs/upstream/addyosmani-agent-skills.md | 2 +- plugins/evals/skills/plugin-eval/SKILL.md | 72 ++++--------------- .../skills/plugin-eval/reference/boundary.md | 10 +++ .../reference/calibrating-a-judge.md | 24 +++++++ .../plugin-eval/reference/case-authoring.md | 2 +- .../skills/plugin-eval/reference/iterating.md | 18 +++++ 7 files changed, 81 insertions(+), 59 deletions(-) create mode 100644 .changes/evals/perf-12661-plugin-eval-sections-to-reference-7d3e91a4.md create mode 100644 plugins/evals/skills/plugin-eval/reference/boundary.md create mode 100644 plugins/evals/skills/plugin-eval/reference/calibrating-a-judge.md create mode 100644 plugins/evals/skills/plugin-eval/reference/iterating.md diff --git a/.changes/evals/perf-12661-plugin-eval-sections-to-reference-7d3e91a4.md b/.changes/evals/perf-12661-plugin-eval-sections-to-reference-7d3e91a4.md new file mode 100644 index 00000000000..e98eca35ab5 --- /dev/null +++ b/.changes/evals/perf-12661-plugin-eval-sections-to-reference-7d3e91a4.md @@ -0,0 +1,12 @@ +--- +bump: patch +--- + +### Changed + +- plugin-eval: the "Calibrating a judge", "Iterating" and "Boundary" sections move from SKILL.md + to `reference/calibrating-a-judge.md`, `reference/iterating.md` and `reference/boundary.md`. + The text moves verbatim except relative links and the script paths, now written `` + as reference files require. SKILL.md keeps the "Calibrating a judge" heading with a pointer and + gains a reference index with a read-when row per file. The body drops from 41,783 to 38,238 + bytes. diff --git a/docs/upstream/addyosmani-agent-skills.md b/docs/upstream/addyosmani-agent-skills.md index 56891192cdc..670a1bf746a 100644 --- a/docs/upstream/addyosmani-agent-skills.md +++ b/docs/upstream/addyosmani-agent-skills.md @@ -130,7 +130,7 @@ recorded miss. The PR that settles a row edits that row. | [`evals/README.md`, lexical routing tier](https://github.com/addyosmani/agent-skills/blob/301b02b12bff98e4688e6984af6556802322bbde/evals/README.md#L22) | Scores prompts against all descriptions with TF-IDF in CI, with a collision check | Descriptions miss users' words or overlap. **Stated** ([source](https://github.com/addyosmani/agent-skills/blob/301b02b12bff98e4688e6984af6556802322bbde/evals/README.md#L22)) | Trigger quality | **already have:** `plugins/skill-quality/skills/check/SKILL.md:227`, near-miss probes at `plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md:254-257`. Upstream's pairwise collision error has no counterpart, and `:273` already holds a lexical score to tripwire status | | [`evals/README.md`, description clause for pasted requests](https://github.com/addyosmani/agent-skills/blob/301b02b12bff98e4688e6984af6556802322bbde/evals/README.md#L65) | Records that models answer a request directly when the content is pasted in, and that naming that shape in a description raised skill firing without new false fires; a later rerun shows the result is tied to a CLI version | A capable model skips a skill when the job looks easy. **Stated** ([source](https://github.com/addyosmani/agent-skills/blob/301b02b12bff98e4688e6984af6556802322bbde/evals/README.md#L65)) | `plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md:254-257` builds probes from plain queries and near-misses, with no pasted-content class; a search of skill bodies and eval prompts finds no description using such a clause | **adapt:** add pasted-content probes to the frozen set in `skill-criteria.md`, measured through `/skill-quality:check measure-invocation`. Copy no upstream rate, and change a description only on our own measured gain, because [the rerun](https://github.com/addyosmani/agent-skills/blob/301b02b12bff98e4688e6984af6556802322bbde/evals/README.md#L71) shows these numbers do not last | | [`evals/README.md`, pressure cases](https://github.com/addyosmani/agent-skills/blob/301b02b12bff98e4688e6984af6556802322bbde/evals/README.md#L43) | Eval cases whose prompt pushes against a procedure through urgency, sunk cost or rank | A process that holds only when nobody pushes is not much of a process. **Stated** ([source](https://github.com/addyosmani/agent-skills/blob/301b02b12bff98e4688e6984af6556802322bbde/evals/README.md#L43)) | `plugins/evals/skills/design/SKILL.md:183-185` lists routing, happy path, guardrail and anti-pattern; a search of `plugins/evals/skills` and the authoring skill finds no such class, though single cases exist (`plugins/source-control/evals/pull-request-ci-mode-escalates-instead-of-weakening-test`) | **take:** name it as a case kind in that list, for skills that hold a procedure | -| [`evals/plugin/` and the behavioral tier](https://github.com/addyosmani/agent-skills/tree/301b02b12bff98e4688e6984af6556802322bbde/evals/plugin) | Paired fires and stays-quiet cases graded in two arms, plugin on and off; only completed runs reach the grader | Firing is not the same as helping. **Stated** ([source](https://github.com/addyosmani/agent-skills/blob/301b02b12bff98e4688e6984af6556802322bbde/evals/README.md#L63)) | Measuring what a skill adds | **already have:** `plugins/evals/skills/plugin-eval/SKILL.md:2`, `:378`, `:390`, `plugins/evals/skills/design/SKILL.md:199-203` | +| [`evals/plugin/` and the behavioral tier](https://github.com/addyosmani/agent-skills/tree/301b02b12bff98e4688e6984af6556802322bbde/evals/plugin) | Paired fires and stays-quiet cases graded in two arms, plugin on and off; only completed runs reach the grader | Firing is not the same as helping. **Stated** ([source](https://github.com/addyosmani/agent-skills/blob/301b02b12bff98e4688e6984af6556802322bbde/evals/README.md#L63)) | Measuring what a skill adds | **already have:** `plugins/evals/skills/plugin-eval/SKILL.md:2`, `plugins/evals/skills/plugin-eval/reference/iterating.md:5`, `:17`, `plugins/evals/skills/design/SKILL.md:199-203` | | [`evals/skill-impact.md`](https://github.com/addyosmani/agent-skills/blob/301b02b12bff98e4688e6984af6556802322bbde/evals/skill-impact.md) | An append-only list of proposals rejected on eval evidence, currently empty | Contributors repeat rejected work. **Stated** ([source](https://github.com/addyosmani/agent-skills/blob/301b02b12bff98e4688e6984af6556802322bbde/evals/skill-impact.md#L3)) | Remembering rejected ideas | **already have:** these `docs/upstream/` records, and the ledger in `/work-items:triage` | | [`scripts/lib/skill-lint.js`, `validate-skills.js`](https://github.com/addyosmani/agent-skills/blob/301b02b12bff98e4688e6984af6556802322bbde/scripts/lib/skill-lint.js) | Lints frontmatter, naming, required sections and a line budget | Rules kept in one tested place. **Stated** ([source](https://github.com/addyosmani/agent-skills/blob/301b02b12bff98e4688e6984af6556802322bbde/scripts/lib/skill-lint.js#L5)) | Skill linting | **already have:** `/skill-quality:check`. Upstream's top-level key allowlist would flag frontmatter we use by design | | [`scripts/run-evals.js`](https://github.com/addyosmani/agent-skills/blob/301b02b12bff98e4688e6984af6556802322bbde/scripts/run-evals.js) | A custom runner for the routing and behavioral tiers | Predates or sits beside the host's evaluator. **Inference** | Running evals | **already have:** `claude plugin eval` through `/evals:plugin-eval`; native-first rules out a custom runner (`docs/plugin-philosophy.md:285`) | diff --git a/plugins/evals/skills/plugin-eval/SKILL.md b/plugins/evals/skills/plugin-eval/SKILL.md index a43d34f6ee9..92b599d350c 100644 --- a/plugins/evals/skills/plugin-eval/SKILL.md +++ b/plugins/evals/skills/plugin-eval/SKILL.md @@ -26,11 +26,19 @@ reads the delta. It never re-implements grading and never simulates a run or a s | `ci` | Answer from the CI section below | | `init []` | Scaffold a suite: `claude plugin eval init --bare ` where there is no terminal | -This page holds every rule the actions use; answer from it. Each upstream specific it states has a -row under [Sources](#sources). Background for a human reader: -[case-authoring.md](reference/case-authoring.md), [reading-results.md](reference/reading-results.md) -(JSON fields), [ci.md](reference/ci.md) (workflow file, exit codes, parser), and -[records.md](reference/records.md) (the observations behind each source row). +This page holds the rules every action uses; answer from it, and read a reference file when its row +below fits the question. Each upstream specific it states has a row under [Sources](#sources). + +| File | Read when | +|---|---| +| [calibrating-a-judge.md](reference/calibrating-a-judge.md) | Calibrating an `llm` grader's judge (Run step 2) | +| [iterating.md](reference/iterating.md) | Tuning one case after a read, or confirming a change | +| [boundary.md](reference/boundary.md) | The question is what the CLI owns, or a skill-creator `evals.json` | +| [case-authoring.md](reference/case-authoring.md) | Background for a human reader on writing cases | +| [reading-results.md](reference/reading-results.md) | JSON fields, denials, what a delta does and does not say | +| [ci.md](reference/ci.md) | Workflow file, exit codes, parser | +| [sandboxes.md](reference/sandboxes.md) | Sandbox start, the link swap, removing kept sandboxes | +| [records.md](reference/records.md) | The observations behind each source row | ## Preflight @@ -371,49 +379,10 @@ noise report's verdict is that the interval excludes 0 with the with-arm ahead, no verdict it disagrees with, every counted run did the task, and a gap claimed beyond one model has its second-model run; otherwise `do not promote`, followed by the first condition that failed. Then remove the run's sandboxes, once its traces are no longer needed: [sandboxes.md](reference/sandboxes.md#removing-a-reads-sandboxes). -## Iterating - -1. Read the delta column first, never the absolute score. The read is done when each case is - either a number or an explicit "not comparable". -2. The usual first finding: a delta near zero with the case's `tool_used: Skill` grader failing means the model is not - choosing the skill on natural phrasing. Fix the skill's `description`, not the case, and re-run - that one case until the grader passes. A passing fired grader shows the trigger works; it is not - evidence that the skill improved, and any claim of a gain still needs a VALID run and its noise - report. -3. That grader passing and a negative delta: suspect the judge before the plugin, since a small - judge marks a correct answer wrong on formatting. Re-run with a larger `--judge-model` and - tighten the rubric until it survives a rubric silent on form. -4. Iterate on one case with `--case --runs 1 --ablation none` (it reports `SCORE` and - `PASS%`, no delta). One run is noisy, so confirm any change at 3 runs per case, the only repeat - count this repository sets ([local-decisions.md, Repeat count](../methodology/reference/local-decisions.md#repeat-count)), - and read the confirm's noise report before trusting it. -5. Give each case one grader on the result and one on how the model got there (`tool_used` or - `tool_order`). That pairing separates "the answer was right" from "the plugin is why". - ## Calibrating a judge -Trust an `llm` grader's scores only after its judge, the model the suite uses, agrees with labeled -answers on at least 90% of runs; a pass says nothing about another model. Before the first full -run, calibrate every LLM-judge grader the harness can reproduce (one with labeled samples, judging -text); the default judge is uncalibrated until it passes. Report any other judge grader as -uncalibrated, never trusted silently. Labels are the must-pass and must-fail answers in the case's -`samples/.json`, set by three agents independently with the user settling every -disagreement: - -```bash -python3 "${CLAUDE_PLUGIN_ROOT}/skills/plugin-eval/scripts/calibrate-judge.py" build --suite --out -claude plugin eval --trust-plugin --ablation none --threshold 0 --runs 3 --judge-model --keep-temp --no-publish --json /results.json -python3 "${CLAUDE_PLUGIN_ROOT}/skills/plugin-eval/scripts/calibrate-judge.py" score --manifest /manifest.json /results.json -``` - -`--threshold 0` keeps the CLI's exit code about errors, since must-fail cases are meant to score 0. - -`score` prints a `FAIL grader` line for each grader under 90% and exits 1; fix that rubric, or move -the whole suite to a stronger judge, and calibrate again before reading its scores. Its false -positives and negatives name the samples to read first. A sample with no reproduced run is listed -`untested`, counts toward no agreement, and shows in the verdict line; raise `--runs` or tighten -the prompt before reading the grader's score. What `build` writes and skips, and which runs -`score` leaves out: [case-authoring.md, Calibration samples](reference/case-authoring.md#calibration-samples). +The agreement bar, the commands and what to do on a miss: +[calibrating-a-judge.md](reference/calibrating-a-judge.md). ## CI @@ -424,17 +393,6 @@ read the JSON in addition to the exit code. Exit 1 still fails the job on a belo but it is overloaded across six causes and exit 2 means partial, so the JSON is what tells a reader which one happened and whether the arms were comparable at all (pointer: the CI pins row under [Sources](#sources)). -## Boundary, the built-in `plugin eval` command - -- The **CLI** (`plugin eval`, a built-in command) owns execution, grading, the ablation arms, the JSON and the HTML report. It is gated - by the version floor under Preflight, so this skill reads the version and never assumes the - command is available. This skill owns what surrounds a spend: the preflight, the no-spend - validation, the estimate and ceiling, target routing and the reading discipline. It never re-implements grading and never simulates a run or a score. -- Success criteria and case design belong to `/evals:design`; come here once a suite exists. -- The skill-creator JSON eval format is not a suite this command reads. Convert its cases to case - directories (`/evals:design skill ` writes them), check those with `/evals:validate`, then - run them here. To move one plugin's `evals.json` cases into its suite, run `python3 "${CLAUDE_PLUGIN_ROOT}/skills/plugin-eval/scripts/convert-evals-json.py" ` (`--dry-run` prints the case count and writes nothing; `--help` prints what it writes), then `/evals:validate` on the plugin's `evals/`. - ## Next - Validator FAIL, or a case file that failed to load: `/evals:validate `. diff --git a/plugins/evals/skills/plugin-eval/reference/boundary.md b/plugins/evals/skills/plugin-eval/reference/boundary.md new file mode 100644 index 00000000000..0934c8d134e --- /dev/null +++ b/plugins/evals/skills/plugin-eval/reference/boundary.md @@ -0,0 +1,10 @@ +# Boundary, the built-in `plugin eval` command + +- The **CLI** (`plugin eval`, a built-in command) owns execution, grading, the ablation arms, the JSON and the HTML report. It is gated + by the version floor under the hub's [Preflight](../SKILL.md#preflight), so this skill reads the version and never assumes the + command is available. This skill owns what surrounds a spend: the preflight, the no-spend + validation, the estimate and ceiling, target routing and the reading discipline. It never re-implements grading and never simulates a run or a score. +- Success criteria and case design belong to `/evals:design`; come here once a suite exists. +- The skill-creator JSON eval format is not a suite this command reads. Convert its cases to case + directories (`/evals:design skill ` writes them), check those with `/evals:validate`, then + run them here. To move one plugin's `evals.json` cases into its suite, run `python3 "/scripts/convert-evals-json.py" ` (`--dry-run` prints the case count and writes nothing; `--help` prints what it writes), then `/evals:validate` on the plugin's `evals/`. diff --git a/plugins/evals/skills/plugin-eval/reference/calibrating-a-judge.md b/plugins/evals/skills/plugin-eval/reference/calibrating-a-judge.md new file mode 100644 index 00000000000..baa7e14b29c --- /dev/null +++ b/plugins/evals/skills/plugin-eval/reference/calibrating-a-judge.md @@ -0,0 +1,24 @@ +# Calibrating a judge + +Trust an `llm` grader's scores only after its judge, the model the suite uses, agrees with labeled +answers on at least 90% of runs; a pass says nothing about another model. Before the first full +run, calibrate every LLM-judge grader the harness can reproduce (one with labeled samples, judging +text); the default judge is uncalibrated until it passes. Report any other judge grader as +uncalibrated, never trusted silently. Labels are the must-pass and must-fail answers in the case's +`samples/.json`, set by three agents independently with the user settling every +disagreement: + +```bash +python3 "/scripts/calibrate-judge.py" build --suite --out +claude plugin eval --trust-plugin --ablation none --threshold 0 --runs 3 --judge-model --keep-temp --no-publish --json /results.json +python3 "/scripts/calibrate-judge.py" score --manifest /manifest.json /results.json +``` + +`--threshold 0` keeps the CLI's exit code about errors, since must-fail cases are meant to score 0. + +`score` prints a `FAIL grader` line for each grader under 90% and exits 1; fix that rubric, or move +the whole suite to a stronger judge, and calibrate again before reading its scores. Its false +positives and negatives name the samples to read first. A sample with no reproduced run is listed +`untested`, counts toward no agreement, and shows in the verdict line; raise `--runs` or tighten +the prompt before reading the grader's score. What `build` writes and skips, and which runs +`score` leaves out: [case-authoring.md, Calibration samples](case-authoring.md#calibration-samples). diff --git a/plugins/evals/skills/plugin-eval/reference/case-authoring.md b/plugins/evals/skills/plugin-eval/reference/case-authoring.md index 4ecfa63b56a..45f720699ab 100644 --- a/plugins/evals/skills/plugin-eval/reference/case-authoring.md +++ b/plugins/evals/skills/plugin-eval/reference/case-authoring.md @@ -180,7 +180,7 @@ Run the static validator before every paid pass; it checks the bounds above with ## Calibration samples -What the hub's "Calibrating a judge" commands do with a case's `samples/.json`: +What the [calibrating-a-judge.md](calibrating-a-judge.md) commands do with a case's `samples/.json`: - `build` writes one case per sample into an empty plugin: the source case's prompt goes out unchanged, and an appended system prompt presents the sample as fixed test material for the agent diff --git a/plugins/evals/skills/plugin-eval/reference/iterating.md b/plugins/evals/skills/plugin-eval/reference/iterating.md new file mode 100644 index 00000000000..388bb7c69a0 --- /dev/null +++ b/plugins/evals/skills/plugin-eval/reference/iterating.md @@ -0,0 +1,18 @@ +# Iterating + +1. Read the delta column first, never the absolute score. The read is done when each case is + either a number or an explicit "not comparable". +2. The usual first finding: a delta near zero with the case's `tool_used: Skill` grader failing means the model is not + choosing the skill on natural phrasing. Fix the skill's `description`, not the case, and re-run + that one case until the grader passes. A passing fired grader shows the trigger works; it is not + evidence that the skill improved, and any claim of a gain still needs a VALID run and its noise + report. +3. That grader passing and a negative delta: suspect the judge before the plugin, since a small + judge marks a correct answer wrong on formatting. Re-run with a larger `--judge-model` and + tighten the rubric until it survives a rubric silent on form. +4. Iterate on one case with `--case --runs 1 --ablation none` (it reports `SCORE` and + `PASS%`, no delta). One run is noisy, so confirm any change at 3 runs per case, the only repeat + count this repository sets ([local-decisions.md, Repeat count](../../methodology/reference/local-decisions.md#repeat-count)), + and read the confirm's noise report before trusting it. +5. Give each case one grader on the result and one on how the model got there (`tool_used` or + `tool_order`). That pairing separates "the answer was right" from "the plugin is why". From d19d1258bef6d08846a90cca841bb781d85bd978 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Tue, 3 Nov 2026 11:30:54 -0500 Subject: [PATCH 2/3] fix(evals): keep the judge calibration rules in the plugin-eval hub The calibration procedure is Run step 2, so the hub's promise that it holds every rule the actions use no longer held once that section moved; the meaning diff called it a loss. It returns verbatim, with case-authoring.md's pointer to it. Iterating and Boundary stay in reference files, which no action runs, and the intro says when to read each, answered from its own file, and defines for their script path. Body bytes (SKILL.md with frontmatter stripped): 41,783 on main, 39,662 here. Refs: #12661 Co-Authored-By: Claude Opus 5.5 --- ...gin-eval-sections-to-reference-7d3e91a4.md | 11 ++--- plugins/evals/skills/plugin-eval/SKILL.md | 46 +++++++++++++------ .../reference/calibrating-a-judge.md | 24 ---------- .../plugin-eval/reference/case-authoring.md | 2 +- 4 files changed, 37 insertions(+), 46 deletions(-) delete mode 100644 plugins/evals/skills/plugin-eval/reference/calibrating-a-judge.md diff --git a/.changes/evals/perf-12661-plugin-eval-sections-to-reference-7d3e91a4.md b/.changes/evals/perf-12661-plugin-eval-sections-to-reference-7d3e91a4.md index e98eca35ab5..907012e8769 100644 --- a/.changes/evals/perf-12661-plugin-eval-sections-to-reference-7d3e91a4.md +++ b/.changes/evals/perf-12661-plugin-eval-sections-to-reference-7d3e91a4.md @@ -4,9 +4,8 @@ bump: patch ### Changed -- plugin-eval: the "Calibrating a judge", "Iterating" and "Boundary" sections move from SKILL.md - to `reference/calibrating-a-judge.md`, `reference/iterating.md` and `reference/boundary.md`. - The text moves verbatim except relative links and the script paths, now written `` - as reference files require. SKILL.md keeps the "Calibrating a judge" heading with a pointer and - gains a reference index with a read-when row per file. The body drops from 41,783 to 38,238 - bytes. +- plugin-eval: the "Iterating" and "Boundary" sections move from SKILL.md to + `reference/iterating.md` and `reference/boundary.md`. No action runs either, so the hub still + holds every rule its actions use; its intro says when to read each file. The text moves verbatim + except relative links and one script path, now written ``, which the intro defines. + The body drops from 41,783 to 39,662 bytes. diff --git a/plugins/evals/skills/plugin-eval/SKILL.md b/plugins/evals/skills/plugin-eval/SKILL.md index 92b599d350c..8036730efb5 100644 --- a/plugins/evals/skills/plugin-eval/SKILL.md +++ b/plugins/evals/skills/plugin-eval/SKILL.md @@ -26,19 +26,15 @@ reads the delta. It never re-implements grading and never simulates a run or a s | `ci` | Answer from the CI section below | | `init []` | Scaffold a suite: `claude plugin eval init --bare ` where there is no terminal | -This page holds the rules every action uses; answer from it, and read a reference file when its row -below fits the question. Each upstream specific it states has a row under [Sources](#sources). - -| File | Read when | -|---|---| -| [calibrating-a-judge.md](reference/calibrating-a-judge.md) | Calibrating an `llm` grader's judge (Run step 2) | -| [iterating.md](reference/iterating.md) | Tuning one case after a read, or confirming a change | -| [boundary.md](reference/boundary.md) | The question is what the CLI owns, or a skill-creator `evals.json` | -| [case-authoring.md](reference/case-authoring.md) | Background for a human reader on writing cases | -| [reading-results.md](reference/reading-results.md) | JSON fields, denials, what a delta does and does not say | -| [ci.md](reference/ci.md) | Workflow file, exit codes, parser | -| [sandboxes.md](reference/sandboxes.md) | Sandbox start, the link swap, removing kept sandboxes | -| [records.md](reference/records.md) | The observations behind each source row | +This page holds every rule the actions use; answer from it. Each upstream specific it states has a +row under [Sources](#sources). In every reference file, `` is this skill's directory, +`${CLAUDE_SKILL_DIR}`. Background for a human reader: +[case-authoring.md](reference/case-authoring.md), [reading-results.md](reference/reading-results.md) +(JSON fields), [ci.md](reference/ci.md) (workflow file, exit codes, parser), and +[records.md](reference/records.md) (the observations behind each source row). Two topics are not +actions and are answered from their own files, never from this page alone: tuning one case after a +read or confirming a change ([iterating.md](reference/iterating.md)), and what the CLI owns or how +to move a skill-creator `evals.json` into a suite ([boundary.md](reference/boundary.md)). ## Preflight @@ -381,8 +377,28 @@ its second-model run; otherwise `do not promote`, followed by the first conditio ## Calibrating a judge -The agreement bar, the commands and what to do on a miss: -[calibrating-a-judge.md](reference/calibrating-a-judge.md). +Trust an `llm` grader's scores only after its judge, the model the suite uses, agrees with labeled +answers on at least 90% of runs; a pass says nothing about another model. Before the first full +run, calibrate every LLM-judge grader the harness can reproduce (one with labeled samples, judging +text); the default judge is uncalibrated until it passes. Report any other judge grader as +uncalibrated, never trusted silently. Labels are the must-pass and must-fail answers in the case's +`samples/.json`, set by three agents independently with the user settling every +disagreement: + +```bash +python3 "${CLAUDE_PLUGIN_ROOT}/skills/plugin-eval/scripts/calibrate-judge.py" build --suite --out +claude plugin eval --trust-plugin --ablation none --threshold 0 --runs 3 --judge-model --keep-temp --no-publish --json /results.json +python3 "${CLAUDE_PLUGIN_ROOT}/skills/plugin-eval/scripts/calibrate-judge.py" score --manifest /manifest.json /results.json +``` + +`--threshold 0` keeps the CLI's exit code about errors, since must-fail cases are meant to score 0. + +`score` prints a `FAIL grader` line for each grader under 90% and exits 1; fix that rubric, or move +the whole suite to a stronger judge, and calibrate again before reading its scores. Its false +positives and negatives name the samples to read first. A sample with no reproduced run is listed +`untested`, counts toward no agreement, and shows in the verdict line; raise `--runs` or tighten +the prompt before reading the grader's score. What `build` writes and skips, and which runs +`score` leaves out: [case-authoring.md, Calibration samples](reference/case-authoring.md#calibration-samples). ## CI diff --git a/plugins/evals/skills/plugin-eval/reference/calibrating-a-judge.md b/plugins/evals/skills/plugin-eval/reference/calibrating-a-judge.md deleted file mode 100644 index baa7e14b29c..00000000000 --- a/plugins/evals/skills/plugin-eval/reference/calibrating-a-judge.md +++ /dev/null @@ -1,24 +0,0 @@ -# Calibrating a judge - -Trust an `llm` grader's scores only after its judge, the model the suite uses, agrees with labeled -answers on at least 90% of runs; a pass says nothing about another model. Before the first full -run, calibrate every LLM-judge grader the harness can reproduce (one with labeled samples, judging -text); the default judge is uncalibrated until it passes. Report any other judge grader as -uncalibrated, never trusted silently. Labels are the must-pass and must-fail answers in the case's -`samples/.json`, set by three agents independently with the user settling every -disagreement: - -```bash -python3 "/scripts/calibrate-judge.py" build --suite --out -claude plugin eval --trust-plugin --ablation none --threshold 0 --runs 3 --judge-model --keep-temp --no-publish --json /results.json -python3 "/scripts/calibrate-judge.py" score --manifest /manifest.json /results.json -``` - -`--threshold 0` keeps the CLI's exit code about errors, since must-fail cases are meant to score 0. - -`score` prints a `FAIL grader` line for each grader under 90% and exits 1; fix that rubric, or move -the whole suite to a stronger judge, and calibrate again before reading its scores. Its false -positives and negatives name the samples to read first. A sample with no reproduced run is listed -`untested`, counts toward no agreement, and shows in the verdict line; raise `--runs` or tighten -the prompt before reading the grader's score. What `build` writes and skips, and which runs -`score` leaves out: [case-authoring.md, Calibration samples](case-authoring.md#calibration-samples). diff --git a/plugins/evals/skills/plugin-eval/reference/case-authoring.md b/plugins/evals/skills/plugin-eval/reference/case-authoring.md index 45f720699ab..4ecfa63b56a 100644 --- a/plugins/evals/skills/plugin-eval/reference/case-authoring.md +++ b/plugins/evals/skills/plugin-eval/reference/case-authoring.md @@ -180,7 +180,7 @@ Run the static validator before every paid pass; it checks the bounds above with ## Calibration samples -What the [calibrating-a-judge.md](calibrating-a-judge.md) commands do with a case's `samples/.json`: +What the hub's "Calibrating a judge" commands do with a case's `samples/.json`: - `build` writes one case per sample into an empty plugin: the source case's prompt goes out unchanged, and an appended system prompt presents the sample as fixed test material for the agent From 13457314cc940a99129a2b726072698e23ecea58 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Tue, 3 Nov 2026 11:56:14 -0500 Subject: [PATCH 3/3] fix(evals): keep the Boundary section in the plugin-eval hub The native-overlap registry (docs/native-surfaces/records.json, row for plugin eval) marks this skill's boundary section as baked into SKILL.md, and its self-check fails when the hub has no `## Boundary` section. The section returns verbatim; only Iterating stays in reference/iterating.md. Body bytes (SKILL.md with frontmatter stripped): 41,783 on main, 40,550 here. Refs: #12661 Co-Authored-By: Claude Opus 5.5 --- ...gin-eval-sections-to-reference-7d3e91a4.md | 9 ++++---- plugins/evals/skills/plugin-eval/SKILL.md | 21 +++++++++++++------ .../skills/plugin-eval/reference/boundary.md | 10 --------- 3 files changed, 19 insertions(+), 21 deletions(-) delete mode 100644 plugins/evals/skills/plugin-eval/reference/boundary.md diff --git a/.changes/evals/perf-12661-plugin-eval-sections-to-reference-7d3e91a4.md b/.changes/evals/perf-12661-plugin-eval-sections-to-reference-7d3e91a4.md index 907012e8769..1922480a773 100644 --- a/.changes/evals/perf-12661-plugin-eval-sections-to-reference-7d3e91a4.md +++ b/.changes/evals/perf-12661-plugin-eval-sections-to-reference-7d3e91a4.md @@ -4,8 +4,7 @@ bump: patch ### Changed -- plugin-eval: the "Iterating" and "Boundary" sections move from SKILL.md to - `reference/iterating.md` and `reference/boundary.md`. No action runs either, so the hub still - holds every rule its actions use; its intro says when to read each file. The text moves verbatim - except relative links and one script path, now written ``, which the intro defines. - The body drops from 41,783 to 39,662 bytes. +- plugin-eval: the "Iterating" section moves verbatim from SKILL.md to `reference/iterating.md` + (only its relative link changes). No action runs it, so the hub still holds every rule its + actions use; its intro says to answer tuning and confirm questions from that file. The body + drops from 41,783 to 40,550 bytes. diff --git a/plugins/evals/skills/plugin-eval/SKILL.md b/plugins/evals/skills/plugin-eval/SKILL.md index 8036730efb5..59b4c1283c4 100644 --- a/plugins/evals/skills/plugin-eval/SKILL.md +++ b/plugins/evals/skills/plugin-eval/SKILL.md @@ -27,14 +27,12 @@ reads the delta. It never re-implements grading and never simulates a run or a s | `init []` | Scaffold a suite: `claude plugin eval init --bare ` where there is no terminal | This page holds every rule the actions use; answer from it. Each upstream specific it states has a -row under [Sources](#sources). In every reference file, `` is this skill's directory, -`${CLAUDE_SKILL_DIR}`. Background for a human reader: +row under [Sources](#sources). Background for a human reader: [case-authoring.md](reference/case-authoring.md), [reading-results.md](reference/reading-results.md) (JSON fields), [ci.md](reference/ci.md) (workflow file, exit codes, parser), and -[records.md](reference/records.md) (the observations behind each source row). Two topics are not -actions and are answered from their own files, never from this page alone: tuning one case after a -read or confirming a change ([iterating.md](reference/iterating.md)), and what the CLI owns or how -to move a skill-creator `evals.json` into a suite ([boundary.md](reference/boundary.md)). +[records.md](reference/records.md) (the observations behind each source row). Tuning one case +after a read, or confirming a change, is not an action and is answered from +[iterating.md](reference/iterating.md), never from this page alone. ## Preflight @@ -409,6 +407,17 @@ read the JSON in addition to the exit code. Exit 1 still fails the job on a belo but it is overloaded across six causes and exit 2 means partial, so the JSON is what tells a reader which one happened and whether the arms were comparable at all (pointer: the CI pins row under [Sources](#sources)). +## Boundary, the built-in `plugin eval` command + +- The **CLI** (`plugin eval`, a built-in command) owns execution, grading, the ablation arms, the JSON and the HTML report. It is gated + by the version floor under Preflight, so this skill reads the version and never assumes the + command is available. This skill owns what surrounds a spend: the preflight, the no-spend + validation, the estimate and ceiling, target routing and the reading discipline. It never re-implements grading and never simulates a run or a score. +- Success criteria and case design belong to `/evals:design`; come here once a suite exists. +- The skill-creator JSON eval format is not a suite this command reads. Convert its cases to case + directories (`/evals:design skill ` writes them), check those with `/evals:validate`, then + run them here. To move one plugin's `evals.json` cases into its suite, run `python3 "${CLAUDE_PLUGIN_ROOT}/skills/plugin-eval/scripts/convert-evals-json.py" ` (`--dry-run` prints the case count and writes nothing; `--help` prints what it writes), then `/evals:validate` on the plugin's `evals/`. + ## Next - Validator FAIL, or a case file that failed to load: `/evals:validate `. diff --git a/plugins/evals/skills/plugin-eval/reference/boundary.md b/plugins/evals/skills/plugin-eval/reference/boundary.md deleted file mode 100644 index 0934c8d134e..00000000000 --- a/plugins/evals/skills/plugin-eval/reference/boundary.md +++ /dev/null @@ -1,10 +0,0 @@ -# Boundary, the built-in `plugin eval` command - -- The **CLI** (`plugin eval`, a built-in command) owns execution, grading, the ablation arms, the JSON and the HTML report. It is gated - by the version floor under the hub's [Preflight](../SKILL.md#preflight), so this skill reads the version and never assumes the - command is available. This skill owns what surrounds a spend: the preflight, the no-spend - validation, the estimate and ceiling, target routing and the reading discipline. It never re-implements grading and never simulates a run or a score. -- Success criteria and case design belong to `/evals:design`; come here once a suite exists. -- The skill-creator JSON eval format is not a suite this command reads. Convert its cases to case - directories (`/evals:design skill ` writes them), check those with `/evals:validate`, then - run them here. To move one plugin's `evals.json` cases into its suite, run `python3 "/scripts/convert-evals-json.py" ` (`--dry-run` prints the case count and writes nothing; `--help` prints what it writes), then `/evals:validate` on the plugin's `evals/`.