diff --git a/.changes/evals/perf-12661-plugin-eval-sections-to-reference-7d3e91a4.md b/.changes/evals/perf-12661-plugin-eval-sections-to-reference-7d3e91a4.md new file mode 100644 index 00000000000..1922480a773 --- /dev/null +++ b/.changes/evals/perf-12661-plugin-eval-sections-to-reference-7d3e91a4.md @@ -0,0 +1,10 @@ +--- +bump: patch +--- + +### Changed + +- plugin-eval: the "Iterating" section moves verbatim from SKILL.md to `reference/iterating.md` + (only its relative link changes). No action runs it, so the hub still holds every rule its + actions use; its intro says to answer tuning and confirm questions from that file. The body + drops from 41,783 to 40,550 bytes. diff --git a/docs/upstream/addyosmani-agent-skills.md b/docs/upstream/addyosmani-agent-skills.md index 56891192cdc..670a1bf746a 100644 --- a/docs/upstream/addyosmani-agent-skills.md +++ b/docs/upstream/addyosmani-agent-skills.md @@ -130,7 +130,7 @@ recorded miss. The PR that settles a row edits that row. | [`evals/README.md`, lexical routing tier](https://github.com/addyosmani/agent-skills/blob/301b02b12bff98e4688e6984af6556802322bbde/evals/README.md#L22) | Scores prompts against all descriptions with TF-IDF in CI, with a collision check | Descriptions miss users' words or overlap. **Stated** ([source](https://github.com/addyosmani/agent-skills/blob/301b02b12bff98e4688e6984af6556802322bbde/evals/README.md#L22)) | Trigger quality | **already have:** `plugins/skill-quality/skills/check/SKILL.md:227`, near-miss probes at `plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md:254-257`. Upstream's pairwise collision error has no counterpart, and `:273` already holds a lexical score to tripwire status | | [`evals/README.md`, description clause for pasted requests](https://github.com/addyosmani/agent-skills/blob/301b02b12bff98e4688e6984af6556802322bbde/evals/README.md#L65) | Records that models answer a request directly when the content is pasted in, and that naming that shape in a description raised skill firing without new false fires; a later rerun shows the result is tied to a CLI version | A capable model skips a skill when the job looks easy. **Stated** ([source](https://github.com/addyosmani/agent-skills/blob/301b02b12bff98e4688e6984af6556802322bbde/evals/README.md#L65)) | `plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md:254-257` builds probes from plain queries and near-misses, with no pasted-content class; a search of skill bodies and eval prompts finds no description using such a clause | **adapt:** add pasted-content probes to the frozen set in `skill-criteria.md`, measured through `/skill-quality:check measure-invocation`. Copy no upstream rate, and change a description only on our own measured gain, because [the rerun](https://github.com/addyosmani/agent-skills/blob/301b02b12bff98e4688e6984af6556802322bbde/evals/README.md#L71) shows these numbers do not last | | [`evals/README.md`, pressure cases](https://github.com/addyosmani/agent-skills/blob/301b02b12bff98e4688e6984af6556802322bbde/evals/README.md#L43) | Eval cases whose prompt pushes against a procedure through urgency, sunk cost or rank | A process that holds only when nobody pushes is not much of a process. **Stated** ([source](https://github.com/addyosmani/agent-skills/blob/301b02b12bff98e4688e6984af6556802322bbde/evals/README.md#L43)) | `plugins/evals/skills/design/SKILL.md:183-185` lists routing, happy path, guardrail and anti-pattern; a search of `plugins/evals/skills` and the authoring skill finds no such class, though single cases exist (`plugins/source-control/evals/pull-request-ci-mode-escalates-instead-of-weakening-test`) | **take:** name it as a case kind in that list, for skills that hold a procedure | -| [`evals/plugin/` and the behavioral tier](https://github.com/addyosmani/agent-skills/tree/301b02b12bff98e4688e6984af6556802322bbde/evals/plugin) | Paired fires and stays-quiet cases graded in two arms, plugin on and off; only completed runs reach the grader | Firing is not the same as helping. **Stated** ([source](https://github.com/addyosmani/agent-skills/blob/301b02b12bff98e4688e6984af6556802322bbde/evals/README.md#L63)) | Measuring what a skill adds | **already have:** `plugins/evals/skills/plugin-eval/SKILL.md:2`, `:378`, `:390`, `plugins/evals/skills/design/SKILL.md:199-203` | +| [`evals/plugin/` and the behavioral tier](https://github.com/addyosmani/agent-skills/tree/301b02b12bff98e4688e6984af6556802322bbde/evals/plugin) | Paired fires and stays-quiet cases graded in two arms, plugin on and off; only completed runs reach the grader | Firing is not the same as helping. **Stated** ([source](https://github.com/addyosmani/agent-skills/blob/301b02b12bff98e4688e6984af6556802322bbde/evals/README.md#L63)) | Measuring what a skill adds | **already have:** `plugins/evals/skills/plugin-eval/SKILL.md:2`, `plugins/evals/skills/plugin-eval/reference/iterating.md:5`, `:17`, `plugins/evals/skills/design/SKILL.md:199-203` | | [`evals/skill-impact.md`](https://github.com/addyosmani/agent-skills/blob/301b02b12bff98e4688e6984af6556802322bbde/evals/skill-impact.md) | An append-only list of proposals rejected on eval evidence, currently empty | Contributors repeat rejected work. **Stated** ([source](https://github.com/addyosmani/agent-skills/blob/301b02b12bff98e4688e6984af6556802322bbde/evals/skill-impact.md#L3)) | Remembering rejected ideas | **already have:** these `docs/upstream/` records, and the ledger in `/work-items:triage` | | [`scripts/lib/skill-lint.js`, `validate-skills.js`](https://github.com/addyosmani/agent-skills/blob/301b02b12bff98e4688e6984af6556802322bbde/scripts/lib/skill-lint.js) | Lints frontmatter, naming, required sections and a line budget | Rules kept in one tested place. **Stated** ([source](https://github.com/addyosmani/agent-skills/blob/301b02b12bff98e4688e6984af6556802322bbde/scripts/lib/skill-lint.js#L5)) | Skill linting | **already have:** `/skill-quality:check`. Upstream's top-level key allowlist would flag frontmatter we use by design | | [`scripts/run-evals.js`](https://github.com/addyosmani/agent-skills/blob/301b02b12bff98e4688e6984af6556802322bbde/scripts/run-evals.js) | A custom runner for the routing and behavioral tiers | Predates or sits beside the host's evaluator. **Inference** | Running evals | **already have:** `claude plugin eval` through `/evals:plugin-eval`; native-first rules out a custom runner (`docs/plugin-philosophy.md:285`) | diff --git a/plugins/evals/skills/plugin-eval/SKILL.md b/plugins/evals/skills/plugin-eval/SKILL.md index a43d34f6ee9..59b4c1283c4 100644 --- a/plugins/evals/skills/plugin-eval/SKILL.md +++ b/plugins/evals/skills/plugin-eval/SKILL.md @@ -30,7 +30,9 @@ This page holds every rule the actions use; answer from it. Each upstream specif row under [Sources](#sources). Background for a human reader: [case-authoring.md](reference/case-authoring.md), [reading-results.md](reference/reading-results.md) (JSON fields), [ci.md](reference/ci.md) (workflow file, exit codes, parser), and -[records.md](reference/records.md) (the observations behind each source row). +[records.md](reference/records.md) (the observations behind each source row). Tuning one case +after a read, or confirming a change, is not an action and is answered from +[iterating.md](reference/iterating.md), never from this page alone. ## Preflight @@ -371,25 +373,6 @@ noise report's verdict is that the interval excludes 0 with the with-arm ahead, no verdict it disagrees with, every counted run did the task, and a gap claimed beyond one model has its second-model run; otherwise `do not promote`, followed by the first condition that failed. Then remove the run's sandboxes, once its traces are no longer needed: [sandboxes.md](reference/sandboxes.md#removing-a-reads-sandboxes). -## Iterating - -1. Read the delta column first, never the absolute score. The read is done when each case is - either a number or an explicit "not comparable". -2. The usual first finding: a delta near zero with the case's `tool_used: Skill` grader failing means the model is not - choosing the skill on natural phrasing. Fix the skill's `description`, not the case, and re-run - that one case until the grader passes. A passing fired grader shows the trigger works; it is not - evidence that the skill improved, and any claim of a gain still needs a VALID run and its noise - report. -3. That grader passing and a negative delta: suspect the judge before the plugin, since a small - judge marks a correct answer wrong on formatting. Re-run with a larger `--judge-model` and - tighten the rubric until it survives a rubric silent on form. -4. Iterate on one case with `--case --runs 1 --ablation none` (it reports `SCORE` and - `PASS%`, no delta). One run is noisy, so confirm any change at 3 runs per case, the only repeat - count this repository sets ([local-decisions.md, Repeat count](../methodology/reference/local-decisions.md#repeat-count)), - and read the confirm's noise report before trusting it. -5. Give each case one grader on the result and one on how the model got there (`tool_used` or - `tool_order`). That pairing separates "the answer was right" from "the plugin is why". - ## Calibrating a judge Trust an `llm` grader's scores only after its judge, the model the suite uses, agrees with labeled diff --git a/plugins/evals/skills/plugin-eval/reference/iterating.md b/plugins/evals/skills/plugin-eval/reference/iterating.md new file mode 100644 index 00000000000..388bb7c69a0 --- /dev/null +++ b/plugins/evals/skills/plugin-eval/reference/iterating.md @@ -0,0 +1,18 @@ +# Iterating + +1. Read the delta column first, never the absolute score. The read is done when each case is + either a number or an explicit "not comparable". +2. The usual first finding: a delta near zero with the case's `tool_used: Skill` grader failing means the model is not + choosing the skill on natural phrasing. Fix the skill's `description`, not the case, and re-run + that one case until the grader passes. A passing fired grader shows the trigger works; it is not + evidence that the skill improved, and any claim of a gain still needs a VALID run and its noise + report. +3. That grader passing and a negative delta: suspect the judge before the plugin, since a small + judge marks a correct answer wrong on formatting. Re-run with a larger `--judge-model` and + tighten the rubric until it survives a rubric silent on form. +4. Iterate on one case with `--case --runs 1 --ablation none` (it reports `SCORE` and + `PASS%`, no delta). One run is noisy, so confirm any change at 3 runs per case, the only repeat + count this repository sets ([local-decisions.md, Repeat count](../../methodology/reference/local-decisions.md#repeat-count)), + and read the confirm's noise report before trusting it. +5. Give each case one grader on the result and one on how the model got there (`tool_used` or + `tool_order`). That pairing separates "the answer was right" from "the plugin is why".