Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
---
bump: patch
---

### Changed

- plugin-eval: the "Iterating" section moves verbatim from SKILL.md to `reference/iterating.md`
(only its relative link changes). No action runs it, so the hub still holds every rule its
actions use; its intro says to answer tuning and confirm questions from that file. The body
drops from 41,783 to 40,550 bytes.
2 changes: 1 addition & 1 deletion docs/upstream/addyosmani-agent-skills.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,7 @@ recorded miss. The PR that settles a row edits that row.
| [`evals/README.md`, lexical routing tier](https://github.com/addyosmani/agent-skills/blob/301b02b12bff98e4688e6984af6556802322bbde/evals/README.md#L22) | Scores prompts against all descriptions with TF-IDF in CI, with a collision check | Descriptions miss users' words or overlap. **Stated** ([source](https://github.com/addyosmani/agent-skills/blob/301b02b12bff98e4688e6984af6556802322bbde/evals/README.md#L22)) | Trigger quality | **already have:** `plugins/skill-quality/skills/check/SKILL.md:227`, near-miss probes at `plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md:254-257`. Upstream's pairwise collision error has no counterpart, and `:273` already holds a lexical score to tripwire status |
| [`evals/README.md`, description clause for pasted requests](https://github.com/addyosmani/agent-skills/blob/301b02b12bff98e4688e6984af6556802322bbde/evals/README.md#L65) | Records that models answer a request directly when the content is pasted in, and that naming that shape in a description raised skill firing without new false fires; a later rerun shows the result is tied to a CLI version | A capable model skips a skill when the job looks easy. **Stated** ([source](https://github.com/addyosmani/agent-skills/blob/301b02b12bff98e4688e6984af6556802322bbde/evals/README.md#L65)) | `plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md:254-257` builds probes from plain queries and near-misses, with no pasted-content class; a search of skill bodies and eval prompts finds no description using such a clause | **adapt:** add pasted-content probes to the frozen set in `skill-criteria.md`, measured through `/skill-quality:check measure-invocation`. Copy no upstream rate, and change a description only on our own measured gain, because [the rerun](https://github.com/addyosmani/agent-skills/blob/301b02b12bff98e4688e6984af6556802322bbde/evals/README.md#L71) shows these numbers do not last |
| [`evals/README.md`, pressure cases](https://github.com/addyosmani/agent-skills/blob/301b02b12bff98e4688e6984af6556802322bbde/evals/README.md#L43) | Eval cases whose prompt pushes against a procedure through urgency, sunk cost or rank | A process that holds only when nobody pushes is not much of a process. **Stated** ([source](https://github.com/addyosmani/agent-skills/blob/301b02b12bff98e4688e6984af6556802322bbde/evals/README.md#L43)) | `plugins/evals/skills/design/SKILL.md:183-185` lists routing, happy path, guardrail and anti-pattern; a search of `plugins/evals/skills` and the authoring skill finds no such class, though single cases exist (`plugins/source-control/evals/pull-request-ci-mode-escalates-instead-of-weakening-test`) | **take:** name it as a case kind in that list, for skills that hold a procedure |
| [`evals/plugin/` and the behavioral tier](https://github.com/addyosmani/agent-skills/tree/301b02b12bff98e4688e6984af6556802322bbde/evals/plugin) | Paired fires and stays-quiet cases graded in two arms, plugin on and off; only completed runs reach the grader | Firing is not the same as helping. **Stated** ([source](https://github.com/addyosmani/agent-skills/blob/301b02b12bff98e4688e6984af6556802322bbde/evals/README.md#L63)) | Measuring what a skill adds | **already have:** `plugins/evals/skills/plugin-eval/SKILL.md:2`, `:378`, `:390`, `plugins/evals/skills/design/SKILL.md:199-203` |
| [`evals/plugin/` and the behavioral tier](https://github.com/addyosmani/agent-skills/tree/301b02b12bff98e4688e6984af6556802322bbde/evals/plugin) | Paired fires and stays-quiet cases graded in two arms, plugin on and off; only completed runs reach the grader | Firing is not the same as helping. **Stated** ([source](https://github.com/addyosmani/agent-skills/blob/301b02b12bff98e4688e6984af6556802322bbde/evals/README.md#L63)) | Measuring what a skill adds | **already have:** `plugins/evals/skills/plugin-eval/SKILL.md:2`, `plugins/evals/skills/plugin-eval/reference/iterating.md:5`, `:17`, `plugins/evals/skills/design/SKILL.md:199-203` |
| [`evals/skill-impact.md`](https://github.com/addyosmani/agent-skills/blob/301b02b12bff98e4688e6984af6556802322bbde/evals/skill-impact.md) | An append-only list of proposals rejected on eval evidence, currently empty | Contributors repeat rejected work. **Stated** ([source](https://github.com/addyosmani/agent-skills/blob/301b02b12bff98e4688e6984af6556802322bbde/evals/skill-impact.md#L3)) | Remembering rejected ideas | **already have:** these `docs/upstream/` records, and the ledger in `/work-items:triage` |
| [`scripts/lib/skill-lint.js`, `validate-skills.js`](https://github.com/addyosmani/agent-skills/blob/301b02b12bff98e4688e6984af6556802322bbde/scripts/lib/skill-lint.js) | Lints frontmatter, naming, required sections and a line budget | Rules kept in one tested place. **Stated** ([source](https://github.com/addyosmani/agent-skills/blob/301b02b12bff98e4688e6984af6556802322bbde/scripts/lib/skill-lint.js#L5)) | Skill linting | **already have:** `/skill-quality:check`. Upstream's top-level key allowlist would flag frontmatter we use by design |
| [`scripts/run-evals.js`](https://github.com/addyosmani/agent-skills/blob/301b02b12bff98e4688e6984af6556802322bbde/scripts/run-evals.js) | A custom runner for the routing and behavioral tiers | Predates or sits beside the host's evaluator. **Inference** | Running evals | **already have:** `claude plugin eval` through `/evals:plugin-eval`; native-first rules out a custom runner (`docs/plugin-philosophy.md:285`) |
Expand Down
23 changes: 3 additions & 20 deletions plugins/evals/skills/plugin-eval/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,9 @@ This page holds every rule the actions use; answer from it. Each upstream specif
row under [Sources](#sources). Background for a human reader:
[case-authoring.md](reference/case-authoring.md), [reading-results.md](reference/reading-results.md)
(JSON fields), [ci.md](reference/ci.md) (workflow file, exit codes, parser), and
[records.md](reference/records.md) (the observations behind each source row).
[records.md](reference/records.md) (the observations behind each source row). Tuning one case
after a read, or confirming a change, is not an action and is answered from
[iterating.md](reference/iterating.md), never from this page alone.
Comment thread
kyle-sexton marked this conversation as resolved.
Resolved

## Preflight

Expand Down Expand Up @@ -371,25 +373,6 @@ noise report's verdict is that the interval excludes 0 with the with-arm ahead,
no verdict it disagrees with, every counted run did the task, and a gap claimed beyond one model has
its second-model run; otherwise `do not promote`, followed by the first condition that failed. Then remove the run's sandboxes, once its traces are no longer needed: [sandboxes.md](reference/sandboxes.md#removing-a-reads-sandboxes).

## Iterating

1. Read the delta column first, never the absolute score. The read is done when each case is
either a number or an explicit "not comparable".
2. The usual first finding: a delta near zero with the case's `tool_used: Skill` grader failing means the model is not
choosing the skill on natural phrasing. Fix the skill's `description`, not the case, and re-run
that one case until the grader passes. A passing fired grader shows the trigger works; it is not
evidence that the skill improved, and any claim of a gain still needs a VALID run and its noise
report.
3. That grader passing and a negative delta: suspect the judge before the plugin, since a small
judge marks a correct answer wrong on formatting. Re-run with a larger `--judge-model` and
tighten the rubric until it survives a rubric silent on form.
4. Iterate on one case with `--case <name> --runs 1 --ablation none` (it reports `SCORE` and
`PASS%`, no delta). One run is noisy, so confirm any change at 3 runs per case, the only repeat
count this repository sets ([local-decisions.md, Repeat count](../methodology/reference/local-decisions.md#repeat-count)),
and read the confirm's noise report before trusting it.
5. Give each case one grader on the result and one on how the model got there (`tool_used` or
`tool_order`). That pairing separates "the answer was right" from "the plugin is why".

## Calibrating a judge

Trust an `llm` grader's scores only after its judge, the model the suite uses, agrees with labeled
Expand Down
18 changes: 18 additions & 0 deletions plugins/evals/skills/plugin-eval/reference/iterating.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# Iterating

1. Read the delta column first, never the absolute score. The read is done when each case is
either a number or an explicit "not comparable".
2. The usual first finding: a delta near zero with the case's `tool_used: Skill` grader failing means the model is not
choosing the skill on natural phrasing. Fix the skill's `description`, not the case, and re-run
that one case until the grader passes. A passing fired grader shows the trigger works; it is not
evidence that the skill improved, and any claim of a gain still needs a VALID run and its noise
report.
3. That grader passing and a negative delta: suspect the judge before the plugin, since a small
judge marks a correct answer wrong on formatting. Re-run with a larger `--judge-model` and
tighten the rubric until it survives a rubric silent on form.
4. Iterate on one case with `--case <name> --runs 1 --ablation none` (it reports `SCORE` and
`PASS%`, no delta). One run is noisy, so confirm any change at 3 runs per case, the only repeat
count this repository sets ([local-decisions.md, Repeat count](../../methodology/reference/local-decisions.md#repeat-count)),
and read the confirm's noise report before trusting it.
5. Give each case one grader on the result and one on how the model got there (`tool_used` or
`tool_order`). That pairing separates "the answer was right" from "the plugin is why".
Loading