Repository navigation
perf(evals): move the plugin-eval iteration loop to a reference file #12875
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
+32
−21
Merged
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
88fdeac
perf(evals): move three plugin-eval sections to reference files
kyle-sexton d19d125
fix(evals): keep the judge calibration rules in the plugin-eval hub
kyle-sexton 1345731
fix(evals): keep the Boundary section in the plugin-eval hub
kyle-sexton e14054b
Merge branch 'main' into host-run/12661-plugin-eval-cut
kyle-sexton File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
10 changes: 10 additions & 0 deletions
10
.changes/evals/perf-12661-plugin-eval-sections-to-reference-7d3e91a4.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| --- | ||
| bump: patch | ||
| --- | ||
|
|
||
| ### Changed | ||
|
|
||
| - plugin-eval: the "Iterating" section moves verbatim from SKILL.md to `reference/iterating.md` | ||
| (only its relative link changes). No action runs it, so the hub still holds every rule its | ||
| actions use; its intro says to answer tuning and confirm questions from that file. The body | ||
| drops from 41,783 to 40,550 bytes. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| # Iterating | ||
|
|
||
| 1. Read the delta column first, never the absolute score. The read is done when each case is | ||
| either a number or an explicit "not comparable". | ||
| 2. The usual first finding: a delta near zero with the case's `tool_used: Skill` grader failing means the model is not | ||
| choosing the skill on natural phrasing. Fix the skill's `description`, not the case, and re-run | ||
| that one case until the grader passes. A passing fired grader shows the trigger works; it is not | ||
| evidence that the skill improved, and any claim of a gain still needs a VALID run and its noise | ||
| report. | ||
| 3. That grader passing and a negative delta: suspect the judge before the plugin, since a small | ||
| judge marks a correct answer wrong on formatting. Re-run with a larger `--judge-model` and | ||
| tighten the rubric until it survives a rubric silent on form. | ||
| 4. Iterate on one case with `--case <name> --runs 1 --ablation none` (it reports `SCORE` and | ||
| `PASS%`, no delta). One run is noisy, so confirm any change at 3 runs per case, the only repeat | ||
| count this repository sets ([local-decisions.md, Repeat count](../../methodology/reference/local-decisions.md#repeat-count)), | ||
| and read the confirm's noise report before trusting it. | ||
| 5. Give each case one grader on the result and one on how the model got there (`tool_used` or | ||
| `tool_order`). That pairing separates "the answer was right" from "the plugin is why". |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.