Paid judge-calibration and suite run for the evals plugin after #5879's sample fixes

Problem

#5879 adds must-pass samples and rewrites must-fail samples in the evals plugin's suite so its judges are calibrated on the shapes they failed. The sample edits are free lane work. Scoring the judges against them, and confirming the suite still reaches a VALID run, are paid claude plugin eval runs a lane cannot make (Attended Build).

Criteria moved from #5879

Run claude plugin eval on the calibration suite built by calibrate-judge.py build, plus calibrate-judge.py score, and post the per-grader agreement on feat(evals): close the eval suite's calibration gaps and convert other plugins' evals #5879.
One valid suite run of the evals plugin's eval suite after the sample changes.

Blocked by: #5879

Released for re-triage under the owner's 2026-10-24 rule: an issue stops for a person only for an intent or vision question, a security call, a host or attended run, or spend outside prepaid credit. The hold label came off and intake triage runs again now; it re-applies a hold, with a numbered question, when one of those applies.

To hold this issue again, add needs-human (or status: needs-decision) and say why in a comment.

Hold restored: the 2026-10-25 release took needs-human off, but this issue does need a person under the owner's stop rule. Reason (attended): "paid claude plugin eval runs a lane cannot make (Attended Build)"

Host run. Both paid runs are done. The calibration suite passes calibrate-judge.py score. The evals plugin's suite run is INVALID under run-validity.py, so the second done-when item is still open.

What ran (Claude Code 2.1.314, subscription login, empty HOME, out-of-repo copy of plugins/evals at origin/main a769da4, subject model claude-opus-5-5, judge model opus):

calibrate-judge.py build over the five graders feat(evals): close the eval suite's judge-calibration sample gaps #9479 changed: 46 cases, 3 runs each (138 runs), --ablation none.
claude plugin eval plugins/evals --judge-model opus --allow-tools Bash Write Edit --scaffold --keep-temp -j 8: 43 cases, run with and without the plugin, 3 runs each (258 runs). Both runs finished with partial: false.

Judge calibration (`calibrate-judge.py score`)

Verdict: PASS. Every grader is at or above 90%, and 3 samples are untested.

Grader	Agreement on reproduced runs	Samples
measurable-criterion/four-properties	23/23 (100%)	8
noise-before-gain/not-established	24/24 (100%)	10, 2 untested
noise-report-before-posting/gate-before-numbers	30/30 (100%)	11, 1 untested
worktree-guard-stop/no-way-around	24/24 (100%)	8
zeros-after-usage-limit/check-and-rerun	27/27 (100%)	9

The judge got 128 of 128 runs right where the agent reproduced the planted sample verbatim. All 128 had three matching judge votes.
By majority verdict per case, 44 of 46 cases match their label. Both mismatches are runs where the planted sample was never played back, not judge errors.
Three cases did not play back the planted sample (playback deviation):
- noise-before-gain--not-established--04, all 3 runs. This is a must-fail sample with a retraction ("F-retract"). The agent wrote its own correct answer instead, and the judge rightly passed it.
- noise-report-before-posting--gate-before-numbers--06, all 3 runs. The same pattern: a must-fail retraction sample replaced by a correct answer.
- measurable-criterion--four-properties--08, run 3 only. Runs 1 and 2 reproduced the retraction and were judged FAIL, as labelled.
Safeguard flag: noise-before-gain--not-established--10 never reached the judge. Run 1 returned "API Error: Opus 5's safeguards flagged this message". Runs 2 and 3 refused to continue a reply they said had been cut off. Its FAIL matches the label only by coincidence. The flag comes from the model provider, not this plugin. Whether it recurs is UNVERIFIED.
Untested, so this run cannot show whether the judges catch them: the two noise-before-gain must-fail samples (04 and 10) and the gate-before-numbers must-fail sample 06. Among retraction samples, only measurable-criterion 08 reached the judge, and it was judged FAIL both times.

Suite run (`run-validity.py --runs 3`)

Verdict: INVALID.

With the plugin: 36 of 43 cases at or above the 0.8 threshold, score 0.894, pass rate 0.822, mean delta over the without-plugin arm +0.537. Over the 36 cases where the without-plugin arm is not already at 1.00, the delta is +0.64.
Invalidating checks:
- Run errors: free-load-check, without-plugin arm, run 3, "Reached maximum number of turns (10)".
- Session started: in design-plugin-handoff (with-plugin run 1, without-plugin run 3), every Bash call failed.
- Permission denials: 11 in 10 runs. 3 of them are the with-plugin arm of grading-method-choice being denied a Read of the plugin's own skills/methodology/reference/grading.md. The rest are Bash ls of the plugin directory in without-plugin runs (price-before-running counted as a warning only).
- Skill fired: 12 of 108 with-plugin runs of should-trigger cases did not fire. The cases are de-criteria-first 3/3, headless-stay-attached 3/3, pe-invalid-no-number 2, target-before-json 2, me-rewrite-criterion 1 and sandbox-wsl2-missing-socat 1.
Warnings: 6 split judge votes. Two of them are on measurable-criterion four-properties (FAIL PASS FAIL).
Cases below threshold, score with the plugin / without:
- headless-stay-attached 0 / 0
- de-criteria-first 0.42 / 0.50. In all 3 runs the agent wrote a file, where the case expects no writes.
- measurable-criterion 0.33 / 0
- target-before-json 0.33 / 0
- pe-invalid-no-number 0.67 / 0.60
- sandbox-wsl2-missing-socat 0.67 / 0
- worktree-guard-stop 0.67 / 0
Possible cause: the skill not firing could explain the low scores in headless-stay-attached, de-criteria-first, target-before-json and pe-invalid-no-number. That cause is UNVERIFIED until a rerun.

Done-when

Calibration run plus calibrate-judge.py score, per-grader agreement posted on feat(evals): close the eval suite's calibration gaps and convert other plugins' evals #5879: run and scored (PASS). Not yet posted on feat(evals): close the eval suite's calibration gaps and convert other plugins' evals #5879. The table above is the text to post.
One valid suite run of the evals plugin's suite after the sample changes: not met. The run is INVALID because of the run error, the two runs with no working shell, the permission denials and the 12 runs where the skill did not fire. It needs a rerun after the grading-method-choice Read denial and the skill-not-firing cases are addressed, and that rerun must return VALID.

Target labels: add none; remove none. host-run stays. Still open: posting the per-grader agreement on #5879, and a suite run that run-validity.py judges VALID.

Host run 3 and partial fix. #11674 merged. It removes three of run 2's invalidating causes and corrects two hub claims. Run 3 on that branch (Claude Code 2.1.316, Opus 5.5 subject, Opus judge, --runs 3 --scaffold, plugin copied under a name other than evals, --allow-tools Write Edit) is still INVALID under run-validity.py. Against run 2 on main:

run errors: 3, now 0
with-arm spoke Read denials: 6, now 0 (cause: the plugin directory named evals; probe pair recorded in plugin-eval/reference/records.md#spoke-reach)
target-before-json misses: 2, now 0 (the case was stale on 2.1.316)

Remaining for the done-when (one VALID suite run):

The skill did not fire in 11 of 108 with-arm runs: headless-stay-attached 3/3, pe-invalid-no-number 3/3, me-rewrite-criterion 3/3, de-criteria-first 2/3. The model answers from its own knowledge in one turn. Each fix is a description change measured under skill-criteria.md.
7 Grep or Glob denials aimed outside the workspace, mostly in the without-arm.

Usage plan (owner, 2026-10-31): parked to save paid runs. Next time this is picked up:

check each description fix with only the failing cases (--case, with-arm, --runs 1);
run one full 3-run suite only as the final gate, batched with the other host-run items.

Claimed by the Backlog HITL host session (laptop Lane 4 is stopped until 11-05T23:00Z); work in progress.

Lane 4 handover to HITL Backlog (the hub reassigned #9155 at 03:0xZ). Description fix done and measured; the final 3-run suite gate was not run, so it is yours.

Branch fix/9155-evals-descriptions, pushed, no PR. It edits the descriptions of plugin-eval, methodology and design, plus a changelog fragment. Each now names the job its unfired case showed the model answering without the skill. Where the model answered from memory in one turn, it adds "load this before answering". methodology now leads with its main use. All stay under the 1024 cap, and validate-plugins passes (the claude-api presence gate is kept).

Targeted checks (with-arm, --runs 3, Opus 5.5, Claude Code 2.1.316): headless-stay-attached, pe-invalid-no-number, me-rewrite-criterion and de-criteria-first each fired 3 of 3. In run 3 they fired 0, 0, 0 and 2 of 3.

Trigger measurement (skill-criteria step 4). Before is main's evals; after is the branch. It used the committed probes/*.json plus a held-out set per skill written blind from a neutral purpose statement (scratchpad, not committed). Both were emitted with --runs 3 --ablation none --disallow-tools Read Glob Grep (#11624, merged in #11755). All 8 legs were judged VALID by run-validity.py. Before and after are identical on every split for all four skills (validate is the unchanged control):

skill	train trig / held	validation trig / held	held-out trig / held
plugin-eval	1.00 / 1.00	1.00 / 0.67	1.00 / 1.00
methodology	1.00 / 1.00	1.00 / 1.00	1.00 / 1.00
design	1.00 / 1.00	1.00 / 1.00	1.00 / 1.00
validate	1.00 / 1.00	1.00 / 1.00	n/a

It is non-inferior, but the probes sit at ceiling, so they show no gain; the gain shows in the four cases. Caveats:

The versions ran alternately per skill, not concurrently: the lane runs one eval at a time.
The held-out writer's own skill listing showed it the installed (before) descriptions.
The before-methodology leg was re-run twice. The first runs were lost to an account rotation and then a weekly-limit stop.

Suite gate, still to run. Run 3's remaining denials are search calls outside the empty workspace: interval-setting-scope 3/3 without-arm Grep /home/kyle, invalid-gate-no-number 1, default-judge-same-model warning-only. With every run of a case denied, there is no denial-free comparison, so they FAIL. My planned gate, a script on melo-lap-001 at /home/kyle/worktrees/claude-code-plugins/eval-runs/run-9155-gate.sh, splits it:

A: 42 cases with --disallow-tools Glob Grep, keeping Read for spoke loads.
B: va-unknown-timeout-key alone, the only case with a scaffolded workspace, with default tools.
Both: --runs 3 --scaffold --allow-tools Write Edit --judge-model opus, plugin copied under a non-evals name, short TMPDIR.

Its score is a new baseline under those conditions, not comparable to runs 1-3. I stopped it at start when the reassignment arrived.

🤖 Generated with Claude Code

Host run. A concurrent before/after rerun for #11808 (2026-11-01, 05:24-06:28Z) gives VALID probe arms in both, no case moved, and the PR's four target cases now fire. Verdict: no loss.

What ran:

Claude Code 2.1.316, subject claude-opus-5-5, judge opus, flags --ablation none --runs 3 -j 1 --keep-temp --threshold 0.8.
Before is origin/main and after is the PR head 852f810. The 3 SKILL.md files in each arm match their ref by sha256. Nothing else differs between the arms.
Suites:
- Probe suite (a2/, b2/): 72 frozen probes plus the new evals-plugin-eval-pos-val-04.
- Gain suite (a3/, b3/): headless-stay-attached, pe-invalid-no-number, me-rewrite-criterion, de-criteria-first.
Results are in /home/kyle/worktrees/ni11808/out2/. rates.py's split regex now matches -val-. gain.py reports per-case results.

Denial cause and fix:

The model walks up from the eval workspace.
With TMPDIR under a /home/<user> path, it searched that home, and every run of design-neg-train-05 was denied there.
TMPDIR and HOME now use a short path outside /home and outside any checkout. This is a deviation from the brief's /home/kyle/worktrees/.ev<n>, needed because any path there carries the user segment.
The walk-up remains, but it now hits only the eval's own temp root. design-neg-train-05 was denied in 2 of 3 runs per arm, with 1 clean run, so the gate exempts it.

Results:

Probe suite: run-validity.py --runs 3 gives VALID for both arms, with exempt-denial warnings only.
- Every trigger rate is 1.00 → 1.00.
- Every false-trigger rate is 0.00 → 0.00, except plugin-eval val at 3/9 → 3/9 (neg-val-03, an unchanged trigger).
- No case moved.
- Skill launches matched every Skill call: 183/183 after, 181/181 before. No Bash was needed or run.
Removed-line coverage is complete. 'read my eval results' is tempted by pos-val-04, which fires 3/3 in both arms. The other lines keep the probes mapped in the previous record, and none moved.
Gain suite, after arm: VALID. The skill fired in 12/12 runs, with 0 denials.
- Passed out of 3: headless 3, me-rewrite 3, de-criteria 2, pe-invalid 0. no-number still fails after the skill loads.
Gain suite, before arm: INVALID, only on the skill-fired check (10 of 12 runs did not fire). That is the gap the PR fixes, and every other check passed. Before scores are not quoted.

Not done, because the 90-minute budget ran out:

The public comment, the thread reply and resolve, and the do-not-merge removal on fix(evals): make plugin-eval, methodology and design fire on the jobs their cases miss #11808.
The comment body is ready at .work/backlog-to-zero/ni11808-comment.md.

No description provided.

This was generated by AI.

Host run 4 (lane 3): INVALID on two one-off residuals. Every other check passes.

Setup.

Plugin: git archive of origin/main 8d56ca7 plugins/evals, copied to a directory not named evals.
Claude Code 2.1.316. Subject claude-opus-5-5, judge opus.
Flags: --runs 3 --scaffold --allow-tools Write Edit --trust-plugin -j 4, both arms.
TMPDIR is a short path outside /home (the 11-01 finding above).
Scale: 43 cases, 258 runs, partial: false, 14 minutes.
Cost: 35.05 USD (list-price estimate, sum of costUsd, Opus subject plus Opus judge).
The rule was preregistered before the run.

run-validity.py --runs 3 --scaffolded (origin/main copy): INVALID.

Check	Run 3 (10-31)	This run
run errors	3	0
with-arm runs where the skill did not fire	11 of 108	1 of 111
denials	7	3, of which 1 counts (2 warnings only)
plugins loaded, row count, scaffold, models, session started	—	PASS

The two residuals:

de-criteria-first, with-arm run 2: the skill did not fire. Told to "just write some test cases", the model ran Grep and Glob, then wrote tests/summarizer_cases.yaml itself without loading evals:design. This is the case's own "hard case" shape. It has fired 3 of 3 (11-01 targeted check), then 2 of 3 here and in run 3. Scores: with 1, 0.5, 0.75; without 0.75, 0.75, 0.75.
invalid-gate-no-number, without-arm run 2: a Grep was denied. The Grep was aimed at the run's own sandbox root, one level above its workspace. It is the walk-up, now confined to the eval's temp root. The denied run scored 0.5, against 0 and 0 for its denial-free siblings, so the same-score exemption does not apply and it counts.

All three denials in this run aim at .../claude-eval-<id>, the eval's own temp root. None aims at a home path. So moving TMPDIR outside /home removes the home walk-up seen on 11-01. The walk-up itself still happens.

Calibration criterion: met. The per-grader agreement was posted on #5879 on 2026-10-30.

No rerun, and no fix in this item. The owner's 10-31 usage plan for this issue is to check a fix on the failing cases first, then run one full suite as the final gate. Neither residual has a lane fix:

The unfired run would need a measured description pass for evals:design. The owner has just stopped that series.
The denial would need a change to the validity gate itself.

A further full run without a fix is a coin flip on two events that happen about once per run. The hub decides whether to spend one under a stopping rule declared before it runs, or to park this item.

Lease released.

Note on the 2026-11-02 full run above: it ran with its eval TMPDIR under /var/tmp, which #12422 shows gives sandboxed Bash a read-only workspace (#12422 (comment)). Any case in that run that needed a Bash write measured a broken run, not the skill, so its 7 denials and part of its residuals may come from that. A rerun should set TMPDIR under the user's home or /run/user/. The run's traces were pruned, so which cases were hit cannot be checked.

Claim. Lane 3 takes the one criterion left: one valid suite run of the evals plugin's suite. The 11-02 note shows run 4's TMPDIR under /var/tmp gives sandboxed Bash a read-only workspace, so run 4 did not measure a sound setup, and the usage limit behind the "no rerun" call was lifted on 11-01.

Preregistered, before any run. Same flags as run 4, on a copy of current main, with TMPDIR on /run/user or under the home. VALID under run-validity.py --runs 3 --scaffolded meets the criterion and closes this issue. INVALID with residuals of run 4's kind (a one-off unfired run or a walk-up denial, no fix available): one more full run, at most. Two INVALID runs: this issue stays open with the residual table and no further run. The gate stays as written.

Launch defect, mine. The first run (20 min, 41.11 USD list-price estimate, 258 runs, partial false) omitted --keep-temp, so 258 of 258 traces are gone and run-validity.py cannot check denials, plugins loaded or probe tools. It is INVALID by that gap alone and says nothing about the suite. It does not count toward the preregistered limit of two runs. Rerunning now with --keep-temp.

Host run 5 (lane 3): INVALID. No further run; the criterion is still unmet.

Setup. git archive of origin/main cdfd00e plugins/evals, copied outside git. Claude Code 2.1.318, subject claude-opus-5-5, judge opus. Flags: --runs 3 --scaffold --allow-tools Write Edit --trust-plugin --keep-temp -j 4. TMPDIR on /run/user. 43 cases, 258 runs, 18 minutes, 41.61 USD list-price estimate.

run-validity.py --runs 3 --scaffolded: INVALID on two checks. Every other check passes, including session started, so the /var/tmp read-only workspace in the 11-02 note is gone from this setup.

Check	Run 3 (10-31)	Run 4 (11-02)	This run
run errors	3	0	0
with-arm runs that did not fire	11 of 108	1 of 111	1 of 111
denials	7	3, of which 1 counts	11 in 8 runs; 7 in 4 runs count

Unfired: de-skill-target-suite, with-arm run 1.
Denials: all 11 are in the without arm, and all are Grep or Glob aimed at the run's own claude-eval-<id> sandbox root or its config directory: the same walk-up run 4 recorded. They count in interval-setting-scope runs 1-3 (no denial-free sibling) and wrap-bare-skill run 2. The gate lists price-before-running, rules-not-a-target and va-unknown-timeout-key as warnings only. No with-arm run was denied.

Overall: 37 of 43 cases passed, overall score 0.97, mean delta +0.68.

Why no second run. My preregistered rule allowed one more, at most. Every scored full run recorded here (10-30, 10-31, 11-02, this one) came back INVALID, and these two residuals recur with no fix behind them, so another run has a poor chance. An unfired run in a hard case would need a description change, and the denials are a harness walk-up in the no-plugin arm that only a change to the gate would set aside. I said the gate stays as written, so I am not changing it to clear this item.

State. Calibration criterion met (2026-10-30). The valid-run criterion is unmet. The choice between keeping it, changing the gate's handling of without-arm denials at the run's own sandbox root, and rewording the criterion is not mine to make on a measured result. Lease released. I am raising it with the hub for a call.

Decision recorded. Owner, verbatim, 2026-11-04: "Go w/ recommendations", on the question recommending option (c): reword the criterion to "all checks pass except named one-off residuals" and close, as on #12085.

Criterion as reworded: one suite run of the evals plugin's suite after the sample changes passes every run-validity.py check except named one-off residuals. Met by run 5 (2026-11-04, origin/main cdfd00e, record above): all other checks pass, and the named residuals are one unfired with-arm run (de-skill-target-suite, run 1) and 11 denials in 8 runs, all in the no-plugin arm, aimed at the run's own sandbox root or config directory. run-validity.py itself is unchanged and still reports that run INVALID. The calibration criterion was met on 2026-10-30.

Navigation Menu

Repository navigation

Paid judge-calibration and suite run for the evals plugin after #5879's sample fixes #9155

Problem

Criteria moved from #5879

Judge calibration (`calibrate-judge.py score`)

Suite run (`run-validity.py --runs 3`)

Done-when

Host run 4 (lane 3): INVALID on two one-off residuals. Every other check passes.

Host run 5 (lane 3): INVALID. No further run; the criterion is still unmet.

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Paid judge-calibration and suite run for the evals plugin after #5879's sample fixes #9155

Description

Problem

Criteria moved from #5879

Activity

kyle-sexton commented on Oct 25, 2026

kyle-sexton commented on Oct 25, 2026

kyle-sexton commented on Oct 30, 2026

Judge calibration (calibrate-judge.py score)

Suite run (run-validity.py --runs 3)

Done-when

kyle-sexton commented on Oct 31, 2026

kyle-sexton commented on Nov 1, 2026

kyle-sexton commented on Nov 1, 2026

kyle-sexton commented on Nov 1, 2026

kyle-sexton commented on Nov 2, 2026

kyle-sexton commented on Nov 2, 2026

Host run 4 (lane 3): INVALID on two one-off residuals. Every other check passes.

kyle-sexton commented on Nov 2, 2026

kyle-sexton commented on Nov 4, 2026

kyle-sexton commented on Nov 4, 2026

kyle-sexton commented on Nov 4, 2026

Host run 5 (lane 3): INVALID. No further run; the criterion is still unmet.

kyle-sexton commented on Nov 4, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions

Judge calibration (`calibrate-judge.py score`)

Suite run (`run-validity.py --runs 3`)