[mdsmithaustin]

Why

Live promise evaluations could refuse legitimate scratch worktrees or bootstrap dependencies, while reporting and oracle gaps could distort results. This change fixes those grading paths and preserves uncertainty when traces cannot prove compliance.

What changed

• Authorize harness scratch roots before turns, remove watch-pr bootstrap dependencies without following links, and continue batches after run-specific grade refusals with exit code 2.
• Correct Grok turn numbering after /compact, record history rewrites, and return INCONCLUSIVE for missing reads erased by compaction or independent-review wording that needs a judge.
• Exclude retained Hermes copies from report totals and add regression coverage, documentation, and live-rerun findings.

Scope

Covers the promise evaluator, Grok trace parsing, and aggregate reports. Remaining oracle and skill-routing gaps are documented in the rerun report; this change does not establish controller-key secrecy across the full pipeline.

Tradeoffs

Compaction-erased reads and unparsed independent-review wording remain INCONCLUSIVE rather than receiving an automatic PASS. Observed edits or commits still fail the read-only promise.

Blast Radius

Touches live evaluation batches and reported promise results across harnesses. Scratch-worktree admission and bootstrap cleanup affect grading inputs; controller-level refusals still stop the batch.

Verification

The committed rerun report records that all seven previously refused cases graded without a refusal. It also records mixed Claude Code results and unresolved oracle gaps. Regression tests were added for scratch worktrees, safe cleanup, refusal handling, compaction, oracle uncertainty, and report deduplication. No tests or live evaluations were run during this PR-drafting phase.

✅ Low: Each fix (A–E) is narrowly scoped to eval grading tooling, matches the stated intent and recorded decisions, and has a red-first behavioral test; traced paths (scratch allocations, link-safe node_modules removal, refusal batching/exit codes, pair de-duplication via original_root, Grok compaction/turn numbering, judge referral) produced no wrong-result case.

Updates from git push no-mistakes

[mdsmithaustin]

Verdict: PASS+NOTES
Head: 081421a
Patch-id: dbb13307248307890065b2729a6b6723d567239d
Docs: pass

PR 289 trail review: fix(evals): correct promise grading refusals and oracle results

Scope

• The review covers three heads:
  • cc7cb9a (merge-base 208ee58, patch-id cbb3f1ffa35bbc3ee2ed08a1380835d9b0fb719f).
  • The rebased 577211f (merge-base fd7973b).
  • The final head 081421a, after the second rebase onto main bfe834d (chore(deps): bump uv from 0.12.18 to 0.13.4 in /tools in the pip group across 1 directory #290).
• Final head 081421a (merge-base bfe834d = current main):
  • git diff bfe834de 081421a9 is identical to git diff fd7973b9 577211f0 with index lines ignored. The patch-id is dbb13307 on both.
  • git diff 577211f0 081421a9 touches only tools/uv-requirements.txt, which is main's uv pin bump.
  • The rerun added no pipeline commits. 081421a is 577211f rebased.
  • The evals/promises tree is byte-identical to 577211f.
• git diff fd7973b9 577211f^ (rebased fdfd643) is byte-identical to git diff 208ee58c cc7cb9a with index lines ignored. Its patch-id is the same cbb3f1ff.
• The final diff fd7973b..577211f equals the reviewed diff with one addition. The added commit is 577211f "no-mistakes(review): Skip bootstrap cleanup when the project is absent". It touches evals/promises/live.py:318 (an os.path.lexists(run.project) guard) and adds test_live.BootstrapDependencies.test_a_project_deleted_during_the_turn_reaches_grading. I read and accept it. It fixes a batch stop I had found at cc7cb9a: an agent that deleted its project made _Root.open(run.project) raise unsafe_root before grading, which stopped the batch.
• git diff cc7cb9a 577211f, excluding the main delta paths, shows only those two files. The main deltas touch nothing under evals/promises:
  • fd7973b touches only evals/harness-routing/{README.md,oracle.py} and tools/test_harness_routing.py.
  • bfe834d touches only tools/uv-requirements.txt, which CI uses to install uv.
  • Nothing under evals/promises references either change, so the docs context is unchanged.
• Files read in context: evals/promises/{live.py, grade_boundary.py, oracles.py, report.py, harnesses/grok.py, README.md, results/2026-10-20-grader-fix.md, results/2026-10-20-sync-7a9a3ddb.md, test_live.py, test_grok.py, test_oracles.py, test_report.py}, both testdata/grok_compaction fixtures, harness tmp setup in harnesses/{claude_code,codex,grok,hermes_evidence}.py.

Commands run and outcomes

• Disposable clone: /private/tmp/claude-501/-Users-msmith1-HMV-System-mds-pstack/429cfdb7-6f12-40e3-9442-f09cfea07c89/scratchpad/review289.V7Nd/pstack. Merge-base worktree: /private/tmp/claude-501/-Users-msmith1-HMV-System-mds-pstack/429cfdb7-6f12-40e3-9442-f09cfea07c89/scratchpad/review289.V7Nd/base (removed after use).
• python3 -m unittest discover -s evals/promises -p 'test_*.py' with python3 3.14.8 from PATH:
  • merge-base 208ee58: 822 tests, errors=5, skipped=1.
  • cc7cb9a: 842 tests, the same 5 errors.
  • 577211f: 843 tests, the same 5 errors.
  • 081421a: 843 tests, the same 5 errors.
  • The 5 errors are test_grade_boundary...test_git_fsmonitor_helper_cannot_access_outside_bytes (known) and four test_claude_code Native tests. All four raise IsolationUnavailable: unsupported executable: /Applications/ChatGPT.app/Contents/Resources/codex-cli/codex-path/rg, a pinned-digest mismatch on this host. They fail identically on the merge-base. The PR does not touch claude_code.py.
• The 26 targeted PR tests (BootstrapDependencies, ScratchWorktrees, GradeRefusalBatch, test_report, CompactedHistory, ReadOnlyPinAfterCompaction, AuthorResultInOtherWords) pass at 577211f. They also pass at 081421a, inside the full suite run.
• tools/gate-check.sh lint: all 8 lint checks PASS at cc7cb9a, 577211f and 081421a.
• Probe /private/tmp/claude-501/-Users-msmith1-HMV-System-mds-pstack/429cfdb7-6f12-40e3-9442-f09cfea07c89/scratchpad/probe289/probe_tmpdir.py (native grade, synthetic Codex turn):
  • untouched TMPDIR: graded.
  • agent recreates <run>/tmp: refusal root_replaced.
  • agent deletes it: refusal unsafe_root.
  • probe_base.py on the merge-base grades all three.
• Grok re-harvest differential: grok_regrade.py ran on scratch copies of baseline runs relay-55ed4sis and relay-6tna36t5 from ~/pstack-live-runs/2026-10-20-sync-7a9a3ddb. These are the only 2 of 35 baseline Grok runs with compaction_meta, and the only case with a /compact turn is name-only-route-after-compact.
  • relay-6tna36t5 read-only-phrase-pins-investigation: FAIL to INCONCLUSIVE. This move is documented.
  • relay-55ed4sis routed-skill-loads-after-compaction: INCONCLUSIVE to PASS. This move is not documented (note 2).
• Saved-trace oracle differential, base vs head, over every trace.json in both run dirs:
  • poteto-runs-documentation-impact-before-completion: only Codex tally-1lw1ob03 moves, FAIL to INCONCLUSIVE with needs_judge. This is the baseline's ORACLE row. The other 3 runs are unchanged.
  • read-only-phrase-pins-investigation: 11 runs, 0 moved.
  • routed-skill-loads-after-compaction: 4 runs, 0 moved.
• gh pr view 289 / gh pr checks 289: see the CI section.

Correctness

• Fail-only-on-proof holds.
  • read_only_pin (oracles.py:1160) moves only FAIL to INCONCLUSIVE, and only for a chat_history compaction recorded in the graded turn. Edits and commits still FAIL. Stream and rollout compactions and other turns still FAIL (tests pin all four).
  • doc_impact (oracles.py:2644) moves only FAIL to an INCONCLUSIVE judge referral when "independent review" appears in the reply or turn texts. A turn without that text still FAILs.
  • Both moves match ORACLE rows in results/2026-10-20-sync-7a9a3ddb.md. The Grok raw summary shows investigation.md was read. Codex wrote "requires independent review".
  • No oracle change creates a PASS.
• Grok turn numbering (grok.py:289) is backed by the relay-55ed4sis record. Its prompt_index: 1 carries turn 2's prompt text. It is the identity mapping for any case without /compact, and only one case has one.
• Refusal flow (live.py:318-406):
  • A GradeRefused from grade() becomes a per-run refusal screen, and the run ends with exit 2.
  • BATCH_REFUSALS stops the batch at once with exit 2 (test pins boundary_unavailable with 1 turn run).
  • LoginRequired still exits 3 (test_live test_a_login_that_dies_mid_wave_stops_the_wave_after_its_error_verdict, unchanged).
  • Harvest, seal and cleanup refusals still propagate and stop the batch, as before.
  • A Hermes pair is still retained on a refused grade.
• _Root.remove (grade_boundary.py:196-213):
  • The walk uses _parent O_NOFOLLOW per component. A link at scripts raises unsafe_link.
  • Final component: lstat refuses a link (unsafe_link) or a non-directory (unsafe_object).
  • shutil.rmtree(dir_fd=...) uses the fd-based walk, which unlinks inner links without following them and rechecks samestat against a swap.
  • An absent path returns. Tests pin both link positions and the outside bytes.
• Report (report.py:11-23) skips a pair's run/ verdict only when original_root/verdict.json exists under the report dir. Tests cover the three report scopes and a missing original. Both live pairs point at hermes/runs/{rollup-0qdv1pvu,tally-fivu9j6c} with verdicts present, so a full-dir report counts each once.

Findings (none blocking)

1. New refusal path for scratch-root replacement. The scratch roots are now allocations, and _approved_trees opens every allocation with its bound identity before it needs one (grade_boundary.py:586). A run whose agent deletes or recreates <run>/tmp now refuses its grade, and so would one that replaces the Hermes hroot/profiles/probe/cache, even with no worktree. The merge-base grades these runs (probe above). The refusal is per-run and non-fatal now. README:233 covers it only through "replacing a bound root". Suggested follow-up: open an allocation only when a registered worktree needs it, or say in README:130 that replacing a scratch root refuses.
2. Undocumented verdict move. The Grok harvest change turns baseline relay-55ed4sis routed-skill-loads-after-compaction from INCONCLUSIVE into PASS. Evidence: "compactions before turn 2: [(0, 'chat_history')]" and "figure-it-out loaded by read at seq 3" in turn 2. The PASS is proven. The results file and PR body do not name it. The body's tradeoff line reads as if this change creates no PASS.
3. Resolved at the final head. At 577211f the no-mistakes appendix still said the cleanup "can turn an agent-deleted project into a batch-stopping refusal". The 081421a appendix replaces that line with a Low summary, which matches the code. The remaining edge is a project replaced by a file or a link, which still stops the batch at cleanup. The merge-base also stopped the batch on that run, through an uncaught grade refusal.
4. Nit: grade_boundary.py:211 maps every non-ENOENT OSError from rmtree, such as a PermissionError, to unsafe_link.
5. Nit: README:136 says "the batch goes on to the next case". With --runs N it goes on to the next run, which can be the same case.
6. Nit: results file line 3 says "two Claude Code FAILs measured once". The baseline had four Claude Code FAIL verdicts across the two cases (sticky-followup and feature-run).

Documentation impact (review mode)

• Examined: README.md and results/2026-10-20-grader-fix.md at cc7cb9a. They are byte-identical at 577211f and at 081421a. 577211f touches only live.py and test_live.py, and 081421a adds only main's uv pin.
• Every changed README claim matches the code: lines 74, 76, 96, 130, 136, 152, 164, 170 and 233. These cover report pair counting, the judge referral, Grok x_compactions semantics and numbering, the allocation list, the refusal screen and exit codes, scratch worktree admission, retention completion with refusal, uncaught-exception export, and node_modules cleanup with an unsafe_link batch stop. No other maintained doc describes these behaviors. PORTING.md:251 needs no change.
• Results file, checked against ~/pstack-live-runs/2026-10-21-grader-fix (read-only reads, no git or graders there):
  • All 13 runs (6 claude-code, 5 grok, 2 hermes) have verdict.json and skills_at 8731b6e. The model is default and effort is high on every run.
  • grader-head.txt 8dc530a9 is a local commit whose grade_boundary.py, live.py, oracles.py and grok.py are byte-identical to head. Only report.py and 577211f differ, and neither affects these runs.
  • Every table row matches its verdicts:
    • tally-tj9eqkjg has the review worktree pstack-tally-review.cupYgb under tmp.
    • rollup-0qdv1pvu has candidate-1..3 under tmp/arena-rollup.AUQAhv.
    • Both babysit leads ran watch-pr, and no node_modules remains.
    • Claude Code counts: worklist PASS 3/3, principles FAIL 1/INC 2, inconclusive-reply FAIL 1/INC 2, sticky and short-prompt FAIL 1/PASS 2, new-task FAIL 2/PASS 1, read-only pin PASS 3.
  • Every FAIL-class claim matches its trace:
    • s27_4et8: turn 1 recommended a fast-forward. Turn 2 read shipping.md and ran git merge --ff-only json-flag. Turn 3 says "uses the Bug fix playbook".
    • ulu4bu_1 Bug fix worklist.
    • cat bug-fix.md; echo ======; cat investigation.md in both runs.
    • 86yk3j_c read 3 principles and named none.
    • r_5omkox Exit code 127 / ALL PASS / shasum: command not found.
    • Grok **Step 8.** overlap is 2 of 5 and ITEM_NUMBER does not match.
  • The ulu4bu_1 author rewrite (--reset-author) is described as "the account identity". The file contains no email, and the diff contains no email or personal path.
• Result: pass. Findings 1 and 5 are wording gaps, not incorrect claims.

CI and head

• Final headRefOid: 081421a. The head moved from cc7cb9a to 577211f, then to 081421a.
• All 7 gh pr checks 289 checks are SUCCESS on 081421a. Its runs (lint, skill-checks, CodeQL) were created at 01:27:30Z with headSha 081421a. The checks are:
  • CodeQL
  • Analyze (actions)
  • Analyze (javascript-typescript)
  • Analyze (python)
  • skill-checks
  • skills, completed 01:35:42Z. It runs the evals/promises suite on Ubuntu, where the Darwin-only classes skip.
  • worktree-audit
• mergeStateStatus is CLEAN.
• 577211f was also all green at 01:20Z.
• PR title and body claims are true at 081421a, apart from note 2. The no-mistakes attestation names head_sha 081421a, with step pr "running", step ci "pending", and live_validation "inconclusive" 5/9. That attestation text predates the green checks.

Not driven

• No paid live model runs.
• No live.py grade or report inside any saved run dir.
• The Hermes profile-cache worktree path was checked only through test_live.ScratchWorktrees.
• Ubuntu CI is the only Linux evidence.

Ran on: claude-opus-5-5@xhigh