diff --git a/evals/promises/README.md b/evals/promises/README.md index 9b9cc2d4..e9d9bd16 100644 --- a/evals/promises/README.md +++ b/evals/promises/README.md @@ -71,7 +71,9 @@ python3 evals/promises/live.py run --harness --c python3 evals/promises/live.py report --upstream-ref upstream/main ``` -Each run makes a fresh git repository from `fixtures//` and `histories//`. Each history step is committed in order, except a step with `"commit": false`, which stays as uncommitted work in the tree. A step with `"branch"` switches to that branch first, creating it at the current commit when it does not exist, so later steps land there and `main` keeps the fixture. It installs `skills/` from `--skills-at` (default `HEAD`) where the harness discovers project skills, and runs the case's turns headless. A run that completes harvest and grading gets `run.json`, `trace.json`, and `verdict.json`. Every adapter writes the same trace schema, with events, files read, worklist snapshots, spawns, and the final reply. A tool event carries its harness call id, so each result pairs with its own call. `oracles.py` grades each promise as `PASS`, `FAIL`, or `INCONCLUSIVE`, and the authorized parent grader adds `ERROR` for a dead login. A verdict is `INCONCLUSIVE` when the trace cannot show the behavior, or when the harness reports `x_host_skill_hits`, which means the user's own skills touched the run. Some oracles also return `INCONCLUSIVE` when only a judge can decide, such as whether a reply's wording reports a review `pass`. That verdict carries `needs_judge: true` and an `excerpt` of the graded reply in `verdict.json`. In a multi-turn case the graded reply can come from an earlier turn than `final_reply`, and the verdict's evidence names that turn. When the excerpt is not enough, read that turn's text events in `trace.json` before you tally the run. +`live.py report` counts a retained Hermes pair from its original run directory when that run is under ``. It counts the pair from its own `run/` copy otherwise. + +Each run makes a fresh git repository from `fixtures//` and `histories//`. Each history step is committed in order, except a step with `"commit": false`, which stays as uncommitted work in the tree. A step with `"branch"` switches to that branch first, creating it at the current commit when it does not exist, so later steps land there and `main` keeps the fixture. It installs `skills/` from `--skills-at` (default `HEAD`) where the harness discovers project skills, and runs the case's turns headless. A run that completes harvest and grading gets `run.json`, `trace.json`, and `verdict.json`. Every adapter writes the same trace schema, with events, files read, worklist snapshots, spawns, and the final reply. A tool event carries its harness call id, so each result pairs with its own call. `oracles.py` grades each promise as `PASS`, `FAIL`, or `INCONCLUSIVE`, and the authorized parent grader adds `ERROR` for a dead login. A verdict is `INCONCLUSIVE` when the trace cannot show the behavior, or when the harness reports `x_host_skill_hits`, which means the user's own skills touched the run. Some oracles also return `INCONCLUSIVE` when only a judge can decide, such as whether a reply's wording reports a review `pass`. When documentation-impact ran but the parser finds no author result, text naming `independent review` in the change turn goes to the judge. That verdict carries `needs_judge: true` and an `excerpt` of the relevant reply or interim text in `verdict.json`. In a multi-turn case the graded reply can come from an earlier turn than `final_reply`, and the verdict's evidence names that turn. When the excerpt is not enough, read that turn's text events in `trace.json` before you tally the run. Before the first case, `live.py run` checks the selected Claude Code, Codex, or Grok prerequisites and login at no model cost. Claude Code refuses an absent native launcher with an installation message and exit 1. An existing launcher, including a dangling symlink, goes through release admission by signature before any login check. Preflight runs `auth status` with the admitted binary; a launcher that cannot be admitted raises `IsolationUnavailable`. A failed login check stops the wave with exit 3 before any case runs, and the message names the CLI and its login command. Claude Code runs `claude auth status` with the host `HOME`, `USER`, a minimal `PATH`, and an empty `CLAUDE_SECURESTORAGE_CONFIG_DIR`, so a host-exported `ANTHROPIC_API_KEY` cannot mask a missing login. It does not set the private `CLAUDE_CONFIG_DIR` each turn gets, since that directory exists only after a run is prepared, so it checks the host login. Codex needs the `auth.json` file and a passing `codex login status`. Grok refuses a host token that expires within five minutes, because the run's read-only provider never refreshes it. Claude Code and Codex refresh their own access tokens when they run, so their checks do not read token expiry. Hermes has no preflight check, and its preparation step still refuses a missing `auth.json`. @@ -91,7 +93,7 @@ The promise is that a name-only skill never loads from its description. A load c The routed-load oracles require a successful Skill tool load on Claude Code. For a killed run with no load attempts, a missing load is `FAIL` only when the trace shows a routing decision, which is a project edit, a question to the user, or a read of another routed skill. A worklist or a playbook read does not count for a killed run, because a lead can open a worklist or read a playbook and still load the routed skill later. Reads of `poteto-mode`, `pstack-harness`, and `principle-*` leaves do not establish a routing decision either. Without a decision, that run is `INCONCLUSIVE`. A file read on Claude Code or a request that hands invocation back to the user still fails. -A turn whose text is `/compact` compacts the session. Claude Code sends it through `--resume` like any later turn. `codex exec` has no `/compact`, so the Codex adapter resumes the session with `model_auto_compact_token_limit=2000`, which compacts before that turn's prompt. Both adapters record each compaction in `x_compactions` with the turn it happened in. +A turn whose text is `/compact` compacts the session. Claude Code sends it through `--resume` like any later turn. `codex exec` has no `/compact`, so the Codex adapter resumes the session with `model_auto_compact_token_limit=2000`, which compacts before that turn's prompt. Both adapters record each compaction in `x_compactions` with the turn it happened in. Grok rewrites its chat history on a compaction and marks the rewrite with `compaction_meta` records. The Grok adapter records one `x_compactions` entry per rewrite, with `source: chat_history` and the turn of the latest prompt the rewrite kept. Grok's `prompt_index` skips a `/compact` turn, and the adapter numbers later turns past it. For `read-only-phrase-pins-investigation`, a missing Investigation playbook read in the `new task` turn is `INCONCLUSIVE` when a `chat_history` compaction rewrote that turn. Observed edits or commits still fail the promise. Live runs spend model budget. Codex and Hermes draw on the same ChatGPT plan when Hermes uses the `openai-codex` provider. A Feature case takes about 5 minutes on Claude Code and over 20 minutes on Codex. @@ -125,13 +127,13 @@ No other listed edge is recorded as moving a verdict. Fix one only after a recor ## Parent grading authority -Trusted setup issues a parent-grade authorization before candidate turns. It captures the selected project identity, baseline order, skill pin, case, reference bytes, oracle code, and owned workspace allocation. It stores this authority in a private controller directory adjacent to `--out`, outside each run root and Hermes's run-root mount. The controller directory contains an issuer key and signed run authorizations. Candidate records, copied manifests, trace paths, and Git discovery output cannot add permission grants. Keep the controller directory at its original path. +Trusted setup issues a parent-grade authorization before candidate turns. It captures the selected project identity, baseline order, skill pin, case, reference bytes, oracle code, and owned workspace allocations. The allocations are `/w` and each harness's scratch roots, `/tmp` and, for Hermes, its profile cache `/hroot/profiles/probe/cache`. It stores this authority in a private controller directory adjacent to `--out`, outside each run root and Hermes's run-root mount. The controller directory contains an issuer key and signed run authorizations. Candidate records, copied manifests, trace paths, and Git discovery output cannot add permission grants. Keep the controller directory at its original path. Controller initializers hold an advisory lock on the controller directory descriptor while reading or atomically publishing the issuer key. Concurrent first initializers adopt the same key. This lock coordinates cooperating controllers and does not protect against same-user processes that ignore it. The issuer key is a file owned by the same host user. Directory permissions do not hide it from an unconfined process running as that user. The grader's file policy blocks controller access during grading. Parent-grade authority requires a pipeline whose candidate launchers also block controller file access before grading. The current native candidate launchers do not establish that requirement. This patch does not establish key secrecy or authority for the whole pipeline. -A completed run prints `authorization_id`. To regrade that registered run, use the same evaluator output directory and its ID: +A completed run prints `authorization_id`. A grade-time refusal prints its receipt under `refusal` in place of `verdicts`, and the batch goes on to the next case. If the batch finishes after any grade-time refusal, `live.py run` exits 2. Refusals before grading, including cleanup and harvest refusals, still stop the batch. A `boundary_unavailable` or `unknown_authorization` refusal stops the batch at once with exit 2, because every later case would refuse the same way. To regrade that registered run, use the same evaluator output directory and its ID: ```sh python3 evals/promises/live.py grade --out @@ -147,7 +149,7 @@ Missing sealed `run.json`, `trace.json`, or retained prior `verdict.json` produc Old raw run directories need explicit trusted retention import. Supplying a directory to `grade` no longer grants its recorded project path. The private `_authorize_retained` operation binds a controller-reviewed original project or reviewed relocation, explicit approved worktrees, the raw metadata and prior-verdict digests, and a separate diagnostic output directory. The trusted retention caller owns that review. New retained issuance admits the exact prior JSON bytes before sealing. A missing or null prior error produces signed null. A non-null error must contain exactly the three nonblank strings, match the reviewed case, harness, and skill pin, and match the harness's existing login identity. The issuer copies the accepted strings and binds the exact prior digest while the handle remains unsealed. Its signature records issuer acceptance, not historical classifier origin. Prior promise labels never become current proof. Old retained handles keep their prior digest check and recover error only from their separate diagnostic output, without importing raw prior error. Raw `run.json`, `trace.json`, the selected skill pin, and the prior `verdict.json` stay unchanged. A relocation cannot rewrite Git pointers or recorded trace paths to manufacture parity. Unsupported Git relationships refuse. -The grader admits worktree members only within the workspace allocations declared before turns, or specific sibling roots named by trusted fixture or retention setup. Each linked member must match the bound common Git directory and its reverse pointer. An unapproved discovered worktree refuses before content or Git access to that tree. An absent project, legacy baseline, incomplete trace, and timed-out turns retain their existing oracle behavior. +The grader admits worktree members only within the workspace allocations declared before turns, or specific sibling roots named by trusted fixture or retention setup. Each linked member must match the bound common Git directory and its reverse pointer. An unapproved discovered worktree refuses before content or Git access to that tree. Trusted setup creates the harness scratch roots and declares them before turns, so an arena or review worktree the agent registers under its `TMPDIR` or the Hermes profile cache is graded. An absent project, legacy baseline, incomplete trace, and timed-out turns retain their existing oracle behavior. Raw run storage must sit outside every candidate writable project, worktree, and workspace allocation, including for retained imports with separate diagnostic output. Missing registered worktree `.git`, `commondir`, or reverse-pointer files produce an `unapproved_git` refusal receipt. Invalid UTF-8 or NUL bytes in these path files also produce `unapproved_git`. Codex and Grok traces may retain either the full trusted turn list or the adapters' exact six-field projection. The projection preserves every turn's index, session ID, argv, exit code, timeout flag, and duration in order. Other subsets, added fields, changed values, and changed turn counts refuse. Sealing preserves the harvested trace. @@ -159,13 +161,13 @@ Use the Hermes retention argument when an external controller may remove the tem python3 evals/promises/live.py run --harness hermes --case principle-steer-run --out /tmp/pstack-live --hermes-retain-out /path/to/retained-evidence ``` -A nonzero exit or missing `hermes_pair` in a successful run's JSON output means retention did not complete. The argument is a destination parent controlled by the evaluator. Other adapters reject it. The Python entry point accepts the same parent as `run_case(..., hermes_retain_out=Path(...))`. +Retention completed exactly when the printed JSON screen names `hermes_pair`. This includes a screen that also carries `refusal` with exit 2. A missing `hermes_pair` means retention did not complete. The argument is a destination parent controlled by the evaluator. Other adapters reject it. The Python entry point accepts the same parent as `run_case(..., hermes_retain_out=Path(...))`. Every ancestor in the retention-parent spelling must be a real directory. A symlinked ancestor refuses binding, even if its target exists. Hermes binds a private directory below that parent before image inspection, preload, version probes, or turns. It keeps host cwd, stdout, stderr, prepared metadata, and the launch ledger outside the writable run mount. The owner chooses resume from its private captured streams. Mutating `run.turns`, `hermes.json`, or old transcript filenames does not change that choice. Without the retention argument, private evidence lives beside the temporary run directory. -After grading, the owner exports `pair-/pair.json`, `run/`, and `hermes-evidence/`. The pair contains the final trace and grade, every acquisition attempt, and all closed private captures. The JSON output's `hermes_pair` names this pair. Preparation, turn, harvest, and grading exceptions also attempt export. Exception pairs record absent `run.json`, `trace.json`, or `verdict.json` members. The original exception remains an exception. An export failure reports the private evidence location and prints no successful screen record. Controllers must retain this pair through the CLI argument. Copying only the temporary run directory loses private Hermes evidence. +After grading, the owner exports `pair-/pair.json`, `run/`, and `hermes-evidence/`. The pair contains the available trace and grade, every acquisition attempt, and all closed private captures. The JSON output's `hermes_pair` names this pair. Uncaught preparation, turn, harvest, and grading exceptions also attempt export. Pairs record absent `run.json`, `trace.json`, or `verdict.json` members. An uncaught exception still propagates after export. An export failure reports the private evidence location and prints no successful screen record. Controllers must retain this pair through the CLI argument. Copying only the temporary run directory loses private Hermes evidence. Run symlinks, including native language-server launchers, remain symlinks in the pair. The optional `symlinks` catalog maps each `run/` member to a `target_hex` descriptor with its exact target bytes. Export and replay read the link itself without opening or traversing its target. Regular-file `members` keep their existing size and SHA-256 receipts. Private evidence, captures, and acquisition inputs still require regular files. @@ -228,7 +230,7 @@ Guarded stat and directory traversal errors produce `unsafe_link` refusals. Miss Admission validates the protected `code/`, `references/`, and `fallback/` trees under the controller's run directory. Earlier attempts' writable scratch does not become input to later grades. A completed check can leave symlinks or sockets there without changing a repeat grade. -Candidate actions can turn a run into a refusal even when an older grader returned a promise verdict. Creating a symlink or hard link under the project, replacing a bound root, or leaving an unsupported worktree relationship refuses the grade. Creating an untracked embedded Git repository can make an oracle encounter its directory as a file. That read produces `unsafe_object` rather than silently ignoring the directory. These refusals have no promise verdict. +Candidate actions can turn a run into a refusal even when an older grader returned a promise verdict. Creating a symlink or hard link under the project, replacing a bound root, or leaving an unsupported worktree relationship refuses the grade. One known link source is cleared first. After the turns and before harvest, `live.py` deletes `poteto-mode/scripts/node_modules` under the installed skills, which the watch-pr bootstrap's `bun install` creates, without following links. A link at `scripts` or `node_modules` refuses with `unsafe_link` and stops the batch. Creating an untracked embedded Git repository can make an oracle encounter its directory as a file. That read produces `unsafe_object` rather than silently ignoring the directory. These refusals have no promise verdict. Git excludes global and system configuration. Fixture checks and their descendants inherit the worker's file policy and receive no controller or authority descriptors. Owned native controls also attempt to hardlink sealed metadata, case references, fallback skills, and selected installed skill copies into the writable project. The kernel denies link creation; the protected bytes and link counts remain unchanged. Python with `LANG=en_US.UTF-8` creates a non-inheritable anonymous runtime socket on the reviewed host, including when no controller launches it. The descriptor test permits that socket and rejects extra file, directory, pipe, or inheritable descriptors. Ordinary completed checks retain their actual return code and captured streams. Exact stdout mismatches, nonzero failures, and genuine timeouts keep the existing oracle text. A directly detected setup or guarded-I/O error produces a separate refusal. A check can catch an OS denial and return normally, so complete denial attribution is not guaranteed. diff --git a/evals/promises/grade_boundary.py b/evals/promises/grade_boundary.py index dc0f8d4b..d7159cc1 100644 --- a/evals/promises/grade_boundary.py +++ b/evals/promises/grade_boundary.py @@ -7,6 +7,7 @@ import platform import pwd import shlex +import shutil import signal import stat import subprocess @@ -192,6 +193,25 @@ def walk(fd, prefix): self.verify() return found + def remove(self, rel): + """Delete the directory at rel without following a link on the way to it or inside it. Absent is done.""" + self.verify() + try: + parent, name = self._parent(rel) + except FileNotFoundError: + return + try: + info = os.stat(name, dir_fd=parent, follow_symlinks=False) + if not stat.S_ISDIR(info.st_mode): + raise GradeRefused("unsafe_link" if stat.S_ISLNK(info.st_mode) else "unsafe_object", self.path / rel) + shutil.rmtree(name, dir_fd=parent) + except FileNotFoundError: + pass + except OSError as error: + raise GradeRefused("unsafe_link", self.path / rel) from error + finally: + os.close(parent) + def write(self, name, data): if Path(name).name != name: raise GradeRefused("output_unsafe", name) @@ -438,12 +458,12 @@ def _lookup(out, run_id): return _Controller(out).load(run_id)[0] -def _before_turns(run): +def _before_turns(run, scratch=()): _runtime() record = {"harness": run.harness, "case": run.case["id"], "skills_at": run.skills_at, "project": str(run.project), "timeout_s": run.timeout_s, "turns": [], "baseline": run.baseline} controller = _Controller(run.root.parent, create=True) - return controller.issue(run.root, run.project, run.case, record, allocations=(run.project.parent,)) + return controller.issue(run.root, run.project, run.case, record, allocations=(run.project.parent, *scratch)) def _write_record(handle, record): diff --git a/evals/promises/harnesses/grok.py b/evals/promises/harnesses/grok.py index 78e2b6b1..43883830 100644 --- a/evals/promises/harnesses/grok.py +++ b/evals/promises/harnesses/grok.py @@ -31,6 +31,7 @@ [init] \tdefaultBranch = main """ +LOCAL_COMMANDS = ("/compact",) SESSION_FILES = ("chat_history.jsonl", "events.jsonl", "usage.json", "summary.json", "tool_definitions.json") AUTH_PROVIDER = """#!{python} @@ -285,6 +286,12 @@ def turn_prompts(run): return prompts +def turn_of_index(prompt_index, prompts): + """Grok counts only the prompts it sends to the model, so a `/compact` turn takes no prompt_index.""" + sent = [index for index, prompt in enumerate(prompts) if prompt.strip() not in LOCAL_COMMANDS] + return sent[prompt_index] if prompt_index < len(sent) else prompt_index + + def turn_of_prompt(prompt, prompts, previous): matches = [index for index, known in enumerate(prompts) if known.strip() == prompt.strip() and index > previous] return matches[0] if matches else previous + 1 @@ -317,6 +324,7 @@ def parse_session(chat, cwd, entry_skill, lead=False, prompts=()): injected = False read_entry = False todos = [] + compactions, rewriting = [], None injection = re.compile(rf']*args=') if entry_skill else None def mark(): @@ -330,17 +338,23 @@ def add(event): for rec in load_jsonl(chat): kind = rec.get("type") + if kind == "user" and rec.get("synthetic_reason") == "compaction_meta": + if not rewriting: + rewriting = {"source": "chat_history"} + compactions.append(rewriting) + rewriting["turn"] = turn if kind == "user" and not rec.get("synthetic_reason") and lead and ( "prompt_index" in rec or USER_QUERY.search(text_of(rec.get("content")))): body = text_of(rec.get("content")) query = USER_QUERY.search(body) prompt = query.group(1) if query else body - turn = rec["prompt_index"] if "prompt_index" in rec else turn_of_prompt(prompt, prompts, turn if events else -1) + turn = turn_of_index(rec["prompt_index"], prompts) if "prompt_index" in rec else turn_of_prompt(prompt, prompts, turn if events else -1) add({"kind": "user", "text": prompt}) turn_skills += [{"turn": turn, "skill": name} for name in SKILL_REF.findall(body)] if kind == "user" and not rec.get("synthetic_reason") and injection and injection.search(text_of(rec.get("content"))): injected = True elif kind == "assistant": + rewriting = None model = rec.get("model_id") or model effort = rec.get("reasoning_effort") or effort content = text_of(rec.get("content")) @@ -387,7 +401,7 @@ def add(event): "output_head": (content or "")[:400], "id": rec.get("tool_call_id")}) return {"events": events, "files_read": list(dict.fromkeys(files)), "worklist": worklist, "spawns": spawns, "final_reply": final, "model": model, "effort": effort, "injected": injected, "read_entry": read_entry, - "first_reply": first_reply, "turn_skills": turn_skills} + "first_reply": first_reply, "turn_skills": turn_skills, "compactions": compactions} def decoded_cwd(chat_path): @@ -454,7 +468,7 @@ def harvest(run): cwd = str(run.project) empty = {"events": [], "files_read": [], "worklist": [], "spawns": [], "final_reply": None, "model": None, "effort": None, "injected": False, "read_entry": False, - "first_reply": None, "turn_skills": []} + "first_reply": None, "turn_skills": [], "compactions": []} lead = parse_session(evidence[leads[0]], decoded_cwd(leads[0]), entry, lead=True, prompts=turn_prompts(run)) if leads else empty launch = json.loads((run.root / "launch.json").read_text()) if (run.root / "launch.json").is_file() else {} @@ -517,6 +531,7 @@ def harvest(run): "x_todo_tools": "off" if run.case.get("env", {}).get("todo_tools") is False else "on", "x_turn_entries": turn_entries(run, lead), "x_turns_without_events": turns_without_events(lead["events"], len(run.turns)), + "x_compactions": lead["compactions"], "x_subagents": subagents, "x_unlinked_children": sorted(s["session_id"] for s in subagents if not s["parent_session_id"]), "x_host_skill_hits": host_skill_hits([*evidence.values(), *streams.values()]), diff --git a/evals/promises/live.py b/evals/promises/live.py index 9acd97fa..356dbf95 100644 --- a/evals/promises/live.py +++ b/evals/promises/live.py @@ -17,7 +17,7 @@ from dataclasses import dataclass, field from pathlib import Path -from grade_boundary import create_file, grade, GradeRefused, read_file, write_file, _before_turns, _json, _lookup, _seal, _write_record +from grade_boundary import create_file, grade, GradeRefused, read_file, write_file, _before_turns, _json, _lookup, _Root, _seal, _write_record from harnesses.login import LoginRequired, auth_death, death_record, error_verdict HERE = Path(__file__).resolve().parent @@ -37,6 +37,18 @@ "grok": "harnesses.grok", } RUN_OVERRIDES = ("claude-code", "codex") +# Run-root directories each harness points the agent's temp writes at (TMPDIR, and Hermes's HERMES_WRITE_SAFE_ROOT +# profile cache). A git worktree the agent registers under one is graded; nothing else under them is read. +SCRATCH_ROOTS = { + "claude-code": ("tmp",), + "codex": ("tmp",), + "grok": ("tmp",), + "hermes": ("tmp", "hroot/profiles/probe/cache"), +} +# The watch-pr bootstrap's `bun install` target. Its node_modules/.bin links would refuse the grade walk. +BOOTSTRAP_MODULES = Path("poteto-mode/scripts/node_modules") +# Grade refusals about the grader or its controller rather than the run: every later case would refuse the same way. +BATCH_REFUSALS = ("boundary_unavailable", "unknown_authorization") MEANINGS = ("eval", "evals", "evaluation", "judge", "experiment", "rubric", "score", "compare", "benchmark", "candidate", "arena", "promise", "promises", "oracle", "verdict") @@ -295,15 +307,27 @@ def run_case(harness, case_id, skills_at, out, index, hermes_retain_out=None, mo run.implicit_off = skill_listing.managed_skills(run.project / module.SKILLS_DIR) exclude = run.project / ".git" / "info" / "exclude" exclude.write_text(exclude.read_text() + "".join(f"{d}\n" for d in module.PRIVATE_DIRS)) - authorization = _before_turns(run) - pair = None + scratch = [run.root / rel for rel in SCRATCH_ROOTS[harness]] + for path in scratch: + _Root.open(path, create=True).close() + authorization = _before_turns(run, scratch) + pair = refusal = None try: died = drive(module, run, authorization) error = died and death_record(*died) + if os.path.lexists(run.project): + project = _Root.open(run.project) + try: + project.remove(module.SKILLS_DIR / BOOTSTRAP_MODULES) + finally: + project.close() if run.turns or not died: trace = {**module.harvest(run), "x_implicit_off": run.implicit_off} _seal(authorization, meta(run), trace, error=error) - verdict = grade(authorization) + try: + verdict = grade(authorization) + except GradeRefused as refused: + refusal = refused.receipt else: verdict = error_verdict(case, harness, commit, error) write_file(root / "verdict.json", _json(verdict)) @@ -328,12 +352,13 @@ def run_case(harness, case_id, skills_at, out, index, hermes_retain_out=None, mo finally: if owner is not None: owner.close() - print(json.dumps({"run": str(root), "authorization_id": authorization._id, "case": case_id, "harness": harness, "n": index, - **({"hermes_pair": str(pair.root)} if pair is not None else {}), - "verdicts": {p: v["verdict"] for p, v in verdict["promises"].items()}})) + screen = {"run": str(root), "authorization_id": authorization._id, "case": case_id, "harness": harness, "n": index, + **({"hermes_pair": str(pair.root)} if pair is not None else {}), + **({"refusal": refusal} if refusal else {"verdicts": {p: v["verdict"] for p, v in verdict["promises"].items()}})} + print(json.dumps(screen)) if died: raise LoginRequired(*died) - return root + return screen def meta(run): @@ -364,15 +389,21 @@ def main(argv=None): if args.cmd == "run": out = Path(args.out) out.mkdir(parents=True, exist_ok=True) + refused = False try: preflight(args.harness) for case_id in args.case: for n in range(args.runs): - run_case(args.harness, case_id, args.skills_at, out, n, args.hermes_retain_out, args.model, args.effort) + screen = run_case(args.harness, case_id, args.skills_at, out, n, args.hermes_retain_out, args.model, args.effort) + if "refusal" in screen: + refused = True + if screen["refusal"]["reason"] in BATCH_REFUSALS: + print(f"batch stopped: {screen['refusal']['reason']}: {screen['refusal']['detail']}", file=sys.stderr) + return 2 except LoginRequired as expired: print(f"wave stopped: {expired}", file=sys.stderr) return 3 - return 0 + return 2 if refused else 0 if args.cmd == "grade": for run_id in args.run_ids: try: diff --git a/evals/promises/oracles.py b/evals/promises/oracles.py index d31a4ee0..d9acea56 100644 --- a/evals/promises/oracles.py +++ b/evals/promises/oracles.py @@ -1157,9 +1157,14 @@ def read_only_pin(view): evidence = [f"turn {turn} playbook reads: {[PLAYBOOK.match(r).group(1) for r in reads if PLAYBOOK.match(r)]}"] failures, more = no_edits_verdict(view, turn) evidence += more - if not routed: + rewritten = any(c.get("source") == "chat_history" and c.get("turn") == turn for c in view.trace.get("x_compactions") or []) + if not routed and not rewritten: failures.insert(0, "investigation.md not read in the `new task` turn") - return failed(failures, *evidence) if failures else passed(*evidence) + if failures: + return failed(failures, *evidence) + if not routed: + return inconclusive(f"a compaction rewrote turn {turn}'s chat history, so its playbook reads may be gone", *evidence) + return passed(*evidence) result = route_verdict(view) failures, evidence = no_edits_verdict(view) if result["verdict"] != PASS: @@ -2634,7 +2639,13 @@ def doc_impact_before_completion(view): if not read_any: return inconclusive("run killed before completion", *evidence) if view.killed else failed("documentation-impact never ran before completion", *evidence) if not result: - return failed("completion reply carries no author result", *evidence) if reply else inconclusive("no completion reply", *evidence) + if not reply: + return inconclusive("no completion reply", *evidence) + named = next((t for t in [reply, *view.texts(turn)] if "independent review" in t.lower()), None) + if named: + return inconclusive("the turn names independent review in words the author-result reader does not parse", *evidence, + needs_judge=True, excerpt=excerpt_of(named)) + return failed("completion reply carries no author result", *evidence) return passed(*evidence) diff --git a/evals/promises/report.py b/evals/promises/report.py index 00bff47d..fa898faa 100644 --- a/evals/promises/report.py +++ b/evals/promises/report.py @@ -8,9 +8,24 @@ HARNESSES = ("claude-code", "codex", "hermes", "grok") +def run_verdicts(out): + out = Path(out) + for path in sorted(out.rglob("verdict.json")): + for d in path.relative_to(out).parents: + pair = out / d.parent / "pair.json" + if d.name == "run" and pair.is_file(): + original_root = json.loads(pair.read_text()).get("original_root") + if original_root is not None: + original = (Path(original_root) / "verdict.json").resolve() + if original.is_relative_to(out.resolve()) and original.is_file(): + break + else: + yield path + + def collect(out): cells = collections.defaultdict(list) - for path in sorted(Path(out).rglob("verdict.json")): + for path in run_verdicts(out): record = json.loads(path.read_text()) for pid, result in record["promises"].items(): if not result.get("not_applicable") and not applicability(pid, record["harness"]).get("not_applicable"): @@ -23,7 +38,7 @@ def collect(out): def listing_states(out): states = collections.defaultdict(collections.Counter) - for path in sorted(Path(out).rglob("verdict.json")): + for path in run_verdicts(out): trace = path.parent / "trace.json" listing = json.loads(trace.read_text()).get("x_skill_listing") if trace.is_file() else None if listing: diff --git a/evals/promises/results/2026-10-20-grader-fix.md b/evals/promises/results/2026-10-20-grader-fix.md new file mode 100644 index 00000000..b6381840 --- /dev/null +++ b/evals/promises/results/2026-10-20-grader-fix.md @@ -0,0 +1,58 @@ +# Grader fix reruns, 2026-10-20 + +**Status: COMPLETE.** [2026-10-20-sync-7a9a3ddb.md](./2026-10-20-sync-7a9a3ddb.md) left 7 runs refused at grading and two Claude Code FAILs measured once. This wave reruns those cases with the grader from this branch and the skills at `8731b6eb`. Every run graded. No run was refused and no batch stopped. + +No `--model` or `--effort` was passed, so each harness ran its configured default at high effort. The run directories are kept outside the repository, under `pstack-live-runs/2026-10-21-grader-fix/`. That directory name is a misdated label kept so the runs stay regradable because each grade authorization stores absolute run paths. + +## Previously refused runs + +| harness | case | refusal on 2026-10-20 | now | what the run exercised | +|---|---|---|---|---| +| grok | `route-code-review` | `unapproved_worktree` | PASS | A review worktree registered under `/tmp` was graded. | +| grok | `feature-run` | `unapproved_worktree` | graded, no FAIL | No worktree under `/tmp` this time. | +| grok | `feature-run-no-todo` | `unapproved_worktree` | 1 PASS, 1 FAIL (ORACLE, below) | No worktree under `/tmp` this time. | +| grok | `route-babysit` | `unsafe_link` | PASS | The lead launched watch-pr. No `node_modules` remained at grading. | +| grok | `route-babysit-check` | `unsafe_link` | PASS | The lead launched watch-pr. No `node_modules` remained at grading. | +| hermes | `route-refactoring` | `unapproved_worktree` | PASS | No worktree this time. | +| hermes | `route-autonomous-run` | `unapproved_worktree` | INCONCLUSIVE | Three arena worktrees registered under `/tmp` were graded. | + +The run that registered worktrees in the Hermes profile cache is covered by `test_live.ScratchWorktrees`, not by this wave. + +## Claude Code repeats + +`sticky-followup` and `feature-run` ran 3 times each. + +| promise | case | verdicts | +|---|---|---| +| `worklist-opens-with-playbook-steps` | `feature-run` | PASS 3 | +| `worklist-in-native-task-tool` | `feature-run` | PASS 3 | +| `poteto-mode-reads-principles-and-names-applied` | `feature-run` | FAIL 1, INCONCLUSIVE 2 | +| `reply-says-inconclusive-when-check-cannot-run` | `feature-run` | FAIL 1, INCONCLUSIVE 2 | +| `poteto-mode-sticky` | `sticky-followup` | FAIL 1, PASS 2 | +| `short-prompt-continues-in-context` | `sticky-followup` | FAIL 1, PASS 2 | +| `new-task-rematches` | `sticky-followup` | FAIL 2, PASS 1 | +| `read-only-phrase-pins-investigation` | `sticky-followup` | PASS 3 (2 are wrong, below) | + +The worklist promise failed on Claude Code in both earlier waves and passed 3 of 3 here. The skills tree changed between them, so this wave does not show which change fixed it. The bisect of `e7f0abe3` that the baseline proposed is no longer needed. + +## FAIL classes + +Each FAIL's trace was read before it was tallied. + +| promise | harness | run | class | finding | +|---|---|---|---|---| +| `poteto-mode-sticky`, `short-prompt-continues-in-context` | claude-code | `relay-s27_4et8` | KNOWN | The baseline's borderline row recurred in 1 of 3 runs. Turn 1 recommended fast-forwarding `main`. On `/poteto-mode do it` the lead read `shipping.md` and ran `git merge --ff-only json-flag`. | +| `new-task-rematches` | claude-code | `relay-ulu4bu_1`, `relay-s27_4et8` | REAL | On "new task … don't change any code yet" the lead re-matched away from Opening a PR but chose Bug fix, not Investigation. `relay-s27_4et8` says "it uses the Bug fix playbook". The broken claim is the read-only pin in `docs/guide/02-poteto-mode.md`. The baseline and `2026-10-03.md` had Claude Code PASS. | +| `poteto-mode-reads-principles-and-names-applied` | claude-code | `tally-86yk3j_c` | KNOWN | The lead read three principle files and applied them without naming any. The other two runs named all three. | +| `reply-says-inconclusive-when-check-cannot-run` | claude-code | `tally-r_5omkox` | ORACLE | `reply_inconclusive` judges the whole command `… verify.py … \| tail -1; git diff … \| shasum`. The exit 127 came from `shasum`, and `verify.py` printed `ALL PASS`. | +| `worklist-opens-with-playbook-steps` | grok | `tally-1o8wftmc` | ORACLE | The worklist holds all eight steps. `item_matches_step` misses `**Step 8.** Run **Opening a PR**. State: pending.`, because `ITEM_NUMBER` does not match a `**Step 8.**` label and token overlap is 2 of 5. | + +`read-only-phrase-pins-investigation` passed in `relay-ulu4bu_1` and `relay-s27_4et8`, and both PASSes are wrong. `read_only_pin` accepts `investigation.md` read anywhere in the turn, including as the second file of `cat bug-fix.md; echo ======; cat investigation.md` before the lead runs Bug fix. + +In `relay-ulu4bu_1`, on turn 1 `continue`, the lead also acted on a turn-0 offer and rewrote both branch commits' author to the account identity. No promise grades this. + +## Follow-ups + +- Oracles: judge only the check segment of a compound command in `reply_inconclusive`; accept a `**Step N.**` label in `item_matches_step`; make `read_only_pin` require that the turn routes to Investigation, not only that it reads it. +- Skills: on "don't change any code yet", Claude Code chose Bug fix in 2 of 3 runs. Find what in the poteto-mode entry steers a defect report past the read-only pin. +- `sticky-followup` still leaves no open step after turn 0, which keeps its sticky verdict ambiguous. diff --git a/evals/promises/test_grok.py b/evals/promises/test_grok.py index 34eb83f6..ed968552 100644 --- a/evals/promises/test_grok.py +++ b/evals/promises/test_grok.py @@ -6,6 +6,7 @@ from types import SimpleNamespace from unittest import mock +import oracles from grade_boundary import GradeRefused from harnesses import grok @@ -299,5 +300,48 @@ def test_harvest_never_rereads_a_capture_rewritten_after_the_copy(self): self.change_captures_after_copy(lambda path, outside: path.write_bytes(outside.read_bytes())) +class CompactedHistory(unittest.TestCase): + """Minimized from Grok relay-6tna36t5 in the 2026-10-20 wave: turn 3 compacted and rewrote the lead's chat history.""" + + FIXTURE = Path(__file__).resolve().parent / "testdata" / "grok_compaction" / "relay-6tna36t5-lead-chat_history.jsonl" + PROMPTS = ["/poteto-mode add a --json flag. text output stays byte-identical. verify both forms.", "continue", "/poteto-mode do it", + "/poteto-mode new task. figure out why the cache entry survives logout. don't change any code yet."] + + def harvest(self, chat, prompts=PROMPTS): + tmp = tempfile.TemporaryDirectory(prefix="pstack-grok-compaction-") + self.addCleanup(tmp.cleanup) + root = Path(tmp.name).resolve() / "run" + lead = root / "grok-home" / "sessions" / "cwd" / "lead" + lead.mkdir(parents=True) + (lead / "chat_history.jsonl").write_bytes(chat) + run = SimpleNamespace(root=root, project=root / "w" / "relay", case={"turns": prompts}, + turns=[{"index": i, "session_id": "lead", "argv": ["/bin/grok", "-p", p]} for i, p in enumerate(prompts)]) + with mock.patch.object(grok, "host_home", return_value=root / "host"): + return grok.harvest(run) + + def test_the_compacted_new_task_turn_is_not_graded_a_failed_route(self): + case = json.loads((Path(__file__).resolve().parent / "cases" / "sticky-followup" / "case.json").read_text()) + result = oracles.check("read-only-phrase-pins-investigation", self.harvest(self.FIXTURE.read_bytes()), case, None) + self.assertEqual(result["verdict"], oracles.INCONCLUSIVE, result) + + def test_a_rewritten_history_records_one_compaction_in_the_turn_whose_prompt_it_kept(self): + trace = self.harvest(self.FIXTURE.read_bytes()) + self.assertEqual((trace["x_compactions"], trace["x_turns_without_events"]), ([{"turn": 3, "source": "chat_history"}], [0, 1, 2])) + + def test_a_compact_turn_does_not_shift_the_turns_after_it(self): + """Minimized from Grok relay-55ed4sis in the 2026-10-20 wave, whose prompt_index skips the `/compact` turn.""" + chat = (self.FIXTURE.parent / "relay-55ed4sis-lead-chat_history.jsonl").read_bytes() + prompts = ["what does relay do? two sentences is plenty.", "/compact", + "/poteto-mode im stepping away. keep going until the migration check reports zero old callers. log your decisions."] + trace = self.harvest(chat, prompts) + reads = [e["turn"] for e in trace["events"] if e.get("name") == "read_file" and "figure-it-out" in e["input"]["target_file"]] + self.assertEqual((reads, trace["x_turns_without_events"]), ([2], [1])) + self.assertEqual(trace["x_compactions"], [{"turn": 0, "source": "chat_history"}]) + + def test_a_history_without_compaction_records_none(self): + chat = b"".join(line + b"\n" for line in self.FIXTURE.read_bytes().splitlines() if b"compaction_meta" not in line) + self.assertEqual(self.harvest(chat)["x_compactions"], []) + + if __name__ == "__main__": unittest.main() diff --git a/evals/promises/test_live.py b/evals/promises/test_live.py index 2dd49f26..1dc02521 100644 --- a/evals/promises/test_live.py +++ b/evals/promises/test_live.py @@ -210,7 +210,7 @@ def turn(run, text, index): with mock.patch.object(live, "load_case", return_value=case), \ mock.patch.object(module, "prepare", side_effect=prepare), \ mock.patch.object(module, "turn", side_effect=turn), redirect_stdout(output): - root = live.run_case(harness, case["id"], "HEAD", out, 0) + root = Path(live.run_case(harness, case["id"], "HEAD", out, 0)["run"]) trace = json.loads((root / "trace.json").read_text()) record = json.loads((root / "run.json").read_text()) self.assertEqual(trace["x_turns"], [ @@ -249,7 +249,7 @@ def turn(run, text, index): with mock.patch.object(live, "load_case", return_value=case), mock.patch.object(codex, "prepare", side_effect=prepare), \ mock.patch.object(codex, "turn", side_effect=turn), redirect_stdout(io.StringIO()): - root = live.run_case("codex", case["id"], "HEAD", out, 0, model="gpt-6-luna", effort="xhigh") + root = Path(live.run_case("codex", case["id"], "HEAD", out, 0, model="gpt-6-luna", effort="xhigh")["run"]) installed = (root / "w" / "rollup" / ".agents/skills/deslop/SKILL.md").read_text() trace = json.loads((root / "trace.json").read_text()) record = json.loads((root / "run.json").read_text()) @@ -298,6 +298,218 @@ def test_a_run_with_a_clean_turn_is_graded_even_when_the_trace_carries_only_the_ DESLOP_STEP_5 = "Comments the rules below do not settle go to `/no-comments`." +def synthetic_turns(harness, during_turn=lambda run, index: None): + """prepare and turn stand-ins that write empty owned streams, so run_case harvests and grades with no model.""" + def prepare(run): + (run.root / "launch.json").write_text(json.dumps({"path": "synthetic-no-model", "source": "test", "version": "test", "rejected": []})) + + def turn(run, text, index): + during_turn(run, index) + (run.root / f"turn-{index}.jsonl").write_bytes(b"") + return {"index": index, "session_id": "owned-session", "argv": [harness, text], "exit_code": 0, "timed_out": False, + "duration_s": 0.25, "stream": str(run.root / f"turn-{index}.jsonl"), "stderr": str(run.root / f"turn-{index}.err"), + f"{harness}_bin": "synthetic-no-model", f"{harness}_bin_source": "test", + **({"last_message": str(run.root / "last.txt")} if harness == "codex" else {})} + return prepare, turn + + +def steer_case(): + case = live.load_case("principle-steer-run") + case["turns"] = ["go"] + case["tmp_lock"] = False + return case + + +def add_worktree(run, path): + live.git_run(run.project, "worktree", "add", "-q", "-b", f"candidate-{path.name}", str(path)) + + +@unittest.skipUnless((platform.system(), platform.release(), platform.machine()) == ("Darwin", "25.6.0", "arm64"), + "native parent grading requires the reviewed Darwin 25.6.0 arm64 runtime") +class ScratchWorktrees(unittest.TestCase): + def test_a_worktree_the_agent_adds_under_the_run_tmpdir_is_graded(self): + case = steer_case() + with tempfile.TemporaryDirectory(prefix="pstack-live-scratch-") as tmp: + out = Path(tmp).resolve() + for harness in ("codex", "grok"): + with self.subTest(harness=harness): + module = live.adapter(harness) + prepare, turn = synthetic_turns(harness, lambda run, index: add_worktree( + run, run.root / "tmp" / "arena-tally-json.yq6S6F" / "candidate-2")) + output = io.StringIO() + with mock.patch.object(live, "load_case", return_value=case), mock.patch.object(module, "prepare", side_effect=prepare), \ + mock.patch.object(module, "turn", side_effect=turn), redirect_stdout(output): + screen = live.run_case(harness, case["id"], "HEAD", out, 0) + self.assertEqual(json.loads(output.getvalue()), screen) + self.assertNotIn("refusal", screen) + self.assertEqual(set(screen["verdicts"]), set(case["promises"])) + self.assertTrue((Path(screen["run"]) / "tmp/arena-tally-json.yq6S6F/candidate-2/.git").is_file()) + + def test_a_worktree_hermes_adds_under_its_write_safe_profile_cache_is_graded(self): + from test_hermes_evidence import _OwnerFixture + from harnesses import hermes + from test_hermes import HermesDatabase + fixture = _OwnerFixture() + fixture.setUp() + self.addCleanup(fixture.doCleanups) + case = live.load_case("principle-steer-run") + case["turns"] = ["go"] + case["entry"] = None + original_turn = hermes.turn + candidates = [] + + def turn(run, text, index): + record = original_turn(run, text, index) + [argv] = [argv for argv, *_ in fixture.calls[-3:] if any(a.startswith("HERMES_WRITE_SAFE_ROOT=") for a in argv)] + safe = next(a for a in argv if a.startswith("HERMES_WRITE_SAFE_ROOT=")).split("=", 1)[1].split(":") + self.assertEqual(safe[0], str(run.project)) + candidates.append(Path(safe[1]) / "scratch" / "tally-architect-arena" / "candidate-2") + add_worktree(run, candidates[-1]) + db = HermesDatabase(hermes.profile(run) / "state.db") + db.session("root") + db.con.execute("update sessions set cwd=?", (str(run.project),)) + db.message("root", "user", "go") + db.message("root", "assistant", "owned CLI reply") + db.done() + return record + + output = io.StringIO() + with mock.patch.object(live, "load_case", return_value=case), mock.patch.object(hermes, "turn", turn), redirect_stdout(output): + result = live.main(["run", "--harness", "hermes", "--case", case["id"], "--out", str(fixture.base / "scratch-runs"), + "--hermes-retain-out", str(fixture.base / "scratch-retained")]) + screen = json.loads(output.getvalue()) + self.assertEqual((result, "refusal" in screen), (0, False)) + self.assertEqual(set(screen["verdicts"]), set(case["promises"])) + self.assertTrue((candidates[0] / ".git").is_file()) + + +@unittest.skipUnless((platform.system(), platform.release(), platform.machine()) == ("Darwin", "25.6.0", "arm64"), + "native parent grading requires the reviewed Darwin 25.6.0 arm64 runtime") +class BootstrapDependencies(unittest.TestCase): + def test_a_project_deleted_during_the_turn_reaches_grading(self): + import shutil + case = steer_case() + module = live.adapter("codex") + prepare, turn = synthetic_turns("codex", lambda run, index: shutil.rmtree(run.project)) + with tempfile.TemporaryDirectory(prefix="pstack-live-bootstrap-deleted-") as tmp: + out = Path(tmp).resolve() + output = io.StringIO() + with mock.patch.object(live, "load_case", return_value=case), mock.patch.object(module, "prepare", side_effect=prepare), \ + mock.patch.object(module, "turn", side_effect=turn), mock.patch.object(live, "grade", wraps=live.grade) as grade, \ + redirect_stdout(output): + screen = live.run_case("codex", case["id"], "HEAD", out, 0) + grade.assert_called_once() + self.assertEqual(json.loads(output.getvalue()), screen) + self.assertIn("refusal", screen) + self.assertTrue((Path(screen["run"]) / "trace.json").is_file()) + + def plant(self, run, index): + modules = run.project / live.adapter(run.harness).SKILLS_DIR / "poteto-mode" / "scripts" / "node_modules" + (modules / "typescript" / "bin").mkdir(parents=True) + (modules / "typescript" / "bin" / "tsc").write_text("#!/usr/bin/env node\n") + (modules / ".bin").mkdir() + (modules / ".bin" / "tsc").symlink_to("../typescript/bin/tsc") + + def test_the_watch_pr_bun_install_under_the_installed_skills_is_removed_before_grading(self): + case = steer_case() + with tempfile.TemporaryDirectory(prefix="pstack-live-bootstrap-") as tmp: + out = Path(tmp).resolve() + for harness in ("codex", "grok"): + with self.subTest(harness=harness): + module = live.adapter(harness) + prepare, turn = synthetic_turns(harness, self.plant) + output = io.StringIO() + with mock.patch.object(live, "load_case", return_value=case), mock.patch.object(module, "prepare", side_effect=prepare), \ + mock.patch.object(module, "turn", side_effect=turn), redirect_stdout(output): + screen = live.run_case(harness, case["id"], "HEAD", out, 0) + scripts = Path(screen["run"]) / "w" / case["fixture"] / module.SKILLS_DIR / "poteto-mode" / "scripts" + self.assertEqual(set(screen["verdicts"]), set(case["promises"])) + self.assertFalse((scripts / "node_modules").exists() or (scripts / "node_modules").is_symlink()) + self.assertTrue((scripts / "bootstrap.ts").is_file()) + + def test_a_link_swapped_into_the_bootstrap_path_is_refused_without_deleting_through_it(self): + import shutil + case = steer_case() + for linked in ("poteto-mode/scripts", "poteto-mode/scripts/node_modules"): + with self.subTest(linked=linked), tempfile.TemporaryDirectory(prefix="pstack-live-bootstrap-link-") as tmp: + out = Path(tmp).resolve() + outside = out.parent / f"{out.name}-outside" + self.addCleanup(shutil.rmtree, outside, True) + + def swap(run, index): + (outside / "node_modules" / ".bin").mkdir(parents=True) + (outside / "node_modules" / ".bin" / "tsc").write_text("outside\n") + target = run.project / live.adapter("codex").SKILLS_DIR / linked + shutil.rmtree(target, ignore_errors=True) + target.parent.mkdir(parents=True, exist_ok=True) + target.symlink_to(outside if linked.endswith("scripts") else outside / "node_modules") + + prepare, turn = synthetic_turns("codex", swap) + with mock.patch.object(live, "load_case", return_value=case), \ + mock.patch.object(live.adapter("codex"), "prepare", side_effect=prepare), \ + mock.patch.object(live.adapter("codex"), "turn", side_effect=turn), redirect_stdout(io.StringIO()): + with self.assertRaises(GradeRefused) as refused: + live.run_case("codex", case["id"], "HEAD", out, 0) + self.assertEqual(refused.exception.receipt["reason"], "unsafe_link") + self.assertEqual(list(out.rglob("trace.json")), []) + self.assertEqual((outside / "node_modules" / ".bin" / "tsc").read_text(), "outside\n") + + +@unittest.skipUnless((platform.system(), platform.release(), platform.machine()) == ("Darwin", "25.6.0", "arm64"), + "native parent grading requires the reviewed Darwin 25.6.0 arm64 runtime") +class GradeRefusalBatch(unittest.TestCase): + def wave(self, during_turn, runs=2): + case = steer_case() + module = live.adapter("codex") + prepare, turn = synthetic_turns("codex", during_turn) + tmp = tempfile.TemporaryDirectory(prefix="pstack-live-batch-") + self.addCleanup(tmp.cleanup) + out = Path(tmp.name).resolve() / "runs" + output, errors = io.StringIO(), io.StringIO() + with mock.patch.object(live, "load_case", return_value=case), mock.patch.object(live, "preflight"), \ + mock.patch.object(module, "prepare", side_effect=prepare), mock.patch.object(module, "turn", side_effect=turn) as turns, \ + redirect_stdout(output), redirect_stderr(errors): + code = live.main(["run", "--harness", "codex", "--case", case["id"], "--runs", str(runs), "--out", str(out)]) + return code, turns.call_count, [json.loads(line) for line in output.getvalue().splitlines()], errors.getvalue() + + def test_a_grade_refusal_in_the_first_case_still_runs_the_second_and_exits_2(self): + def first_run_strays(run, index): + if not list(run.root.parent.glob("*/elsewhere")): + add_worktree(run, run.root / "elsewhere" / "candidate-2") + + code, turns, lines, _ = self.wave(first_run_strays) + self.assertEqual((code, turns, len(lines)), (2, 2, 2)) + refused, graded = lines + self.assertEqual(refused["refusal"]["reason"], "unapproved_worktree") + self.assertEqual(refused["refusal"]["run_id"], refused["authorization_id"]) + self.assertNotIn("verdicts", refused) + self.assertEqual((graded["n"], "refusal" in graded), (1, False)) + self.assertEqual(set(graded["verdicts"]), set(live.load_case("principle-steer-run")["promises"])) + + def test_a_refused_run_stays_sealed_for_a_later_grade(self): + code, _, [refused], _ = self.wave(lambda run, index: add_worktree(run, run.root / "elsewhere" / "candidate-2"), runs=1) + self.assertEqual(code, 2) + out = Path(refused["run"]).parent + with redirect_stderr(io.StringIO()) as again: + self.assertEqual(live.main(["grade", refused["authorization_id"], "--out", str(out)]), 2) + self.assertEqual(json.loads(again.getvalue())["refusal"]["reason"], "unapproved_worktree") + + def test_a_grader_that_cannot_launch_stops_the_batch_after_the_first_case(self): + import grade_boundary + popen = subprocess.Popen + + def no_sandbox(argv, *args, **kwargs): + if argv[0] == "/usr/bin/sandbox-exec": + raise OSError("sandbox-exec is gone") + return popen(argv, *args, **kwargs) + + with mock.patch.object(grade_boundary.subprocess, "Popen", side_effect=no_sandbox): + code, turns, lines, errors = self.wave(lambda run, index: None) + self.assertEqual((code, turns, len(lines)), (2, 1, 1)) + self.assertEqual(lines[0]["refusal"]["reason"], "boundary_unavailable") + self.assertIn("batch stopped", errors) + + class SkillOverlays(unittest.TestCase): def tree(self, text): tmp = tempfile.TemporaryDirectory(prefix="pstack-overlay-") diff --git a/evals/promises/test_oracles.py b/evals/promises/test_oracles.py index e463672f..7becfddf 100644 --- a/evals/promises/test_oracles.py +++ b/evals/promises/test_oracles.py @@ -1626,6 +1626,41 @@ def test_preparation_after_compaction_is_inconclusive_with_or_without_a_refusal( (INCONCLUSIVE, ["run killed before a routing decision"])) + +class ReadOnlyPinAfterCompaction(unittest.TestCase): + """Grok relay-6tna36t5: a compaction in the `new task` turn rewrote the history that held its playbook reads.""" + + case = load_case("sticky-followup") + + def grade(self, compactions, more=lambda project: ()): + tmp = tempfile.TemporaryDirectory() + self.addCleanup(tmp.cleanup) + project = make_repo(tmp.name, 2) + events = in_turn(3, [{"seq": 0, "kind": "user", "text": "new task"}, + {"seq": 1, "kind": "tool_call", "name": "read_file", "input": {"target_file": "/w/.agents/skills/how/SKILL.md"}}, + *more(project)]) + trace = minimal(events=events, harness="grok", x_compactions=compactions, x_turns_without_events=[0, 1, 2]) + return grade("read-only-phrase-pins-investigation", trace, self.case, project) + + def test_a_compaction_in_the_graded_turn_leaves_a_missing_route_unproven(self): + result = self.grade([{"turn": 3, "source": "chat_history"}]) + self.assertEqual(result["verdict"], INCONCLUSIVE, result) + + def test_a_missing_route_still_fails_without_that_compaction(self): + for compactions in ([], [{"turn": 1, "source": "chat_history"}], [{"turn": 3, "source": "stream"}], [{"turn": 3, "source": "rollout"}]): + with self.subTest(compactions=compactions): + result = self.grade(compactions) + self.assertEqual((result["verdict"], result["failures"]), (FAIL, ["investigation.md not read in the `new task` turn"]), result) + + def test_an_edit_or_commit_after_the_compaction_still_fails(self): + for failure, more in (("edited project files", lambda project: [edit(2, str(project / "f.py"))]), + ("1 commit(s) made in turn 3", lambda project: bash(2, f"git -C {project} commit -am 'sneak'"))): + with self.subTest(failure=failure): + result = self.grade([{"turn": 3, "source": "chat_history"}], more) + self.assertEqual(result["verdict"], FAIL, result) + self.assertTrue(any(f.startswith(failure) for f in result["failures"]), result) + + class NameOnlyStaysUnloaded(unittest.TestCase): case = load_case("name-only-unnamed-request") hidden = ["figure-it-out", "poteto-mode", "show-me-your-work", "swarm"] @@ -2178,6 +2213,37 @@ def test_a_plan_to_get_any_required_review_is_not_an_author_result(self): self.assertEqual(result["verdict"], FAIL, result) +class AuthorResultInOtherWords(unittest.TestCase): + """Codex tally-1lw1ob03 in the 2026-10-20 wave states its author result in words the parser does not read.""" + + INTERIM = ("The README now documents the JSON shape and shows output verified against `sample.txt`. The documentation " + "author pass requires independent review, so a separate reviewer will check those claims before I finish.") + REPLY = ("Added `--json` with ordered `word` and `count` records.\n\nEight regression tests and eleven additional CLI checks " + "passed. Independent code and documentation review passed.") + REPLY_NAMING_IT = REPLY + "\n\nOne worker owned the coupled files, followed by independent review." + + def grade_turn(self, interim, reply): + events = in_turn(0, [{"seq": 0, "kind": "user", "text": "add --json"}, read(7, "documentation-impact/SKILL.md"), + text(113, interim), text(151, reply)]) + events += in_turn(1, [{"seq": 160, "kind": "user", "text": "review the docs"}, text(170, "Pass. No documentation gaps found.")]) + return grade("poteto-runs-documentation-impact-before-completion", minimal(events=events, harness="codex"), load_case("doc-impact-run")) + + def test_an_interim_report_naming_independent_review_goes_to_the_judge(self): + result = self.grade_turn(self.INTERIM, self.REPLY) + self.assertEqual((result["verdict"], result.get("needs_judge")), (INCONCLUSIVE, True), result) + self.assertIn("requires independent review", result["excerpt"]) + + def test_a_reply_naming_independent_review_goes_to_the_judge(self): + result = self.grade_turn("README updated.", self.REPLY_NAMING_IT) + self.assertEqual((result["verdict"], result.get("needs_judge")), (INCONCLUSIVE, True), result) + self.assertEqual(result["excerpt"], self.REPLY_NAMING_IT) + + def test_a_turn_that_never_names_independent_review_still_fails(self): + result = self.grade_turn("README updated.", self.REPLY) + self.assertEqual(result["verdict"], FAIL, result) + self.assertEqual(result["failures"], ["completion reply carries no author result"]) + + class ProjectRelativePaths(unittest.TestCase): def project(self): tmp = tempfile.TemporaryDirectory() diff --git a/evals/promises/test_report.py b/evals/promises/test_report.py index a4c74ffc..e443bc04 100644 --- a/evals/promises/test_report.py +++ b/evals/promises/test_report.py @@ -56,6 +56,44 @@ def test_a_run_whose_transcript_could_not_be_read_counts_only_the_promises_that_ self.assertIn("| `persona-delivery-hermes-grok` | | | INCONCLUSIVE | | |", table) self.assertNotIn("claude-effort-applies-via-effort-agents", table) + def test_a_retained_hermes_pair_counts_its_run_once(self): + with tempfile.TemporaryDirectory() as out: + for rel in ("hermes/runs/tally-a", "hermes-retained/pair-1/run"): + run = Path(out) / rel + run.mkdir(parents=True) + (run / "verdict.json").write_text(json.dumps({"harness": "hermes", "promises": { + "p1": {"verdict": "PASS", "failures": [], "evidence": []}}})) + (run / "trace.json").write_text(json.dumps({"x_skill_listing": {"step": "installed"}})) + pair = Path(out) / "hermes-retained" / "pair-1" + (pair / "pair.json").write_text(json.dumps({"original_root": str(Path(out) / "hermes/runs/tally-a")})) + for scope in (Path(out), pair.parent, pair): + with self.subTest(scope=scope): + table = report.render(scope, upstream=Path(out)) + self.assertIn("hermes pass rate: 1 of 1\n", table + "\n") + self.assertIn("| `p1` | | | PASS | |", table) + self.assertIn("hermes skill listing: installed 1", table) + + def test_a_retained_hermes_pair_counts_when_its_original_run_is_missing(self): + with tempfile.TemporaryDirectory() as out: + pair = Path(out) / "hermes-retained" / "pair-1" + pair.mkdir(parents=True) + self.run_dir(pair, "run", "hermes", {"p1": "PASS"}) + (pair / "run" / "trace.json").write_text(json.dumps({"x_skill_listing": {"step": "installed"}})) + (pair / "pair.json").write_text(json.dumps({"original_root": str(Path(out) / "hermes/runs/tally-a")})) + for scope in (Path(out), pair.parent, pair): + with self.subTest(scope=scope): + table = report.render(scope, upstream=Path(out)) + self.assertIn("hermes pass rate: 1 of 1\n", table + "\n") + self.assertIn("hermes skill listing: installed 1", table) + + def test_a_run_directory_without_a_pair_beside_it_still_counts(self): + with tempfile.TemporaryDirectory() as out: + self.run_dir(out, "a", "grok", {"p1": "PASS"}) + (Path(out) / "b").mkdir() + self.run_dir(Path(out) / "b", "run", "grok", {"p1": "FAIL"}) + table = report.render(Path(out), upstream=Path(out)) + self.assertIn("grok pass rate: 1 of 2\n", table + "\n") + def test_a_harness_with_no_error_prints_a_plain_pass_rate(self): with tempfile.TemporaryDirectory() as out: self.run_dir(out, "a", "grok", {"p1": "PASS", "p2": "PASS", "p3": "FAIL"}) diff --git a/evals/promises/testdata/grok_compaction/relay-55ed4sis-lead-chat_history.jsonl b/evals/promises/testdata/grok_compaction/relay-55ed4sis-lead-chat_history.jsonl new file mode 100644 index 00000000..282f4d23 --- /dev/null +++ b/evals/promises/testdata/grok_compaction/relay-55ed4sis-lead-chat_history.jsonl @@ -0,0 +1,6 @@ +{"type": "system", "content": "You are Grok 4.7 released by xAI. You are an autonomous agent that completes software engineering tasks. There is no human operator in this session. Your main goal is to complete the user's request in the tag.\n\n\nOS Version: macos\nShell: /bin/zsh\nWorkspace Path: /run/w/relay\nToday's date: 2026-10-20\n\n\n\nThe rules section has a number of possibl"}], "synthetic_reason": "compaction_meta"} +{"type": "user", "content": [{"type": "text", "text": "\nwhat does relay do? two sentences is plenty.\n"}]} +{"type": "user", "content": [{"type": "text", "text": "This session is being continued from a previous conversation that ran out of context. The summary below covers the earlier portion of the conversation.\n\nSummary:\n1. Primary Request and Intent: The user asked \"what does relay do? two sentenc"}], "synthetic_reason": "compaction_meta"} +{"type": "user", "content": [{"type": "text", "text": "\n/poteto-mode im stepping away. keep going until the migration check reports zero old callers. log your decisions.\n\n\n\n tag.\n\n\nOS Version: macos\nShell: /bin/zsh\nWorkspace Path: /run/w/relay\nToday's date: 2026-10-20\n\n\n\nThe rules section has a number of possibl"}], "synthetic_reason": "compaction_meta"} +{"type": "user", "content": [{"type": "text", "text": "\n/poteto-mode new task. figure out why the cache entry survives logout. don't change any code yet.\n"}]} +{"type": "user", "content": [{"type": "text", "text": "This session is being continued from a previous conversation that ran out of context. The summary below covers the earlier portion of the conversation.\n\nSummary:\n1. Primary Request and Intent: The user first asked, via `/poteto-mode`, to ad"}], "synthetic_reason": "compaction_meta"} +{"type": "assistant", "content": "The cache question is still read-only. I'll load the explainer and investigator prompts, then send both out with the code anchor and no file edits.", "tool_calls": [{"id": "call-d58c3b5a-408c-4836-9cc1-e80bdbd78ee7-0", "name": "read_file", "arguments": "{\"target_file\":\"/run/w/relay/.agents/skills/how/references/explainer-prompt.md\"}"}, {"id": "call-d58c3b5a-408c-4836-9cc1-e80bdbd78ee7-1", "name": "read_file", "arguments": "{\"target_file\":\"/run/w/relay/.agents/skills/why/references/investigator-prompt.md\"}"}, {"id": "call-d58c3b5a-408c-4836-9cc1-e80bdbd78ee7-2", "name": "read_file", "arguments": "{\"target_file\":\"/run/w/relay/.agents/skills/why/references/sources/code-archaeology.md\",\"limit\":120}"}, {"id": "call-d58c3b5a-408c-4836-9cc1-e80bdbd78ee7-3", "name": "read_file", "arguments": "{\"target_file\":\"/run/w/relay/.agents/skills/how/SKILL.md\",\"limit\":80}"}], "model_id": "grok-4.7-build", "reasoning_effort": "high"} +{"type": "assistant", "content": "The cache entry stays for the life of one Python process. This repository has no logout. `clear()` would drop the entry, and no caller invokes it. A new process starts with an empty dict. The git history does not say why any of that is so.\n", "model_id": "grok-4.7-build", "reasoning_effort": "high"}