Repository navigation
sync: upstream 88d4751 (/sharpen returns only the rewritten prompt) - #283
mdsmithaustin wants to merge 2 commits into
Conversation
…n prompt Upstream-SHA: 88d47514b82e7c44f00f2b2a72ff6dd12158038c
|
Closing as superseded. That commit keeps the four upstream asks and adds rewrite-only and original-deliverable boundaries, because live runs showed the exact upstream wording let Hermes execute the task and let Claude Code and Grok add deliverables. PORTING.md records those boundaries for future syncs. It also keeps the stable promise ID This branch conflicts with |
|
Prototype variant 1 of 3, throwaway. Compact recap of PR 289, posted here only to test GitHub rendering. Recap. Risk: low (judgment). Eval tooling only. Base
|
| Area | Files | Lines | What changed |
|---|---|---|---|
| Batch control | live.py |
+41 −10 | A grade refusal prints as refusal, the batch continues, and main exits 2. boundary_unavailable and unknown_authorization stop it at once. |
| Grading boundary | grade_boundary.py |
+22 −2 | _before_turns admits harness scratch roots. New _Root.remove deletes a directory without following links. |
| Grok trace | harnesses/grok.py |
+18 −3 | Turn numbers skip /compact. History rewrites land in x_compactions. |
| Oracles | oracles.py |
+14 −3 | Two FAIL paths now return INCONCLUSIVE, one of them with needs_judge. |
| Report | report.py |
+17 −2 | A retained Hermes run/ copy no longer counts twice. |
| Docs, results | README.md, results/2026-10-20-grader-fix.md |
+68 −8 | Behavior docs and the rerun report. |
| Tests | 4 test files, 2 Grok fixtures | +374 −2 | Each fix has its own tests. |
Figure 1. A grade-time refusal now returns to main as data, and only two controller-level reasons stop the batch early.
sequenceDiagram
participant M as main
participant R as run_case
participant B as grade_boundary
participant G as grade
M->>R: runs one case
R->>B: declares tmp scratch roots as allocations
R->>R: drives the harness turns
R->>B: removes poteto-mode/scripts/node_modules without following links
R->>G: grades the sealed run
alt grade refuses
G-->>R: raises GradeRefused
R-->>M: returns a screen with refusal
M->>M: stops with exit 2 on a controller-level reason
M->>M: otherwise runs the next case and exits 2 at the end
else grade finishes
G-->>R: returns verdicts
R-->>M: returns a screen with verdicts
end
|
Prototype variant 2 of 3, throwaway. Rich recap of PR 289, posted here only to test GitHub rendering. Verdict. Grading refusals that ended a live-eval batch are either prevented or recorded, and three oracle and report paths stop overstating results. Risk: low (judgment). Base Files
Key changes
Grading path, before and afterFigure 1. Two refusal sources are cleared before grading, and any refusal left over is recorded instead of ending the batch. flowchart LR
subgraph before["Before, bfe834de"]
direction TB
b1["authorize project parent only"]
b2["drive turns"]
b3["harvest and seal"]
b4["grade"]
b5["[-] GradeRefused escapes run_case"]
b6["[-] batch ends"]
end
subgraph after["After, 081421a9"]
direction TB
a1["[~] authorize project parent and scratch roots"]
a2["drive turns"]
a3["[+] remove bootstrap node_modules"]
a4["harvest and seal"]
a5["grade"]
a6["[+] screen carries refusal"]
a7{"[+] controller-level reason?"}
a8["[+] stop batch, exit 2"]
a9["[+] run next case, exit 2 at end"]
end
b1 -- "starts" --> b2
b2 -- "leaves tree for" --> b3
b3 -- "hands sealed run to" --> b4
b4 -- "raises" --> b5
b5 -- "ends" --> b6
a1 -- "starts" --> a2
a2 -- "leaves tree for" --> a3
a3 -- "clears links for" --> a4
a4 -- "hands sealed run to" --> a5
a5 -- "records refusal in" --> a6
a6 -- "checks" --> a7
a7 -- "yes, stops" --> a8
a7 -- "no, continues" --> a9
Risks
Review order
|
|
Prototype variant 3 of 3, throwaway. Component diff recap of PR 289, posted here only to test GitHub rendering of Mermaid C4. Base Figure 1. The PR adds 3 components and changes 7, all inside C4Component
title Component diff for PR 289, evals/promises grading path
Container_Boundary(livepy, "live.py") {
Component(main, "[~] main", "Python", "continues after a grade refusal, exits 2")
Component(runcase, "[~] run_case", "Python", "creates scratch roots, clears bootstrap modules, returns a screen")
}
Container_Boundary(gb, "grade_boundary.py") {
Component(beforeturns, "[~] _before_turns", "Python", "declares scratch allocations")
Component(remove, "[+] _Root.remove", "Python", "deletes a directory without following links")
Component(grade, "grade", "Python", "raises GradeRefused")
}
Container_Boundary(grokpy, "harnesses/grok.py") {
Component(parse, "[~] parse_session, harvest", "Python", "records compaction rewrites")
Component(turnidx, "[+] turn_of_index", "Python", "skips /compact in turn numbers")
}
Container_Boundary(orc, "oracles.py") {
Component(pin, "[~] read_only_pin", "Python", "INCONCLUSIVE after a rewrite")
Component(doc, "[~] doc_impact_before_completion", "Python", "routes independent review wording to the judge")
}
Container_Boundary(rep, "report.py") {
Component(collect, "[~] collect, listing_states", "Python", "iterate run_verdicts")
Component(rv, "[+] run_verdicts", "Python", "skips retained Hermes copies")
}
Rel(main, runcase, "[~] reads refusal from")
Rel(runcase, beforeturns, "[~] authorizes scratch roots via")
Rel(runcase, remove, "[+] clears node_modules with")
Rel(runcase, parse, "harvests Grok trace through")
Rel(parse, turnidx, "[+] numbers turns with")
Rel(runcase, grade, "[~] catches GradeRefused from")
Rel(grade, pin, "runs")
Rel(grade, doc, "runs")
Rel(collect, rv, "[+] lists verdicts through")
UpdateLayoutConfig($c4ShapeInRow="2", $c4BoundaryInRow="2")
Evidence per changed node
|
|
Story prototype 1 of 4, throwaway. A story strip for PR 289, built from its commits, its verdict comment, and the results files on main. Variant A is Mermaid Story. 7 runs refused at grading now grade, and one refusal no longer ends the batch.A. timeline
title PR 289, grading refusals
Problem : 7 runs refused at grading
Root cause : tmp/ worktrees never approved
: watch-pr left a symlink
Fix : sign tmp/ in before turns
: drop bootstrap links unfollowed
Proof : rerun graded all 7
For you : batches grade and go on
The strip leaves out three smaller fixes in the same PR. It renumbers Grok turns after B.
|
|
Story prototype 2 of 4, throwaway. A proof trail for PR 293, still open, built from its 27 commits, CI, and two local trail-review verdict files that were never posted on the PR. Runs of commits are collapsed. An X marks red test commits, a square marks no-mistakes gate rounds. Story. Two trail reviews failed, the same class of miss came back, and round 2 changed the premise.---
config:
gitGraph:
mainBranchName: "PR 293"
---
gitGraph LR:
commit id: "build 429cef7"
commit id: "gate x2 47d40c6" type: HIGHLIGHT tag: "review 1 FAIL"
commit id: "5 red tests" type: REVERSE
commit id: "5 fixes 785bd55"
commit id: "docs ab16d2a"
commit id: "gate x3 d47b97a" type: HIGHLIGHT tag: "review 2 FAIL"
commit id: "4 red tests" type: REVERSE
commit id: "fix c78d87e" tag: "premise: scan every block"
commit id: "3 fixes e6c53d0"
commit id: "docs 23c1d9f" tag: "review 3 pending"
|
|
Story prototype 3 of 4, throwaway. What the user sees for PR 289. An evaluator runs Story. Before, the first refused run ended the batch. After, all 7 runs grade in one batch.Before. The grader refuses the run, and nothing catches the refusal. sequenceDiagram
participant E as Evaluator
participant L as live.py run
participant A as Agent
participant G as Grader
E->>L: run the batch
L->>A: case 1 turns
A-->>L: tmp/ worktree or watch-pr link
L->>G: grade case 1
G-->>L: GradeRefused
L-->>E: uncaught, batch ends
Note over E,G: later cases never run
After. Scratch roots are signed in before the turns, and bootstrap links are gone before grading. sequenceDiagram
participant E as Evaluator
participant L as live.py run
participant A as Agent
participant G as Grader
E->>L: run the batch
L->>L: sign tmp/ in as allocation
L->>A: case 1 turns
A-->>L: tmp/ worktree or watch-pr link
L->>L: drop node_modules, no follow
L->>G: grade case 1
G-->>L: verdicts
Note over L,G: same for cases 2 to 7
L-->>E: 7 of 7 graded
|
|
Story prototype 4 of 4, throwaway. The number that moved in PR 289, from the 2026-10-20 sync results and the grader-fix results on main. Variant A is a Mermaid Story. Runs refused at grading went from 7 to 0 on the same 7 cases.A. ---
config:
xyChart:
width: 560
height: 220
showDataLabel: true
showDataLabelOutsideBar: true
---
xychart-beta horizontal
title "Runs refused at grading, same 7 cases"
x-axis ["2026-10-20 sync", "grader-fix rerun"]
y-axis "runs refused" 0 --> 10
bar [7, 0]
Claim. With PR 289's grader, none of the 7 previously refused runs was refused. Measured with one run per case, the grader refused 5 Grok runs and 2 Hermes runs on 2026-10-20 and none in the rerun of those cases. What the number does not show. Only 4 of the 7 reruns went through a fixed path, a worktree under B. No chart
|
|
Recap trial, PR #285, version 1. Throwaway. Written by an agent following a draft recipe, posted here only to check how it reads and renders. Recap. In one Claude Code session, compaction kept only the first 20,000 characters of the poteto-mode entry and cut its Babysit and Shipping routes (issue 265), so the entry now opens with a recovery rule and the playbook index and moves the trigger, delegation, and writing rules into references under a 16,000-byte lint budget, and a posted PASS+NOTES verdict and seven green checks cover the head.Risk. Medium. The trigger list moved out of Base
Reading order.
These pointers are a start, not the review. Read the whole diff, because a defect can sit in a file this recap ranks low.
Before. At base, the lint tier of sequenceDiagram
participant M as Maintainer
participant G as gate-check.sh
M->>G: runs the lint tier
G-->>M: prints 7 PASS or FAIL lines
Note over G: no check reads the entry size
After. At head, the lint tier also runs sequenceDiagram
participant M as Maintainer
participant G as gate-check.sh
participant C as check-mode-entry-size.py
M->>G: runs the lint tier
G->>C: [+] runs the byte budget check
C-->>G: exits 1 above 16,000 bytes
G-->>M: prints a FAIL line and exits 1
Headline. The entry fell from 24,079 bytes to 15,756 bytes, under the new 16,000-byte budget. The playbook routes that start past the issue's 20,000-character cut fell from 20 of 25 to 0 of 25, and the last route now starts at byte 8,134. What it does not show. Bytes are not tokens. Claude Code documents its limit in tokens, and the repository says this budget is not a token conversion or a retention guarantee. No run in this PR shows Claude Code keeping the new entry after compaction. Retention evidence Proof trail. Three review rounds ran on three heads, and the last one posted PASS+NOTES on
|
| Round | Head | Verdict | What changed after it |
|---|---|---|---|
| 1 | dfa2003c |
Review finding that the task triggers now read as optional | 336e5d88 requires reading and applying the task triggers on every invocation, resume, and task scope change. |
| 2 | 67f966a8 |
Low, which the comment itself labels historical and not the current verdict | dc5ce83a adds the task triggers to the recovery rule. bc43c972 replaces the Session pickup recovery checklist with a link to that rule. |
| 3 | bc43c972 |
PASS+NOTES | Nothing. The PR merged at this head. |
The no-mistakes note in the PR body rates the change Medium because the recovery rule did not require rereading the task triggers, with 177 bytes of headroom. That headroom matches the 15,823-byte entry at 336e5d88, so the note probably describes that commit, before dc5ce83a added the triggers to the recovery rule (a guess from the byte count). The verdict names two open gaps. Standalone recovery reads are not enforced (#287), and an earlier routing-oracle defect is unchanged (#279). Both issues are now closed.
|
Recap trial, PR #285, version 2. Throwaway. Written by an agent following a draft recipe, posted here only to check how it reads and renders. Recap. Compaction can hide workflow routes, so this change puts recovery and playbook selection first and requires complete guidance before action, with a posted final-head PASS+NOTES review.Risk. Medium. Recovery changes the agent's workflow contract, and the final-head trial did not fully follow the standalone-read procedure. Check the recovery rule against the posted verification limits. Base
Reading order.
These pointers are a start, not the review. Read the whole diff, because a defect can sit in a file this recap ranks low. Story. The fix addresses lost workflow guidance after compaction while keeping recovery evidence bounded.
Before. Base guidance tells the user to invoke the mode again if it stops applying. This diagram shows the documented fallback. Base README sequenceDiagram
participant U as User
participant A as Agent
participant F as Skill files
U->>A: Invokes mode again after it stops
A->>F: Reads mode instructions
F-->>A: Returns mode entry
After. The head directs the agent to reread complete governing files and restore task state before its next governed action after known compaction. This diagram shows the required procedure. Head recovery rule sequenceDiagram
participant U as User
participant A as Agent
participant F as Skill files
U->>A: Continues the current task
A->>F: Rereads mode on its own [+]
F-->>A: Returns complete entry [+]
A->>F: Reads triggers and active step [+]
F-->>A: Returns complete guidance [+]
A->>A: Restores task state and gates [+]
Headline. The whole mode entry fell from 24,079 UTF-8 bytes, n=1 base snapshot, to 15,756 UTF-8 bytes, n=1 head snapshot. Base file, head file What it does not show. The inclusive 16,000-byte budget is repository policy, not a token conversion or a runtime retention guarantee. The posted comparison also reports that the base recovered successfully, so it establishes no general recovery improvement. Budget scope, comparison limits Proof trail. Posted reviews span three heads, and the final verdict preserves the procedural compliance gap.Proof trail. The graph groups the current twelve commits into ten guidance edits and two final fixes. The highlighted commit has the posted final-head review. Earlier review heads remain in the table below. ---
config:
gitGraph:
mainBranchName: "PR 285"
---
gitGraph LR:
commit id: "[~] edits x10 336e5d88"
commit id: "[~] triggers dc5ce83a"
commit id: "[~] pickup bc43c972" type: HIGHLIGHT tag: "PASS+NOTES"
The first two review submissions have empty bodies. The trigger objection and replies and historical record supply their context. Current trigger fix and pickup fix are in the current commit sequence. The final posted verdict reports 7 of 7 GitHub checks succeeded at |
|
Recap trial, PR #274, version 1. Throwaway. Written by an agent following a draft recipe, posted here only to check how it reads and renders. Recap. Read-only scan, parse, and watch helpers inherited the lead's model, so this PR gives them a configurable
|
| Area | Files | Lines | What changed |
|---|---|---|---|
| Role configuration | skills/setup-pstack/SKILL.md, skills/setup-pstack/examples/pstack-models.md, skills/setup-pstack/references/models-config.md, skills/setup-pstack/scripts/check-models-config.py |
+5 -2 | Adds routine to the accepted roles, setup instructions, scope reference, and shipped model defaults. |
| Scan and watch callers | skills/automate-me/SKILL.md, skills/poteto-mode/SKILL.md, skills/poteto-mode/playbooks/autonomous-run.md, skills/poteto-mode/playbooks/runtime-forensics.md, skills/poteto-mode/playbooks/session-pickup.md, skills/poteto-mode/playbooks/trace-forensics.md, skills/poteto-mode/playbooks/worktree-cleanup.md, skills/principle-guard-the-context-window/SKILL.md |
+8 -8 | Routes read-only scans, artifact parsing, and watchers through routine and the existing Spawn a role procedure. |
| Orchestration contract | skills/poteto-mode/scripts/orch/role-contract.json |
+1 -0 | Adds routine to singleRoles. |
| Tests | tests/skills/setup-pstack/scripts/test_check_models_config.py, tools/test_orch_role_contract.py |
+29 -0 | Adds cases for four CLI defaults, a user override, and single-role contract membership. |
| User and porting docs | PORTING.md, docs/guide/01-setup.md |
+5 -4 | Explains the role, its port-specific routing, and the exception to unsuffixed flat defaults. |
| Promise ledger | evals/promises/ledger.json |
+59 -22 | Adds the routine-role promise and updates budget wording, watcher evidence, and guide-unit mappings. |
Reading order.
- Core logic. Read the accepted role set and shipped defaults.
- Wiring. Trace poteto-mode spawn defaults, transcript mining, then the playbook callers in the table.
- Boilerplate. Check the orchestration contract against its membership tests, then skim the docs and ledger.
These pointers are a start, not the review. Read the whole diff, because a defect can sit in a file this recap ranks low.
Evidence and limits. The new resolver cases cover Claude Code, Codex, Grok, Hermes, and a user override. On this head, the lint workflow and skill-checks workflow succeeded, and the posted review reports PASS+NOTES. Those records do not establish live scan quality, cost savings, or latency gains.
Before. With shipped defaults at base, these helpers fall through to default and inherit the lead's model, per the spawn rule and default entry.
sequenceDiagram
participant L as Lead
participant R as Resolver
participant W as Helper
L->>R: Resolves default for a scan
R-->>L: Returns inherit-parent
L->>W: Spawns with the lead model
W-->>L: Returns a reduced finding
After. The new spawn rule selects routine, whose shipped bindings are sonnet@medium and, under Codex, gpt-6-luna@medium.
sequenceDiagram
participant L as Lead
participant R as Resolver
participant W as Helper
L->>R: Resolves routine for a scan [+]
R-->>L: Returns the routine binding [+]
L->>W: Spawns with resolved settings [~]
W-->>L: Returns a reduced finding
|
Recap trial, PR #274, version 2. Throwaway. Written by an agent following a draft recipe, posted here only to check how it reads and renders. Recap. Read-only scan, parse, and watch subagents named no role and fell through to
|
| Area | Files | Lines | What changed |
|---|---|---|---|
| Role roster | skills/setup-pstack/scripts/check-models-config.py, skills/poteto-mode/scripts/orch/role-contract.json, skills/setup-pstack/examples/pstack-models.md |
+4 -1 | The resolver and the role contract accept routine, and the shipped defaults set it to sonnet@medium and to gpt-6-luna@medium under ## codex. |
| Spawn sites | skills/poteto-mode/SKILL.md, skills/poteto-mode/playbooks/*.md, skills/automate-me/SKILL.md, skills/principle-guard-the-context-window/SKILL.md |
+8 -8 | Each site puts its scan, parse, or watcher subagent on the routine role, and seven of the eight add a pointer to Spawn a role. |
| Setup and porting docs | skills/setup-pstack/SKILL.md, skills/setup-pstack/references/models-config.md, docs/guide/01-setup.md, PORTING.md |
+7 -5 | The docs define the role's scope, say that routine is the one flat line with a written effort, and record it as a port addition. |
| Tests | tests/skills/setup-pstack/scripts/test_check_models_config.py, tools/test_orch_role_contract.py |
+29 -0 | New tests pin the resolved routine arm on all four harnesses, a user override, and its place in singleRoles. |
| Promise ledger | evals/promises/ledger.json |
+59 -22 | The ledger adds the setup-configures-routine-role promise, rewords one claim and one quote, and replaces two guide units with three new ones. |
Reading order.
- Core logic.
ROLESincheck-models-config.py(line 22), the tworoutinelines inexamples/pstack-models.md(lines 27 and 49), and the teststest_routine_ships_a_medium_effort_scan_model_on_every_harnessandtest_routine_is_a_single_role_in_the_artifact. - Wiring. The
routinesentence at each of the eight sites. Start with the Task defaults paragraph inskills/poteto-mode/SKILL.md(line 98). - Boilerplate.
evals/promises/ledger.json,docs/guide/01-setup.md, andPORTING.md.
These pointers are a start, not the review. Read the whole diff, because a defect can sit in a file this recap ranks low.
Before. At base, a scan site named no role. Step 1 of Spawn a role sent it to default, which ships as inherit-parent, so the subagent ran on the lead's model.
sequenceDiagram
participant L as Lead
participant R as Resolver
participant S as Scan subagent
L->>R: resolves default
R-->>L: returns inherit-parent
L->>S: spawns with no model field
S-->>L: returns finding from lead model
After. The site names routine. On Claude Code the resolver returns sonnet at medium. On Codex it returns gpt-6-luna at medium, on Grok Build grok-4.7 at medium, and on Hermes inherit-parent at medium, as the new test pins. Because the role now resolves to a written effort on every harness, the fork rule no longer allows a fork at these sites. No test in this PR exercises a spawn site, so its tests prove the resolver and the defaults, not that each site names the role.
sequenceDiagram
participant L as Lead
participant R as Resolver
participant S as Scan subagent
L->>R: [~] resolves routine
R-->>L: [~] returns sonnet@medium
L->>S: [~] spawns on sonnet at medium
S-->>L: returns reduced finding
|
Recap trial, PR #293, version 1. Throwaway. Written by an agent following a draft recipe, posted here only to check how it reads and renders. Recap. A native recovery trial batched complete guidance into one outer response, so this PR adds an offline evaluator that checks packaging alongside content and timing, and the posted head review reports passing CLI tests and CI.Risk. Medium. Verdicts depend on reviewer-supplied requirements, boundaries, and action classifications. Check those inputs before treating a result as recovery compliance. Scope contract Base
Reading order.
These pointers are a start, not the review. Read the whole diff, because a defect can sit in a file this recap ranks low. Story. Complete guidance in separate inner blocks can still violate the standalone outer-response rule.
Before. Base sequenceDiagram
participant R as Reviewer
participant S as Saved recording
participant E as Recovery evaluator
R->>S: inspects recovery responses
S-->>R: shows bodies and call boundaries
Note over E: [-] CLI absent at base
After. The new CLI reports sequenceDiagram
participant R as Reviewer
participant S as Saved recording
participant E as Recovery evaluator
R->>E: [+] supplies trace and manifest
E->>S: [+] checks bytes and call boundaries
E-->>R: [+] reports FAIL for mixed control
R->>E: [+] supplies synthetic repair control
E-->>R: [+] reports SATISFIED_WITH_REPAIRS
Review limits. The posted PASS+NOTES review leaves documentation precision notes about canonical JSON escapes and notice matching, plus a missing metadata-copy regression case. Its local full test gate hit four host executable-isolation errors that also occurred on base, despite green CI. The public controls are sanitized projections and a synthetic contrast, not fresh native trials. The review accepts the scan's two-level JSON escape limit. The manifest still determines which guidance and actions count. Scan scope, posted limit assessment |
|
Recap trial, PR #293, version 2. Throwaway. Written by an agent following a draft recipe, posted here only to check how it reads and renders. Recap. Issue 287 found a Codex recovery that read two governing files in one outer tool response, so
|
| Area | Files | Lines | What changed |
|---|---|---|---|
| Evaluator | evals/skill-recovery/evaluate.py (new) |
+663 -0 | A standard-library CLI parses native Codex call and output records, binds them to a reviewed manifest by SHA-256, and grades each recovery window up to its first governed call. |
| Tests | tools/test_skill_recovery.py (new) |
+801 -0 | 68 test methods run the CLI as a subprocess against generated recordings and the three committed fixtures, and assert exit codes and report fields. |
| Fixtures | evals/skill-recovery/fixtures/ (new) |
+332 -0 | Three hand-built JSONL recordings with manifests (mixed-full and pickup-late-entry expect FAIL, repaired-entry expects SATISFIED_WITH_REPAIRS), two one-line guidance files, and a provenance note. |
| Docs | evals/skill-recovery/README.md (new), evals/README.md |
+100 -1 | A new README states the CLI, manifest, and native-input contract, and evals/README.md gains an index row and a third fixture exception. |
Reading order.
- Core logic.
_evaluatedecides each window's status, marks repaired attempts, and picks the overall exit code._measurereturns the packaging and the transport flags for one outer response. The claim checks at the end of_bindraiseundeclared_governing_body,inconsistent_read_claims, andsplit_read_claim. They use_encodings,_hits, and_region. - Wiring.
_bodiesand_tracepair native call records with their outputs and classify each block's carrier.mainbinds the inputs, turns any input error into exit 2, and prints one JSON report. The manifest and native-input reference is the contract the tests check. - Boilerplate. The JSONL recordings and manifests under
evals/skill-recovery/fixtures/, theprepareandrebindhelpers intools/test_skill_recovery.py, and the two edits toevals/README.md.
These pointers are a start, not the review. Read the whole diff, because a defect can sit in a file this recap ranks low.
|
Recap trial, PR #282, version 1. Throwaway. Written by an agent following a draft recipe, posted here only to check how it reads and renders. Recap. The partial post-sync screen needed a replacement, so this PR records the reruns at
|
| Area | Files | Lines | What changed |
|---|---|---|---|
| Evaluation record | evals/promises/results/2026-10-20-sync-7a9a3ddb.md (new) |
+140 -0 | Records the verdict matrix, failure classifications, grading refusals, historical comparisons, and follow-ups. |
The posted PASS+NOTES review verifies the model and effort values against run records and leaves optional wording notes. The head's lint, skill-checks, and PR workflow runs succeeded. These checks do not establish live behavior at the PR head.
Reading order.
- Core evidence. Read the report's scope and grading limits first. The runs tested
7a9a3ddb; later changes to owner files are outside that measurement. - Results and interpretation. Read the matrix and totals, then the FAIL classifications. The report distinguishes known gaps, borderline behavior, and oracle errors.
- Comparisons and next steps. Finish with the baseline limits and follow-ups. This PR records proposed grader, oracle, and case changes without implementing them.
These pointers are a start, not the review. Read the whole diff, because a defect can sit in a file this recap ranks low.
Headline. Claude Code's graded coverage rose from 5 of 35 runnable changed-owner cases in the October 19 screen to 35 of 35 in the October 20 screen. See the prior scope and the new scope.
What it does not show. Greater coverage does not establish a pass-rate improvement. The new screen covers 35 of 55 runnable live cases and 53 of 112 live promises. It records 150 PASS, 58 INCONCLUSIVE, and 14 FAIL verdicts, with seven runs refused at grading across Hermes and Grok. The run directories remain outside the repository. See the totals, coverage limit, and run locations and refusals.
|
Recap trial, PR #282, version 2. Throwaway. Written by an agent following a draft recipe, posted here only to check how it reads and renders. Recap. Step 8 of Syncing from upstream needs a rerun of each live promise whose owner files changed, and the 2026-10-19 record was only PARTIAL, so this PR adds
|
| Area | Files | Lines | What changed |
|---|---|---|---|
| Promise results | evals/promises/results/2026-10-20-sync-7a9a3ddb.md (new) |
+140 -0 | Records the post-sync live reruns pinned to 7a9a3ddb, with a 53-promise matrix, a class for each FAIL, the 7 refused runs, a comparison with earlier results, and follow-ups. |
Reading order.
- Core logic. The status and the claim that the sync caused no FAIL (lines 3-5). Then the FAIL table, which classes 10 FAILs as KNOWN, 2 as REAL and borderline, and 2 as ORACLE (lines 90-109).
- Wiring. What ran, which runs the grader refused, and why each refusal is a grader gap (lines 7-23). Then the comparison with
2026-10-19-sync-d457567.mdand with the "Fixed in this stack" rows of2026-10-03.md(lines 111-132). - Boilerplate. The matrix rows (lines 27-81). A recount of those rows matches the totals on lines 83-87. Then the open follow-ups (lines 134-140).
The round-3 verdict lists optional notes that this head leaves open. One says that the claim on line 128, that no fixed row is worse than recorded, covers only measured rows.
These pointers are a start, not the review. Read the whole diff, because a defect can sit in a file this recap ranks low.
Headline. The post-sync record grows from 102 verdicts on three harnesses to 222 verdicts on four. Of the 222, 150 are PASS, 58 are INCONCLUSIVE, and 14 are FAIL.
| Harness | Pass rate in 2026-10-19-sync-d457567.md |
Pass rate in this PR |
|---|---|---|
| Claude Code | 10 of 19 | 44 of 60 |
| Codex | 42 of 51 | 50 of 59 |
| Hermes | 4 of 32 | 22 of 58 |
| Grok | not measured | 34 of 45 |
Sources are the 2026-10-19 totals at the base and this PR's totals at the head.
What it does not show. The two files ran different case sets. Claude Code ran 5 cases on 2026-10-19 and 35 here (line 113), so a change in rate does not compare the same cases. Each cell holds one graded run per case and harness, and the 7 refused runs have no verdict. On Hermes, the harvest guard caused 28 of its 34 INCONCLUSIVE verdicts (line 132). The matrix measures 7a9a3ddb. It does not measure fab7600d or ded31f3e, which merged later and changed owner files of promises in this matrix (line 9).
|
Hero prototype 285 Mermaid, throwaway. One GitHub-rendered ---
config:
flowchart:
wrappingWidth: 330
nodeSpacing: 20
rankSpacing: 26
padding: 8
---
flowchart TB
accTitle: PR 285 hero, Mermaid variant
accDescr: Compaction kept only the first 20,000 characters of poteto-mode, which cut 20 of 25 playbook routes including Babysit and Shipping and let an agent merge without the watcher, so PR 285 moves the recovery rule and playbook index to the top, extracts three references, and adds a 16,000-byte lint budget, leaving 0 of 25 routes past the cut.
classDef why fill:#fff8c5,stroke:#9a6700,color:#1f2328
classDef del fill:#ffebe9,stroke:#cf222e,color:#1f2328
classDef add fill:#dafbe1,stroke:#1a7f37,color:#1f2328
classDef chg fill:#ddf4ff,stroke:#0969da,color:#1f2328
classDef note fill:#f6f8fa,stroke:#8c959f,color:#59636e,stroke-dasharray:4 3
subgraph WHY["WHY · issue 265"]
direction TB
W1["In one session, compaction kept only the first<br/>20,000 chars of poteto-mode. The copy ended:<br/>“…Refactoring (rename, extract, inline, de”"]
W2["Then an agent merged 3 PRs with no watcher<br/>(Babysit: watch-pr READY before merge)"]
W1 --> W2
end
subgraph WHAT["WHAT · 21 files, +236 −129"]
direction TB
E["~ SKILL.md −90 +49<br/>recovery rule + playbook index moved first"]
R["+ 3 references: task-triggers,<br/>delegation, writing"]
L["+ check-mode-entry-size.py<br/>16,000-byte lint budget"]
E -- "rules move to" --> R
E -- "guarded by" --> L
end
subgraph HOW["HOW · where the 20,000-char cut lands"]
direction LR
B["− BEFORE 8731b6eb<br/>24,079 bytes<br/>index at chars 17,994–24,033<br/><b>20 of 25 routes cut</b><br/>incl. Babysit, Shipping"]
A["+ AFTER bc43c972<br/>15,756 bytes<br/>index at chars 1,795–8,229<br/><b>0 of 25 routes cut</b><br/>last route at char 8,134"]
B --> A
end
subgraph PROOF["PROOF"]
direction TB
P1["head's check, base: FAIL 24079/16000, exit 1<br/>head's check, head: PASS 15756/16000, exit 0<br/>PASS+NOTES review · 7 of 7 checks green"]
N["not shown: the cut is one reported session, and bytes are not tokens. The recovery run was Codex. No Claude Code compaction ran."]
P1 --> N
end
WHY --> WHAT --> HOW --> PROOF
class W1,W2 why
class E chg
class R,L,A,P1 add
class B del
class N note
style WHY fill:#ffffff,stroke:#d0d7de
style WHAT fill:#ffffff,stroke:#d0d7de
style HOW fill:#ffffff,stroke:#d0d7de
style PROOF fill:#ffffff,stroke:#d0d7de
|
|
Hero prototype 285 story card, throwaway. A hand-authored HTML card for PR 285, rendered to PNG at 2x with pinned Chromium and attached with
|
|
Hero prototype 285 annotated artifact, throwaway. The real base and head text of
|
|
Hero prototype 285 to-scale 2-up, throwaway. Both versions of the entry drawn to scale from their real section offsets, side by side against the same cut, for PR 285. Facts come from
Format probe: the same diagram as an SVG attachment
|
|
Hero prototype 274 Mermaid, throwaway. One GitHub-rendered ---
config:
flowchart:
wrappingWidth: 330
nodeSpacing: 20
rankSpacing: 26
padding: 8
---
flowchart TB
accTitle: PR 274 hero, Mermaid variant
accDescr: Eight read-only scan, parse, and watch spawn sites named no role, so they fell through to default and ran on the lead's model, and PR 274 names them on a new routine role that resolves to sonnet at medium on Claude Code and gpt-6-luna at medium on Codex, proven by resolver tests and green CI.
classDef why fill:#fff8c5,stroke:#9a6700,color:#1f2328
classDef del fill:#ffebe9,stroke:#cf222e,color:#1f2328
classDef add fill:#dafbe1,stroke:#1a7f37,color:#1f2328
classDef chg fill:#ddf4ff,stroke:#0969da,color:#1f2328
classDef note fill:#f6f8fa,stroke:#8c959f,color:#59636e,stroke-dasharray:4 3
subgraph WHY["WHY · commit 53e7815"]
direction TB
W1["8 read-only spawn sites named no role:<br/>transcript mining, artifact parsing, watchers.<br/>So they took default = inherit-parent and<br/>ran on the lead's model and effort."]
end
subgraph WHAT["WHAT · 18 files, +107 −36"]
direction TB
S["~ 8 caller sentences now say<br/>“… on the routine role”"]
C["+ routine in ROLES and<br/>role-contract singleRoles"]
D["+ shipped defaults<br/>routine: sonnet@medium<br/>codex: gpt-6-luna@medium"]
S -- "resolved by" --> C
C -- "reads" --> D
end
subgraph HOW["HOW · resolver output, base vs head"]
direction LR
B["− BEFORE f7a27730<br/>resolve routine:<br/>unknown role, exit 2<br/>so: default, inherit-parent<br/>on all 4 harnesses"]
A["+ AFTER e822fe64<br/>resolve routine:<br/>Claude Code sonnet@medium<br/>Codex gpt-6-luna@medium<br/>Grok grok-4.7@medium *<br/>Hermes inherit-parent@medium *"]
B --> A
end
subgraph PROOF["PROOF"]
direction TB
P1["3 new tests: 4 harness defaults, a user<br/>override, contract membership<br/>7 of 7 checks green · PASS+NOTES review"]
N["not shown: no live cost, latency, or quality run. * Grok spawns take no effort, and Hermes keeps the parent model. 6 of 8 site edits have no test or ledger quote."]
P1 --> N
end
WHY --> WHAT --> HOW --> PROOF
class W1 why
class S chg
class C,D,A,P1 add
class B del
class N note
style WHY fill:#ffffff,stroke:#d0d7de
style WHAT fill:#ffffff,stroke:#d0d7de
style HOW fill:#ffffff,stroke:#d0d7de
style PROOF fill:#ffffff,stroke:#d0d7de
|
|
Hero prototype 274 story card, throwaway. A hand-authored HTML card for PR 274, rendered to PNG at 2x with pinned Chromium and attached with
|
|
Hero prototype 274 annotated artifact, throwaway. Real
|
|
Hero prototype 274 rewiring 2-up, throwaway. The same 8 spawn sites wired to
|
|
Design A for PR 274, throwaway Read-only scan, parse, and watch helpers named no role, so they resolved through
Not covered. No live run measured cost or scan quality, Grok Build runs the role at session effort, and six of the eight call-site edits have no test or ledger quote. |
|
Design B for PR 274, throwaway Read-only scan, parse, and watch helpers named no role, so they resolved through
Not covered. No live run measured cost or scan quality, Grok Build runs the role at session effort, and six of the eight call-site edits have no test or ledger quote. |
|
Design C for PR 274, throwaway Read-only scan, parse, and watch helpers named no role, so they resolved through
Not covered. No live run measured cost or scan quality, Grok Build runs the role at session effort, and six of the eight call-site edits have no test or ledger quote. |
|
Design D for PR 274, throwaway Read-only scan, parse, and watch helpers named no role, so they resolved through
Not covered. No live run measured cost or scan quality, Grok Build runs the role at session effort, and six of the eight call-site edits have no test or ledger quote. |
1 participant
Syncs upstream cursor/plugins
pstack/fromd457567to88d4751.Upstream commits
/sharpenno longer frames a task and then does it. It now asks four things (what a top professional would add, the rawest material, the methods and tools, and an outside check), rewrites the prompt, and leaves the user's choices open.skills/sharpen/SKILL.md: took upstream's new description and body. Droppeddisable-model-invocation: trueper the substitution map. The existingagents/openai.yamlgate stays.README.md: took upstream's new/sharpenrow. In place of upstream's "runs only when you type/sharpen", it uses the port's gated wording ("it doesn't load from its description (on claude code, once/setup-pstacksets the skill listing), so type/sharpen"), the same as the/poteto-helpand/in-your-own-wordsrows.skills/poteto-help/SKILL.md,skills/poteto-help/references/recipes.md,docs/guide/02-poteto-mode.md,docs/guide/10-recipes-and-pitfalls.md: took upstream's text verbatim. The port's text before the sync matched upstream's old text.tools/skill-trigger-cases.json: changed the sharpen row'sdescription_containsfromSharpen a tasktodeeper, not biggerso it matches the new description, and reworded its example request.implicit_allowedstays false..cursor-plugin/plugin.jsonandplugin.jsonversion bumps: not ported (substitution map)..github/upstream-sha→88d47514b82e7c44f00f2b2a72ff6dd12158038c.No single-model cuts, agent changes, or skill or playbook additions, renames, or removals.
Guide promises
0f77d985e3fc,25321c29d8ee,370107715792, and924614a9e8a9(all promise), plus6c303f8ac2d4and8616da1694e0(context). Both guides dropped them.b1e90eb13b4e,9ead3f17f124,1dd47905086a,123a352a331d,3e74944d2a28. Context units:ff45621c7b54,b78bf6d55700,eb1880162cb5.sharpen-frames-task-before-workwithsharpen-rewrites-prompt-without-doing-task(static, ownerskills/sharpen/SKILL.md). Its six evidence quotes come from the new skill body. The old id said the skill does the work, which is no longer true, and no test or case referenced it.Checks
Tools: real
bunandrgfrom mise installs ahead of the shims, with Python chosen bytools/gate-check.sh.tools/gate-check.sh lint: PASS (rerun after the results file was added)tools/gate-check.sh test: PASS (tools, promise, and setup-pstack unit tests, poteto-mode bun install, tests, and typecheck)python3 evals/promises/ledger.py audit --upstream-ref upstream/main: 0 problemsskill-ci check: 4 of 4 checks, 0 failedNothing failed, so no
mainbaseline comparison was needed.Live (step 8): the only live promise whose owners changed is
poteto-help-finds-need-and-routes. Its two cases ran once each on Claude Code, Codex, Hermes, and Grok Build: 8 of 8 PASS. The matrix is inevals/promises/results/2026-10-20-sync-88d4751.md.CI: none had run when this PR opened. Only CI results that actually run count. The local checks above do not stand in for the Docker suites or other CI-only checks.
Published as a draft under the scheduled-sync exception to no-mistakes, which covers this branch only.