test(adaptive): prove the repaired-run continue end to end - #67
Merged
sanil-23 merged 4 commits intoAug 19, 2026
Merged
Conversation
…tinue-after-repair-proof
The gate's unit tests ask whether an edit is safe. The engine's e2e asks
whether a continue re-enters the right node. Neither can catch the chain
being wired up wrong — a gate that says yes to a `ResumePoint` nobody
threads through, a runner handed one for a workflow the chooser did not
pick — because each of them is one joint.
This drives `Loop::run` end to end against a real engine, a real
checkpointer and a real store. Only the model and the tool are doubles,
and both have to be: the model so the repair is a known edit rather than
a guess, the tool so "did the prefix run again" is answerable at all.
The workflow is `start → post_comment → tally`, named for the argument.
`tally` calls a slug that is down; the scripted repair points it at one
that answers, which is an edit to the failed node and nothing else. Two
attempts, and the assertion is a **count**:
post_comment 1 the effectful prefix, across both attempts
tally_broken 1 the attempt that broke
tally 1 the continue
plus both legs running under one thread, which is what "continued" means
here. A state comparison would pass whether the prefix ran once or twice.
The second test is the one that makes the gate load-bearing rather than
decorative: the same episode with a repair that also edits the node
UPSTREAM of the failure. The loop must start over, and the observable
cost of starting over is `post_comment` running twice.
Checked by falsification. With the runner ignoring `Attempt::resume` and
always starting fresh, the first test reports `left: 2, right: 1` on the
effectful node — so it is measuring the continue, not agreeing with it.
Writing it also reproduced a defect worth naming, because it is the same
one in the same shape a real host hit: a runner that reports no steps and
an empty `changed` for a failed run gets settled mechanically as terminal
`MissingEvidence` before the judge is ever asked, and the episode stands
down after one attempt. A failed run still did whatever it did before it
broke; the report has to say so. The runner here reads its steps back out
of the failure boundary's committed state for exactly that reason.
Stacked: needs `may_continue` and `Attempt::resume` (tinyhumansai#65) and
`retry_with_checkpointer` / `failure_boundary` (tinyhumansai#66). This branch is the
two merged plus the test, so it is also the first place CI runs them
together.
|
Important Review skippedThis review includes 1 billable file. This on-demand review is free during your promotion. Your included review limit has been reached. Run
⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
You can disable this status message by setting the Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
How this change flows0 changed behaviours across 3 relationships. 4 surrounding behaviours are shown (60 graph nodes walked). 35 further behaviours left out to keep the diagram readable. flowchart LR
n0["run"]:::impacted
n1["LlmProvider"]:::impacted
n2["...air_starts_over_and_the_prefix_runs_again"]:::impacted
n3["...from_the_break_without_redoing_the_prefix"]:::impacted
n2 -->|calls| n0
n2 -->|uses| n1
n3 -->|calls| n0
classDef changed fill:#0d4429,stroke:#238636,color:#e6edf3
classDef impacted fill:#161b22,stroke:#6e7681,color:#c9d1d9
classDef flagged fill:#5a1e02,stroke:#d93f0b,color:#ffffff
classDef blocking fill:#67060c,stroke:#f85149,color:#ffffff
Green: changed behaviour. Grey: surrounding behaviour. Arrows name the call, use, implementation, or test relationship. Orange: has findings. Red: has a finding that blocks the merge. |
![tinysweeper[bot]](https://avatars.githubusercontent.com/in/4526585?s=60&v=4){kind=small}
tinysweeper
Bot
added
the
priority: p3
…nyflows into feat/continue-after-repair-proof
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
test(adaptive): prove the continue end to end, on an invocation counter
The gate's unit tests ask whether an edit is safe. The engine's e2e asks
whether a continue re-enters the right node. Neither can catch the chain
being wired up wrong — a gate that says yes to a
ResumePointnobodythreads through, a runner handed one for a workflow the chooser did not
pick — because each of them is one joint.
This drives
Loop::runend to end against a real engine, a realcheckpointer and a real store. Only the model and the tool are doubles,
and both have to be: the model so the repair is a known edit rather than
a guess, the tool so "did the prefix run again" is answerable at all.
The workflow is
start → post_comment → tally, named for the argument.tallycalls a slug that is down; the scripted repair points it at onethat answers, which is an edit to the failed node and nothing else. Two
attempts, and the assertion is a count:
plus both legs running under one thread, which is what "continued" means
here. A state comparison would pass whether the prefix ran once or twice.
The second test is the one that makes the gate load-bearing rather than
decorative: the same episode with a repair that also edits the node
UPSTREAM of the failure. The loop must start over, and the observable
cost of starting over is
post_commentrunning twice.Checked by falsification. With the runner ignoring
Attempt::resumeandalways starting fresh, the first test reports
left: 2, right: 1on theeffectful node — so it is measuring the continue, not agreeing with it.
Writing it also reproduced a defect worth naming, because it is the same
one in the same shape a real host hit: a runner that reports no steps and
an empty
changedfor a failed run gets settled mechanically as terminalMissingEvidencebefore the judge is ever asked, and the episode standsdown after one attempt. A failed run still did whatever it did before it
broke; the report has to say so. The runner here reads its steps back out
of the failure boundary's committed state for exactly that reason.
Stacked: needs
may_continueandAttempt::resume(#65) andretry_with_checkpointer/failure_boundary(#66). This branch is thetwo merged plus the test, so it is also the first place CI runs them
together.