feat(core): add a temporal durable-execution mode - #2
Draft
moedash wants to merge 103 commits into
Draft
Conversation
|
|
|
Hey! Your PR title Please update it to start with one of:
Where See CONTRIBUTING.md for details. |
6 tasks
moedash
changed the title
1 task
|
Thanks for updating your PR! It now meets our contributing guidelines. 👍 |
6 tasks
moedash
marked this pull request as draft
August 15, 2026 08:00
moedash
force-pushed
the
2026/08/opencode-temporal
branch
from
August 17, 2026 08:30
15eef17 to
f85e59a
Compare
6 tasks
8 tasks
Phase 1 of a drop-in durability layer: an opencode session becomes a durable Temporal workflow that owns the conversation and prompt queue and drives each turn against the shipping opencode server over HTTP. The turn runs server-side (prompt_async), so it survives a worker crash; recovery re-attaches and is idempotent on the user-message count. A crash demo kills the worker mid-turn and the session still completes: the runTurn activity re-drives on a fresh worker (attempt 2) with a single prompt sent. opencode's loop, tools, model, storage, and API are untouched. Phase 2 (a durable SessionExecution on the v2 engine) is scoped in the package README.
Phase 2: make each v2 session a durable Temporal workflow. SessionExecutionTemporal implements the substitutable SessionExecution service (wake -> signalWithStart, resume -> forced signalWithStart, interrupt -> cancel signal), with the local coordinator's drain (SessionRunner.run for the whole turn) moved into a runContinuation activity that runs against the durable event log. The Temporal client and an embedded worker are co-hosted inside the server process (both run under bun). It is opt-in via OPENCODE_SESSION_EXECUTION=temporal; the one-line swap is at routes.ts, and the loop, tools, model, storage, and HTTP API are untouched. Verified: with it enabled, prompting a v2 session drove a full turn to completion (step.ended) and Temporal recorded a completed per-session workflow (session-exec-<id>).
Added the engine-level crash-recovery harness (kill the whole server mid-turn, restart, the turn continues from the event log) and updated the README: Phase 2 is built and verified, with the run recipe, the code layout, and the known limits (resume does not yet return the typed RunError; active is process-local; startup needs Temporal reachable).
resume previously drove a forced run fire-and-forget and returned void, so a run failure was swallowed. It now drives the run through a Temporal Update-with-Start and awaits the result: a genuine run error is thrown non-retryable by the activity (so only crashes/timeouts still retry), rejects the update, and the layer maps it to a RunError (carried as ContextSnapshotDecodeError with the original text in details). The per-session workflow is now long-lived with an idle timeout so update-with-start can always reach it. Verified (scripts/resume-check.ts): resume resolves on a healthy session and rejects on a failing one. Full typecheck stays green (30/30).
active listed a process-local set of sessions this process had started, so it was empty after a restart. It now queries Temporal for the open per-session workflows (WorkflowType 'sessionExecution', Running) and maps their ids back to session ids, so it reflects durable state and survives a restart. The process-local set is gone.
resume previously surfaced run failures as a generic ContextSnapshotDecodeError carrier. The activity now encodes the error through a Schema.Union of every RunError member (run-error-codec.ts) into the non-retryable failure details, and the layer walks the failure chain and decodes it back into the exact tagged instance (e.g. LLMError with its reason), falling back to the carrier only if decoding fails. Verified: a unit round-trip reconstructs an LLMError faithfully, and scripts/resume-check.ts shows a failing resume rejects with the encoded _tag = LLM.Error reaching the caller.
The v2 engine event-sources to SQLite, so a session was resumable only on the host with the file. Point every worker at one shared store and any worker resumes: a new libSQL SqlClient backend (sqlite.libsql.ts, over @libsql/client) selected by OPENCODE_DB_URL gives a networked SQLite (sqld/Turso) across hosts, and OPENCODE_DB already shares a file same-host. Same SQLite dialect, so the schema and all migrations are unchanged; the local-only PRAGMAs are skipped for the shared backend. Verified: a full turn runs against a libSQL file: store (migrations + event log), and scripts/shared-store-failover.sh shows turn 1 on worker A, A killed, then a fresh worker B recalls turn 1's code word (two distinct worker identities) purely from the shared store. Remote-URL transaction atomicity is documented as the remaining networked-writer step. Full typecheck 30/30.
OPENCODE_SESSION_EXECUTION=temporal ran a whole turn as one activity. New temporal-turn mode drives the turn one step at a time: SessionRunner gains runStep (one iteration of run's loop, reusing runTurn), the sessionTurn workflow loops a runTurnStep activity, and each step (one provider attempt + its tools) is its own activity with its own retry/timeout/visibility. The step loop is workflow control flow; turn semantics are unchanged. Selected by one env check in routes.ts. Verified: a create-then-read-then-reply turn recorded three runTurnStep activities under a sessionTurn workflow and completed. Finer per-model-call / per-tool granularity is a larger rewrite left for later. Full typecheck 30/30.
The libSQL client runs each statement as its own auto-commit request, so a multi-statement event append (the `event_sequence` upsert plus the `event` insert) could tear on a crash against a remote store. The transaction connection now drives a real interactive libSQL transaction, so those writes commit all-or-nothing. Verified against a `file:` store; networked crash-atomicity still needs a live `sqld`/Turso to test.
Remote libSQL writes now go through interactive transactions, so the caveat about torn multi-statement writes no longer applies; only the networked crash test stays pending.
In temporal-turn mode a Temporal step retry re-invokes `runStep` with `first=false`, so `failInterruptedTools` never ran and the re-drive re-streamed a request with a `tool_use` and no `tool_result`. The provider rejects that, and with `maximumAttempts: 100` it becomes a poison loop. It now runs before every turn; on a healthy step (whose prior tools already settled) it is a no-op.
A Temporal step retry re-invokes `runStep` on the same durable log. If the in-flight step already dispatched tools (`Tool.Called` is recorded before the side effect runs), re-streaming would re-run those side effects and append a duplicate assistant message. `runStep` now finalizes such a step from the log: completed tools keep their results, still-unsettled ones are failed, and a synthesized `Step.Ended` closes it without re-calling the model. A step with no dispatched tools is still re-streamed, which is safe. Token metering is 0 for the resumed step; faithful metering would need a durable sealed marker.
Covers the every-entry tool-close fix and the log-resume of a crashed step, with the in-flight-tool honest limit.
The runner rebuilds context only from the shared DB, so the conversation resumes on any worker. The working tree is the one host-local correctness constraint; snapshots and retained tool-output files are viewing-only. Corrects the earlier overstated tool-output gap.
The AI-399 evaluation and the worktree design memo are internal decision records, not part of the change this branch proposes. The shipped worktree mechanism stays documented in the README and the code; the one load-bearing note (warm-path alternatives) moved inline.
The claims stay documented and verified in the README; the runnable reproductions and the tmux demo ride a stacked PR so the branch being curated for upstreaming carries only the change itself.
The base branch runs local mode on a hand-written per-session supervisor (execution/local-driver.ts over workflow-core.ts). Independent review (Codex, several rounds) found repeated lifecycle races in that path and in alternative hand-written coordinators: concurrent successors during interrupt cleanup, a completion barrier that did not cover resume-started drains, and fresh-resume-vs-wake intent confusion. All are things opencode's existing SessionRunCoordinator already handles correctly and has direct tests for. So local mode now delegates to it: - routes.ts selects SessionExecutionLocal (execution/local.ts) for the default mode. It maps active/wake/resume/interrupt onto the coordinator and drains with SessionRunner.run -- the same lifecycle the v1 server uses. Temporal mode (execution/temporal.ts) is unchanged. - Removed the supervisor-based local-driver.ts. workflow-core.ts is now Temporal-only; comments in it, temporal-workflow.ts, drain.ts, and temporal.ts no longer claim a shared local supervisor. - Repointed the local integration test to SessionExecutionLocal (session-execution-local.test.ts), adjusted for the coordinator's retire-when-idle semantics (it holds no idle timer). - README: two modes drive one SessionRunner over one durable event log; the "one supervisor, two drivers" framing is replaced. Net: local mode reuses well-exercised code instead of a second hand-written coordination loop. drain.ts/SessionRunner.runStep remain the Temporal per-step path.
Stood up an @temporalio/testing (time-skipping) harness that runs the real sessionTurn workflow with a mock activity, in-process and deterministic (test/temporal-harness-smoke.test.ts). It immediately caught a blocker: On @temporalio/workflow 1.21, condition(fn, timeout) called when fn is already true leaves the current CancellationScope cancelled. The supervisor starts with pendingWake=true, so the first idle-wait condition returns true, and the NEXT condition (in drainTurn) throws CancelledFailure -- which the loop reads as an interrupt. Result: the workflow completes without ever scheduling a runTurnStep activity. Temporal mode never drained a turn in this SDK version. (This, not the HTTP "steer" delivery, was the real cause of the "0 activities" seen end-to-end.) Fix: temporal-workflow.ts's condition adapter short-circuits an already-true predicate, keeping the timeout timer and its scope off that path. Verified in the harness (activity now scheduled, one drain, clean idle completion) and live against a dev server (a real gpt-5-mini turn completes: assistant reply recorded, one runTurnStep activity completed).
The token was runId#attempt, but Temporal activity attempt numbers restart at 1 for every step, so step 1 attempt 1 and step 2 attempt 1 both minted `run#1`. A zombie attempt left over from an earlier step could therefore re-match the current owner and append stale events past the fence. Include the per-execution activity id so every step's tokens are disjoint; a retry of the same step still differs by attempt, so it still fences its prior attempt. Unit-tested in temporal-owner-token.test.ts.
The interrupt path treated any signal failure other than "already completed"/"not found" as success (logged a warning, returned void), so a real control-plane failure -- the user's stop never delivered -- read as a successful stop. Classification is now a tested pure helper (classifyInterruptError); a genuine failure is surfaced as a defect rather than false success, while an already-closed idle workflow stays a no-op.
The bound only counted wake-loop drains, so a resume-heavy workflow never continued-as-new and its history grew until it hit Temporal's limit. The counter now increments inside drainTurn (every drain), and a `rolloverPending` flag lets the main loop trigger continueAsNew -- from the workflow's main method, never an update handler -- once the bound is crossed, even if the crossing drain came from a resume. Verified with a fake-runtime unit test (a resume drain crosses maxDrainsPerRun and rolls over).
Records what's fixed (condition blocker, owner-token collision, interrupt failure reporting, continue-as-new counting) and the interlocking deep items left as one coherent pass (resume/wake lost at interrupt, concurrent resumes duplicating turns, fresh-resume spurious drain), each with a fix sketch and a harness test to validate it.
Addresses the deep coordination findings from independent review, for Temporal mode (local mode uses SessionRunCoordinator and is unaffected): - resume JOINS the single in-flight drain instead of queueing a second forced one (concurrent resumes no longer duplicate provider turns / tool side effects), mirroring SessionRunCoordinator.run. - interrupt stops the CURRENT turn, not the session: it cancels only the turn's child cancellation scope (runInDrainScope) and the long-lived workflow keeps serving, so a wake/resume that races the interrupt drives a fresh turn on the same workflow instead of being lost to a doomed one. - A real workflow (root) cancellation is detected via the root scope and stops the supervisor -- never keeps serving or continue-as-news. - Explicit start intent: resume-with-start passes startWithWake=false, so a fresh resume no longer does a spurious wake drain; carried across continue-as-new. - continue-as-new counts every drain and gates on allHandlersFinished() so an in-flight update's result is never abandoned; a resume-driven rollover carries no spurious wake. Also fixes a blocker uncovered while validating this: the SDK's condition(fn, timeout) on @temporalio/workflow 1.21 leaks its timer-scope cancellation into the root scope when it resolves, which poisoned the next drain -- a session could serve only one turn. The timed wait now races a no-timeout condition against a bare sleep and abandons the loser, cancelling no scope, so nothing leaks (and root-cancellation detection stays reliable).
Fake-runtime unit tests (session-supervisor.test.ts): resume joins one drain, concurrent resumes join, a wake that only joins a resume drain still gets a follow-up, interrupt keeps the supervisor serving, a root cancellation stops it, and a fresh resume-with-start does exactly one drain. Rollover test asserts a resume-driven rollover carries startWithWake=false. Real-Temporal harness (@temporalio/testing), each in its own file since two native servers per bun process segfault: - interrupt: a per-turn interrupt cancels the turn but a later wake runs a second turn on the same workflow. - multiturn: turn 1 completes, the supervisor parks in the idle timed wait (asserted via a TimerStarted history event), then a wake drives turn 2 -- the exact path the condition-leak broke.
…ot a quirk. The 0-activities symptom was the condition-timeout scope leak, not the delivery mode. A default (steer) prompt drives a turn to completion once the leak is fixed.
The supervisor redesign reverted the OPENCODE_SESSION_IDLE_TIMEOUT forwarding and removed the file the contract lib typed against. The override rides as a third workflow argument now (continue-as-new keeps it), the lib types against the Temporal node, and the interrupt scenario expects idle retirement since an interrupted supervisor keeps serving.
The suite is the executable definition of the executor seam, so it lives where any driver package can import it; the local run is now a one-line conformance call. The old test-lib path re-exports the moved harness so the rest of the test tree keeps its import.
Core now carries only the executor seam, the built-in local executor, and the executor-agnostic toolkit (step runner, event fencing, error codec, worktree materializer); everything Temporal, including the per-step activity drain, lives in @opencode-ai/temporal, which the server wires in as one dependency. The step contract types moved into the drain where they belonged.
protocol.ts is the one source of truth for the workflow type, signal and update names, the workflow-id scheme, and the connection defaults; config.ts is a service read at layer build, so an embedder or a test injects settings instead of racing module-load env reads.
Positional arguments turn signature evolution into a breaking change mid-history; a single record lets new settings ride along, and a continue-as-new run carries it forward unchanged.
workflow-core became supervisor.ts when local mode stopped driving it; SupervisorRuntime and makeSupervisor say what the interface and the factory actually build.
The package reads as what it is now: one executor behind core's seam, held to the conformance suite, with a porting recipe for engines that want the same shape.
The suite mocks the LLM client, so the model descriptor's endpoint and token are never used for I/O; a real-looking OpenAI URL invited the wrong conclusion. An RFC 2606 .invalid host cannot be mistaken for a live dependency and cannot resolve if the mock is ever removed.
The resume scenarios left a durable executor's session running forever: the test's task queue dies with the process, so nothing ever processes the idle timer's task. Each scenario now waits out the retirement while its worker still exists.
With OPENCODE_TEMPORAL_ROLE=client the turn runs in a separate worker process, and its session.next.* events live on that process's bus; the TUI only observes its own daemon's admission. A reply therefore rendered one prompt late, when the next admission triggered a re-read. A followed session now re-reads until its latest assistant message settles.
The prompt bar syncs its selection from the last user message; the invented values clobbered the user's choice and the empty model rendered a 'Model / is not valid' toast on every turn. An empty agent short-circuits that sync.
The read model can show an assistant's completed flag before the settled row's rewritten content, so a follow that trusted one read stopped with the reply's tail missing. It now stops only after two identical settled reads, and it emits part removals when the settled rewrite coalesces streamed parts under new ids, so no stale text lingers.
A question raised inside an activity on one worker was parked on an in-memory deferred, so no other process could list or answer it and the step stayed blocked. Deterministic ids let a re-driven attempt adopt the same row or return answers that already landed, which also makes the question tool safe to mark idempotent for crash-resume.
The sync store only knew the v1 question event names, so the v2 engine's question.v2.* events populated nothing and the dialog never opened. The reply path also called a v1 route that 404s on the v2 daemon. In temporal mode the question tool runs inside a worker activity, so its asked event never reaches the daemon's stream. The session projector now polls the durable question list during a follow and projects the ask and answer lifecycle, so a worker-raised question opens the dialog and the reply routes back to unblock the turn.
moedash
force-pushed
the
2026/08/opencode-temporal-tui
branch
from
September 7, 2026 23:27
809d705 to
b2a9f42
Compare
moedash
force-pushed
the
2026/08/opencode-temporal
branch
from
September 7, 2026 23:27
289f93b to
3434d80
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue for this PR
Closes anomalyco#42678
Status: WIP. Review continues, but the correctness issues independent review (Codex, several rounds) found are folded in as of
82e1b69ca5: local mode runs on the provenSessionRunCoordinator; the Temporal supervisor was redesigned (interrupt cancels only the turn and keeps serving, resume joins the in-flight drain, the timed-condition scope leak is gone); the event-log owner token is unique per activity execution; interrupt delivery failures surface as defects. A deterministic harness (fake-runtime plus@temporalio/testing) and the two-driver contract suite back it. Fine to share internally as WIP; not ready for external delivery while review is open.Type of change
What does this PR do?
This PR adds a Temporal durable-execution layer to opencode (a fork of anomalyco/opencode dev at 4643e65).
It makes an opencode session a durable Temporal workflow: a coding session survives worker loss, runs detached, and resumes on any worker from a shared store. The shape is a plugin: core carries the
SessionExecutionseam, the built-in local executor (the provenSessionRunCoordinator), the executor-agnostic toolkit (step runner, event fencing, error codec, worktree materializer), and a conformance suite that defines the seam executably; everything Temporal lives in@opencode-ai/temporal, one dependency the server wires in, selected byOPENCODE_SESSION_EXECUTION=temporalwith one activity per step. On top of that: a shared libSQL event store with verified atomic writes, standalone workers (OPENCODE_TEMPORAL_ROLE), cross-process migrations, log-based crash resume with per-tool idempotency, durable permission asks replyable from any process, worktree materialization from snapshot packs, and turn loop bounds. The serve-wrapper increment lives in its own PR.The base branch
2026/08/opencode-temporal-tuicarries the TUI bridge (upstream anomalyco#42658) on the upstream commit this work sits on, so the diff shows only the Temporal work. Start atpackages/temporal/README.mdfor run recipes, verified claims, and the honest limits (thequestiontool, remote streaming throughput).How did you verify your code works?
Typecheck passes on
core,server, andcli. The driver-contract suite passes against both drivers (the Temporal run is opt-in against a dev server); the worktree-materializer and runner suites pass. The runnable reproductions (crash test, failover, worker smoke, demo) live in the stacked #4.Screenshots / recordings
Not a UI change.
Checklist