fix(runtime): prune and persist the tool DeferralTracker #4685

Activity

1. aheritier

   added

   area/sessionsFor features/issues/fixes related to session lifecycle (resume, persistence, export)For features/issues/fixes related to session lifecycle (resume, persistence, export)

   area/runtimeRuntime engine, agent loop execution, tool dispatch, loop detectionRuntime engine, agent loop execution, tool dispatch, loop detection

   on Oct 21, 2026
2. aheritier

   added the

   Bug

   issue type on Oct 21, 2026
3. aheritier

   assigned

   dgageot

   and

   aheritier

   and unassigned

   dgageot

   on Oct 21, 2026
4. aheritier commented on Oct 22, 2026

   

   aheritier

   on Oct 22, 2026

   Last edited by aheritier

   CollaboratorAuthor

   More actions

   Implementation plan

   Planned now, implemented later. The survey behind it covered every serve mode, both providers that defer tools, and the session store.

   Goal:

   • A session's tool-deferral baseline stays the same for its whole life, across turns, runtimes, pooled servers and reloads, so its request prefix doesn't change.
   • Nothing accumulates for sessions that are gone.

   Architecture:

   1. State on the session. The tracker's state (the tool names present on the first model call, plus each later tool's load point) moves from the runtime's toolDeferrals map onto session.Session, as a small ToolDeferrals value with its own lock.
      • It goes away with its session, so it can't grow.
      • Pooled serve chat runtimes then share each conversation's baseline.
   2. Persistence. A new SQLite migration adds a tool_deferrals TEXT (JSON) column, read and written like instruction_context.
      • The runtime saves the session when the state changes: the first call's baseline, or a new load point. It uses UpdateSession, as the harness state does, and skips sub-sessions, which are saved when they complete.
   3. Lifecycle.
      • Native compaction resets the state, as it does today.
      • Clone keeps it. Branches and forks copy it, because they replay the same history and its load points.
      • Sub-sessions start empty.

   Decisions (2026-10-22): the state lives on the session; it is persisted in a new column added by a migration.

   Branch: fix/session-tool-deferrals from main.

   Findings that shape the plan

   • Growth by mode:
     • serve chat pools runtimes across conversations, so entries grow without bound, and one conversation can see different baselines on different pooled runtimes.
     • serve api keeps a runtime per session until it is deleted.
     • The TUI keeps one runtime across /new and session loads.
     • serve mcp and serve a2a build a runtime per call or turn.
   • The reset is broader than the issue says. serve a2a builds a new runtime on every turn, so it loses the baseline every turn, not only after a restart.
   • What a reset changes on the wire:
     • Anthropic sends deferred tools with defer_loading after the inline ones, and replaces the load-point tool_result content with tool_reference blocks.
     • OpenAI leaves them out of tools and injects tool_search_call / tool_search_output after the load point.
     • A reset therefore changes both the tool list and past messages. Only these two providers use Deferred.
   • History can't rebuild the baseline. Messages record only the tools that were called, and Deferred / DeferredAtToolCallID are json:"-".
   • session_end hooks fire per run (per user turn), so they are the wrong place to prune.

   Global Constraints

   • Same marking logic. Mark / MarkAt semantics don't change: search-only tools are ignored, load points come from the last successful tool call, and a tool seen before any tool call stays inline until one exists. The existing pkg/tools/deferrals_test.go cases keep passing against the new owner.
   • The migration only adds a nullable column. Existing sessions read as "no baseline yet": their next call records it, as today. Say so in the PR.
   • The state is private. It isn't surfaced in session summaries or API responses.
   • Test cost:
     • unit tests for the state and its JSON;
     • a SQLite round-trip test;
     • one runtime test, where the integration is the point: two runtimes, a reload in between, the same deferred request.
   • Each task ends green; chain lint and commit with &&.

   Review Focus

   1. Locking. The runtime marks from the run loop while other goroutines may clone or persist the session. The state needs its own mutex; Clone deep-copies it under that lock.
   2. Save frequency. Save only when the state changes (first call, new load point), not on every model call.
   3. Branch/fork copy. Load points are tool-call IDs in the copied history. Copy the state only when those IDs survive the branch point; otherwise start empty. Check branch.go's cut-off logic.

   ---

   Phase 1: The state

   Task 1.1: ToolDeferrals value

   Files: pkg/tools/deferrals.go (or pkg/session, wherever the imports don't cycle), deferrals_test.go.

   • Step 1: Turn DeferralTracker's per-session maps into one value, ToolDeferrals{Initial []string, LoadPoints map[string]string}.
     • It has MarkAt(toolCallID, tools) (marked []Tool, changed bool) and Reset(), with its own mutex and JSON tags.
     • Port the existing tests; add changed assertions and a JSON round trip.
   • Step 2: Run the tests. Expected: PASS.
   • Step 3: Commit refactor(tools): make tool deferrals a per-session value.

   Task 1.2: On the session, persisted

   Files:

   • pkg/session/session.go (field, accessor, Clone), branch.go (copy rule);

   • pkg/session/migrations.go (032 tool_deferrals TEXT), store.go (select, insert, update, scan);

   • tests next to each.

   • Step 1: Write failing tests.

     • SQLite: a session with a baseline and load points survives UpdateSession → reopen → GetSession.
     • An old row with no column value reads as empty.
     • Clone is a deep copy.
     • A branch copies the state when the load points are in the branch's history, and starts empty otherwise.

   • Step 2: Run them. Expected: FAIL.

   • Step 3: Implement.

   • Step 4: Run them again, plus go test ./pkg/session/.... Expected: PASS.

   • Step 5: Commit feat(session): persist the tool deferral baseline.

   Phase 2: The runtime

   Task 2.1: Use the session's state

   Files: pkg/runtime/runtime.go (drop toolDeferrals), loop.go (mark), session_compaction.go (reset), the runtime tests that reference the tracker.

   • Step 1: Write a failing runtime test with the existing reprobe harness (TestReprobe_NewToolsAvailableAfterToolCall shape) and a SQLite store:
     • turn 1 calls a tool, after which a new tool appears, so it is deferred with load point call_1;
     • save, reopen the store, and build a new runtime;
     • turn 2 sends the new tool still Deferred, at call_1, and the first-call tools inline.
   • Step 2: Run it. Expected: FAIL (a new runtime re-baselines).
   • Step 3: Implement:
     • MarkAt on sess's state;
     • when changed and the session isn't a sub-session, UpdateSession (skip without a store);
     • compaction resets the session's state.
   • Step 4: Run the runtime tests, including TestReprobe_*. Expected: PASS.
   • Step 5: Mutation check: don't persist on change. The reload test must fail. Revert with the inverse edit.
   • Step 6: Commit fix(runtime): keep the tool deferral baseline with the session.

   Task 2.2: No runtime-held state

   • Step 1: git grep -n "DeferralTracker\|toolDeferrals" finds no runtime-held tracker.
   • Step 2: A serve chat-style test: two pooled runtimes serving the same conversation in turn send the same deferred request. Reuse the chatserver test helpers if cheap; otherwise rely on Task 2.1 and note it in the PR.

   Phase 3: Validation and PR

   Task 3.1

   • Step 1: Run task build, task test, lint and go run ./lint ..
   • Step 2: Run go test -race -shuffle=on ./pkg/runtime/ ./pkg/session/... ./pkg/tools/....
   • Step 3: Get a fresh review of the branch.
   • Step 4: Ask the user, then push over SSH and open the PR: "Closes fix(runtime): prune and persist the tool DeferralTracker #4685". Include:
     • the per-mode findings;
     • the migration note;
     • a References table (Anthropic tool search defer_loading and tool_reference; prompt caching prefix).
   • Step 5: Check the PR for review comments.

   ---

   Checklist

   • Phase 1: The state
     • Task 1.1: ToolDeferrals value
     • Task 1.2: On the session, persisted
   • Phase 2: The runtime
     • Task 2.1: Use the session's state
     • Task 2.2: No runtime-held state
   • Phase 3: Validation and PR
     • Task 3.1: Validate, review, PR, comments

   Success Criteria

   • No runtime holds per-session deferral state, so nothing grows with sessions that ended.
   • A session sends the same tools, deferred at the same load points, across turns, new runtimes (serve a2a), pooled runtimes (serve chat) and SQLite reloads.
   • Existing sessions keep working: no baseline means the next call records one.
   • Native compaction still resets the baseline.
   • Build, tests, lint and race/shuffle pass. The reload test failed under its mutation check.