Repository navigation
feat(Claude): end-to-end server fallback for Claude 5.x #4584
Copy link
Copy link
Open
Labels
area/providersFor features/issues/fixes related to LLM providers (Bedrock, LiteLLM, Qwen, custom, etc.)For features/issues/fixes related to LLM providers (Bedrock, LiteLLM, Qwen, custom, etc.)area/providers/anthropicFor features/issues/fixes related to the usage of Anthropic modelsFor features/issues/fixes related to the usage of Anthropic modelsarea/runtimeRuntime engine, agent loop execution, tool dispatch, loop detectionRuntime engine, agent loop execution, tool dispatch, loop detectionarea/sessionsFor features/issues/fixes related to session lifecycle (resume, persistence, export)For features/issues/fixes related to session lifecycle (resume, persistence, export)
Description
Goal
Support Claude API server-side fallback with 5.x models end to end, then remove the temporary block added in #4583 (sub-issue #4652).
This is not a client-side retry loop. Fallback chains with no 5.x model keep working as they do today.
Why
With server-side fallback (audited 2026-10-13), one response can come from several models:
- A fallback marker can appear mid-output. It arrives as a whole content block with no deltas, and the serving model can change after
message_start. - Client tool calls made before the marker are superseded and must not be executed.
- The next request must keep markers in place but leave out superseded thinking and client tool-use blocks.
- Later requests can go straight to the fallback for about an hour (sticky routing), without any marker. Only the response model and
usage.iterationsshow it. - Billing is per attempt in
usage.iterations. The top-levelusagecovers only the last attempt and must not be added again.
Today fallbacks.go only forwards the list with the June beta header (fallbacks.go).
Scope
- Support the July beta (
server-side-fallback-2026-07-01):defaultand explicit lists (up to 3 distinct targets). Keep June explicit lists working. - Only on first-party Claude API hosts. Reject non-empty
fallbackson Vertex, Converse and unverified custom endpoints or gateways. - Haiku 5.5:
defaultis accepted with a warning that it has no effect; an explicit list with a Haiku primary fails locally. - Apply the documented Sonnet 5.5
between_tools→ Sonnet 5 conversion exactly, and nothing more general. - A stream state machine that tracks the serving model per segment, plus a next-turn projection that follows the rules above. Raw history stays intact.
- Build per-attempt cost on the per-stream serving model from fix: price routed turns using the serving model #4579. Missing or conflicting attribution is unpriced, not zero.
Acceptance criteria
- Fixtures for July
default/list, June lists, malformed input, target count and uniqueness, and incompatible request/target pairs. - Stream tests: fallback before output, fallback mid-output, multiple handovers, interruption, refusal while a tool call is open, completed server tools.
- Superseded client tools are never executed. Save, reload and continue gives the same projection.
- Sticky routing without a marker is detected and persisted. Two interleaved streams on a shared client never swap models or costs.
- Cost tests: per-attempt totals with no double counting, billed vs. non-billed refusals, unchanged producer with a display alias, unknown producer → unpriced. Direct no-fallback and non-5.x fallback costs unchanged.
- Temporary block from feat(Claude): Claude 5.x core — request policy and signed history #4583 removed. Docs and examples updated.
-
task build,task test,task lint,go test -racepass.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
area/providersFor features/issues/fixes related to LLM providers (Bedrock, LiteLLM, Qwen, custom, etc.)For features/issues/fixes related to LLM providers (Bedrock, LiteLLM, Qwen, custom, etc.)area/providers/anthropicFor features/issues/fixes related to the usage of Anthropic modelsFor features/issues/fixes related to the usage of Anthropic modelsarea/runtimeRuntime engine, agent loop execution, tool dispatch, loop detectionRuntime engine, agent loop execution, tool dispatch, loop detectionarea/sessionsFor features/issues/fixes related to session lifecycle (resume, persistence, export)For features/issues/fixes related to session lifecycle (resume, persistence, export)
Activity
[-]feat(Claude): enable end-to-end 5.x server fallback [P5; depends on #4583, #4576 contract][/-][+]feat(Claude): end-to-end server fallback for Claude 5.x[/+]feat(anthropic): reject server-side fallback involving Claude 5.x until
aheritier commentedon Oct 21, 2026
Delivery plan
The docs were re-checked on 2026-10-21 (Refusals and fallback). The scope above still holds. Delivery is in three PRs; the temporary Claude 5.x block stays until the last one.
fallbacks: defaultor up to three distinct models, sent withserver-side-fallback-2026-07-01through the beta Messages API. Validated at startup: Haiku 5.5 rules; Claude API only (rejected on Vertex AI, Bedrock, gateways and otherbase_urlhosts)fallbackmarker reaches the runtime, which drops the superseded client tool calls and thinking at that point; tool calls keep streaming live. Serving model per segment, plus sticky routing. Next-turn projection (keep markers, drop pre-fallback thinking and clienttool_use). Save, reload and continueusage.iterationsat each attempt's model rates (an unknown model is unpriced, no double counting); remove the temporary block; docs and examplesDecisions:
between_tools→ Sonnet 5 conversion itself.max_tokens,thinking, …) are out of scope.