Skip to content

feat(Claude): end-to-end server fallback for Claude 5.x #4584

Description

@aheritier

Goal

Support Claude API server-side fallback with 5.x models end to end, then remove the temporary block added in #4583 (sub-issue #4652).

This is not a client-side retry loop. Fallback chains with no 5.x model keep working as they do today.

Why

With server-side fallback (audited 2026-10-13), one response can come from several models:

  • A fallback marker can appear mid-output. It arrives as a whole content block with no deltas, and the serving model can change after message_start.
  • Client tool calls made before the marker are superseded and must not be executed.
  • The next request must keep markers in place but leave out superseded thinking and client tool-use blocks.
  • Later requests can go straight to the fallback for about an hour (sticky routing), without any marker. Only the response model and usage.iterations show it.
  • Billing is per attempt in usage.iterations. The top-level usage covers only the last attempt and must not be added again.

Today fallbacks.go only forwards the list with the June beta header (fallbacks.go).

Scope

  • Support the July beta (server-side-fallback-2026-07-01): default and explicit lists (up to 3 distinct targets). Keep June explicit lists working.
  • Only on first-party Claude API hosts. Reject non-empty fallbacks on Vertex, Converse and unverified custom endpoints or gateways.
  • Haiku 5.5: default is accepted with a warning that it has no effect; an explicit list with a Haiku primary fails locally.
  • Apply the documented Sonnet 5.5 between_tools → Sonnet 5 conversion exactly, and nothing more general.
  • A stream state machine that tracks the serving model per segment, plus a next-turn projection that follows the rules above. Raw history stays intact.
  • Build per-attempt cost on the per-stream serving model from fix: price routed turns using the serving model #4579. Missing or conflicting attribution is unpriced, not zero.

Acceptance criteria

  • Fixtures for July default/list, June lists, malformed input, target count and uniqueness, and incompatible request/target pairs.
  • Stream tests: fallback before output, fallback mid-output, multiple handovers, interruption, refusal while a tool call is open, completed server tools.
  • Superseded client tools are never executed. Save, reload and continue gives the same projection.
  • Sticky routing without a marker is detected and persisted. Two interleaved streams on a shared client never swap models or costs.
  • Cost tests: per-attempt totals with no double counting, billed vs. non-billed refusals, unchanged producer with a display alias, unknown producer → unpriced. Direct no-fallback and non-5.x fallback costs unchanged.
  • Temporary block from feat(Claude): Claude 5.x core — request policy and signed history #4583 removed. Docs and examples updated.
  • task build, task test, task lint, go test -race pass.

Epic: #4580 · Depends on #4583

Activity

  1. added
    area/providers/anthropicFor features/issues/fixes related to the usage of Anthropic models
    area/providersFor features/issues/fixes related to LLM providers (Bedrock, LiteLLM, Qwen, custom, etc.)
    area/runtimeRuntime engine, agent loop execution, tool dispatch, loop detection
    area/sessionsFor features/issues/fixes related to session lifecycle (resume, persistence, export)
    on Oct 13, 2026
  2. self-assigned this
    on Oct 13, 2026
  3. changed the title [-]feat(Claude): enable end-to-end 5.x server fallback [P5; depends on #4583, #4576 contract][/-] [+]feat(Claude): end-to-end server fallback for Claude 5.x[/+] on Oct 19, 2026
  4. aheritier commented on Oct 21, 2026

    @aheritier
    CollaboratorAuthor

    Delivery plan

    The docs were re-checked on 2026-10-21 (Refusals and fallback). The scope above still holds. Delivery is in three PRs; the temporary Claude 5.x block stays until the last one.

    PR Scope
    1 Request side: fallbacks: default or up to three distinct models, sent with server-side-fallback-2026-07-01 through the beta Messages API. Validated at startup: Haiku 5.5 rules; Claude API only (rejected on Vertex AI, Bedrock, gateways and other base_url hosts)
    2 Stream and continuation. The fallback marker reaches the runtime, which drops the superseded client tool calls and thinking at that point; tool calls keep streaming live. Serving model per segment, plus sticky routing. Next-turn projection (keep markers, drop pre-fallback thinking and client tool_use). Save, reload and continue
    3 Per-attempt cost from usage.iterations at each attempt's model rates (an unknown model is unpriced, no double counting); remove the temporary block; docs and examples

    Decisions:

    • July header only. It accepts lists too, so June configs keep working. Under July the API applies the Sonnet 5.5 between_tools → Sonnet 5 conversion itself.
    • Per-entry overrides (max_tokens, thinking, …) are out of scope.
    • Superseded tool calls are retracted when the marker arrives, rather than buffered until the end of the stream.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area/providersFor features/issues/fixes related to LLM providers (Bedrock, LiteLLM, Qwen, custom, etc.)area/providers/anthropicFor features/issues/fixes related to the usage of Anthropic modelsarea/runtimeRuntime engine, agent loop execution, tool dispatch, loop detectionarea/sessionsFor features/issues/fixes related to session lifecycle (resume, persistence, export)

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions