feat(llm): enable Bedrock/Anthropic prompt caching for Claude models

[seanturner83]

Summary

Enables Anthropic/Bedrock prompt caching for Claude models by passing LiteLLM's cache_control_injection_points through ModelSettings.extra_args in make_model_settings.

A Strix scan is a long, multi-turn agentic loop that re-sends a large, stable prefix every turn — the system prompt plus the tool schemas — while only the conversation tail changes. Without a caching breakpoint that whole prefix is re-tokenised and billed at the full input rate on every turn. On Bedrock Claude this is the single biggest lever on scan cost.

Measured on a real scan: cache-read went 0% → 57% with these injection points set — roughly halving input-token cost, and the ratio climbs on longer scans where the stable prefix dominates more turns.

Why this shape (not a ModelSettings flag)

This is deliberately kept at the LiteLLM-config layer rather than a general caching flag on ModelSettings. That's the direction the Agents SDK maintainer explicitly prescribed when declining a native cache_system_prompt field:

"For LiteLLM, the preferred direction is to keep this at the LiteLLM configuration layer. LiteLLM already supports cache_control_injection_points, and LitellmModel passes ModelSettings.extra_args through to litellm.acompletion()."
— openai/openai-agents-python#3008 (PR #3009 closed on this basis; earlier #905, #1257 wontfix)

The SDK owner's rationale: caching is LiteLLM/provider-specific, and a ModelSettings flag would let strict OpenAI-compatible paths emit non-standard cache_control parts (not a guaranteed no-op). Applying it here, in the Strix SDK-consumer layer, is exactly that prescribed pattern — and it's where the provider is already known.

How it works

LiteLLM implements this end to end: when cache_control_injection_points is present in the call kwargs, its AnthropicCacheControlHook fires and emits the provider-appropriate breakpoint (Anthropic cache_control; Bedrock Converse cachePoint), honouring Anthropic's 4-breakpoint cap. LitellmModel forwards ModelSettings.extra_args straight into litellm.acompletion(), so passing the points there is all that's required.

We mark the two big stable segments (2 of the 4 allowed breakpoints, headroom left):

• system prompt (role: system) — the largest repeated span
• tool schemas (tool_config) — sizeable and identical every turn

Safety / scope

• Gated on Claude (_is_claude_model) → a strict no-op for every other provider: no injection points means the hook never fires, so nothing non-standard reaches OpenAI / AnyLLM / non-Claude endpoints.
• Only Claude-family routes (Anthropic native, Bedrock, Vertex, OpenRouter → Claude) honour the marker.
• Conversation-tail turns left uncached (they change every turn — a breakpoint there would never hit).

Tests

tests/test_inputs.py — parametrised, non-vacuous:

• Claude routes (bedrock / anthropic-native / openrouter) get the two injection points.
• Non-Claude (gpt-5 / gemini / o3) get extra_args=None.

ruff clean; no new dependencies (LiteLLM support is already present).

[greptile-apps]

Greptile Summary

This PR enables prompt caching for Claude models through LiteLLM. The main changes are:

• Detect Claude routes from the configured model name.
• Add system-prompt and tool-schema cache injection points.
• Test Claude and non-Claude settings generation.

Confidence Score: 4/5

The Claude settings path should preserve existing LiteLLM arguments before merging.

• Existing extra_args can be replaced when caching is enabled.
• The tool cache marker may be ignored by the locked LiteLLM runtime.
• Non-Claude routes remain unchanged.

strix/core/inputs.py

Important Files Changed

Filename	Overview
strix/core/inputs.py	Adds Claude-specific LiteLLM cache settings, but can replace existing extra arguments and depends on support for the tool cache location.
tests/test_inputs.py	Tests the generated settings shape for representative Claude and non-Claude model names.

Prompt To Fix All With AI

Fix the following 2 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 2
strix/core/inputs.py:144-147
**Existing Extra Arguments Are Replaced**

When a Claude caller supplies `ModelSettings.extra_args`, this final `resolve` replaces that dictionary with the caching dictionary. Unrelated LiteLLM options then disappear from the completion request, so the cache key should be merged into the existing arguments.

```suggestion
    if _is_claude_model(model_name):
        extra_args = {
            **(model_settings.extra_args or {}),
            **_claude_prompt_cache_extra_args(),
        }
        model_settings = model_settings.resolve(
            ModelSettings(extra_args=extra_args),
        )
```

### Issue 2 of 2
strix/core/inputs.py:188
**Tool Cache Point Can Be Ignored**

The locked LiteLLM path may not recognize `tool_config` as an injection-point location. In that runtime, this entry is silently ignored, so tool schemas are re-tokenized on every turn even though the settings test passes; use a location supported by the locked version or update the dependency with integration coverage.

_{Reviews (1): Last reviewed commit: "feat(llm): enable Bedrock/Anthropic prom..." | Re-trigger Greptile}

[seanturner83]

Thanks @greptile-apps — addressed both in af77695, with a note on where each held up vs. not:

Issue 1 (extra_args replaced): ModelSettings.resolve() actually merges extra_args (there's explicit dict-merge logic in model_settings.py, and unrelated keys like num_retries/timeout survive a resolve()), and make_model_settings builds ModelSettings from scratch so the base extra_args is None here regardless — so nothing was being dropped. That said, the suggestion is a good readability call: I now build {**existing, **cache} explicitly so preservation is obvious at the call site rather than relying on resolve()'s non-obvious merge semantics. No behaviour change.

Issue 2 (tool_config ignored): on current LiteLLM the Bedrock Converse transform does handle location: "tool_config" (converse_transformation.py appends a cachePoint to the tools; verified on litellm 1.90.1), so it isn't silently dropped. The real nuance is graceful degradation: an unrecognised injection-point location on an older pin is simply not injected — no error — so a stale LiteLLM still gets the system-prompt breakpoint (the dominant win). Documented that in the docstring rather than changing behaviour.

Verified live before filing: cache-read 0% → 57% on a real multi-turn scan with these points set.

[greptile-apps]

Both issues are cleanly addressed. The explicit {**existing, **cache} merge is the right call — it's self-documenting and future-proof regardless of what resolve() does under the hood. The graceful-degradation note in the docstring correctly captures the real behavior: unrecognised locations are silently skipped, so an older LiteLLM pin still benefits from the system-prompt breakpoint.

A few small observations on the final state:

1. _is_claude_model check — "claude" in model_name.lower() will also match hypothetical non-Anthropic model names containing "claude" (e.g. "some-provider/claude-knockoff"). Almost certainly fine in practice, but worth being aware of.

2. location: "message" vs "system" — the injection point is {"location": "message", "role": "system"}. Just confirming you've verified this is the correct key LiteLLM's AnthropicCacheControlHook expects for the system prompt slot — the 57% cache-read number confirms it's working, so this is just a doc note.

3. Test for extra_args preservation — the comment in the implementation calls out "unrelated LiteLLM options are preserved" as a maintenance invariant, but there's no test exercising it (e.g. passing in a ModelSettings with a pre-existing extra_args key and asserting it survives the merge). Not a blocker, but given the comment specifically highlights this concern, a test would lock it in.

Otherwise this looks solid. The gating, the merge pattern, the docstring, and the parametrised tests are all in good shape. LGTM.

[seanturner-zh]

Pushed a follow-up (cc2b335) hardening this against a hard crash I hit running it live.

Problem: the injection is gated on the model name containing "claude", but LiteLLM's AnthropicCacheControlHook only consumes cache_control_injection_points for models it recognises as cache-capable via its (statically bundled) model map. On a Bedrock route whose model isn't in that map, the marker passes straight through to the Converse API, which rejects unknown fields outright:

ValidationException: cache_control_injection_points: Extra inputs are not permitted

That fails the whole run at the first LLM call. It bites any Bedrock Claude model LiteLLM hasn't mapped yet (a just-released model), and is worse when LiteLLM can't refresh its remote model map — e.g. behind a TLS-intercepting corporate proxy — and falls back to a stale local copy. I hit it on bedrock/global.anthropic.claude-sonnet-5.

Fix: withhold the marker only for a Bedrock route LiteLLM can't confirm supports prompt caching (_bedrock_route_without_cache_support). The scope is deliberately narrow — Anthropic-native, Vertex, and OpenRouter Claude tolerate/ignore the marker (or LiteLLM keys them under names we don't resolve), so gating those on confirmed support would disable caching for capable models, the opposite of this PR's intent. Only Bedrock hard-rejects, so only Bedrock is guarded. Unmapped Bedrock model → runs uncached instead of crashing.

Added two tests covering both directions (unmapped Bedrock → no marker; unmapped non-Bedrock → keeps marker).