feat(providers): prompt caching capability + usage-based cache pricing#5922
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
Keep staging's SSO migration at 0268 and regenerate include_tool_calls as 0269.
|
bugbot run |
PR SummaryMedium Risk Overview Agent-events protocol is split: Public chat SSE redacts tool arguments/results in Agent block adds an advanced Prompt caching switch for supported Anthropic models; the executor forwards Billing consistency: evaluator/router/Pi paths use proxied or centralized billable cost instead of recomputing from tokens. Provider streaming responses omit unsettled content from Docs and add-model guidance are updated for prompt caching, OpenAI live tools, Anthropic summary thinking, and the new protocol/toggle behavior. Reviewed by Cursor Bugbot for commit ca00374. Configure here. |
Greptile SummaryThis PR adds live agent-event streaming and cache-aware provider pricing.
Confidence Score: 5/5The PR appears safe to merge. No blocking failures remain.
|
| Filename | Overview |
|---|---|
| apps/sim/providers/openai/streaming-tool-loop.ts | Implements a live OpenAI Responses tool loop with start events emitted before execution and end events emitted after completion. |
| apps/sim/lib/workflows/streaming/agent-stream-protocol.ts | Defines the versioned thinking, tool-lifecycle, answer, and reset frames consumed by streaming clients. |
| apps/sim/providers/cost-policy.ts | Centralizes cache-aware model-usage pricing across normalized provider usage shapes. |
| apps/sim/app/api/chat/[identifier]/route.ts | Negotiates agent-event framing and independently applies thinking and tool-call deployment policies. |
| packages/db/schema.ts | Adds persisted provider and chat capability fields with compatibility behavior for existing deployments. |
| apps/docs/content/docs/en/workflows/blocks/agent.mdx | Documents prompt caching and regenerated provider streaming capabilities. |
Sequence Diagram
sequenceDiagram
participant Client
participant Chat as Chat API
participant Agent as Agent Executor
participant OpenAI
participant Tool
Client->>Chat: Request with agent-events-v1
Chat->>Agent: Execute with event policies
Agent->>OpenAI: Start Responses stream
OpenAI-->>Agent: response.output_item.added(function_call)
Agent-->>Client: tool start
Agent->>Tool: Execute tool call
Tool-->>Agent: Tool result
Agent-->>Client: tool end
Agent->>OpenAI: Continue with tool result
OpenAI-->>Agent: Final answer deltas
Agent-->>Client: Answer chunks
Reviews (5): Last reviewed commit: "refactor(providers): drop unreachable bi..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Replace the arbitrary cached-rate heuristic with a single cache-aware pricing function, and add prompt caching as an opt-in capability for Anthropic. Pricing: priceModelUsage in cost-policy.ts is now the only place cache arithmetic happens. Provider adapters normalize their wire shape into ModelUsage (input always excludes cache buckets); the pricing function never branches on provider. This removes five divergent behaviors, including the !!request.context heuristic that gave Router and Evaluator an unearned 10x input discount, and the overwrite that silently billed Anthropic cache reads and writes at zero. Also parses OpenAI cache_write_tokens, previously ignored. Caching: Anthropic gets a capability-gated advanced switch that places cache_control on the last tool and last system block; system is now always a TextBlockParam array. OpenAI gets a stable per-block prompt_cache_key with no UI, since its caching is automatic.
…ol-stream # Conflicts: # apps/docs/content/docs/en/workflows/blocks/agent.mdx
icecrasher321
changed the title
|
@cursor review |
…e pricing Cache-aware pricing only reached trace segments. The billable block cost still called calculateCost on the cache-inclusive prompt total, so OpenAI cache hits and Gemini implicit-cache hits were charged at the full input rate and GPT-5.6+ cache writes went unbilled. Both providers now accumulate cache buckets and price through priceModelUsage, matching the Anthropic token convention where input excludes cache reads and writes. Cached counts are clamped to the prompt total so an over-reporting payload cannot bill more input than the request contained.
|
@cursor review |
Redaction only ran on the empty-selection branch, but a deployment almost always selects outputs, so it was dead in the case it exists for. Selecting toolCalls streamed the raw arguments and results to a public chat client in a chunk frame, and providerTiming carried thinking content the same way. Both paths now extract from the sanitized block output rather than the raw log: the streamed selected output, which is the reachable vector, and the final envelope. Sanitizing the source rather than per selected path means a newly selectable field cannot reopen the hole.
|
@cursor review |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit fbca58a. Configure here.
This comment has been minimized.
This comment has been minimized.
Every provider pricing helper took a policy parameter no caller passed. Worse than dead: passing one would have double-applied the margin the central layer already applies. Removed, so providers can only price at list. Also removed guards that cannot fire. The central fallback normalized cache buckets no provider can reach it with (all three that report cache usage price themselves) and did so at a 1x write multiplier no vendor charges. priceModelUsage re-validated token counts the adapter had already clamped, and applyModelCostPolicy defaulted a required total field. Validation now happens once, in the adapter that parses the vendor payload and is the only layer that knows cache buckets are a subset of the prompt total.
|
@cursor review |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit ca00374. Configure here.
1 participant
Summary
Three things, in dependency order:
includeThinking/includeToolCallstoggles.priceModelUsageincost-policy.tsis now the only place cache-aware arithmetic happens. Provider adapters normalize their wire shape intoModelUsage; the pricing function never branches on provider.cache_controlbreakpoints), always-onprompt_cache_keyfor OpenAI, billing correctness only for Gemini.Type of Change
Testing
bun run lint,bun run type-check,bun run check:api-validation:strict,bun run check:migrations origin/staging, and the full suite (1111 files / 14478 tests) all pass locally.New coverage: cross-provider pricing parity (Anthropic's exclusive vs OpenAI's inclusive token shapes must price identically), per-bucket write multipliers,
cache_write_tokensparsing and clamping, multi-turn cache accumulation for OpenAI and Gemini, Anthropic breakpoint placement on both the streaming and non-streaming paths, structured-output + caching together, stable/distinctprompt_cache_key, and the two tool-cost invariants.Checklist