Repository navigation
feat(bedrock,vertex): serve /v1/messages to Claude natively on Bedrock and Vertex AI - #1318
Merged
Merged
Conversation
…k and Vertex AI /v1/messages and /v1/messages/count_tokens addressed to Claude on Amazon Bedrock or Vertex AI no longer go through the chat translation. The caller's Anthropic body goes to Bedrock InvokeModel / InvokeModelWithResponseStream / CountTokens and to Vertex AI rawPredict / streamRawPredict / the count-tokens model, and the answer comes back in the Anthropic wire: thinking signatures (and signature_delta), redacted_thinking, images inside tool_result, stop reasons, message_start usage with the cache counters, and the upstream message id. On the way the body is adapted per platform: model leaves the body, the platform's anthropic_version goes in, top-level fields are filtered to what the platform accepts, metadata keeps user_id only, anthropic-beta values are filtered to the platform's known set (body anthropic_beta on Bedrock, header on Vertex AI), the effort beta is added before Claude 4.6, Bedrock context_management edits are filtered to the ones it runs, cache_control keeps type+ttl (Bedrock) / drops scope (Vertex), and media URLs are downloaded through upstream.media_fetch. A Bedrock stream exception surfaces as an Anthropic error event (rate_limit_error, overloaded_error, api_error, ...); one raised before any content fails the attempt so routing can fail over. The shared passthrough pipeline (guardrails, usage event, cost, access log, model echo, failover) is reused.
|
Warning Review limit reached
This review includes 28 billable files and costs up to $7.00.
Reviews can continue after your included limit without a manual trigger. An admin must approve usage-based billing. Or wait 6 minutes for your next included review. View limit details
Comment |
…errides, counts tokens strictly - The provider key's request overrides are applied by the proxy as on the Anthropic passthrough; content_list_to_string (a response override) no longer rewrites the forwarded Messages body. - The platform bridge takes the body value instead of re-parsing its bytes. - Vertex AI count-tokens sends no anthropic-beta and treats an answer without input_tokens as a decode error rather than a count of 0. - A downloaded text/* body becomes a text source only for a document block. - The count_tokens 400 names Claude on Bedrock / Vertex AI as a valid target.
…up's next target Bedrock CountTokens and Vertex AI count-tokens serve fewer models and regions than the Messages routes, so a non-retryable refusal from one no longer ends a count_tokens a later Anthropic-wire target can answer. Media downloads on a streamed platform call are bounded by the request's end-to-end budget rather than the stream chunk gap.
… /v1/messages gets The CountTokens invokeModel body and the Vertex AI count-tokens body are shaped like the Messages passthrough toward the same platform: top-level field allowlist, metadata.user_id only, cache_control type+ttl on Bedrock and without scope on Vertex AI. The invokeModel body's max_tokens is raised above a thinking budget so InvokeModel's schema accepts it; the value does not change the count.
…drock Mantle
bedrock-runtime's CountTokens refuses Claude models Bedrock serves only
through cross-region inference (Claude Opus 5.5 among them); AWS counts
them on Bedrock Mantle's Anthropic count_tokens. When the runtime call
answers 400/404 for a Claude model, count_tokens is retried on
POST {mantle}/anthropic/v1/messages/count_tokens with the base model id,
messages/system/tools and the thinking/tool_choice/output_config options,
signed SigV4 (service bedrock) from the provider key's access-key
credentials.
New startup key upstream.bedrock.mantle_endpoint (default null: the
public regional host https://bedrock-mantle.{region}.api.aws derived from
the provider key's region) for VPC-endpoint and GovCloud deployments.
jarvis9443
added a commit
to api7/api7-helm-chart
that referenced
this pull request
Oct 20, 2026
Pairs with api7/aisix#1318: the Bedrock Mantle endpoint count_tokens falls back to for Claude models bedrock-runtime cannot count. Null derives https://bedrock-mantle.{region}.api.aws from the provider key's region.
…ative-messages # Conflicts: # crates/aisix-gateway/src/lib.rs # crates/aisix-gateway/src/media_fetch.rs # crates/aisix-provider-bedrock/Cargo.toml # crates/aisix-provider-bedrock/src/bridge.rs # crates/aisix-provider-bedrock/src/lib.rs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
/v1/messagesand/v1/messages/count_tokensaddressed to Claude on Amazon Bedrock or Vertex AI now go to the platform's own Anthropic Messages routes with the caller's body, instead of being parsed into the chat form and rebuilt for InvokeModel (non-stream) or ConverseStream (stream). That translation was lossy for agent clients such as Claude Code: every thinking block came back withsignature: ""and nosignature_delta(so replayed thinking got stripped and the model lost its own reasoning each tool loop, which also invalidated message-level prompt-cache breakpoints), an image inside atool_resultnever reached the model, streamingmessage_startusage was 0/0 with no cache fields, the streamed message id was gateway-made and no provider request id was recorded for streams,metadata.user_id/stop_sequencesleaked as unknown keys, everyanthropic-betavalue was dropped,context_management/is_error/citations were dropped, several stop reasons collapsed toend_turn, and count_tokens answered 400.What is sent now:
InvokeModel/InvokeModelWithResponseStream(/model/{id}/invoke,/invoke-with-response-stream) andCountTokens(/model/{base id}/count-tokens,invokeModelinput when the body carries typed content blocks,converseinput when every message is a plain string).:rawPredict/:streamRawPredictonpublishers/anthropic/models/{id}, andpublishers/anthropic/models/count-tokens:rawPredictwith the version suffix (@…) dropped frommodel.invokeModelbody carries amax_tokensabove any thinking budget (it does not change the count). A platform refusing to count a model moves a routing group on to its next Anthropic-wire target instead of ending the request.POST {mantle}/anthropic/v1/messages/count_tokenswith the base model id,messages/system/toolsand thethinking/tool_choice/output_configoptions,anthropic-version: 2023-06-01, signed SigV4 (servicebedrock) from the provider key's access-key credentials. New startup keyupstream.bedrock.mantle_endpoint(defaultnull: the public regional hosthttps://bedrock-mantle.{region}.api.awsfrom the provider key's region) for VPC-endpoint / GovCloud deployments; the chart declares it in feat(aisix): declare upstream.media_fetch, upstream.bedrock and outbound.ssrf_protection in values.yaml api7-helm-chart#409.bedrockadapter with ananthropic.model id (region / inference-profile prefix allowed) or an ARN naming a Claude model, and thevertexadapter with aclaude-*model. An ARN that hides its model (application inference profile) keeps the translation./v1/chat/completionsand/v1/responsesare unchanged.What the body goes through on the way, and nothing else:
modelleaves the body;anthropic_versionisbedrock-2023-05-31/vertex-2023-10-16;streamis removed on Bedrock and kept on Vertex AI.anthropic_version, anthropic_beta, max_tokens, messages, system, stop_sequences, temperature, top_p, top_k, tools, tool_choice, thinking, metadata, output_config, safeguards, context_management. Vertex AI:anthropic_version, max_tokens, messages, metadata, stop_sequences, stream, system, temperature, thinking, tool_choice, tools, top_k, top_p, inference_geo, mcp_servers, context_management, compaction, container, output_config, speed, cache_control, safeguards. A legacy top-leveloutput_formatmoves intooutput_config.format.metadatakeepsuser_idonly.anthropic-betavalues (the caller's header, plus ones the body's own features need) are filtered to each platform's known set and mapped where the platform names them differently (advanced-tool-use-2025-11-20→tool-search-tool-2025-10-19). Bedrock reads them from the body'santhropic_beta; Vertex AI from theanthropic-betaheader, which is filtered the same way whatever set it. Bedrock getseffort-2025-11-24added whenoutput_config.effortis set on a family before Claude 4.6,tool-search-tool-2025-10-19when a tool-search tool is present, and itscontext_management.editsfiltered to the ones InvokeModel runs (compact_20260112,clear_tool_uses_20250919, each with its beta; the field is dropped when none is left). Vertex AI gets the context-management, web-search, tool-search, safeguards and per-turn-control betas when the body uses those features.cache_controlkeepstypeandttlon Bedrock, dropsscopeon Vertex AI.tool_result) is downloaded and inlined throughupstream.media_fetch(same SSRF, size and per-request limits); with media fetching off, a URL source is a 400.The answer is relayed in the Anthropic wire through the same passthrough pipeline Anthropic's own API uses, so guardrails on request and streamed response, the usage event and cost (including cache read / creation), the access log, the provider request id (the upstream message id, stream and non-stream), model-name projection, failover before the first byte and mid-stream handling all hold. Bedrock's event stream is re-framed as Anthropic SSE; an exception Bedrock raises inside it becomes an Anthropic
errorevent with an Anthropic error type (throttlingException→rate_limit_error,serviceUnavailableException→overloaded_error,modelStreamErrorExceptionby its original status,internalServerException→api_error,validationException→invalid_request_error,modelTimeoutException→timeout_error), so an Anthropic client's retry logic reads it. An exception raised before any content fails the attempt instead, so routing can fail over. Errors carry the upstream's own message text, as every Bedrock and Vertex AI error does since #1319 (a status phrase only when the upstream sent none).Behavior change for existing callers: a
/v1/messagescaller on Claude-on-Bedrock/Vertex now gets the platform's response verbatim (signatures,redacted_thinking, upstream message id, native stop reasons,amazon-bedrock-invocationMetricson Bedrock'smessage_stop), forwardedanthropic-beta/metadata.user_id/context_management, and a working count_tokens. Nothing has to be edited in configuration.Tests:
claude-platform-native-messages-e2edrives mock Bedrock (InvokeModel, event stream incl. an in-bandthrottlingException, CountTokens) and mock Vertex AI upstreams and asserts the outbound body/headers and the client-visible output: signatures both directions (non-stream + stream), the image inside a tool result,message_startusage with cache counters, upstream message id and provider request id on the usage event, beta filtering, field allowlist, metadata, URL image inlining, count_tokens on both, the throttle mapped torate_limit_error, and an output guardrail on the passthrough stream. All of them fail with the platform branch disabled.claude-platform-prompt-caching,bedrock-converse-stream-reasoningandanthropic-stream-tool-callsare updated for the streamed/v1/messagescases that pinned ConverseStream / the translated stream.🤖 Generated with Claude Code
Fixes api7/AISIX-Cloud#1885