{part.name}
; + } + if (part.type === "structured-output") { + const report = part.data ?? part.partial; + return report?.name ?{report.name}
: null; + } + return null; + }), + )} + {final ?{final.oneLiner}
: null} + + ); } ``` -## Workspace skills +You can also read the object from `await chat()` when you do not stream. See [Harness Agents](../structured-outputs/harnesses) for the full part list and the repo-report example. + +## Models and options + +`acpCompatible` has no built-in model list. The model is any string that your agent takes, or one of the `models` that you declare. -When you provision a [workspace](../sandbox/workspace) via `withSandbox`, -`acpCompatible` projects its skills into the harness — each kind lands where that -harness expects it: +### Configuration -| Workspace input | How `acpCompatible` projects it | +Pass these to `acpCompatible` or as the second argument of `acpCompatibleText`: + +| Field | What it does | +| --- | --- | +| `name` (required) | The harness label, the log prefix, and the name of the `
+
+
+ {isLoading &&
+ );
+}
+```
+
+The job record stays for seven days. The video URL expires 24 hours after the job completes.
+
+### Seedance options per model
+
+Seedance options are different for each model. Ark rejects an option that does not apply with a `400`, for example: "the specified parameter `draft` is not supported for model seedance-1-0-pro in t2v, must be empty".
+
+| Option | Models that accept it |
+| --- | --- |
+| `service_tier` (`"default"` or `"flex"`) | Seedance 1.x only |
+| `camera_fixed` | Seedance 1.x only |
+| `frames` (`25 + 4n`, from `29` to `289`) | `seedance-1-0-pro-250528`, `seedance-1-0-pro-fast-251015` |
+| `draft` (a cheap preview) | `seedance-1-5-pro-251215` only |
+| `priority` (`0` to `9`) | Seedance 2.5 and the `dreamina-seedance-2-0-*` models |
+| `duration: -1` (the model picks the length) | Seedance 2.5, Seedance 2.0, `seedance-1-5-pro-251215` |
+| `output_format` (`"mp4"` or `"mov"`) | Seedance 2.5 (default `mp4`) |
+| `seed`, `watermark`, `generate_audio`, `return_last_frame`, `callback_url` | All models |
+
+- `watermark` is `false` by default for video. For Seedream images, it is `true`.
+- All models accept `generate_audio`. Only Seedance 2.5, Seedance 2.0, and `seedance-1-5-pro-251215` add an audio track.
+
+Resolutions also change by model:
+
+- No Seedance model has a 2K tier.
+- Only `dreamina-seedance-2-0-260128` has `4k`.
+- Seedance 2.5 accepts `480p`, `720p`, and `1080p`.
+- `seedance-1-0-pro-fast-251015` accepts `1080p`.
+
+The adapter finds a combination that a model does not support before it sends the request, and throws a clear error.
+
+Reference media uses the shared [role hints](../media/video-generation#role-hints): `start_frame`, `end_frame`, and `reference`.
+
+- Seedance 2.5 and Seedance 2.0 take reference images, video, and audio.
+- Seedance 2.5 also accepts audio-only reference input. It takes up to 30 images, 10 videos, and 10 audio clips. Seedance 2.0 takes 9, 3, and 3.
+- The 1.x models take start and end frames only. `seedance-1-0-pro-fast-251015` takes a start frame only.
+
+### Seedance 2.5
+
+`dreamina-seedance-2-5-260628` is the newest Seedance model. Use it for long clips (up to 30 seconds), audio-only reference input, and `output_format: "mov"`.
+
+```typescript
+import { generateVideo } from "@tanstack/ai";
+import { byteplusVideo } from "@tanstack/ai-byteplus";
+
+const { jobId } = await generateVideo({
+ adapter: byteplusVideo("dreamina-seedance-2-5-260628"),
+ prompt: "a guitar being played in a store",
+ threadId: "thread-1",
+ size: "16:9_1080p",
+ duration: 10,
+ modelOptions: {
+ generate_audio: true,
+ priority: 5,
+ output_format: "mp4",
+ },
+});
+
+console.log(jobId);
+```
+
+| Capability | Seedance 2.5 |
+| --- | --- |
+| Duration | 4 to 30 seconds, or `-1` (the model picks; required for video edit tasks) |
+| Resolution | `480p`, `720p`, `1080p` (default `720p`). No 4k. |
+| Reference media | 1 to 30 images, 0 to 10 videos, 0 to 10 audio clips. Audio only is allowed. |
+| First and last frame | Yes |
+| `priority`, `generate_audio` | Yes |
+| `service_tier`, `camera_fixed`, `frames`, `draft` | No |
+
+The account can need to activate the model, or buy a resource pack, in the Ark Console. Until then, a call fails with `404 ModelNotOpen`.
+
+`byteplusVideo()` accepts any string, so you can use an id that BytePlus releases before this package lists it. For an unknown id, the adapter skips its local checks and lets Ark validate the request. Known ids keep their checks.
+
+### Seedance through BytePlus or fal
+
+You can also reach Seedance through [`@tanstack/ai-fal`](./fal). This adapter talks to BytePlus directly, so you get:
+
+- BytePlus billing and rate limits.
+- The model ids of BytePlus.
+- All the Seedance fields above (`camera_fixed`, `draft`, `priority`, reference media roles). fal has a smaller, shared set.
+
+Use the account that you already have. You do not need both packages for Seedance.
+
+## Seedream reference images
+
+To edit an image, or to guide the result with images, put image parts in the `prompt` array. Seedream takes up to 14 reference images, and `dola-seedream-5-0-pro-260628` takes up to 10:
+
+```typescript
+import { generateImage } from "@tanstack/ai";
+import { byteplusImage } from "@tanstack/ai-byteplus";
+
+const result = await generateImage({
+ adapter: byteplusImage("seedream-5-0-260128"),
+ prompt: [
+ { type: "text", content: "Put this guitar on a concert stage" },
+ {
+ type: "image",
+ source: { type: "url", value: "https://example.com/guitar.png" },
+ },
+ ],
+ threadId: "thread-1",
+});
+
+console.log(result.images.length);
+```
+
+Two Seedream behaviors to know:
+
+- The adapter never sets `watermark` for you, so the BytePlus default (`true`) applies.
+- Seedream has no `n` option. When you ask for more than one image, the adapter uses the group-image mode (`sequential_image_generation: "auto"` with `max_images`). The model then decides how many images the prompt needs. A request for four can return two. The adapter logs a warning when this occurs.
+
+## Seed Speech
+
+### Voice ids
+
+Seed Speech has no top-level `speaker` field. The voice goes inside a `references` entry. The adapter turns `voice` into `references: [{ speaker: "en_female_stokie_uranus_bigtts" }]` for you.
+
+`modelOptions.references` replaces that entry. It does not merge with it. So if you send `references` for voice cloning, the adapter drops `voice`. To keep a stock voice with your clips, add a `speaker` entry yourself.
+
+The end of a voice id tells you its generation:
+
+- `_uranus_bigtts`: TTS 2.0 voices, the newest generation.
+- `_mars_bigtts` and `_moon_bigtts`: TTS 1.0 voices.
+- `*_emo_v2_*`: TTS 1.0 voices that also accept emotion tags.
+
+The full list is in the [BytePlus voice list](https://docs.byteplus.com/en/docs/byteplusvoice/voicelist). It changes more often than this package, so the adapter accepts any string. `speaker` also takes a voice that you cloned with Voice Replication.
+
+### Speech options
+
+Use `modelOptions` for these:
+
+- `ogg_opus` output and an exact `sample_rate`.
+- `references`: voice cloning, with up to three 30-second audio clips. Refer to them from the text as `@Audio1` to `@Audio3`.
+- `watermark`, `speech_rate`, and the raw `enable_subtitle` timings.
+
+```typescript
+import { generateSpeech } from "@tanstack/ai";
+import { byteplusSpeech, type BytePlusTTSResult } from "@tanstack/ai-byteplus";
+
+const result: BytePlusTTSResult = await generateSpeech({
+ adapter: byteplusSpeech("seed-audio-1.0"),
+ text: "welcome to the guitar store",
+ threadId: "thread-1",
+ modelOptions: {
+ format: "ogg_opus",
+ sample_rate: 48000,
+ speech_rate: 20,
+ enable_subtitle: true,
+ },
+});
+
+for (const sentence of result.subtitle?.sentences ?? []) {
+ console.log(sentence.text, sentence.start_time);
+}
+```
+
+### Timings
+
+The shared `timestamps: true` option is easier than `modelOptions.enable_subtitle`. It sets the same flag. Then it maps the timings to `result.alignment` (per word) and `result.segments` (per sentence), in seconds:
+
+```typescript
+import { generateSpeech } from "@tanstack/ai";
+import { byteplusSpeech } from "@tanstack/ai-byteplus";
+
+const result = await generateSpeech({
+ adapter: byteplusSpeech("seed-audio-1.0"),
+ text: "welcome to the guitar store",
+ threadId: "thread-1",
+ timestamps: true,
+});
+
+console.log(result.alignment?.unit); // "word"
+console.log(result.alignment?.endSeconds.at(-1)); // where speech stops
+```
+
+### Dialogue
+
+Seed Audio 1.0 makes a dialogue with several voices in one request. Pass `turns`, and the adapter builds the `references` array and the role prompt for you. A request can have up to three different voices:
+
+```typescript
+import { generateSpeech } from "@tanstack/ai";
+import { byteplusSpeech } from "@tanstack/ai-byteplus";
+
+// Replace SECOND_VOICE with another voice id from the BytePlus voice list.
+const SECOND_VOICE = "your-second-voice-id";
+
+const result = await generateSpeech({
+ adapter: byteplusSpeech("seed-audio-1.0"),
+ turns: [
+ { text: "Do you sell picks?", voice: "en_female_stokie_uranus_bigtts" },
+ { text: "By the till.", voice: SECOND_VOICE },
+ ],
+ timestamps: true,
+ threadId: "thread-1",
+});
+
+for (const segment of result.segments ?? []) {
+ console.log(segment.startSeconds, segment.text);
+}
+```
+
+### Watermarks
+
+For speech, `watermark` is an object. It has two markers, and both are off by default:
+
+- `aigc_watermark`: adds a sound marker at the end of the clip.
+- `aigc_metadata`: writes the source of the audio into the audio header. It writes nothing unless `enable` is `true`.
+
+```typescript
+import { generateSpeech } from "@tanstack/ai";
+import { byteplusSpeech } from "@tanstack/ai-byteplus";
+
+const result = await generateSpeech({
+ adapter: byteplusSpeech("seed-audio-1.0"),
+ text: "welcome to the guitar store",
+ threadId: "thread-1",
+ modelOptions: {
+ watermark: {
+ aigc_watermark: true,
+ aigc_metadata: { enable: true, content_producer: "guitar-store" },
+ },
+ },
+});
+
+console.log(result.format);
+```
+
+`watermark: true` is short for `{ aigc_watermark: true }`. Images and video use a boolean `watermark`. Only speech takes the object.
+
+### Limits
+
+- One request makes 120 seconds of audio at most. The limit applies to `originalDuration`, the length before `speech_rate`. BytePlus also bills on this length.
+- The raw `subtitle` timings are in milliseconds. `duration` and `originalDuration` are in seconds. `alignment` and `segments` are always in seconds.
+- The `url` on the result expires about two hours after generation. Keep the base64 `audio`, not the link.
+
+## Transcription options
+
+Set these in `modelOptions` of `byteplusTranscription`:
+
+- `enable_itn`: writes spoken numbers and dates as digits.
+- `enable_punc`: adds punctuation.
+- `enable_ddc`: removes filler words.
+- `enable_speaker_info`: adds speaker labels, which come back as `segment.speaker`.
+
+## API reference
+
+Each factory pair has the same shape. The short factory reads the key from the environment. The `create*` factory takes an explicit API key. Both take `model` as the first argument.
+
+| Factories | Creates | Key |
+| --- | --- | --- |
+| `byteplusText(model, config?)` / `createBytePlusText(model, apiKey, config?)` | Chat adapter | `ARK_API_KEY` or `BYTEPLUS_API_KEY` |
+| `byteplusVideo(model, config?)` / `createBytePlusVideo(model, apiKey, config?)` | Seedance video adapter (experimental) | `ARK_API_KEY` |
+| `byteplusImage(model, config?)` / `createBytePlusImage(model, apiKey, config?)` | Seedream image adapter | `ARK_API_KEY` |
+| `byteplusSpeech(model, config?)` / `createBytePlusSpeech(model, apiKey, config?)` | Seed Speech text-to-speech adapter | `BYTEPLUS_VOICE_API_KEY` |
+| `byteplusTranscription(model, config?)` / `createBytePlusTranscription(model, apiKey, config?)` | Seed Speech transcription adapter | `BYTEPLUS_VOICE_API_KEY` |
+
+- `config.baseURL` sets the region endpoint of the Ark adapters.
+- `adapter.inputModalities` lists the input kinds of the selected chat model. It is `undefined` for a model that this package does not know.
+- BytePlus has no provider tools. Define your tools with `toolDefinition()`. Ark uses the OpenAI tool-call format, so the usual [tools flow](../tools/tools.md) works.
diff --git a/docs/adapters/byteplus.md b/docs/adapters/byteplus.md
index 7d00c036db..fbbf712ae8 100644
--- a/docs/adapters/byteplus.md
+++ b/docs/adapters/byteplus.md
@@ -2,7 +2,7 @@
title: BytePlus
id: byteplus-adapter
order: 8
-description: "Use BytePlus ModelArk with TanStack AI — Seed chat models with reasoning, Seedance video generation, Seedream image generation, and Seed Speech text-to-speech and transcription via @tanstack/ai-byteplus."
+description: "Use BytePlus ModelArk in TanStack AI with @tanstack/ai-byteplus: Seed chat models with reasoning, Seedance video, Seedream images, and Seed Speech text-to-speech and transcription."
keywords:
- tanstack ai
- byteplus
@@ -20,14 +20,9 @@ keywords:
- adapter
---
-The BytePlus adapter connects TanStack AI to [BytePlus](https://www.byteplus.com/), ByteDance's international model platform, across four generation kinds:
+`@tanstack/ai-byteplus` calls Seed chat models with reasoning, plus Seedance video, Seedream images, and Seed Speech.
-- **Chat** — the Seed model family (plus GLM, DeepSeek and gpt-oss) on ModelArk's OpenAI-compatible `/chat/completions` endpoint, with reasoning on by default.
-- **Video** — Seedance, an asynchronous task API.
-- **Image** — Seedream.
-- **Speech** — Seed Speech text-to-speech and transcription, which live on a **different host with a different API key** (see [Two products, two keys](#two-products-two-keys)).
-
-## Installation
+## 1. Install
@@ -42,591 +37,281 @@ octane: @tanstack/ai-byteplus
-## Two products, two keys
-
-BytePlus splits these models across two products, and they do not share credentials:
-
-| Adapters | Product | Env var | Auth header |
-| --- | --- | --- | --- |
-| `byteplusText`, `byteplusVideo`, `byteplusImage` | ModelArk (Ark) | `ARK_API_KEY` (falls back to `BYTEPLUS_API_KEY`) | `Authorization: Bearer` |
-| `byteplusSpeech`, `byteplusTranscription` | Seed Speech | `BYTEPLUS_VOICE_API_KEY` | `X-Api-Key` |
+## 2. Set the API keys
-BYOK uses two slugs because these are two keys: `byteplusByok` (Ark) and `byteplusVoiceByok` (Seed Speech), both from `@tanstack/ai-byteplus/byok`.
+BytePlus sells these models as two products, and each product has its own key. Put both keys in the environment of your server:
```bash
-# ModelArk: chat, Seedance video, Seedream image
+# ModelArk: chat, Seedance video, Seedream images
ARK_API_KEY=...
-# Seed Speech: TTS and transcription — a separate product key
+# Seed Speech: text-to-speech and transcription
BYTEPLUS_VOICE_API_KEY=...
```
-Passing an Ark key to the speech adapters fails with `45000010 Invalid X-Api-Key`.
+- `byteplusText`, `byteplusVideo`, and `byteplusImage` read `ARK_API_KEY`. If it is not set, they read `BYTEPLUS_API_KEY`.
+- `byteplusSpeech` and `byteplusTranscription` read `BYTEPLUS_VOICE_API_KEY`. An Ark key fails here with `45000010 Invalid X-Api-Key`.
-Ark keys are also **region-isolated**: the default base URL is the Asia-Pacific south-east endpoint (`https://ark.ap-southeast.bytepluses.com/api/v3`), and a key issued for one region will not authenticate against another. Point the adapter elsewhere with `baseURL`:
+To pass the key in code, use the `create*` factory of the same adapter:
```typescript
-import { createBytePlusText } from '@tanstack/ai-byteplus'
-import { arkApiKey } from './config'
+import { createBytePlusText } from "@tanstack/ai-byteplus";
-const adapter = createBytePlusText('dola-seed-2-1-turbo-260628', arkApiKey, {
- baseURL: 'https://ark.eu-west.bytepluses.com/api/v3',
-})
+const adapter = createBytePlusText("dola-seed-2-1-turbo-260628", process.env.ARK_API_KEY!);
```
-Per the BytePlus docs the EU endpoint serves chat and image only — Seedance video is Asia-Pacific only.
-
-## Chat
-
-The adapter carries the model, so there is no separate `model` option.
+Ark keys work in one region only. The default endpoint is Asia-Pacific south-east. For a key from another region, see [Regions](./byteplus-advanced#regions).
-**Server** — an endpoint that streams the reply back over SSE:
+## 3. Stream a chat from the server
```typescript
-import { chat, toServerSentEventsResponse } from '@tanstack/ai'
-import { byteplusText } from '@tanstack/ai-byteplus'
+import {
+ chat,
+ chatParamsFromRequest,
+ toServerSentEventsResponse,
+} from "@tanstack/ai";
+import { byteplusText } from "@tanstack/ai-byteplus";
export async function POST(request: Request) {
- const { messages } = await request.json()
+ const { messages, threadId, runId } = await chatParamsFromRequest(request);
const stream = chat({
- adapter: byteplusText('dola-seed-2-1-turbo-260628'),
+ adapter: byteplusText("dola-seed-2-1-turbo-260628"),
messages,
- })
+ threadId,
+ runId,
+ });
- return toServerSentEventsResponse(stream)
+ return toServerSentEventsResponse(stream);
}
```
-**Client** — the same `useChat` hook as every other provider:
+## 4. Render the chat
+
+The client is the same for every provider. Point `useChat` at your route:
```tsx
-import { useState } from 'react'
-import { fetchServerSentEvents, useChat } from '@tanstack/ai-react'
+import { useState } from "react";
+import { useChat, fetchServerSentEvents } from "@tanstack/ai-react";
export function Chat() {
- const [input, setInput] = useState('')
-
- const { messages, sendMessage, isLoading } = useChat({
- connection: fetchServerSentEvents('/api/chat'),
- })
+ const [input, setInput] = useState("");
+ const { messages, sendMessage } = useChat({
+ connection: fetchServerSentEvents("/api/chat"),
+ threadId: "support-chat",
+ });
return (
- Status: {videoStatus?.status ?? "starting"}
} + {error &&Error: {error.message}
} + {result && } +
+
-
- )
+ setInput(event.target.value)} />
+
+ );
}
```
-### Model options
-
-Ark's chat endpoint is OpenAI-compatible, so sampling parameters keep their OpenAI snake_case names and live in `modelOptions`. `repetition_penalty` and `service_tier` are the Ark-only additions. The thinking level goes in `reasoning`:
-
-```typescript
-import { chat, toServerSentEventsResponse } from '@tanstack/ai'
-import { byteplusText } from '@tanstack/ai-byteplus'
-
-export async function POST(request: Request) {
- const { messages } = await request.json()
-
- const stream = chat({
- adapter: byteplusText('dola-seed-2-1-turbo-260628'),
- messages,
- modelOptions: {
- temperature: 0.7,
- top_p: 0.9,
- max_tokens: 2048,
- },
- reasoning: 'medium',
- })
-
- return toServerSentEventsResponse(stream)
-}
-```
+For Vue, Solid, Svelte, Angular, and the other frameworks, see [Quick Start](../getting-started/quick-start#3-render-the-chat).
-`max_tokens` and `max_completion_tokens` are mutually exclusive. The types cannot express this, and Ark returns a `400` when you send both.
-
-`service_tier: 'flex'` routes the request to the cheaper offline batch queue with no latency guarantee.
-
-Need a header on each request, or a log of each model call? The chat adapter supports [`wrapFetch`](../advanced/middleware#change-the-http-requests-of-a-call).
-
-## Reasoning and `encrypted_content`
-
-Seed models reason by default. Reasoning arrives as its own stream of `reasoning_content` deltas and is surfaced as reasoning content rather than answer text, so `useChat` renders it separately from the reply. Turn it off per request, on a model that can stop thinking:
-
-```typescript
-import { chat, toServerSentEventsResponse } from '@tanstack/ai'
-import { byteplusText } from '@tanstack/ai-byteplus'
-
-export async function POST(request: Request) {
- const { messages } = await request.json()
-
- const stream = chat({
- adapter: byteplusText('glm-5-2-260617'),
- messages,
- reasoning: 'off',
- })
-
- return toServerSentEventsResponse(stream)
-}
-```
-
-`off` sends `thinking: { type: 'disabled' }` and no effort. The other levels send `thinking: { type: 'enabled' }` and `reasoning_effort`. `deepseek-v3-2-251201` is the one model that does not think by default.
-
-The levels of each model:
-
-- `dola-seed-2-1-turbo-260628` and the `seed-*` models: `minimal` to `high`. They cannot stop thinking.
-- `glm-5-2-260617`: `off` to `high`. `glm-4-7-251222`, `deepseek-v3-2-251201`: `off`, `high`.
-- `deepseek-v4-pro-260425`, `deepseek-v4-flash-260425`: `off`, `high`, `max`.
-- `gpt-oss-120b-250805`: `low`, `medium`, `high`.
-
-The four "thinking summary" models — `dola-seed-2-1-turbo-260628`, `seed-2-0-lite-260428`, `seed-2-0-mini-260428` and `seed-2-0-pro-260328` — also emit an opaque `encrypted_content` blob alongside the reasoning trace. It is a signature over that trace, and BytePlus's docs ask for it back verbatim on the assistant message in the next turn.
-
-**The adapter round-trips it for you**, over the same seam Anthropic's thinking signatures use: the blob is captured off the stream and attached to the reasoning step as its `signature`, which lands on the thinking part of the assistant message and is echoed back on the next request. Two consequences worth knowing:
-
-- If you persist and replay conversation history yourself, keep the thinking parts' `signature` — dropping it costs you the reasoning-cache hit. It is not fatal: a live probe confirmed Ark accepts a turn whose assistant message omits `encrypted_content`.
-- A structured-output turn doesn't capture the blob, for the same reason — again a lost cache hit, not a failed request.
-
-## Structured output
-
-Ten of the eighteen chat models accept `response_format: { type: 'json_schema' }`:
-
-`dola-seed-2-1-turbo-260628`, `seed-2-0-pro-260328`, `seed-2-0-lite-260228`, `seed-2-0-mini-260215`, `seed-1-8-251228`, `seed-1-6-250915`, `seed-1-6-250615`, `seed-1-6-flash-250715`, `seed-1-6-flash-250615`, `glm-5-2-260617`.
-
-```typescript
-import { chat } from '@tanstack/ai'
-import { byteplusText } from '@tanstack/ai-byteplus'
-import { z } from 'zod'
-
-const RecipeSchema = z.object({
- name: z.string().meta({ description: 'Name of the dish' }),
- minutes: z.number().meta({ description: 'Total cooking time in minutes' }),
- ingredients: z.array(z.string()),
-})
-
-const recipe = await chat({
- adapter: byteplusText('dola-seed-2-1-turbo-260628'),
- messages: [{ role: 'user', content: 'Give me a recipe for carbonara' }],
- outputSchema: RecipeSchema,
-})
-
-console.log(recipe.name, recipe.minutes)
-```
-
-On the other eight models the adapter **fails loudly** rather than degrading: `chat({ outputSchema })` throws, and the streaming form emits a `RUN_ERROR` naming the models that do work. There is no JSON-mode fallback to fall back *to* — Ark rejects `response_format: { type: 'json_object' }` on every one of them too.
-
-Two traps worth knowing, both found by probing the live API rather than reading the tables:
-
-- **The published capability tables are wrong in both directions.** `seed-2-0-lite-260428` — the obvious default model — is one of the eight that reject a JSON schema. Reach for `seed-2-0-lite-260228` or `dola-seed-2-1-turbo-260628` when you need typed output.
-- **`glm-4-7-251222` accepts a schema and then ignores it**, answering in prose with a `200`. It is deliberately excluded from the supported list, so the adapter rejects it up front instead of handing you unparseable output.
-
-`BYTEPLUS_STRUCTURED_OUTPUT_CHAT_MODELS` is exported if you want to gate a model picker on it.
+You now have a streaming chat on BytePlus. Seed models think before they answer, and `useChat` shows that thinking as separate thinking parts. [Tools](../tools/tools) work the same as with every adapter. BytePlus has no provider tools of its own.
## Video generation (Seedance)
> Video generation is an [experimental feature](../media/video-generation).
-Seedance is an asynchronous task API: `generateVideo()` opens the job and returns a `jobId`, and the video URL arrives with the terminal status.
+Seedance runs as a job. `generateVideo()` starts the job and returns a `jobId`. Poll the job until it has a video URL:
```typescript
-import { generateVideo, getVideoJobStatus } from '@tanstack/ai'
-import { byteplusVideo } from '@tanstack/ai-byteplus'
+import { generateVideo, getVideoJobStatus } from "@tanstack/ai";
+import { byteplusVideo } from "@tanstack/ai-byteplus";
-const adapter = byteplusVideo('dreamina-seedance-2-0-260128')
+const adapter = byteplusVideo("dreamina-seedance-2-5-260628");
const { jobId } = await generateVideo({
adapter,
- prompt: 'a guitar being played in a store',
- size: '16:9_720p',
+ prompt: "a guitar being played in a store",
+ threadId: "thread-1",
+ size: "16:9_720p",
duration: 5,
-})
-
-let status = await getVideoJobStatus({ adapter, jobId })
-while (status.status === 'pending' || status.status === 'processing') {
- await new Promise((resolve) => setTimeout(resolve, 5000))
- status = await getVideoJobStatus({ adapter, jobId })
-}
-
-console.log(status.status === 'completed' ? status.url : status.error)
-```
-
-**Generated video URLs expire 24 hours after the task completes** (the task record itself is kept for seven days), so download anything you intend to keep — see [Keeping generated files](../persistence/keep-generated-files).
-
-`size` is a `ratio` or `ratio_resolution` template (`'16:9'`, `'16:9_720p'`). Ratios are `16:9`, `9:16`, `4:3`, `3:4`, `1:1`, `21:9` and `adaptive`.
-
-### Driving the whole lifecycle from the client
-
-**Server** — hand the polling to the core with `stream: true` and pipe the chunks out:
-
-```typescript
-import { generateVideo, toServerSentEventsResponse } from '@tanstack/ai'
-import { byteplusVideo } from '@tanstack/ai-byteplus'
+});
-export async function POST(request: Request) {
- const { prompt } = await request.json()
-
- const stream = generateVideo({
- adapter: byteplusVideo('dreamina-seedance-2-0-260128'),
- prompt,
- size: '16:9_720p',
- duration: 5,
- stream: true,
- pollingInterval: 5000,
- })
-
- return toServerSentEventsResponse(stream)
+let status = await getVideoJobStatus({ adapter, jobId });
+while (status.status === "pending" || status.status === "processing") {
+ await new Promise((resolve) => setTimeout(resolve, 5000));
+ status = await getVideoJobStatus({ adapter, jobId });
}
-```
-
-**Client** — `useGenerateVideo` tracks the job for you:
-
-```tsx
-import { fetchServerSentEvents, useGenerateVideo } from '@tanstack/ai-react'
-export function SeedanceGenerator() {
- const { generate, result, videoStatus, isLoading, error } = useGenerateVideo({
- connection: fetchServerSentEvents('/api/generate/video'),
- })
-
- return (
-
-
-
- {isLoading &&
- )
-}
+console.log(status.status === "completed" ? status.url : status.error);
```
-### Per-model options
-
-Seedance options are model-specific, and **Ark rejects an inapplicable field with a `400` rather than ignoring it** — "the specified parameter `draft` is not supported for model seedance-1-0-pro in t2v, must be empty". The applicable set:
+- The video URL expires 24 hours after the job completes. If you want to keep the video, download it. See [Keeping generated files](../persistence/keep-generated-files).
+- `size` is a ratio (`"16:9"`) or a ratio with a resolution (`"16:9_720p"`). The ratios are `16:9`, `9:16`, `4:3`, `3:4`, `1:1`, `21:9`, and `adaptive`.
+- The Seedance 2 models can need activation in the Ark Console first. Until then, a call fails with `404 ModelNotOpen`.
-| Option | Models that accept it |
-| --- | --- |
-| `service_tier` (`'default'` \| `'flex'`) | Seedance 1.x only — Seedance 2.5 and the 2.0 family reject it |
-| `camera_fixed` | Seedance 1.x only |
-| `frames` (fractional-second output, `25 + 4n` in `[29, 289]`) | `seedance-1-0-pro-250528`, `seedance-1-0-pro-fast-251015` |
-| `draft` (cheap low-fidelity preview) | `seedance-1-5-pro-251215` only |
-| `priority` (`0`–`9`) | Seedance 2.5 and the `dreamina-seedance-2-0-*` family |
-| `duration: -1` (model picks the length) | Seedance 2.5, 2.0 and `seedance-1-5-pro-251215` |
-| `output_format` (`'mp4'` \| `'mov'`) | Seedance 2.5 (default `mp4`) |
-| `seed`, `watermark`, `generate_audio`, `return_last_frame`, `callback_url` | every model |
-
-`watermark` defaults to `false` for video (the opposite of Seedream images). `generate_audio` is accepted everywhere but only Seedance 2.5, 2.0 and 1.5-pro actually produce an audio track.
-
-Resolutions are per model too: there is **no 2K tier on any Seedance model**, `4k` exists only on `dreamina-seedance-2-0-260128`, Seedance 2.5 accepts **480p/720p/1080p**, and `seedance-1-0-pro-fast-251015` does accept `1080p` despite older prose listing it as 480p/720p only. An unsupported combination is caught locally with a clear error before the request goes out.
-
-Reference media follows the shared [role hints](../media/video-generation#role-hints) — `start_frame`, `end_frame` and `reference`. Seedance 2.5 and the 2.0 family take full multimodal references (reference images, video and audio); Seedance 2.5 additionally accepts **audio-only** reference input and up to 30 reference images / 10 videos / 10 audio clips (2.0 is 9 / 3 / 3). The 1.x models take start/end frames only, and `seedance-1-0-pro-fast-251015` takes a start frame only.
-
-### Seedance 2.5
-
-Seedance 2.5 (`dreamina-seedance-2-5-260628`) is a first-class model in this package. It is the current multimodal flagship for longer clips (up to **30 seconds**), audio-only reference input, and `output_format: 'mov'`. Like the 2.0 series it may still require model activation / a resource pack in the Ark Console before the account can call it (`404 ModelNotOpen` until then).
-
-```typescript
-import { generateVideo } from '@tanstack/ai'
-import { byteplusVideo } from '@tanstack/ai-byteplus'
-
-const { jobId } = await generateVideo({
- adapter: byteplusVideo('dreamina-seedance-2-5-260628'),
- prompt: 'a guitar being played in a store',
- size: '16:9_1080p',
- duration: 10,
- modelOptions: {
- generate_audio: true,
- priority: 5,
- output_format: 'mp4',
- },
-})
-```
-
-| Capability | Seedance 2.5 |
-| --- | --- |
-| Duration | 4–30s, or `-1` (model chooses; required for video-editing tasks) |
-| Resolution | `480p`, `720p`, `1080p` (default `720p`) — no 4k |
-| Reference media | images 1–30, videos 0–10, audio 0–10; **audio-only allowed** |
-| First + last frame | yes |
-| `priority` / `generate_audio` | yes |
-| `service_tier` / `camera_fixed` / `frames` / `draft` | no |
-
-`byteplusVideo()` still accepts any string for ids BytePlus ships between package releases. An unknown id relaxes local guards so Ark validates the request; known ids (including 2.5) keep their documented narrowing.
-
-### Seedance here vs. Seedance via fal
-
-Seedance is also reachable through [`@tanstack/ai-fal`](./fal), which proxies it alongside hundreds of other hosted models. This adapter talks to BytePlus directly, which means BytePlus billing and rate limits, model ids in BytePlus's own naming, and the first-class Seedance request fields above (`camera_fixed`, `draft`, `priority`, reference-media roles) rather than fal's normalized subset. Pick whichever matches the account you already have; there is no reason to install both just for Seedance.
+To run the job from a React component, and for the options of each model, see [BytePlus: Advanced](./byteplus-advanced#run-a-seedance-job-from-the-client).
## Image generation (Seedream)
```typescript
-import { generateImage } from '@tanstack/ai'
-import { byteplusImage } from '@tanstack/ai-byteplus'
+import { generateImage } from "@tanstack/ai";
+import { byteplusImage } from "@tanstack/ai-byteplus";
const result = await generateImage({
- adapter: byteplusImage('dola-seedream-5-0-pro-260628'),
- prompt: 'a guitar being played in a store',
- size: '2K',
+ adapter: byteplusImage("dola-seedream-5-0-pro-260628"),
+ prompt: "a guitar being played in a store",
+ threadId: "thread-1",
+ size: "2K",
modelOptions: { watermark: false },
-})
-
-console.log(result.images[0]?.url)
-```
+});
-`size` takes either a token (`1K`, `2K`, `4K`) or explicit pixels (`2048x2048`) — never a mix of the two. Pass image parts in the `prompt` array to edit or condition on existing images (up to 14 references, 10 on `dola-seedream-5-0-pro-260628`):
-
-```typescript
-import { generateImage } from '@tanstack/ai'
-import { byteplusImage } from '@tanstack/ai-byteplus'
-
-const result = await generateImage({
- adapter: byteplusImage('seedream-5-0-260128'),
- prompt: [
- { type: 'text', content: 'Put this guitar on a concert stage' },
- {
- type: 'image',
- source: { type: 'url', value: 'https://example.com/guitar.png' },
- },
- ],
-})
-
-console.log(result.images.length)
+console.log(result.images[0]?.url);
```
-Two behaviours surprise people:
-
-- **`watermark` defaults to `true`.** BytePlus stamps "AI generated" into the bottom-right corner unless you pass `watermark: false`. The adapter never sets the field implicitly, so the provider default applies.
-- **`numberOfImages` is an upper bound, not a count.** Seedream has no `n` parameter; asking for more than one image maps onto its group-image mode (`sequential_image_generation: 'auto'` with `max_images`), where the model decides how many images the prompt actually warrants. A request for four can come back with two, and the adapter logs a warning when it does.
+- `watermark` is `true` by default. BytePlus then writes "AI generated" in the bottom-right corner. Set `watermark: false` to stop this.
+- `size` is a token (`1K`, `2K`, `4K`) or pixels (`2048x2048`). Do not mix the two.
+- `numberOfImages` is a maximum, not an exact count. The model decides how many images to return.
+- Result URLs expire after 24 hours. To get the bytes in the response, set `response_format: "b64_json"` in `modelOptions`.
-Result URLs expire after 24 hours. Pass `response_format: 'b64_json'` in `modelOptions` to get the bytes inline instead.
+To edit an existing image, see [Seedream reference images](./byteplus-advanced#seedream-reference-images).
## Text-to-speech (Seed Speech)
-Seed Speech uses `BYTEPLUS_VOICE_API_KEY`, not the Ark key.
+This adapter uses `BYTEPLUS_VOICE_API_KEY`:
```typescript
-import { generateSpeech } from '@tanstack/ai'
-import { byteplusSpeech } from '@tanstack/ai-byteplus'
+import { generateSpeech } from "@tanstack/ai";
+import { byteplusSpeech } from "@tanstack/ai-byteplus";
const result = await generateSpeech({
- adapter: byteplusSpeech('seed-audio-1.0'),
- text: 'welcome to the guitar store',
- voice: 'en_female_stokie_uranus_bigtts',
- format: 'mp3',
-})
-
-console.log(result.contentType, result.audio.length)
-```
-
-Seed Speech has **no top-level `speaker` field** — the voice travels inside a
-`references` entry, so the adapter turns `voice` into
-`references: [{ speaker: 'en_female_stokie_uranus_bigtts' }]` for you. The
-consequence: `modelOptions.references` **replaces** that entry rather than
-merging with it, so a request that passes `references` for voice cloning
-silently drops `voice`. Include a `speaker` member yourself if you still want a
-stock voice alongside your clips.
-
-The suffix on a voice id tells you which generation you are asking for:
-
-- `_uranus_bigtts` — TTS 2.0 voices, the current generation.
-- `_mars_bigtts` / `_moon_bigtts` — TTS 1.0 voices.
-- `*_emo_v2_*` — TTS 1.0 voices that additionally accept emotion tags.
-
-The full roster lives in the [BytePlus voice list](https://docs.byteplus.com/en/docs/byteplusvoice/voicelist) and changes far more often than this package ships, so any string is accepted.
-
-`speaker` also takes a voice you cloned through Voice Replication, not only a stock id. That field is the join between replication and synthesis.
-
-Output formats are `wav`, `mp3`, `pcm` and `ogg_opus`. Reach for `modelOptions` for `ogg_opus`, for an explicit `sample_rate`, for `references` (voice cloning, up to three 30-second audio clips, addressed from the text as `@Audio1` to `@Audio3`), for `watermark`, or for word-level timings:
-
-```typescript
-import { generateSpeech } from '@tanstack/ai'
-import { byteplusSpeech, type BytePlusTTSResult } from '@tanstack/ai-byteplus'
-
-const result: BytePlusTTSResult = await generateSpeech({
- adapter: byteplusSpeech('seed-audio-1.0'),
- text: 'welcome to the guitar store',
- modelOptions: {
- format: 'ogg_opus',
- sample_rate: 48000,
- speech_rate: 20,
- enable_subtitle: true,
- },
-})
-
-for (const sentence of result.subtitle?.sentences ?? []) {
- console.log(sentence.text, sentence.start_time)
-}
+ adapter: byteplusSpeech("seed-audio-1.0"),
+ text: "welcome to the guitar store",
+ threadId: "thread-1",
+ voice: "en_female_stokie_uranus_bigtts",
+ format: "mp3",
+});
+
+console.log(result.contentType, result.audio.length);
```
-Prefer the cross-provider `timestamps: true` over `modelOptions.enable_subtitle`
-unless you want the raw BytePlus block. It sets the same flag and maps the
-response onto `result.alignment` (per word) and `result.segments` (per
-sentence), already converted to seconds:
+- Formats: `wav`, `mp3`, `pcm`, and `ogg_opus`.
+- One request makes 120 seconds of audio at most.
+- `timestamps: true` gives you word timings, and `turns` makes a dialogue with up to three voices.
-```typescript
-import { generateSpeech } from '@tanstack/ai'
-import { byteplusSpeech } from '@tanstack/ai-byteplus'
-
-const result = await generateSpeech({
- adapter: byteplusSpeech('seed-audio-1.0'),
- text: 'welcome to the guitar store',
- timestamps: true,
-})
+The voice ids, voice cloning, timings, dialogue, and watermarks are in [Seed Speech](./byteplus-advanced#seed-speech). The client half is the standard [`useGenerateSpeech` hook](../media/text-to-speech#quick-setup).
-console.log(result.alignment?.unit) // 'word'
-console.log(result.alignment?.endSeconds.at(-1)) // where speech stops
-```
+## Transcription (Seed Speech ASR)
-Seed Audio 1.0 also synthesizes multi-role dialogue in one pass. Pass `turns`
-and the adapter builds the `references` array and the role-structured prompt
-for you, up to **three distinct voices** per request:
+This adapter also uses `BYTEPLUS_VOICE_API_KEY`. The call is synchronous, so you get the transcript back with no polling:
```typescript
-import { generateSpeech } from '@tanstack/ai'
-import { byteplusSpeech } from '@tanstack/ai-byteplus'
+import { generateTranscription } from "@tanstack/ai";
+import { byteplusTranscription } from "@tanstack/ai-byteplus";
-// Replace SECOND_VOICE with another voice id from the BytePlus voice list.
-const SECOND_VOICE = 'your-second-voice-id'
-
-const result = await generateSpeech({
- adapter: byteplusSpeech('seed-audio-1.0'),
- turns: [
- { text: 'Do you sell picks?', voice: 'en_female_stokie_uranus_bigtts' },
- { text: 'By the till.', voice: SECOND_VOICE },
- ],
- timestamps: true,
-})
+const result = await generateTranscription({
+ adapter: byteplusTranscription("seed-asr"),
+ audio: "https://example.com/recording.mp3",
+ modelOptions: { enable_punc: true, enable_speaker_info: true },
+ threadId: "thread-1",
+});
+console.log(result.text);
for (const segment of result.segments ?? []) {
- console.log(segment.startSeconds, segment.text)
+ console.log(segment.speaker, segment.text);
}
```
-### Watermarking speech
+`audio` takes a `File`, base64, a data URL, or a public URL. The limit is two hours and 100 MB. For the other options, see [Transcription options](./byteplus-advanced#transcription-options).
-Seed Audio takes an object here, not a flag. Two markers, both off by default:
+## Models and options
-- `aigc_watermark`: appends an audible rhythm marker to the end of the clip.
-- `aigc_metadata`: writes provenance into the audio header. It writes nothing unless `enable` is `true`.
+### Model options
+
+Ark speaks the OpenAI chat format, so the sampling options keep their OpenAI names. They go in `modelOptions`:
```typescript
-import { generateSpeech } from '@tanstack/ai'
-import { byteplusSpeech } from '@tanstack/ai-byteplus'
+import { chat } from "@tanstack/ai";
+import { byteplusText } from "@tanstack/ai-byteplus";
-const result = await generateSpeech({
- adapter: byteplusSpeech('seed-audio-1.0'),
- text: 'welcome to the guitar store',
+const stream = chat({
+ adapter: byteplusText("dola-seed-2-1-turbo-260628"),
+ messages: [{ role: "user", content: "Hello!" }],
+ threadId: "thread-1",
modelOptions: {
- watermark: {
- aigc_watermark: true,
- aigc_metadata: { enable: true, content_producer: 'guitar-store' },
- },
+ temperature: 0.7,
+ top_p: 0.9,
+ max_tokens: 2048,
},
-})
+});
```
-`watermark: true` is shorthand for `{ aigc_watermark: true }`, so the audible marker is what a boolean gives you. Images and video keep their own boolean `watermark`. Only speech takes the object.
+- Send `max_tokens` or `max_completion_tokens`, not both. Ark returns a `400` when you send both. The types cannot catch this.
+- `repetition_penalty` and `service_tier` are Ark options. `service_tier: "flex"` sends the request to a cheaper batch queue with no latency guarantee.
-Three things to plan around:
-
-- **Synthesis is capped at 120 seconds of output.** The cap applies to `originalDuration` (the length before `speech_rate` is applied), which is also what BytePlus bills on.
-- **Subtitle timings are milliseconds** on the raw `subtitle` block, while `duration` and `originalDuration` are seconds. `alignment` and `segments` are always seconds.
-- The `url` on the result expires roughly two hours after generation — persist the base64 `audio`, not the link.
-
-The client half is the standard [`useGenerateSpeech` hook](../media/text-to-speech#streaming-mode-server-route--client-hook); nothing about it is BytePlus-specific.
+Need a header on each request, or a log of each model call? The chat adapter supports [`wrapFetch`](../advanced/middleware#change-the-http-requests-of-a-call).
-## Transcription (Seed Speech ASR)
+### Reasoning
-Also on the voice key. The endpoint is synchronous — audio in, transcript out, no polling:
+Seed models think by default. Set how hard they think with `reasoning` on `chat()`:
```typescript
-import { generateTranscription } from '@tanstack/ai'
-import { byteplusTranscription } from '@tanstack/ai-byteplus'
-import { audioFile } from './audio'
-
-const result = await generateTranscription({
- adapter: byteplusTranscription('seed-asr'),
- audio: audioFile,
- modelOptions: { enable_punc: true, enable_speaker_info: true },
-})
-
-console.log(result.text)
-for (const segment of result.segments ?? []) {
- console.log(segment.speaker, segment.text)
-}
+import { chat } from "@tanstack/ai";
+import { byteplusText } from "@tanstack/ai-byteplus";
+
+const stream = chat({
+ adapter: byteplusText("glm-5-2-260617"),
+ messages: [{ role: "user", content: "Hello!" }],
+ threadId: "thread-1",
+ reasoning: "off",
+});
```
-Audio can be a `File`, base64, a data URL or a public URL, up to two hours and 100 MB. `enable_itn` renders spoken numbers and dates as digits, `enable_punc` inserts punctuation, `enable_ddc` strips fillers, and `enable_speaker_info` attaches speaker labels that surface as `segment.speaker`.
-
-## Supported models
-
-- **Chat** — `dola-seed-2-1-turbo-260628`, `seed-2-0-lite-260428`, `seed-2-0-mini-260428`, `seed-2-0-pro-260328`, `seed-2-0-lite-260228`, `seed-2-0-mini-260215`, `seed-2-0-code-preview-260328`, `seed-1-8-251228`, `seed-1-6-250915`, `seed-1-6-250615`, `seed-1-6-flash-250715`, `seed-1-6-flash-250615`, `glm-5-2-260617`, `glm-4-7-251222`, `deepseek-v4-pro-260425`, `deepseek-v4-flash-260425`, `deepseek-v3-2-251201`, `gpt-oss-120b-250805`.
-- **Video** — `dreamina-seedance-2-5-260628`, `dreamina-seedance-2-0-260128`, `dreamina-seedance-2-0-fast-260128`, `dreamina-seedance-2-0-mini-260615`, `seedance-1-5-pro-251215`, `seedance-1-0-pro-250528`, `seedance-1-0-pro-fast-251015`.
-- **Image** — `dola-seedream-5-0-pro-260628`, `seedream-5-0-260128`, `seedream-5-0-lite-260128`, `seedream-4-5-251128`, `seedream-4-0-250828`.
-- **Speech** — `seed-audio-1.0` (TTS) and `seed-asr` (transcription).
-
-BytePlus retires model ids aggressively and its published lists include ids that no longer resolve, so this package ships only dated ids that answered a live request. `BYTEPLUS_CHAT_MODELS`, `BYTEPLUS_VIDEO_MODELS`, `BYTEPLUS_IMAGE_MODELS`, `BYTEPLUS_TTS_MODELS` and `BYTEPLUS_TRANSCRIPTION_MODELS` are the authoritative lists and are exported for building model pickers.
-
-## API Reference
-
-Every factory has an environment-variable form and an explicit-key form.
-
-### `byteplusText(model, config?)` / `createBytePlusText(model, apiKey, config?)`
-
-Chat adapter for the Seed, GLM, DeepSeek and gpt-oss models. Reads `ARK_API_KEY` (or `BYTEPLUS_API_KEY`). `config.baseURL` overrides the region endpoint.
-
-`adapter.inputModalities` lists the input kinds of the selected model. It is `undefined` for a model that this package does not know. See [Check what a model accepts](../advanced/extend-adapter#check-what-a-model-accepts).
+The levels of each model:
-### `byteplusVideo(model, config?)` / `createBytePlusVideo(model, apiKey, config?)`
+- `dola-seed-2-1-turbo-260628` and the `seed-*` models: `minimal` to `high`. They cannot stop thinking.
+- `glm-5-2-260617`: `off` to `high`.
+- `glm-4-7-251222`, `deepseek-v3-2-251201`: `off`, `high`.
+- `deepseek-v4-pro-260425`, `deepseek-v4-flash-260425`: `off`, `high`, `max`.
+- `gpt-oss-120b-250805`: `low`, `medium`, `high`.
-Seedance video adapter (experimental). Reads `ARK_API_KEY`.
+What goes over the wire, and how the adapter keeps the `encrypted_content` of the reasoning, is in [Reasoning and `encrypted_content`](./byteplus-advanced#reasoning-and-encrypted_content).
-### `byteplusImage(model, config?)` / `createBytePlusImage(model, apiKey, config?)`
+### Structured output
-Seedream image adapter. Reads `ARK_API_KEY`.
+Ten of the 18 chat models support `chat({ outputSchema })`. Use `dola-seed-2-1-turbo-260628` or `seed-2-0-lite-260228`. On the other models, the call fails with an error that names the models that work. See [Structured output](./byteplus-advanced#structured-output).
-### `byteplusSpeech(model, config?)` / `createBytePlusSpeech(model, apiKey, config?)`
+### Models
-Seed Speech TTS adapter. Reads **`BYTEPLUS_VOICE_API_KEY`**.
+Newest first. The package exports the full lists: `BYTEPLUS_CHAT_MODELS`, `BYTEPLUS_VIDEO_MODELS`, `BYTEPLUS_IMAGE_MODELS`, `BYTEPLUS_TTS_MODELS`, and `BYTEPLUS_TRANSCRIPTION_MODELS`.
-### `byteplusTranscription(model, config?)` / `createBytePlusTranscription(model, apiKey, config?)`
+| Activity | Factory | Models |
+| --- | --- | --- |
+| Chat | `byteplusText` | `dola-seed-2-1-turbo-260628`, `seed-2-0-lite-260428`, `seed-2-0-mini-260428`, `seed-2-0-pro-260328`, `seed-2-0-lite-260228`, `seed-2-0-mini-260215`, `seed-2-0-code-preview-260328`, `seed-1-8-251228`, `seed-1-6-250915`, `seed-1-6-250615`, `seed-1-6-flash-250715`, `seed-1-6-flash-250615`, `glm-5-2-260617`, `glm-4-7-251222`, `deepseek-v4-pro-260425`, `deepseek-v4-flash-260425`, `deepseek-v3-2-251201`, `gpt-oss-120b-250805` |
+| Video | `byteplusVideo` | `dreamina-seedance-2-5-260628`, `dreamina-seedance-2-0-260128`, `dreamina-seedance-2-0-fast-260128`, `dreamina-seedance-2-0-mini-260615`, `seedance-1-5-pro-251215`, `seedance-1-0-pro-250528`, `seedance-1-0-pro-fast-251015` |
+| Images | `byteplusImage` | `dola-seedream-5-0-pro-260628`, `seedream-5-0-260128`, `seedream-5-0-lite-260128`, `seedream-4-5-251128`, `seedream-4-0-250828` |
+| Speech | `byteplusSpeech` | `seed-audio-1.0` |
+| Transcription | `byteplusTranscription` | `seed-asr` |
-Seed Speech transcription adapter. Reads **`BYTEPLUS_VOICE_API_KEY`**.
+BytePlus retires model ids often, and its published lists contain ids that do not work. This package lists only dated ids that answered a live request.
-## Provider Tools
+`adapter.inputModalities` lists the input kinds of the selected model. See [Check what a model accepts](../advanced/extend-adapter#check-what-a-model-accepts).
-BytePlus does not expose provider-specific tool factories. Define your own with `toolDefinition()` from `@tanstack/ai` — Ark's tool calling is the standard OpenAI shape and works with the usual [tools flow](../tools/tools.md).
+## Advanced
-## Next Steps
+These topics are on [BytePlus: Advanced](./byteplus-advanced):
-- [Video Generation](../media/video-generation) — the full jobs/polling flow and the `useGenerateVideo` hook
-- [Image Generation](../media/image-generation) — image-conditioned generation and role hints
-- [Structured Outputs](../structured-outputs/one-shot) — schemas, validation and streaming
-- [Other Adapters](./openai) — explore other providers
+- Regions, and the two BYOK providers.
+- Reasoning, `encrypted_content`, and structured output.
+- Seedance from the client, the options of each model, and Seedance 2.5.
+- Seedream reference images, Seed Speech details, and transcription options.
+- The API reference.
diff --git a/docs/adapters/claude-code.md b/docs/adapters/claude-code.md
index ce41f96c36..35d0002beb 100644
--- a/docs/adapters/claude-code.md
+++ b/docs/adapters/claude-code.md
@@ -2,7 +2,7 @@
title: Claude Code
id: claude-code-adapter
order: 11
-description: "Use Claude Code as a chat backend in TanStack AI — agent harness with local tool execution, stateful coding sessions, and tool bridging via @tanstack/ai-claude-code."
+description: "Run Anthropic's Claude Code agent inside a sandbox from TanStack AI. It reads, edits, and runs code in a real repo and streams each step to your UI."
keywords:
- tanstack ai
- claude code
@@ -14,131 +14,185 @@ keywords:
- adapter
---
-The Claude Code adapter runs [Claude Code](https://docs.anthropic.com/en/docs/claude-code) (via the `@anthropic-ai/claude-agent-sdk`) as a chat backend. Unlike HTTP provider adapters, this is a **harness adapter**: Claude Code runs its own agent loop and executes its own tools — bash, file reads and edits, glob/grep search, web search — locally on your server. Each `chat()` call runs one full harness turn; the harness's tool activity streams back as already-resolved tool-call events your UI can render.
+The Claude Code adapter runs the `claude` CLI in a sandbox and streams each step to `chat()`. The agent reads files, edits them, and runs commands in a repo.
-> **Server-only.** The harness spawns the Claude Code runtime as a subprocess, so this adapter only works in a Node.js server environment — never in the browser. Treat it like giving Claude a shell on the machine it runs on, and configure permissions accordingly.
+You need:
-## Installation
+- A sandbox provider. The samples use Docker, so Docker must run on your machine.
+- An Anthropic API key, or a `claude login` on the machine.
+- The `claude` CLI inside the sandbox. A setup step in the sandbox installs it.
+
+Do you only want a Claude chat, with no repo and no shell? Use the [Anthropic adapter](./anthropic).
+
+## 1. Install
-react: @tanstack/ai-claude-code
-vue: @tanstack/ai-claude-code
-solid: @tanstack/ai-claude-code
-svelte: @tanstack/ai-claude-code
-preact: @tanstack/ai-claude-code
-angular: @tanstack/ai-claude-code
-vanilla: @tanstack/ai-claude-code
-octane: @tanstack/ai-claude-code
+react: @tanstack/ai-claude-code @tanstack/ai-sandbox @tanstack/ai-sandbox-docker
+vue: @tanstack/ai-claude-code @tanstack/ai-sandbox @tanstack/ai-sandbox-docker
+solid: @tanstack/ai-claude-code @tanstack/ai-sandbox @tanstack/ai-sandbox-docker
+svelte: @tanstack/ai-claude-code @tanstack/ai-sandbox @tanstack/ai-sandbox-docker
+preact: @tanstack/ai-claude-code @tanstack/ai-sandbox @tanstack/ai-sandbox-docker
+angular: @tanstack/ai-claude-code @tanstack/ai-sandbox @tanstack/ai-sandbox-docker
+vanilla: @tanstack/ai-claude-code @tanstack/ai-sandbox @tanstack/ai-sandbox-docker
+octane: @tanstack/ai-claude-code @tanstack/ai-sandbox @tanstack/ai-sandbox-docker
-A runnable demo lives at [`examples/sandbox-cloudflare`](https://github.com/TanStack/ai/tree/main/examples/sandbox-cloudflare) — pick Claude Code, Codex, or Grok Build in the UI, with session resume, the harness tool timeline, and tool bridging, wired into a TanStack Start app on Workers. For the same wiring on plain Node with durable, refresh-surviving runs (this adapter on Docker), see [`examples/sandbox-web`](https://github.com/TanStack/ai/tree/main/examples/sandbox-web).
+## 2. Set the API key
-## Authentication
+Put the key in the environment of your server:
-Your laptop can already have `claude login`. A CI runner only has
-`ANTHROPIC_API_KEY`. The default `authMode` is `'api-key'`. Set `'host'`
-when you want `claude login`. See [Harness Auth](../sandbox/auth).
-
-```ts
-import { claudeCodeText } from "@tanstack/ai-claude-code"
-
-claudeCodeText("claude-opus-4-8")
-claudeCodeText("claude-opus-4-8", { authMode: "host" })
+```bash
+ANTHROPIC_API_KEY=sk-ant-...
```
-- `'api-key'` (default): inject `ANTHROPIC_API_KEY` (or pass `apiKey`).
-- `'host'`: use `claude login`. Do not inject `ANTHROPIC_API_KEY`.
+The sandbox in step 3 gives this key to the `claude` process as a workspace secret.
-## Basic Usage
+Did the machine already run `claude login`? Then you can use that login and no key. See [Harness Auth](../sandbox/auth).
-```typescript
-import { chat } from "@tanstack/ai";
-import { claudeCodeText } from "@tanstack/ai-claude-code";
+## 3. Run the agent from the server
-const stream = chat({
- adapter: claudeCodeText("claude-opus-4-8", {
- cwd: "/path/to/project",
- permissionMode: "acceptEdits",
+Define the sandbox once. It clones your repo into a Docker container and installs the `claude` CLI:
+
+```ts
+// sandbox.ts
+import {
+ createSecrets,
+ defineSandbox,
+ defineWorkspace,
+ githubRepo,
+} from "@tanstack/ai-sandbox";
+import { dockerSandbox } from "@tanstack/ai-sandbox-docker";
+
+export const sandbox = defineSandbox({
+ id: "claude-code-agent",
+ provider: dockerSandbox({ image: "node:22" }),
+ workspace: defineWorkspace({
+ source: githubRepo({ repo: "owner/app" }),
+ setup: [
+ "npm install -g @anthropic-ai/claude-code --include=optional && claude --version",
+ ],
+ secrets: createSecrets({
+ ANTHROPIC_API_KEY: process.env.ANTHROPIC_API_KEY ?? "",
+ }),
}),
- messages: [{ role: "user", content: "Fix the failing test in utils.test.ts" }],
});
```
-`RUN_FINISHED.usage.promptTokens` counts the full input, cached tokens included. See [Token usage](../chat/stream-events#token-usage).
+Then pass the adapter and the sandbox to `chat()` in your route:
-## Configuration
+```ts
+// api/agent.ts
+import {
+ chat,
+ chatParamsFromRequest,
+ toServerSentEventsResponse,
+} from "@tanstack/ai";
+import { claudeCodeText } from "@tanstack/ai-claude-code";
+import { withSandbox } from "@tanstack/ai-sandbox";
+import { sandbox } from "./sandbox";
-| Option | Description |
-| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `cwd` | Working directory for the harness session. Defaults to `process.cwd()`. |
-| `permissionMode` | Claude Code permission mode (`'default'`, `'acceptEdits'`, `'bypassPermissions'`, `'plan'`, `'dontAsk'`, `'auto'`). See the permissions note below. |
-| `allowedTools` | Built-in tools the harness may use without prompting (e.g. `['Read', 'Grep', 'Bash(npm test:*)']`). |
-| `disallowedTools` | Built-in tools removed from the harness entirely. |
-| `maxTurns` | Maximum harness-internal turns per run. |
-| `systemPromptMode` | `'append'` (default) keeps Claude Code's preset system prompt and appends your `systemPrompts`; `'replace'` sends yours as the entire prompt. |
-| `mcpServers` | Extra MCP servers passed through to the harness untouched. |
-| `authMode` | `'api-key'` (default) injects `ANTHROPIC_API_KEY`. `'host'` uses `claude login`. Also valid on `modelOptions`. See [Harness Auth](../sandbox/auth). |
-| `apiKey` | Anthropic API key for the harness subprocess. |
-| `env` | Extra environment variables for the harness subprocess. |
-| `pathToClaudeCodeExecutable` | Use a specific Claude Code executable instead of the SDK's bundled one. |
-| `streamPartials` | Emit true token-level deltas for text and tool-call arguments (default `true`). When `false`, both emit whole from the complete message. |
-| `canUseTool` | Custom permission handler; replaces the adapter's default handler. |
-| `settingSources` | Claude Code settings tiers to load. Default `['project']`: the workspace's `CLAUDE.md`, `.claude/skills`, and `.mcp.json` apply, and the host's `~/.claude` (plugins, hooks, skills) is not loaded. Pass `['user', 'project', 'local']` for CLI-equivalent behavior, or `[]` to load no settings. |
+export async function POST(request: Request) {
+ const { messages, threadId, runId } = await chatParamsFromRequest(request);
-**Permissions on headless servers.** Without an explicit `permissionMode` or `canUseTool`, the adapter installs a safe default handler: bridged TanStack tools always run, and any built-in tool call that would normally prompt a human is denied with guidance instead of hanging the request. To let the harness edit files or run commands, set `permissionMode: 'acceptEdits'` / `'bypassPermissions'`, or enumerate `allowedTools`.
+ const stream = chat({
+ adapter: claudeCodeText("claude-opus-4-8"),
+ messages,
+ threadId,
+ runId,
+ middleware: [withSandbox(sandbox)],
+ });
-## Reasoning
+ return toServerSentEventsResponse(stream);
+}
+```
-Set `reasoning` on `chat()`. The adapter turns it into a Claude Code setting:
+Replace `owner/app` with your repo.
-```typescript
-import { chat } from "@tanstack/ai";
-import { claudeCodeText } from "@tanstack/ai-claude-code";
+## 4. Render the agent's work
-const stream = chat({
- adapter: claudeCodeText("claude-opus-4-8"),
- messages: [{ role: "user", content: "Refactor the auth module" }],
- reasoning: "xhigh",
-});
+The client is the same as for any chat. Render the text, and show each tool that the agent ran:
+
+```tsx
+import { useState } from "react";
+import { useChat, fetchServerSentEvents } from "@tanstack/ai-react";
+
+export function CodingAgent() {
+ const [input, setInput] = useState("");
+ const { messages, sendMessage } = useChat({
+ connection: fetchServerSentEvents("/api/agent"),
+ threadId: "support-chat",
+ });
+
+ return (
+
+ );
+}
```
-- `claude-opus-4-8`, `claude-opus-4-7`, `opus`: `--effort`, `low` to `max`.
-- `claude-opus-4-6`, `claude-sonnet-4-6`, `sonnet`: `--effort`, `low`, `medium`, `high`, `max`.
-- `claude-haiku-4-5`, `haiku`: `MAX_THINKING_TOKENS`, `off` to `high`. The level picks the budget, or you set it with `budgetTokens`. `off` sends `0`.
+Ask it "Fix the failing test in utils.test.ts". You see the agent's text, plus a line for each tool it used: `Read`, `Edit`, `Bash`, and more. The agent edited the files inside the container.
-## Stateful Sessions
+## How it works
-Claude Code sessions are stateful — the harness keeps the full working context (files read, commands run, conclusions reached) between turns. The adapter surfaces the session id of every run as a custom stream event named `claude-code.session-id`; thread it back via `modelOptions.sessionId` to resume the session. When resuming, only the latest user message is sent — the harness already holds the prior context.
+- `claudeCodeText` declares that it needs a sandbox. `withSandbox(sandbox)` gives it one. Without a sandbox, `chat()` fails at the call site.
+- Each `chat()` call runs one full agent turn. Claude Code runs its own loop and its own tools.
+- The tools of the agent stream back as `tool-call` parts with the results attached. Your code does not run them.
+- The agent runs with `permissionMode: 'bypassPermissions'` by default. The sandbox is the safety boundary. To limit what the agent can do, add a sandbox [Policy](../sandbox/policy).
+- When the run ends, the adapter sends a `file.changed` custom event with the `git diff` of the workspace.
-The `claude-code.session-id` event payload also includes `skills`, an array of skills loaded during SDK initialization (or an empty array when none are reported).
+To learn more about sandboxes, see [Sandboxes](../sandbox/overview).
-Server endpoint:
+## Resume a session
-```typescript
+Claude Code keeps its working context between turns: the files it read, the commands it ran, and what it found. The adapter sends the session id of each run as a custom event named `claude-code.session-id`. Send it back in `modelOptions.sessionId` to continue the session.
+
+On the server, read the id from the request:
+
+```ts
import {
chat,
chatParamsFromRequest,
toServerSentEventsResponse,
} from "@tanstack/ai";
import { claudeCodeText } from "@tanstack/ai-claude-code";
+import { withSandbox } from "@tanstack/ai-sandbox";
+import { sandbox } from "./sandbox";
export async function POST(request: Request) {
const params = await chatParamsFromRequest(request);
- // Extra fields the client puts in the connection `body` arrive here.
+ // The client sends the id in `forwardedProps`.
const sessionId =
typeof params.forwardedProps.sessionId === "string"
? params.forwardedProps.sessionId
: undefined;
const stream = chat({
- adapter: claudeCodeText("claude-opus-4-8", {
- cwd: "/path/to/project",
- permissionMode: "acceptEdits",
- }),
+ adapter: claudeCodeText("claude-opus-4-8"),
messages: params.messages,
+ threadId: params.threadId,
+ runId: params.runId,
+ middleware: [withSandbox(sandbox)],
modelOptions: { sessionId },
});
@@ -146,20 +200,19 @@ export async function POST(request: Request) {
}
```
-Client (React) — capture the session id from the custom event and send it back on subsequent requests:
+On the client, keep the id from the custom event and pass it to `forwardedProps`. `useChat` sends the latest value with each request:
-```typescript
+```tsx
import { useState } from "react";
-import { useChat } from "@tanstack/ai-react";
-import { fetchServerSentEvents } from "@tanstack/ai-client";
+import { useChat, fetchServerSentEvents } from "@tanstack/ai-react";
-function CodingAssistant() {
+export function useCodingAgent() {
const [sessionId, setSessionId] = useStateStatus: {videoStatus?.status ?? 'starting…'}
} - {error &&Error: {error.message}
} - {result && } -The agent is inspecting the repo.
- if (!final) return null - return{final.summary}
+ if (isLoading) returnThe agent is inspecting the repo.
; + if (!final) return null; + return{final.summary}
; } ``` -If you only need to extract JSON from a prompt and do not need a sandbox, use `@tanstack/ai-anthropic`. That path is faster. +- Claude Code runs one turn with its own tools. Tool activity and text stream as usual. +- The adapter passes the schema to `--json-schema` as inline JSON. The CLI does not take a file path. +- The object arrives as `structured-output.complete`. This also works when Claude sends it through its built-in `StructuredOutput` tool. + +Do you only need JSON from a prompt, with no repo? Use `@tanstack/ai-anthropic`. That is faster. For the full walkthrough, see [Harness Agents](../structured-outputs/harnesses). + +## Models and options + +### Models + +| Model | Reasoning levels | What `reasoning` sets | +| --- | --- | --- | +| `claude-opus-4-8`, `claude-opus-4-7`, `opus` | `low`, `medium`, `high`, `xhigh`, `max` | `--effort` | +| `claude-opus-4-6`, `claude-sonnet-4-6`, `sonnet` | `low`, `medium`, `high`, `max` | `--effort` | +| `claude-haiku-4-5`, `haiku` | `off`, `minimal`, `low`, `medium`, `high` | `MAX_THINKING_TOKENS` | + +The package exports this list as `CLAUDE_CODE_MODELS`. The adapter also takes any other Claude model id as a string. + +### Reasoning + +Set `reasoning` on `chat()`: + +```ts +import { chat } from "@tanstack/ai"; +import { claudeCodeText } from "@tanstack/ai-claude-code"; +import { withSandbox } from "@tanstack/ai-sandbox"; +import { sandbox } from "./sandbox"; + +const stream = chat({ + adapter: claudeCodeText("claude-opus-4-8"), + messages: [{ role: "user", content: "Refactor the auth module" }], + threadId: "thread-1", + reasoning: "xhigh", + middleware: [withSandbox(sandbox)], +}); +``` + +For `claude-haiku-4-5`, the level picks a token budget. You can also set the budget with `budgetTokens`. `off` sends `0`. See [Reasoning](../chat/reasoning). + +### Adapter options + +Pass these as the second argument of `claudeCodeText`: + +| Option | What it does | +| --- | --- | +| `cwd` | The working directory inside the sandbox. Default `/workspace`. | +| `permissionMode` | `'default'`, `'acceptEdits'`, `'bypassPermissions'`, or `'plan'`. Default `'bypassPermissions'`. | +| `allowedTools` | Built-in tools that the agent can use (`--allowedTools`), for example `['Read', 'Grep', 'Bash(npm test:*)']`. | +| `disallowedTools` | Built-in tools that the agent cannot use (`--disallowedTools`). | +| `addDirs` | More directories that the agent can access (`--add-dir`). | +| `maxTurns` | The maximum number of agent turns in one run. | +| `settingSources` | Which Claude Code settings to load. Default `['project']`. See [Settings sources](#settings-sources). | +| `systemPromptMode` | `'append'` (default) adds your `systemPrompts` to the Claude Code prompt. `'replace'` sends only yours. | +| `authMode` | `'api-key'` (default) gives `ANTHROPIC_API_KEY` to the process. `'host'` uses `claude login`. See [Harness Auth](../sandbox/auth). | +| `claudeExecutable` | The path or name of the `claude` executable in the sandbox. Default `claude`. | +| `streamPartials` | Stream text and tool arguments token by token. Default `true`. With `false`, each arrives whole. | +| `env` | More environment variables for the `claude` process. | +| `emitDiff` | Send the `file.changed` event with the `git diff` after the run. Default `true`. | + +### Per-call options + +Pass these in `modelOptions` on `chat()`. They override the adapter options for one call: + +- `sessionId`: resume a session. See [Resume a session](#resume-a-session). +- `cwd`, `maxTurns`, `permissionMode`, `allowedTools`, `disallowedTools`. +- `authMode`: `'host'` or `'api-key'`. + +## Advanced + +### Permissions and policy + +A headless run has no person to answer a permission prompt. So the default `permissionMode` is `'bypassPermissions'`, and the sandbox is the boundary. + +- To allow fewer actions, set `permissionMode`, `allowedTools`, or `disallowedTools`. +- To allow, ask, or deny each command and capability, add a sandbox [Policy](../sandbox/policy). The adapter maps the policy to Claude Code flags and answers the permission prompts of Claude Code from it. +- If the policy says `ask`, the adapter sends an approval-requested event and denies the action. The client approves, then runs the turn again. + +### Settings sources + +By default, the adapter loads only the project settings (`--setting-sources project`): + +- The workspace `CLAUDE.md`, `.claude/skills`, and `.mcp.json` apply to the run. +- A cloned repo's `.claude/settings.json` and hooks also apply. +- The host `~/.claude` (your personal plugins, hooks, and skills) does not load. This matters on the local-process provider. + +For the same behavior as the CLI, pass `settingSources: ['user', 'project', 'local']`. To load no settings, pass `[]`. + +The adapter does not pass `--bare`, because that flag ignores a host `claude login`. + +### How tool choice changes the CLI flags + +For `'none'` and a named tool, the adapter passes `--tools ""` and `--strict-mcp-config`. Then other MCP servers, for example from the workspace or a settings file, do not load. + +### Token usage + +`RUN_FINISHED.usage.promptTokens` counts the full input, cached tokens included. See [Token usage](../chat/stream-events#token-usage). -Full walkthrough, including the client: [Harness Agents](../structured-outputs/harnesses). +### Limitations -## Limitations +- **A sandbox is required.** Every call needs `withSandbox(...)`. +- **The agent owns the loop.** TanStack agent-loop strategies and per-iteration middleware do not apply inside a turn. Use `maxTurns` to limit the turn. +- **No sampling options.** Options such as `temperature` do not exist here. +- **Sessions stay in the sandbox.** Resume needs the same sandbox. +- **Cold starts.** Each call starts a CLI turn, so the first token arrives later than with an HTTP adapter. +- **No `wrapFetch`.** This adapter ignores [`wrapFetch`](../advanced/middleware#change-the-http-requests-of-a-call), because the `claude` process sends the model requests. -- **Server-only (Node).** The harness spawns a subprocess; Windows support is untested. -- **The harness owns the agent loop.** TanStack's agent-loop strategies and per-iteration middleware don't apply inside a harness turn; `maxTurns` is the equivalent control. -- **No sampling controls.** `temperature`-style options don't exist here. -- **Sessions are machine-local.** Resume requires hitting the same server instance. -- **Cold starts.** Each call spawns a harness turn; expect higher first-token latency than HTTP adapters. +### Example apps -This adapter ignores [`wrapFetch`](../advanced/middleware#change-the-http-requests-of-a-call), because the harness process sends the model requests. +- [`examples/sandbox-cloudflare`](https://github.com/TanStack/ai/tree/main/examples/sandbox-cloudflare): a TanStack Start app on Workers. You pick Claude Code, Codex, or Grok Build in the UI. It shows session resume, the tool timeline, and tool bridging. +- [`examples/sandbox-web`](https://github.com/TanStack/ai/tree/main/examples/sandbox-web): this adapter on Docker, with durable runs that continue after a page refresh. diff --git a/docs/adapters/cloudflare.md b/docs/adapters/cloudflare.md index 9abf504050..7f0fc20247 100644 --- a/docs/adapters/cloudflare.md +++ b/docs/adapters/cloudflare.md @@ -1,7 +1,7 @@ --- title: Cloudflare id: cloudflare-adapter -description: "Run Workers AI chat, embeddings, images, speech, transcription, and evaluate from a Cloudflare Worker or any server, and route other providers through AI Gateway, with TanStack AI." +description: "Run Workers AI chat, embeddings, images, speech, transcription, and evaluate from a Cloudflare Worker or any server, and send other providers through AI Gateway, with TanStack AI." keywords: - tanstack ai - cloudflare @@ -14,9 +14,13 @@ keywords: - jev --- -You have a Cloudflare Worker and want an AI route without managing API keys. Or you have a server elsewhere and want to call Workers AI models. Or you already use OpenAI or Anthropic and want AI Gateway caching and logs in front of them. This adapter does all three. +The Cloudflare adapter calls Workers AI, and it can put AI Gateway in front of another provider. -## Installation +- You have a Cloudflare Worker and want an AI route with no API keys to manage. +- You have a server somewhere else and want to call Workers AI models. +- You use OpenAI or Anthropic and want AI Gateway caching and logs in front of them. + +## 1. Install @@ -31,11 +35,9 @@ octane: @tanstack/ai-cloudflare -## Chat from a Worker - -Pass `env.AI` as the binding. No token, no account id. +## 2. Connect to Cloudflare -1. Add the binding to `wrangler.jsonc`: +Inside a Worker, use the `env.AI` binding. You need no token and no account id. Add the binding to `wrangler.jsonc`: ```jsonc { @@ -43,10 +45,25 @@ Pass `env.AI` as the binding. No token, no account id. } ``` -2. Stream a chat response from your Worker: +On any other server, the adapter calls the Cloudflare REST API. Put your account id and an API token with Workers AI access in the environment of your server: + +```bash +CLOUDFLARE_ACCOUNT_ID=your-account-id +CLOUDFLARE_API_TOKEN=your-api-token +``` + +Get the account id from `wrangler whoami`. Create the token in the Cloudflare dashboard, under Workers AI, then Use REST API. + +## 3. Stream a chat from the server + +From a Worker, pass `env.AI` as the binding: ```typescript -import { chat, toServerSentEventsResponse } from "@tanstack/ai"; +import { + chat, + chatParamsFromRequest, + toServerSentEventsResponse, +} from "@tanstack/ai"; import { createCloudflareText } from "@tanstack/ai-cloudflare"; import type { Ai } from "@cloudflare/workers-types"; @@ -56,13 +73,15 @@ interface Env { export default { async fetch(request: Request, env: Env) { - const { messages } = await request.json(); + const { messages, threadId, runId } = await chatParamsFromRequest(request); const stream = chat({ adapter: createCloudflareText("@cf/zai-org/glm-5.3-flash", { binding: env.AI, }), messages, + threadId, + runId, }); return toServerSentEventsResponse(stream); @@ -70,36 +89,86 @@ export default { }; ``` -That is a working AI route. Run `wrangler dev` and post `{ "messages": [...] }` to it. +Run `wrangler dev` to start the route. -## Chat from anywhere else +From any other server, `cloudflareText` reads `CLOUDFLARE_ACCOUNT_ID` and `CLOUDFLARE_API_TOKEN`: -Outside a Worker, the adapter talks to the Cloudflare REST API. You need your account id and an API token with Workers AI access. +```typescript +import { + chat, + chatParamsFromRequest, + toServerSentEventsResponse, +} from "@tanstack/ai"; +import { cloudflareText } from "@tanstack/ai-cloudflare"; + +export async function POST(request: Request) { + const { messages, threadId, runId } = await chatParamsFromRequest(request); + + const stream = chat({ + adapter: cloudflareText("@cf/zai-org/glm-5.3-flash"), + messages, + threadId, + runId, + }); + + return toServerSentEventsResponse(stream); +} +``` + +To pass the values in code, use `createCloudflareText`: ```typescript -import { chat } from "@tanstack/ai"; import { createCloudflareText } from "@tanstack/ai-cloudflare"; const adapter = createCloudflareText("@cf/zai-org/glm-5.3-flash", { accountId: process.env.CLOUDFLARE_ACCOUNT_ID!, apiKey: process.env.CLOUDFLARE_API_TOKEN!, }); - -const stream = chat({ - adapter, - messages: [{ role: "user", content: "Hello!" }], -}); ``` -`cloudflareText(model)` does the same and reads `CLOUDFLARE_ACCOUNT_ID` and `CLOUDFLARE_API_TOKEN` from the environment for you. +## 4. Render the chat + +The client is the same for every provider. Point `useChat` at your route: + +```tsx +import { useState } from "react"; +import { useChat, fetchServerSentEvents } from "@tanstack/ai-react"; -The text and summarize REST config also accepts the OpenAI SDK client options (`baseURL`, `defaultHeaders`, `fetch`, `timeout`, `maxRetries`). The other adapters call the native `/ai/run` endpoint and take `fetch` only. +export function Chat() { + const [input, setInput] = useState(""); + const { messages, sendMessage } = useChat({ + connection: fetchServerSentEvents("/api/chat"), + threadId: "support-chat", + }); + + return ( + + ); +} +``` + +For Vue, Solid, Svelte, Angular, and the other frameworks, see [Quick Start](../getting-started/quick-start#3-render-the-chat). -Need a header on each request, or a log of each model call? The chat adapter supports [`wrapFetch`](../advanced/middleware#change-the-http-requests-of-a-call) over REST. In binding mode, the wrapper still runs. The binding reads only the request body, so it drops the URL and the headers that the wrapper sets. +You now have a streaming chat on Workers AI. [Tools](../tools/tools) and [Structured Outputs](../structured-outputs/overview) work the same as with every adapter. ## Route through AI Gateway -AI Gateway gives you caching, logs, rate limits, and one bill across providers. Add `gateway` to any config to send requests through it. Use `"default"` for the gateway Cloudflare creates for your account. +AI Gateway gives you caching, logs, rate limits, and one bill for all providers. To send requests through it, add `gateway` to any config. Use `"default"` for the gateway that Cloudflare makes for your account: ```typescript import { createCloudflareText } from "@tanstack/ai-cloudflare"; @@ -117,7 +186,7 @@ export function makeAdapter(env: Env) { } ``` -With a gateway you can also run third-party models by their catalog id, billed through Cloudflare: +With a gateway, you can also run third-party models by their catalog id. Cloudflare bills them: ```typescript import { createCloudflareText } from "@tanstack/ai-cloudflare"; @@ -131,7 +200,7 @@ const adapter = createCloudflareText("openai/gpt-5.5", { ### Keep your existing provider adapter -Already using `@tanstack/ai-openai` with your own OpenAI key? `cloudflareGateway()` returns the `baseURL` and headers that point that adapter at your gateway. The rest of your code stays the same. +Do you use `@tanstack/ai-openai` with your own OpenAI key? `cloudflareGateway()` returns the `baseURL` and the headers that point that adapter at your gateway. The rest of your code stays the same: ```typescript import { createOpenaiChat } from "@tanstack/ai-openai"; @@ -143,22 +212,27 @@ const gateway = cloudflareGateway("openai", { cacheTtl: 300, }); -const adapter = createOpenaiChat("gpt-5.5", process.env.OPENAI_API_KEY!, { +const adapter = createOpenaiChat("gpt-6.1-sol", process.env.OPENAI_API_KEY!, { baseURL: gateway.baseURL, defaultHeaders: gateway.headers, }); ``` -The first argument is the provider slug from your gateway dashboard (`openai`, `anthropic`, `groq`, and so on). If the gateway has authentication turned on, pass `cfApiKey` too. That token needs the `AI Gateway Run` permission. +- The first argument is the provider slug from your gateway dashboard, for example `openai`, `anthropic`, or `groq`. +- If the gateway has authentication on, also pass `cfApiKey`. That token needs the `AI Gateway Run` permission. -The `ts-react-chat` example does this. Set `CLOUDFLARE_ACCOUNT_ID`, `CLOUDFLARE_API_TOKEN`, and `CLOUDFLARE_AI_GATEWAY_ID` in its `.env`, and the Cloudflare, OpenAI, Anthropic, and Groq models in its picker all go through your gateway. +The `ts-react-chat` example does this. Set `CLOUDFLARE_ACCOUNT_ID`, `CLOUDFLARE_API_TOKEN`, and `CLOUDFLARE_AI_GATEWAY_ID` in its `.env`. Then the Cloudflare, OpenAI, Anthropic, and Groq models in its picker all go through your gateway. ### Claude and GPT from a Worker -Inside a Worker, the AI Gateway `anthropic/...` and `openai/...` models use their provider's own API, not Chat Completions. Keep the provider adapter, and give it `cloudflareBindingFetch` as its `fetch`. The binding signs the requests, so the Worker needs no provider key: +Inside a Worker, the AI Gateway `anthropic/...` and `openai/...` models use the API of their own provider, not Chat Completions. Keep the provider adapter, and give it `cloudflareBindingFetch` as its `fetch`. The binding signs the requests, so the Worker needs no provider key: ```typescript -import { chat, toServerSentEventsResponse } from "@tanstack/ai"; +import { + chat, + chatParamsFromRequest, + toServerSentEventsResponse, +} from "@tanstack/ai"; import { createAnthropicChat } from "@tanstack/ai-anthropic"; import { cloudflareBindingFetch } from "@tanstack/ai-cloudflare"; import type { Ai } from "@cloudflare/workers-types"; @@ -169,7 +243,7 @@ interface Env { export default { async fetch(request: Request, env: Env) { - const { messages } = await request.json(); + const { messages, threadId, runId } = await chatParamsFromRequest(request); // The SDK needs a key value. The binding does not use it. const adapter = createAnthropicChat("claude-opus-5-5", "cloudflare-binding", { @@ -180,7 +254,7 @@ export default { }), }); - const stream = chat({ adapter, messages, reasoning: "high" }); + const stream = chat({ adapter, messages, threadId, runId, reasoning: "high" }); return toServerSentEventsResponse(stream); }, }; @@ -188,33 +262,22 @@ export default { - `vendor: "anthropic"` sends Anthropic Messages requests to `anthropic/The agent is inspecting the repo.
- if (!final) return null - return{final.summary}
+ if (isLoading) returnThe agent is inspecting the repo.
; + if (!final) return null; + return{final.summary}
; } ``` -If you only need to extract JSON from a prompt and do not need a sandbox, use `@tanstack/ai-openai`. That path is faster. +- Codex runs one turn and limits its last message with `--output-schema`. +- Tool activity and text stream as Codex writes them. +- The adapter parses the last message as the schema object. It arrives as `structured-output.complete`. + +Do you only need JSON from a prompt, with no repo? Use `@tanstack/ai-openai`. That is faster. For the full walkthrough, see [Harness Agents](../structured-outputs/harnesses). + +## Models and options + +### Models + +| Model | Reasoning levels | +| --- | --- | +| `gpt-5.3-codex` | `off`, `low`, `medium`, `high`, `xhigh` | +| `gpt-5.2-codex` | `low`, `medium`, `high`, `xhigh` | +| `gpt-5.1-codex`, `gpt-5.1-codex-mini` | `low`, `medium`, `high` | +| `gpt-5.1` | `off`, `low`, `medium`, `high` | + +The package exports this list as `CODEX_MODELS`. The adapter also takes any other model id that Codex accepts, as a string. + +### Reasoning + +Set `reasoning` on `chat()`. Codex gets it as `model_reasoning_effort`: + +```ts +import { chat } from "@tanstack/ai"; +import { codexText } from "@tanstack/ai-codex"; +import { withSandbox } from "@tanstack/ai-sandbox"; +import { sandbox } from "./sandbox"; + +const stream = chat({ + adapter: codexText("gpt-5.3-codex"), + messages: [{ role: "user", content: "Fix the failing test in utils.test.ts" }], + threadId: "thread-1", + reasoning: "xhigh", + middleware: [withSandbox(sandbox)], +}); +``` + +A call with no `reasoning` uses `modelReasoningEffort` from the adapter options. See [Reasoning](../chat/reasoning). + +### Adapter options + +Pass these as the second argument of `codexText`: + +| Option | What it does | +| --- | --- | +| `cwd` | The working directory inside the sandbox. Default `/workspace`. | +| `sandboxMode` | The inner Codex sandbox: `'read-only'`, `'workspace-write'`, or `'danger-full-access'`. See [Inner sandbox mode](#inner-sandbox-mode). | +| `approvalPolicy` | `'never'` (default), `'on-failure'`, `'on-request'`, or `'untrusted'`. A headless run has no approval UI, so another value can stop a turn. | +| `modelReasoningEffort` | The effort when a call sets no `reasoning`: `'minimal'`, `'low'`, `'medium'`, or `'high'`. | +| `skipGitRepoCheck` | Skip the git repo check of Codex. Default `true`. | +| `networkAccessEnabled` | Allow network access inside the `workspace-write` sandbox. | +| `webSearchMode` | `'disabled'` or `'live'`. | +| `additionalDirectories` | More writable directories next to `cwd`. | +| `authMode` | `'api-key'` (default) expects `CODEX_API_KEY`. `'host'` uses `codex login`. See [Harness Auth](../sandbox/auth). | +| `codexExecutable` | The path or name of the `codex` executable in the sandbox. Default `codex`. | +| `env` | More environment variables for the `codex` process. | +| `config` | More `--config key=value` overrides for the Codex CLI, for example more `mcp_servers` entries. Values go through as TOML. | + +### Per-call options + +Pass these in `modelOptions` on `chat()`. They override the adapter options for one call: + +- `sessionId`: resume a thread. See [Resume a session](#resume-a-session). +- `sandboxMode`, `approvalPolicy`, `skipGitRepoCheck`. +- `workingDirectory`: the working directory for this call. +- `authMode`: `'host'` or `'api-key'`. + +## Advanced + +### Inner sandbox mode + +The default `sandboxMode` depends on the provider: + +- Local-process and Docker: `'workspace-write'`. +- Daytona and Cloudflare: `'danger-full-access'`. These providers cannot create a nested bubblewrap namespace. The outer VM and your sandbox [Policy](../sandbox/policy) are then the isolation. + +The inner Codex sandbox does not work on Windows and on some Linux images. There it fails with `os error 2`. If you see this error, set `sandboxMode: 'danger-full-access'`. The outer sandbox is still the boundary. + +### Text streaming + +Codex reports text as items that grow. The adapter streams each new piece of an item, so text can arrive in larger pieces than with an HTTP adapter. Tool activity, such as a command that starts or ends, streams live. -Full walkthrough, including the client: [Harness Agents](../structured-outputs/harnesses). +### Limitations -## Limitations +- **A sandbox is required.** Every call needs `withSandbox(...)`. +- **The agent owns the loop.** TanStack agent-loop strategies and per-iteration middleware do not apply inside a turn. +- **No sampling options.** Options such as `temperature` do not exist here. +- **Sessions stay in the sandbox.** Resume needs the same sandbox. +- **Cold starts.** Each call starts a CLI turn, so the first token arrives later than with an HTTP adapter. +- **No `wrapFetch`.** This adapter ignores [`wrapFetch`](../advanced/middleware#change-the-http-requests-of-a-call), because the `codex` process sends the model requests. -- **No token-level text streaming.** The Codex SDK reports assistant text and reasoning only as completed items, so text arrives message-at-a-time. Tool activity (commands starting/finishing) still streams live, which keeps the UI feeling alive during long turns. -- **Server-only (Node).** The harness spawns a subprocess. -- **The harness owns the agent loop.** TanStack's agent-loop strategies and per-iteration middleware don't apply inside a harness turn. -- **No sampling controls.** `temperature`-style options don't exist here. -- **Sessions are machine-local.** Resume requires hitting the same server instance. -- **Cold starts.** Each call spawns a harness turn; expect higher first-token latency than HTTP adapters. +### Example apps -This adapter ignores [`wrapFetch`](../advanced/middleware#change-the-http-requests-of-a-call), because the harness process sends the model requests. +- [`examples/sandbox-cloudflare`](https://github.com/TanStack/ai/tree/main/examples/sandbox-cloudflare): a TanStack Start app on Workers. You pick Claude Code, Codex, or Grok Build in the UI. It shows session resume, the tool timeline, and tool bridging. +- [`examples/sandbox-web`](https://github.com/TanStack/ai/tree/main/examples/sandbox-web): Claude Code on Docker, with durable runs that continue after a page refresh. To use Codex there, change the adapter in `src/sandbox-agent.ts`. diff --git a/docs/adapters/cohere.md b/docs/adapters/cohere.md index ce13f129c3..17301998cb 100644 --- a/docs/adapters/cohere.md +++ b/docs/adapters/cohere.md @@ -2,7 +2,7 @@ title: Cohere id: cohere-adapter order: 11 -description: "Use Cohere with TanStack AI via @tanstack/ai-cohere: embed-v4.0 multimodal embeddings for semantic search and RAG, plus rerank models for relevance." +description: "Use Cohere in TanStack AI with @tanstack/ai-cohere: embed-v4.0 multimodal embeddings for semantic search and RAG, and rerank models to sort results by relevance." keywords: - tanstack ai - cohere @@ -16,14 +16,11 @@ keywords: - adapter --- -The Cohere adapter covers the two retrieval steps of a RAG pipeline: +The Cohere adapter embeds text and images, and it reranks documents for a query. This package has no `chat()`, `summarize()`, or media generation. -- **Embeddings** (`cohereEmbedding`): turn text, images, and fused text+image inputs into vectors with `embed()`. -- **Reranking** (`cohereRerank`): reorder candidate documents by relevance to a query with `rerank()`. +For those, use [OpenAI](./openai), [Anthropic](./anthropic), or [Gemini](./gemini). The adapter calls the HTTP API of Cohere with `fetch`, with no SDK dependency. -It does not support `chat()`, `summarize()`, or media generation. Use OpenAI, Anthropic, or Gemini for those. The adapter talks to Cohere's HTTP API directly over `fetch`, with no SDK dependency. - -## Installation +## 1. Install @@ -38,7 +35,17 @@ octane: @tanstack/ai @tanstack/ai-cohere -## Embeddings +## 2. Set the API key + +Get a key from the [Cohere dashboard](https://dashboard.cohere.com/api-keys). Put it in the environment of your server: + +```bash +COHERE_API_KEY=your-cohere-api-key +``` + +`cohereEmbedding` and `cohereRerank` read `COHERE_API_KEY`. To pass the key in code, see [Explicit API keys](#explicit-api-keys). + +## 3. Embed documents ```typescript import { embed } from "@tanstack/ai"; @@ -54,11 +61,37 @@ console.log(result.embeddings[0]?.vector); console.log(result.usage?.promptTokens); ``` -`inputType` is required by Cohere's API. Use `search_document` at index time and `search_query` at query time (or `classification` / `clustering` for those workloads). TanStack AI enforces this at the type level, so `modelOptions` is required for Cohere embedding calls. +The Cohere API requires `inputType`, and the types require `modelOptions` on every Cohere embedding call: -### Multimodal Embeddings +- `search_document`: when you index documents. +- `search_query`: when you embed the search query. +- `classification` or `clustering`: for those workloads. -embed-v4.0 embeds images alongside text. An image part produces an image vector. A nested array of parts (`[textPart, imagePart]`) fuses text and image into one vector, which suits product catalogs and screenshot search. The outer array is the item list, so nest to fuse: +## 4. Rerank results + +```typescript +import { rerank } from "@tanstack/ai"; +import { cohereRerank } from "@tanstack/ai-cohere"; + +const { rerankedDocuments } = await rerank({ + adapter: cohereRerank("rerank-v3.5"), + query: "talk about rain", + documents: ["sunny day at the beach", "rainy afternoon in the city"], +}); + +console.log(rerankedDocuments[0]); // 'rainy afternoon in the city' +``` + +You now have both retrieval steps. For object documents, RAG pipelines, the options, and the result shape, see [Reranking](../rerank/rerank). + +## Multimodal embeddings + +embed-v4.0 embeds images next to text: + +- An image part gives an image vector. +- A nested array of parts (`[textPart, imagePart]`) joins text and image into one vector. Use it for product catalogs and screenshot search. + +The outer array is the list of items, so nest the parts to join them: ```typescript import { embed } from "@tanstack/ai"; @@ -96,27 +129,11 @@ const result = await embed({ console.log(result.embeddings.length); // 2 ``` -Cohere's API does not fetch remote image URLs. Pass base64 data (or a `data:` URI), or opt into adapter-side downloading: - -```typescript -import { embed } from "@tanstack/ai"; -import { cohereEmbedding } from "@tanstack/ai-cohere"; - -const adapter = cohereEmbedding("embed-v4.0", { allowUrlFetch: true }); - -const result = await embed({ - adapter, - input: { - type: "image", - source: { type: "url", value: "https://example.com/guitar.png" }, - }, - modelOptions: { inputType: "search_document" }, -}); -``` +The Cohere API does not fetch remote image URLs. Pass base64 data (or a `data:` URI), or turn on [downloads in the adapter](#image-urls). -### Requesting Dimensions +## Shorter vectors -embed-v4.0 supports Matryoshka output dimensions via the top-level `dimensions` option: +embed-v4.0 takes the top-level `dimensions` option (`256`, `512`, `1024`, or `1536`): ```typescript import { embed } from "@tanstack/ai"; @@ -130,24 +147,9 @@ const result = await embed({ }); ``` -## Reranking - -```typescript -import { rerank } from "@tanstack/ai"; -import { cohereRerank } from "@tanstack/ai-cohere"; - -const { rerankedDocuments } = await rerank({ - adapter: cohereRerank("rerank-v3.5"), - query: "talk about rain", - documents: ["sunny day at the beach", "rainy afternoon in the city"], -}); - -console.log(rerankedDocuments[0]); // 'rainy afternoon in the city' -``` - -For the full reranking guide, with object documents, RAG pipelines, options, and the result shape, see [Reranking](../rerank/rerank). +## Rerank options -Per-request rerank options go on `modelOptions`: +Rerank options for one request go in `modelOptions`: ```typescript import { rerank } from "@tanstack/ai"; @@ -167,30 +169,18 @@ console.log(ranking); ## Models -| Model | Capability | Description | -| -------------------------- | ---------- | ----------------------------------------------------------------- | -| `embed-v4.0` | Embeddings | Multimodal (text + images), Matryoshka `dimensions` support | -| `rerank-v3.5` | Reranking | Latest multilingual reranker (recommended) | -| `rerank-english-v3.0` | Reranking | English-optimized reranker | -| `rerank-multilingual-v3.0` | Reranking | Multilingual reranker | +| Model | Activity | Notes | +| --- | --- | --- | +| `embed-v4.0` | Embeddings | Text and images, `dimensions` support | +| `rerank-v3.5` | Reranking | Newest multilingual reranker. Recommended | +| `rerank-english-v3.0` | Reranking | Tuned for English | +| `rerank-multilingual-v3.0` | Reranking | Multilingual reranker | -## Environment Variables +## Advanced -Both adapters read your API key from the environment: +### Explicit API keys -```bash -COHERE_API_KEY=your-cohere-api-key -``` - -| Variable | Required | Description | -| ---------------- | -------- | ------------------- | -| `COHERE_API_KEY` | Yes | Your Cohere API key | - -Get a key from the [Cohere dashboard](https://dashboard.cohere.com/api-keys). - -## Explicit API Keys - -To pass a key directly instead of reading the environment, use the `create*` factories: +To pass a key in code, not from the environment, use the `create*` factories: ```typescript import { @@ -205,9 +195,29 @@ const embedAdapter = createCohereEmbedding( const rerankAdapter = createCohereRerank("rerank-v3.5", "your-cohere-api-key"); ``` -## Behind a proxy +### Image URLs -Route every request through a gateway, such as Cloudflare AI Gateway or a corporate proxy, with `baseURL` and `defaultHeaders`. These two option names are the same on every TanStack AI adapter, so one gateway config works for all of them. +To let the adapter download `http(s)` image URLs and send them as base64, set `allowUrlFetch`: + +```typescript +import { embed } from "@tanstack/ai"; +import { cohereEmbedding } from "@tanstack/ai-cohere"; + +const adapter = cohereEmbedding("embed-v4.0", { allowUrlFetch: true }); + +const result = await embed({ + adapter, + input: { + type: "image", + source: { type: "url", value: "https://example.com/guitar.png" }, + }, + modelOptions: { inputType: "search_document" }, +}); +``` + +### Behind a proxy + +Send every request through a gateway, such as Cloudflare AI Gateway or a corporate proxy, with `baseURL` and `defaultHeaders`. These two option names are the same on every TanStack AI adapter, so one gateway config works for all of them. ```typescript import { createCohereEmbedding } from "@tanstack/ai-cohere"; @@ -220,35 +230,19 @@ const adapter = createCohereEmbedding("embed-v4.0", process.env.COHERE_API_KEY!, `baseUrl` and `headers` are aliases of the same two options. If you set both forms, `baseURL` and `defaultHeaders` win. -## API Reference +### API reference -### `cohereEmbedding(model, config?)` +| Factory | Creates | +| --- | --- | +| `cohereEmbedding(model, config?)` | Embedding adapter. Reads `COHERE_API_KEY` | +| `createCohereEmbedding(model, apiKey, config?)` | Embedding adapter with an explicit API key | +| `cohereRerank(model, config?)` | Rerank adapter. Reads `COHERE_API_KEY` | +| `createCohereRerank(model, apiKey, config?)` | Rerank adapter with an explicit API key | -Creates an embedding adapter using `COHERE_API_KEY` from the environment. +Config for both adapters: -- `model`: `"embed-v4.0"` -- `config.baseURL`: override the API base URL (default `https://api.cohere.com`). `baseUrl` is an alias. +- `config.baseURL`: the API base URL (default `https://api.cohere.com`). `baseUrl` is an alias. - `config.defaultHeaders`: extra request headers. `headers` is an alias. -- `config.allowUrlFetch`: download `http(s)` image URLs and inline them as base64 (default `false`) - -### `createCohereEmbedding(model, apiKey, config?)` - -Same as `cohereEmbedding` with an explicit API key. - -### `cohereRerank(model, config?)` - -Creates a rerank adapter using `COHERE_API_KEY` from the environment. - -- `model`: one of the rerank models above -- `config.baseURL`: override the API base URL (default `https://api.cohere.com`). `baseUrl` is an alias. -- `config.defaultHeaders`: extra request headers. `headers` is an alias. - -### `createCohereRerank(model, apiKey, config?)` - -Same as `cohereRerank` with an explicit API key. - -## Next Steps +- `config.allowUrlFetch` (embeddings only): download `http(s)` image URLs and send them as base64. Default `false`. -- [Embeddings guide](../embeddings.md): the full `embed()` API -- [Reranking guide](../rerank/rerank): full walkthrough including RAG pipelines -- [Generation Hooks](../media/generation-hooks.md): usage and lifecycle middleware +For the full APIs, see the [Embeddings guide](../embeddings.md), the [Reranking guide](../rerank/rerank), and [Generation Hooks](../media/generation-hooks.md) for usage and lifecycle middleware. diff --git a/docs/adapters/elevenlabs.md b/docs/adapters/elevenlabs.md index 17d3ecbf0d..a2454e109e 100644 --- a/docs/adapters/elevenlabs.md +++ b/docs/adapters/elevenlabs.md @@ -2,7 +2,7 @@ title: ElevenLabs id: elevenlabs-adapter order: 9 -description: "Build realtime voice-to-voice conversational AI with ElevenLabs agents in TanStack AI via the @tanstack/ai-elevenlabs adapter." +description: "Use ElevenLabs in TanStack AI with @tanstack/ai-elevenlabs: realtime voice agents, text-to-speech, music and sound effects, transcription, and voice design." keywords: - tanstack ai - elevenlabs @@ -13,175 +13,114 @@ keywords: - adapter --- -The ElevenLabs adapter is **voice-focused**. It exposes five capabilities: +`@tanstack/ai-elevenlabs` calls ElevenLabs for voice agents, speech, music, sound effects, transcription, and voice design. It has no text chat. -- **Realtime voice agents** (`elevenlabsRealtime` / `elevenlabsRealtimeToken`) — full-duplex voice-to-voice conversations powered by ElevenLabs Conversational AI agents. -- **Text-to-speech** (`elevenlabsSpeech`) — one-shot speech generation via `generateSpeech()`. -- **Music & sound effects** (`elevenlabsAudio`) — one-shot audio generation via `generateAudio()`. -- **Transcription** (`elevenlabsTranscription`) — speech-to-text via `generateTranscription()`. -- **Voice design** (`elevenlabsVoiceDesign`): create a new voice via `generateVoice()`. +The quick setup below builds a live voice conversation with an ElevenLabs agent. -It does not support text `chat()` or `summarize()` — use OpenAI, Anthropic, or Gemini for those. - -The realtime adapter uses an **agent-based architecture** where you configure your conversational AI agent in the [ElevenLabs dashboard](https://elevenlabs.io/) (voice, personality, knowledge base, tools) and then connect to it at runtime. The adapter wraps the `@elevenlabs/client` SDK for seamless integration with `useRealtimeChat` and `RealtimeClient`. - -## Installation +## 1. Install -react: @tanstack/ai-elevenlabs -vue: @tanstack/ai-elevenlabs -solid: @tanstack/ai-elevenlabs -svelte: @tanstack/ai-elevenlabs -preact: @tanstack/ai-elevenlabs -angular: @tanstack/ai-elevenlabs -vanilla: @tanstack/ai-elevenlabs -octane: @tanstack/ai-elevenlabs +react: @tanstack/ai @tanstack/ai-client @tanstack/ai-elevenlabs +vue: @tanstack/ai @tanstack/ai-client @tanstack/ai-elevenlabs +solid: @tanstack/ai @tanstack/ai-client @tanstack/ai-elevenlabs +svelte: @tanstack/ai @tanstack/ai-client @tanstack/ai-elevenlabs +preact: @tanstack/ai @tanstack/ai-client @tanstack/ai-elevenlabs +angular: @tanstack/ai @tanstack/ai-client @tanstack/ai-elevenlabs +vanilla: @tanstack/ai @tanstack/ai-client @tanstack/ai-elevenlabs +octane: @tanstack/ai @tanstack/ai-client @tanstack/ai-elevenlabs -Peer dependencies: +`@tanstack/ai` and `@tanstack/ai-client` are peer dependencies of the adapter. Step 4 also uses the React package, `@tanstack/ai-react`. - +## 2. Set the API key and the agent -react: @tanstack/ai @tanstack/ai-client -vue: @tanstack/ai @tanstack/ai-client -solid: @tanstack/ai @tanstack/ai-client -svelte: @tanstack/ai @tanstack/ai-client -preact: @tanstack/ai @tanstack/ai-client -angular: @tanstack/ai @tanstack/ai-client -vanilla: @tanstack/ai @tanstack/ai-client -octane: @tanstack/ai @tanstack/ai-client +1. Get an API key from the [ElevenLabs dashboard](https://elevenlabs.io/). +2. In the **Conversational AI** section of the dashboard, create an agent. Set its voice, personality, knowledge, and tools there. +3. Put both values in the environment of your server: - +```bash +ELEVENLABS_API_KEY=your-elevenlabs-api-key +ELEVENLABS_AGENT_ID=your-agent-id +``` + +All adapters read `ELEVENLABS_API_KEY`. If you pass `agentId` in code, `ELEVENLABS_AGENT_ID` is optional. -## Server Setup +## 3. Create a voice token on the server -The server generates a **signed WebSocket URL** so your API key never reaches the client. The signed URL is valid for 30 minutes. +The browser must not see your API key. So the server makes a signed WebSocket URL, and the browser connects with it. The URL is valid for 30 minutes. -```typescript group=elevenlabs-1 -import { realtimeToken } from '@tanstack/ai' -import { elevenlabsRealtimeToken } from '@tanstack/ai-elevenlabs' +```typescript +import { realtimeToken } from "@tanstack/ai"; +import { elevenlabsRealtimeToken } from "@tanstack/ai-elevenlabs"; -// In your API route (Express, Hono, TanStack Start, etc.) export async function POST() { const token = await realtimeToken({ adapter: elevenlabsRealtimeToken({ agentId: process.env.ELEVENLABS_AGENT_ID!, }), - }) + }); - return Response.json(token) + return Response.json(token); } ``` -### With Overrides +Serve this handler at `/api/realtime-token`. -You can override agent settings at token generation time without changing your dashboard configuration: +## 4. Talk to the agent -```typescript group=elevenlabs-1 -const token = await realtimeToken({ - adapter: elevenlabsRealtimeToken({ - agentId: process.env.ELEVENLABS_AGENT_ID!, - overrides: { - voiceId: 'custom-voice-id', - systemPrompt: 'You are a helpful voice assistant.', - firstMessage: 'Hello! How can I help you today?', - language: 'en', - }, - }), -}) -``` - -## Client Setup - -### React (useRealtimeChat) +`useRealtimeChat` gets the token, opens the microphone, and plays the voice of the agent: ```tsx -import { useRealtimeChat } from '@tanstack/ai-react' -import { elevenlabsRealtime } from '@tanstack/ai-elevenlabs' +import { useRealtimeChat } from "@tanstack/ai-react"; +import { elevenlabsRealtime } from "@tanstack/ai-elevenlabs"; -function VoiceChat() { - const { - status, - mode, - messages, - connect, - disconnect, - pendingUserTranscript, - pendingAssistantTranscript, - inputLevel, - outputLevel, - } = useRealtimeChat({ - getToken: () => - fetch('/api/realtime-token', { method: 'POST' }).then((r) => r.json()), - adapter: elevenlabsRealtime(), - }) +export function VoiceChat() { + const { status, messages, connect, disconnect, pendingAssistantTranscript } = + useRealtimeChat({ + getToken: () => + fetch("/api/realtime-token", { method: "POST" }).then((r) => r.json()), + adapter: elevenlabsRealtime(), + }); return (
-
- )
+ );
}
```
-### Non-React (RealtimeClient)
+Click **Start conversation** and speak. The agent answers out loud, and the transcript shows up below the button. For the other frameworks and for the full realtime API, see [Realtime Voice Chat](../media/realtime-chat).
-```typescript
-import { RealtimeClient } from '@tanstack/ai-client'
-import { elevenlabsRealtime } from '@tanstack/ai-elevenlabs'
+## Client tools
-const client = new RealtimeClient({
- getToken: () =>
- fetch('/api/realtime-token', { method: 'POST' }).then((r) => r.json()),
- adapter: elevenlabsRealtime(),
- onMessage: (message) => {
- console.log(`${message.role}:`, message.parts)
- },
- onStatusChange: (status) => {
- console.log('Status:', status)
- },
- onModeChange: (mode) => {
- console.log('Mode:', mode)
- },
-})
-
-await client.connect()
-```
-
-## Client Tools
-
-ElevenLabs supports client-side tools that execute in the browser. Define tools using the standard `toolDefinition()` API:
+The agent can call tools that run in the browser. Define them with `toolDefinition()` and pass them to the hook:
```typescript
-import { toolDefinition } from '@tanstack/ai'
-import { useRealtimeChat } from '@tanstack/ai-react'
-import { elevenlabsRealtime } from '@tanstack/ai-elevenlabs'
-import { z } from 'zod'
+import { toolDefinition } from "@tanstack/ai";
+import { useRealtimeChat } from "@tanstack/ai-react";
+import { elevenlabsRealtime } from "@tanstack/ai-elevenlabs";
+import { z } from "zod";
const getWeatherDef = toolDefinition({
- name: 'getWeather',
- description: 'Get weather for a location',
+ name: "getWeather",
+ description: "Get weather for a location",
inputSchema: z.object({
location: z.string(),
}),
@@ -189,105 +128,28 @@ const getWeatherDef = toolDefinition({
temperature: z.number(),
conditions: z.string(),
}),
-})
+});
const getWeather = getWeatherDef.client(async ({ location }) => {
- const res = await fetch(`/api/weather?location=${location}`)
- return res.json()
-})
-
-// Pass tools to the hook
-const chat = useRealtimeChat({
- getToken: () =>
- fetch('/api/realtime-token', { method: 'POST' }).then((r) => r.json()),
- adapter: elevenlabsRealtime(),
- tools: [getWeather],
-})
-```
-
-Tool results are automatically serialized to strings and returned to the ElevenLabs agent. The adapter converts TanStack tool definitions into the `@elevenlabs/client` clientTools format internally.
-
-## Configuration
-
-### `elevenlabsRealtimeToken` Options
-
-Used on the **server** to generate a signed WebSocket URL.
-
-| Option | Type | Required | Description |
-|--------|------|----------|-------------|
-| `agentId` | `string` | No\* | Agent ID configured in the ElevenLabs dashboard. \*Falls back to `ELEVENLABS_AGENT_ID`; required only if that env var is unset |
-| `overrides.voiceId` | `string` | No | Custom voice ID to override the agent's default voice |
-| `overrides.systemPrompt` | `string` | No | Custom system prompt to override the agent's default |
-| `overrides.firstMessage` | `string` | No | First message the agent speaks when the session starts |
-| `overrides.language` | `string` | No | Language code (e.g., `'en'`, `'es'`, `'fr'`) |
-
-### `elevenlabsRealtime` Options
-
-Used on the **client** to establish the connection.
-
-| Option | Type | Default | Description |
-|--------|------|---------|-------------|
-| `connectionMode` | `'websocket' \| 'webrtc'` | auto-detect | Transport protocol for the connection |
-| `debug` | `boolean \| DebugConfig` | `false` | Enable debug logging — pass `true` for all categories, or a `DebugConfig` to select categories/sink |
-
-## Differences from OpenAI Realtime
-
-ElevenLabs and OpenAI take different approaches to realtime voice:
-
-| | ElevenLabs | OpenAI |
-|---|---|---|
-| **Configuration** | Agent-based. Configure voice, personality, and knowledge in the ElevenLabs dashboard or via `overrides` at token time. | Session-based. Configure `instructions`, `voice`, `temperature`, etc. per session via `useRealtimeChat` options. |
-| **Token type** | Signed WebSocket URL (valid 30 minutes) | Ephemeral API token (valid ~10 minutes) |
-| **Transport** | WebSocket (default) or WebRTC | WebRTC |
-| **Audio handling** | `@elevenlabs/client` SDK manages audio capture and playback automatically | TanStack AI manages WebRTC peer connection and audio tracks |
-| **VAD** | Handled by ElevenLabs server-side | Supports `server`, `semantic`, and `manual` modes |
-| **Runtime updates** | Session config is set at creation time and cannot be changed mid-session | Supports `updateSession()` for mid-session config changes |
-| **Image input** | Not supported | Supported via `sendImage()` |
-| **Time domain data** | Not available from the SDK | Available for waveform visualizations |
-
-## Audio Visualization
-
-The ElevenLabs adapter provides audio visualization data through the same interface as other realtime adapters:
-
-```typescript
-import { useRealtimeChat } from '@tanstack/ai-react'
-import { elevenlabsRealtime } from '@tanstack/ai-elevenlabs'
-
-const {
- inputLevel, // 0-1 normalized microphone volume
- outputLevel, // 0-1 normalized speaker volume
- getInputFrequencyData, // Uint8Array frequency spectrum
- getOutputFrequencyData,
-} = useRealtimeChat({
- getToken: () =>
- fetch('/api/realtime-token', { method: 'POST' }).then((r) => r.json()),
- adapter: elevenlabsRealtime(),
-})
-```
-
-**Note:** ElevenLabs provides volume levels and frequency data but does not expose time-domain data. The `getInputTimeDomainData()` and `getOutputTimeDomainData()` methods return static placeholder arrays. The default audio sample rate is 16kHz.
-
-## Environment Variables
-
-Set these in your server environment:
+ const res = await fetch(`/api/weather?location=${location}`);
+ return res.json();
+});
-```bash
-ELEVENLABS_API_KEY=your-elevenlabs-api-key
-ELEVENLABS_AGENT_ID=your-agent-id
+export function useWeatherVoiceChat() {
+ return useRealtimeChat({
+ getToken: () =>
+ fetch("/api/realtime-token", { method: "POST" }).then((r) => r.json()),
+ adapter: elevenlabsRealtime(),
+ tools: [getWeather],
+ });
+}
```
-| Variable | Required | Description |
-|----------|----------|-------------|
-| `ELEVENLABS_API_KEY` | Yes | Your ElevenLabs API key, used server-side for generating signed URLs |
-| `ELEVENLABS_AGENT_ID` | No | Default agent ID. Can also be passed directly to `elevenlabsRealtimeToken()` |
+The adapter turns each result into a string and sends it to the agent.
-Get your API key from the [ElevenLabs dashboard](https://elevenlabs.io/). Create and configure agents in the **Conversational AI** section of the dashboard.
+## Text-to-speech
-## Text-to-Speech
-
-For one-shot speech generation (not realtime), use `elevenlabsSpeech` with `generateSpeech()`:
-
-The `format` option supports `mp3` (default), `pcm`, `opus`, and `wav`. WAV output contains 44.1 kHz, 16-bit mono PCM with a RIFF header. Requests for `aac` or `flac` throw before the API call. An explicit `modelOptions.outputFormat` overrides `format` and returns the selected provider format without WAV wrapping.
+`elevenlabsSpeech` makes one audio clip from text. `voice` is an ElevenLabs voice id:
```typescript
import { generateSpeech } from "@tanstack/ai";
@@ -296,84 +158,64 @@ import { elevenlabsSpeech } from "@tanstack/ai-elevenlabs";
const result = await generateSpeech({
adapter: elevenlabsSpeech("eleven_v4"),
text: "Hello from ElevenLabs!",
- voice: "Rachel",
+ threadId: "thread-1",
+ voice: "21m00Tcm4TlvDq8ikWAM",
format: "mp3",
});
console.log(result.audio); // Base64-encoded audio
```
-### Dialogue and timings
-
-ElevenLabs has a separate dialogue endpoint that takes up to **10 distinct
-voices**, and a timestamped twin of each endpoint. `turns` and `timestamps`
-pick between them, so you never choose an endpoint by hand:
-
-```typescript
-import { generateSpeech } from "@tanstack/ai";
-import { elevenlabsSpeech } from "@tanstack/ai-elevenlabs";
-
-const result = await generateSpeech({
- adapter: elevenlabsSpeech("eleven_v4"),
- turns: [
- { text: "Knock knock.", voice: "bYTqZQo3Jz7LQtmGTgwi" },
- { text: "Who is there?", voice: "6lCwbsX1yVjD49QmpkTR" },
- ],
- timestamps: true,
-});
-
-// Character timings for the whole clip.
-console.log(result.alignment?.unit); // 'character'
-
-// One entry per turn, with the voice that spoke it.
-for (const segment of result.segments ?? []) {
- console.log(segment.turnIndex, segment.voice, segment.text);
-}
-```
+- `format` takes `mp3` (the default), `pcm`, `opus`, or `wav`. The adapter throws for `aac` and `flac` before it calls the API.
+- For a dialogue with several voices, or for timings, see [Dialogue and timings](#dialogue-and-timings).
-`segments` only comes back from the dialogue endpoint. A single-voice request
-with `timestamps: true` returns `alignment` alone.
+See [Text-to-Speech](../media/text-to-speech).
-## Music & Sound Effects
+## Music and sound effects
-`elevenlabsAudio` covers both music generation and sound effects depending on the model:
+`elevenlabsAudio` makes music or a sound effect. The model picks which one:
```typescript
import { generateAudio } from "@tanstack/ai";
import { elevenlabsAudio } from "@tanstack/ai-elevenlabs";
-// Music generation
const music = await generateAudio({
adapter: elevenlabsAudio("music_v2_5"),
prompt: "An upbeat synthwave track for a product launch",
+ threadId: "thread-1",
});
-// Sound effects
const sfx = await generateAudio({
adapter: elevenlabsAudio("eleven_text_to_sound_v2"),
prompt: "A glass shattering on concrete",
+ threadId: "thread-1",
});
+
+console.log(music.audio, sfx.audio);
```
-## Listing Voices
+See [Audio Generation](../media/audio-generation).
-`elevenlabsSpeech` can read the account's voice catalog, which includes anything `generateVoice()` has saved:
+## Transcription
```typescript
-import { listVoices } from "@tanstack/ai";
-import { elevenlabsSpeech } from "@tanstack/ai-elevenlabs";
+import { generateTranscription } from "@tanstack/ai";
+import { elevenlabsTranscription } from "@tanstack/ai-elevenlabs";
-const { voices } = await listVoices({
- adapter: elevenlabsSpeech("eleven_v4"),
- origins: ["generated", "cloned"],
+const result = await generateTranscription({
+ adapter: elevenlabsTranscription("scribe_v2"),
+ audio: "https://example.com/recording.mp3",
+ threadId: "thread-1",
});
+
+console.log(result.text);
```
-ElevenLabs has no origin filter on `GET /v1/voices`, so the adapter fetches the catalog and narrows in memory. Its `famous` and `high_quality` categories both read as `professional`.
+See [Transcription](../media/transcription).
-## Voice Design
+## Voice design
-`elevenlabsVoiceDesign` creates a new voice from a description. The voice IDs it returns go straight into `generateSpeech()`:
+`elevenlabsVoiceDesign` makes a new voice from a description. Use the voice id that it returns in `generateSpeech()`:
```typescript
import { generateVoice } from "@tanstack/ai";
@@ -382,6 +224,7 @@ import { elevenlabsVoiceDesign } from "@tanstack/ai-elevenlabs";
const result = await generateVoice({
adapter: elevenlabsVoiceDesign("eleven_ttv_v3"),
prompt: "A warm, gravelly narrator in his sixties with a slight Irish lilt",
+ threadId: "thread-1",
name: "Irish Narrator",
});
@@ -392,112 +235,215 @@ console.log(voice.voiceId);
console.log(voice.saved); // true, because a name was given
```
-Without a `name`, you get preview voices to audition. With a `name`, the best candidate is kept in your ElevenLabs voice library.
+- Without a `name`, you get preview voices to listen to.
+- With a `name`, ElevenLabs keeps the best voice in your voice library.
-`eleven_ttv_v3` also accepts `referenceAudio`, a clip of a real speaker used as a design reference — it still needs a `prompt`, and `modelOptions.promptStrength` balances the two. `eleven_multilingual_ttv_v2` takes no reference audio. See [Voice Creation](../media/voice-creation) for the full guide.
+For reference audio and the other options, see [Voice Creation](../media/voice-creation).
-## Models
+## Models and options
-| Family | Models |
-| --- | --- |
-| Text-to-speech | `eleven_v4`, `eleven_v4_turbo`, `eleven_v3`, `eleven_v3_conversational`, `eleven_multilingual_v2`, `eleven_flash_v2_5`, `eleven_flash_v2` |
-| Music | `music_v2_5`, `music_v2` |
-| Sound effects | `eleven_text_to_sound_v2` |
-| Transcription | `scribe_v2`, `scribe_v2_medical` |
-| Voice design | `eleven_ttv_v3`, `eleven_multilingual_ttv_v2` |
+| Activity | Factory | Models |
+| --- | --- | --- |
+| Text-to-speech | `elevenlabsSpeech` | `eleven_v4`, `eleven_v4_turbo`, `eleven_v3`, `eleven_v3_conversational`, `eleven_multilingual_v2`, `eleven_flash_v2_5`, `eleven_flash_v2` |
+| Music | `elevenlabsAudio` | `music_v2_5`, `music_v2` |
+| Sound effects | `elevenlabsAudio` | `eleven_text_to_sound_v2` |
+| Transcription | `elevenlabsTranscription` | `scribe_v2`, `scribe_v2_medical` |
+| Voice design | `elevenlabsVoiceDesign` | `eleven_ttv_v3`, `eleven_multilingual_ttv_v2` |
-ElevenLabs deprecated `eleven_turbo_v2`, `eleven_turbo_v2_5`, `eleven_monolingual_v1`, `scribe_v1`, and `music_v1`. The adapter still accepts them, and each is commented as deprecated in `model-meta.ts`. Move to the current model in the same row.
+The adapter also accepts the deprecated ids `eleven_turbo_v2`, `eleven_turbo_v2_5`, `eleven_monolingual_v1`, `scribe_v1`, and `music_v1`. Use the newest model in the same row.
-## Transcription
+The package exports the lists as `ELEVENLABS_TTS_MODELS`, `ELEVENLABS_AUDIO_MODELS`, `ELEVENLABS_TRANSCRIPTION_MODELS`, and `ELEVENLABS_VOICE_MODELS`.
-Transcribe audio with `elevenlabsTranscription`:
+## Advanced
+
+### Agent overrides
+
+To change the agent for one session, set `overrides` when you make the token. Your dashboard settings do not change:
```typescript
-import { generateTranscription } from "@tanstack/ai";
-import { elevenlabsTranscription } from "@tanstack/ai-elevenlabs";
-import { audioFile } from "./audio";
+import { realtimeToken } from "@tanstack/ai";
+import { elevenlabsRealtimeToken } from "@tanstack/ai-elevenlabs";
-const result = await generateTranscription({
- adapter: elevenlabsTranscription("scribe_v2"),
- audio: audioFile,
+const token = await realtimeToken({
+ adapter: elevenlabsRealtimeToken({
+ agentId: process.env.ELEVENLABS_AGENT_ID!,
+ overrides: {
+ voiceId: "custom-voice-id",
+ systemPrompt: "You are a helpful voice assistant.",
+ firstMessage: "Hello! How can I help you today?",
+ language: "en",
+ },
+ }),
});
-console.log(result.text);
+console.log(token.expiresAt);
```
-## Behind a proxy
+| Option | Type | Description |
+| --- | --- | --- |
+| `agentId` | `string` | The agent id from the dashboard. If you do not set it, the adapter reads `ELEVENLABS_AGENT_ID`. |
+| `overrides.voiceId` | `string` | A voice id that replaces the default voice of the agent. |
+| `overrides.systemPrompt` | `string` | A system prompt that replaces the default prompt of the agent. |
+| `overrides.firstMessage` | `string` | The first thing the agent says when the session starts. |
+| `overrides.language` | `string` | A language code, for example `"en"`, `"es"`, or `"fr"`. |
+
+### Client adapter options
+
+`elevenlabsRealtime(options?)` runs in the browser:
-Route every request through a gateway, such as Cloudflare AI Gateway or a corporate proxy, with `baseURL` and `defaultHeaders`. These two option names are the same on every TanStack AI adapter, so one gateway config works for all of them.
+| Option | Type | Default | Description |
+| --- | --- | --- | --- |
+| `connectionMode` | `"websocket"` or `"webrtc"` | auto-detect | The transport of the connection. |
+| `debug` | `boolean` or `DebugConfig` | `false` | Debug logs. `true` turns on all categories. A `DebugConfig` selects categories or a sink. |
+
+### Without React
+
+Use `RealtimeClient` from `@tanstack/ai-client`:
```typescript
-import { createElevenLabsSpeech } from "@tanstack/ai-elevenlabs";
+import { RealtimeClient } from "@tanstack/ai-client";
+import { elevenlabsRealtime } from "@tanstack/ai-elevenlabs";
-const adapter = createElevenLabsSpeech("eleven_v4", process.env.ELEVENLABS_API_KEY!, {
- baseURL: "https://gateway.example.com/elevenlabs",
- defaultHeaders: { "cf-aig-authorization": `Bearer ${process.env.GATEWAY_TOKEN}` },
+const client = new RealtimeClient({
+ getToken: () =>
+ fetch("/api/realtime-token", { method: "POST" }).then((r) => r.json()),
+ adapter: elevenlabsRealtime(),
+ onMessage: (message) => {
+ console.log(`${message.role}:`, message.parts);
+ },
+ onStatusChange: (status) => {
+ console.log("Status:", status);
+ },
+ onModeChange: (mode) => {
+ console.log("Mode:", mode);
+ },
});
+
+await client.connect();
```
-This applies to the speech, audio, and transcription adapters. `baseUrl` and `headers` are aliases of the same two options. If you set both forms, `baseURL` and `defaultHeaders` win.
+### Audio levels for a visualizer
-## API Reference
+The hook gives you the audio data of the microphone and the speaker:
-### `elevenlabsRealtimeToken(options)`
+```typescript
+import { useRealtimeChat } from "@tanstack/ai-react";
+import { elevenlabsRealtime } from "@tanstack/ai-elevenlabs";
-Creates an ElevenLabs realtime token adapter for server-side use with `realtimeToken()`.
+export function useVoiceLevels() {
+ const {
+ inputLevel, // 0 to 1, microphone volume
+ outputLevel, // 0 to 1, speaker volume
+ getInputFrequencyData, // Uint8Array frequency spectrum
+ getOutputFrequencyData,
+ } = useRealtimeChat({
+ getToken: () =>
+ fetch("/api/realtime-token", { method: "POST" }).then((r) => r.json()),
+ adapter: elevenlabsRealtime(),
+ });
+
+ return { inputLevel, outputLevel, getInputFrequencyData, getOutputFrequencyData };
+}
+```
+
+ElevenLabs gives volume levels and frequency data, but no time-domain data. `getInputTimeDomainData()` and `getOutputTimeDomainData()` return fixed placeholder arrays. The default sample rate is 16 kHz.
+
+### ElevenLabs and OpenAI realtime
+
+| | ElevenLabs | OpenAI |
+|---|---|---|
+| Configuration | Agent-based. Set voice, personality, and knowledge in the dashboard, or with `overrides` on the token. | Session-based. Set `instructions`, `voice`, and more for each session in the `useRealtimeChat` options. |
+| Token | Signed WebSocket URL, valid for 30 minutes | Ephemeral API token, valid for about 10 minutes |
+| Transport | WebSocket (default) or WebRTC | WebRTC |
+| Audio | The `@elevenlabs/client` SDK records and plays the audio | TanStack AI runs the WebRTC connection and the audio tracks |
+| Voice activity detection | ElevenLabs does it on its server | `server`, `semantic`, and `manual` modes |
+| Session changes | Fixed when the session starts | `updateSession()` changes the session while it runs |
+| Image input | Not supported | `sendImage()` |
+| Time-domain data | Not available | Available for waveforms |
-**Parameters:**
+### Dialogue and timings
+
+ElevenLabs has a dialogue endpoint that takes up to 10 different voices. Each endpoint also has a version with timestamps. `turns` and `timestamps` pick the endpoint for you:
+
+```typescript
+import { generateSpeech } from "@tanstack/ai";
+import { elevenlabsSpeech } from "@tanstack/ai-elevenlabs";
-- `options.agentId` - Agent ID from the ElevenLabs dashboard
-- `options.overrides?.voiceId` - Custom voice ID
-- `options.overrides?.systemPrompt` - Custom system prompt
-- `options.overrides?.firstMessage` - First message the agent speaks
-- `options.overrides?.language` - Language code
+const result = await generateSpeech({
+ adapter: elevenlabsSpeech("eleven_v4"),
+ turns: [
+ { text: "Knock knock.", voice: "bYTqZQo3Jz7LQtmGTgwi" },
+ { text: "Who is there?", voice: "6lCwbsX1yVjD49QmpkTR" },
+ ],
+ timestamps: true,
+ threadId: "thread-1",
+});
-**Returns:** A `RealtimeTokenAdapter` for use with `realtimeToken()`.
+// Character timings for the whole clip.
+console.log(result.alignment?.unit); // "character"
-### `elevenlabsRealtime(options?)`
+// One entry per turn, with the voice that spoke it.
+for (const segment of result.segments ?? []) {
+ console.log(segment.turnIndex, segment.voice, segment.text);
+}
+```
-Creates an ElevenLabs realtime client adapter for use with `useRealtimeChat` or `RealtimeClient`.
+Only the dialogue endpoint returns `segments`. A request with one voice and `timestamps: true` returns `alignment` only.
-**Parameters:**
+### Speech output format
-- `options.connectionMode?` - `'websocket'` or `'webrtc'` (default: auto-detect)
-- `options.debug?` - Enable debug logging
+- `wav` output is 44.1 kHz, 16-bit mono PCM with a RIFF header.
+- `modelOptions.outputFormat` overrides `format`. The adapter then returns the format of ElevenLabs as it is, with no WAV header.
-**Returns:** A `RealtimeAdapter` for use with `useRealtimeChat()` or `RealtimeClient`.
+### List voices
-### `elevenlabsSpeech(model, config?)` / `createElevenLabsSpeech(model, apiKey, config?)`
+`elevenlabsSpeech` reads the voices of your account. This includes the voices that `generateVoice()` saved:
-Creates an ElevenLabs text-to-speech adapter for use with `generateSpeech()`.
+```typescript
+import { listVoices } from "@tanstack/ai";
+import { elevenlabsSpeech } from "@tanstack/ai-elevenlabs";
-### `elevenlabsAudio(model, config?)` / `createElevenLabsAudio(model, apiKey, config?)`
+const { voices } = await listVoices({
+ adapter: elevenlabsSpeech("eleven_v4"),
+ origins: ["generated", "cloned"],
+});
-Creates an ElevenLabs audio adapter that covers both music generation and sound effects (selected via the model id) for use with `generateAudio()`.
+console.log(voices.length);
+```
-### `elevenlabsTranscription(model, config?)` / `createElevenLabsTranscription(model, apiKey, config?)`
+`GET /v1/voices` has no origin filter, so the adapter gets all voices and filters them in memory. The `famous` and `high_quality` categories both come back as `professional`.
-Creates an ElevenLabs transcription adapter for use with `generateTranscription()`.
+### Behind a proxy
-### `elevenlabsVoiceDesign(model, config?)` / `createElevenLabsVoiceDesign(model, apiKey, config?)`
+To send each request through a gateway, such as Cloudflare AI Gateway or a company proxy, set `baseURL` and `defaultHeaders`. All TanStack AI adapters use these two option names, so one gateway configuration works for all of them.
-Creates an ElevenLabs voice-design adapter for use with `generateVoice()`.
+```typescript
+import { createElevenLabsSpeech } from "@tanstack/ai-elevenlabs";
-### `elevenlabsSpeech(...).listVoices(options?)`
+const adapter = createElevenLabsSpeech("eleven_v4", process.env.ELEVENLABS_API_KEY!, {
+ baseURL: "https://gateway.example.com/elevenlabs",
+ defaultHeaders: { "cf-aig-authorization": `Bearer ${process.env.GATEWAY_TOKEN}` },
+});
+```
-Reads the account's voice catalog via `GET /v1/voices`, for use with `listVoices()`.
+- This works for the speech, audio, transcription, and voice design adapters.
+- `baseUrl` and `headers` are aliases. If you set both forms, `baseURL` and `defaultHeaders` win.
-## Limitations
+### Limits
-- **No text chat support** -- Use OpenAI, Anthropic, Gemini, or another text adapter for `chat()`.
-- **No summarization** -- Use a text adapter for `summarize()`.
-- **No image input** (realtime) -- ElevenLabs realtime does not support sending images during a conversation.
-- **No runtime session updates** (realtime) -- Session configuration is fixed at connection time.
-- **No time-domain audio data** (realtime) -- Frequency data and volume levels are available, but waveform data is not.
-- **Agent required** (realtime) -- You must create and configure an agent in the ElevenLabs dashboard before using the realtime adapter.
+- Realtime does not take images during a conversation.
+- Realtime fixes the session configuration when the connection starts.
+- Realtime needs an agent that you made in the ElevenLabs dashboard.
-## Next Steps
+### API reference
-- [Realtime Voice Chat Guide](../media/realtime-chat) - Complete guide to building realtime voice applications
-- [OpenAI Adapter](./openai) - Alternative realtime voice provider with WebRTC
-- [Tools Guide](../tools/tools) - Learn about the isomorphic tool system
+| Factory | Use it with |
+| --- | --- |
+| `elevenlabsRealtimeToken(options)` | `realtimeToken()` on the server. Returns a `RealtimeTokenAdapter`. |
+| `elevenlabsRealtime(options?)` | `useRealtimeChat()` or `RealtimeClient` in the browser. Returns a `RealtimeAdapter`. |
+| `elevenlabsSpeech(model, config?)` / `createElevenLabsSpeech(model, apiKey, config?)` | `generateSpeech()` and `listVoices()` |
+| `elevenlabsAudio(model, config?)` / `createElevenLabsAudio(model, apiKey, config?)` | `generateAudio()`, for music and sound effects |
+| `elevenlabsTranscription(model, config?)` / `createElevenLabsTranscription(model, apiKey, config?)` | `generateTranscription()` |
+| `elevenlabsVoiceDesign(model, config?)` / `createElevenLabsVoiceDesign(model, apiKey, config?)` | `generateVoice()` |
+
+The short factories read `ELEVENLABS_API_KEY`. The `create*` factories take an explicit API key.
diff --git a/docs/adapters/fal.md b/docs/adapters/fal.md
index c8fc2503ec..e921c9a51c 100644
--- a/docs/adapters/fal.md
+++ b/docs/adapters/fal.md
@@ -1,7 +1,7 @@
---
title: fal.ai
id: fal-adapter
-description: "Generate images and videos with 600+ models on fal.ai using TanStack AI — Nano Banana Pro, FLUX, and more via the @tanstack/ai-fal adapter."
+description: "Use fal.ai in TanStack AI with @tanstack/ai-fal: 600+ hosted models for images, video, live video, speech, transcription, music, and sound effects."
keywords:
- tanstack ai
- fal.ai
@@ -13,11 +13,11 @@ keywords:
- adapter
---
-The fal.ai adapter provides access to 600+ models on the fal.ai platform for image, video, live, audio, speech, and transcription. Unlike text-focused adapters, the fal adapter is **media-focused**. It supports `generateImage()`, `generateVideo()`, `generateLiveVideo()`, `generateAudio()`, `generateSpeech()`, and `generateTranscription()`. It does not support `chat()` or tools.
+The fal.ai adapter calls hosted models for images, video, live video, speech, transcription, music, and sound effects. It has no text chat and no tools.
-For a full working example, see the [fal.ai example app](https://github.com/TanStack/ai/tree/main/examples/ts-react-media).
+The quick setup below generates an image on your server and shows it in React. For a full app, see the [media example](https://github.com/TanStack/ai/tree/main/examples/ts-react-media).
-## Installation
+## 1. Install
@@ -32,110 +32,99 @@ octane: @tanstack/ai-fal
-## Type Safety with String Literals
+## 2. Set the API key
-The fal adapter provides full type safety when you pass the model ID as a **string literal**. This gives you autocomplete for `size` and `modelOptions` specific to that model. Always use string literals — not variables — when creating adapters:
+Create a key at [fal.ai](https://fal.ai). Put it in the environment of your server:
-```typescript
-import { falImage } from "@tanstack/ai-fal";
-
-// Good — full type safety and autocomplete
-const adapter = falImage("fal-ai/z-image/turbo");
-```
-
-```typescript
-import { falImage } from "@tanstack/ai-fal";
-
-// Bad — no type inference for model-specific options
-const modelId = "fal-ai/z-image/turbo";
-const adapter = falImage(modelId);
+```bash
+FAL_KEY=your-fal-api-key
```
-You can also pass any string for new models that fal.ai hasn't provided types for yet — you just won't get type safety on those endpoints.
-
-## Basic Usage
+All fal adapters read `FAL_KEY`. To pass the key in code, set `apiKey` in the config:
```typescript
-import { generateImage } from "@tanstack/ai";
import { falImage } from "@tanstack/ai-fal";
-const result = await generateImage({
- adapter: falImage("fal-ai/flux/dev"),
- prompt: "A futuristic cityscape at sunset",
- numberOfImages: 1,
+const adapter = falImage("fal-ai/nano-banana-pro", {
+ apiKey: process.env.FAL_KEY!,
});
-
-console.log(result.images);
```
-## Basic Usage - Custom API Key
+## 3. Generate an image on the server
+
+Add a `POST` route at `/api/generate/image`:
```typescript
-import { generateImage } from "@tanstack/ai";
+import {
+ generateImage,
+ generationParamsFromRequest,
+ toServerSentEventsResponse,
+} from "@tanstack/ai";
import { falImage } from "@tanstack/ai-fal";
-const adapter = falImage("fal-ai/flux/dev", {
- apiKey: process.env.FAL_KEY!,
-});
-
-const result = await generateImage({
- adapter,
- prompt: "A futuristic cityscape at sunset",
- numberOfImages: 1,
-});
-```
-
-## Configuration
-
-```typescript
-import { falImage, type FalClientConfig } from "@tanstack/ai-fal";
+export async function POST(request: Request) {
+ const { input, threadId } = await generationParamsFromRequest("image", request);
+ if (typeof input.prompt !== "string") {
+ return new Response("Send a text prompt.", { status: 400 });
+ }
-// Direct API key
-const adapter = falImage("fal-ai/flux/dev", {
- apiKey: "your-api-key",
-});
+ const stream = generateImage({
+ adapter: falImage("fal-ai/nano-banana-pro"),
+ prompt: input.prompt,
+ threadId,
+ stream: true,
+ });
-// Using a proxy URL (for client-side usage)
-const proxiedAdapter = falImage("fal-ai/flux/dev", {
- apiKey: "your-api-key",
- proxyUrl: "https://your-server.com/api/fal/proxy",
-});
+ return toServerSentEventsResponse(stream);
+}
```
-## Example: Image Generation
-
-From the [fal.ai example app](https://github.com/TanStack/ai/tree/main/examples/ts-react-media):
-
-```typescript
-import { generateImage } from "@tanstack/ai";
-import { falImage } from "@tanstack/ai-fal";
-
-// Use string literals for the model to get full type safety
-const result = await generateImage({
- adapter: falImage("fal-ai/nano-banana-pro"),
- prompt: "A futuristic cityscape at sunset",
- numberOfImages: 1,
- size: "16:9_4K",
- modelOptions: {
- output_format: "jpeg",
- },
-});
+## 4. Show the image
+
+```tsx
+import { fetchServerSentEvents, useGenerateImage } from "@tanstack/ai-react";
+
+export function ImageGenerator() {
+ const { generate, result, isLoading, error } = useGenerateImage({
+ connection: fetchServerSentEvents("/api/generate/image"),
+ threadId: "image-studio",
+ });
+
+ return (
+ Status: {status}
-Mode: {mode}
-
+
+ {error && 
+ ))}
+
+ );
+}
```
-## Example: Image with Model Options
+Click the button. The image shows up below it. For the other frameworks, see [Generation Hooks](../media/generation-hooks).
-Each fal.ai model has its own type-safe options. The adapter uses fal.ai's types to provide autocomplete and type checking for model-specific parameters. There are 1000s of combinations. If you provide the proper model endpoint id as as string literal, you will then only be able to provide `size` and `modelOptions` that the model supports.
+## Model options
-_IMPORTANT_: It is possible to pass strings and new endpoint ids that Fal has not provided types for. You just won't get type safety on those endpoints. You'll find this happens for very new models.
+Each fal model has its own options. Write the model id as a string literal, and TypeScript completes `size` and `modelOptions` for that model:
```typescript
import { generateImage } from "@tanstack/ai";
import { falImage } from "@tanstack/ai-fal";
-// Model-specific options are type-safe
const result = await generateImage({
adapter: falImage("fal-ai/z-image/turbo"),
prompt: "A serene mountain landscape",
+ threadId: "thread-1",
numberOfImages: 1,
size: "landscape_16_9",
modelOptions: {
@@ -143,69 +132,41 @@ const result = await generateImage({
enable_prompt_expansion: true,
},
});
-```
-
-## Image Size Options
-
-The fal adapter supports a flexible `size` parameter that maps either to `image_size` or to `aspect_ratio` and `resolution` parameters:
-
-| | `size` | Maps To |
-|--------|---------|---------|
-| named | `"landscape_16_9"` | `image_size: "landscape_16_9"` |
-| width x height (OpenAI) | `"1536x1024"` | | `image_size: "1536x1024"` |
-| aspect ratio & resolution | `"16:9_4K"` | `aspect_ratio: "16:9"`, `resolution: "4K"` |
-| aspect ratio only | `"16:9"` | `aspect_ratio: "16:9"` |
-
-```typescript ignore
-// Aspect ratio only
-size: "16:9"
-
-// Aspect ratio with resolution
-size: "16:9_4K"
-// Named size (model-specific)
-size: "landscape_16_9"
+console.log(result.images);
```
-## Video Generation (Experimental)
-
-> **Note:** Video generation is an experimental feature and may change in future releases.
+- Pass the id directly, not through a `string` variable. A variable loses the types of the model.
+- Any string works as a model id. A model that is newer than the fal SDK types runs, but has no autocomplete.
-Video generation uses a queue-based workflow: submit a job, poll for status, then retrieve the video URL when complete.
+For the `size` formats, see [Image sizes](#image-sizes).
-```typescript
-import { generateVideo, getVideoJobStatus } from "@tanstack/ai";
-import { falVideo } from "@tanstack/ai-fal";
-```
+## Video
-## Example: Text-to-Video
+> Video generation is an [experimental feature](../media/video-generation).
-From the [fal.ai example app](https://github.com/TanStack/ai/tree/main/examples/ts-react-media):
+Video runs as a job. `generateVideo()` starts the job, and `getVideoJobStatus()` reads its status:
```typescript
import { generateVideo, getVideoJobStatus } from "@tanstack/ai";
import { falVideo } from "@tanstack/ai-fal";
-// 1. Submit the video generation job
const adapter = falVideo("fal-ai/kling-video/v2.6/pro/text-to-video");
const job = await generateVideo({
adapter,
prompt: "A timelapse of a flower blooming",
+ threadId: "thread-1",
size: "16:9",
duration: "5",
});
-// 2. Poll for status
-const status = await getVideoJobStatus({
- adapter,
- jobId: job.jobId,
-});
+const status = await getVideoJobStatus({ adapter, jobId: job.jobId });
console.log(status.status); // "pending" | "processing" | "completed"
```
-## Example: Image-to-Video
+To animate an image, use an image-to-video model and set the start image in `modelOptions`:
```typescript
import { generateVideo } from "@tanstack/ai";
@@ -214,68 +175,26 @@ import { falVideo } from "@tanstack/ai-fal";
const job = await generateVideo({
adapter: falVideo("fal-ai/kling-video/v2.6/pro/image-to-video"),
prompt: "Animate this scene with gentle wind",
+ threadId: "thread-1",
duration: "5",
modelOptions: {
start_image_url: "https://example.com/image.jpg",
generate_audio: true,
},
});
-```
-
-`duration` is typed per model from `@fal-ai/client`'s `EndpointTypeMap`. Popular models also implement `availableDurations()` / `snapDuration()` for UI sliders:
-
-| Model | `duration` type | `availableDurations()` |
-| --- | --- | --- |
-| `fal-ai/kling-video/v1.6/{standard,pro}/text-to-video`, `fal-ai/kling-video/v2.6/pro/{text,image}-to-video` | `'5' \| '10'` | discrete |
-| `fal-ai/kling-video/v3/pro/{text,image}-to-video` | `'3'` … `'15'` | discrete |
-| `fal-ai/pika/v2.2/text-to-video` | `'5' \| '10'` | discrete |
-| `fal-ai/ltx-2.3/{text,image}-to-video` (+ `/fast`) | `'6' \| '8' \| '10'` | discrete |
-| `fal-ai/luma-dream-machine/ray-2` | `'5s' \| '9s'` | discrete |
-| `fal-ai/veo3.1`, `fal-ai/veo3.1/fast` (+ `/image-to-video`), `fal-ai/veo3` (+ `/image-to-video`) | `'4s' \| '6s' \| '8s'` | discrete |
-| `fal-ai/wan-25-preview/text-to-video` | `'2'` … `'15'` | discrete |
-| `fal-ai/minimax/video-01` | not accepted | `{ kind: 'none' }` |
-| `fal-ai/hunyuan-video-v1.5/text-to-video` | not accepted (`num_frames`) | `{ kind: 'none' }` |
-
-```typescript
-import { generateVideo } from "@tanstack/ai";
-import { falVideo } from "@tanstack/ai-fal";
-const adapter = falVideo("fal-ai/veo3.1");
-adapter.availableDurations(); // { kind: 'discrete', values: ['4s', '6s', '8s'] }
-adapter.snapDuration(7); // '6s'
-
-await generateVideo({
- adapter,
- prompt: "A timelapse of a city skyline at dusk",
- duration: adapter.snapDuration(7),
-});
+console.log(job.jobId);
```
-Uncurated models still type `duration` from the SDK when the endpoint declares the field, but `availableDurations()` returns `{ kind: 'none' }` until they are added to the runtime map.
-
-## Text-to-Speech
-
-Text-to-speech uses `falSpeech()` with the `generateSpeech()` activity. The adapter fetches the generated audio from fal's CDN and returns it as base64-encoded data to match the `TTSResult` contract.
+The `duration` values of each model are in [Video durations](#video-durations). To poll from a React component, see [Video Generation](../media/video-generation).
-```typescript
-import { generateSpeech } from "@tanstack/ai";
-import { falSpeech } from "@tanstack/ai-fal";
-
-const result = await generateSpeech({
- adapter: falSpeech("fal-ai/kokoro/american-english"),
- text: "Hello from fal!",
- voice: "af_heart",
- speed: 1.0,
-});
+## Live video
-// result.audio is a base64-encoded string
-console.log(result.format); // e.g. "wav"
-console.log(result.contentType); // e.g. "audio/wav"
-```
+`falLiveVideo("minimax/h3-max/director")` starts a live video stream that you can steer. The browser connects through a proxy on your server. See [Live Generation](../media/live-generation#fal-h3-max-director).
-### Google Gemini 3.1 Flash TTS
+## Speech
-Google's newest TTS model (`fal-ai/gemini-3.1-flash-tts`) supports 80+ languages and introduces **granular audio tags** for expressive control — you can embed speaker tags and style cues directly in the text.
+`falSpeech` gets the audio from the fal CDN and returns it as base64:
```typescript
import { generateSpeech } from "@tanstack/ai";
@@ -284,13 +203,16 @@ import { falSpeech } from "@tanstack/ai-fal";
const result = await generateSpeech({
adapter: falSpeech("fal-ai/gemini-3.1-flash-tts"),
text: "[warm, enthusiastic] Welcome to TanStack AI! [pause] Let's build something great.",
+ threadId: "thread-1",
voice: "Kore",
});
+
+console.log(result.format, result.contentType);
```
-> **Note:** This model is newer than `@fal-ai/client@1.9.1`'s type map, so `modelOptions` won't autocomplete. The call still works — the fal adapter accepts any model ID as a string. Type-safe autocomplete will land when fal's SDK types catch up.
+`fal-ai/gemini-3.1-flash-tts` is the TTS model of Google. It supports 80+ languages. You can put speaker tags and style tags, such as `[pause]`, in the text. Its `modelOptions` have no autocomplete yet.
-### ElevenLabs v3
+Other models put the voice in `modelOptions`:
```typescript
import { generateSpeech } from "@tanstack/ai";
@@ -299,16 +221,19 @@ import { falSpeech } from "@tanstack/ai-fal";
const result = await generateSpeech({
adapter: falSpeech("fal-ai/elevenlabs/tts/eleven-v3"),
text: "Welcome to TanStack AI.",
+ threadId: "thread-1",
modelOptions: {
voice: "Rachel",
stability: 0.5,
},
});
+
+console.log(result.audio); // Base64-encoded audio
```
-## Transcription
+See [Text-to-Speech](../media/text-to-speech).
-Speech-to-text uses `falTranscription()` with the `generateTranscription()` activity. The `audio` input accepts a URL string, `Blob`, `File`, or `ArrayBuffer` — `ArrayBuffer` is automatically wrapped in a `Blob` for upload.
+## Transcription
```typescript
import { generateTranscription } from "@tanstack/ai";
@@ -318,234 +243,232 @@ const result = await generateTranscription({
adapter: falTranscription("fal-ai/whisper"),
audio: "https://example.com/recording.mp3",
language: "en",
+ threadId: "thread-1",
});
console.log(result.text);
console.log(result.language);
-// When the model returns word/segment timestamps, they're mapped to result.segments
+// When the model returns timestamps, they are in result.segments
for (const segment of result.segments ?? []) {
- console.log(`[${segment.start}s → ${segment.end}s] ${segment.text}`);
+ console.log(`[${segment.start}s to ${segment.end}s] ${segment.text}`);
}
```
-## Audio Generation (Music & Sound Effects)
+`audio` takes a URL string, a `Blob`, a `File`, or an `ArrayBuffer`. The adapter wraps an `ArrayBuffer` in a `Blob` for the upload.
+
+## Music and sound effects
-Music and sound-effect generation uses `falAudio()` with the `generateAudio()` activity. Unlike TTS, the result is returned as a URL in `result.audio.url` (you can fetch it yourself if you need raw bytes).
+`falAudio` returns a URL in `result.audio.url`. If you need the bytes, fetch the URL:
```typescript
import { generateAudio } from "@tanstack/ai";
import { falAudio } from "@tanstack/ai-fal";
-// Music generation with MiniMax Music 2.6 (latest)
const music = await generateAudio({
adapter: falAudio("fal-ai/minimax-music/v2.6"),
prompt: "City Pop, 80s retro, groovy synth bass, warm female vocal, 104 BPM, nostalgic urban night",
+ threadId: "thread-1",
+});
+
+const sfx = await generateAudio({
+ adapter: falAudio("fal-ai/elevenlabs/sound-effects/v2"),
+ prompt: "Thunderclap with rain",
+ threadId: "thread-1",
+ duration: 5,
});
-console.log(music.audio.url);
+console.log(music.audio.url, sfx.audio.url);
```
+Some music models take lyrics in `modelOptions`:
+
```typescript
import { generateAudio } from "@tanstack/ai";
import { falAudio } from "@tanstack/ai-fal";
-// DiffRhythm with explicit lyrics
-const lyrical = await generateAudio({
+const song = await generateAudio({
adapter: falAudio("fal-ai/diffrhythm"),
prompt: "An upbeat electronic track with synths",
+ threadId: "thread-1",
modelOptions: {
lyrics: "[verse]\nHello world\n[chorus]\nLa la la",
},
});
+
+console.log(song.audio.url);
```
-```typescript
-import { generateAudio } from "@tanstack/ai";
-import { falAudio } from "@tanstack/ai-fal";
+See [Audio Generation](../media/audio-generation).
-// Sound effects
-const sfx = await generateAudio({
- adapter: falAudio("fal-ai/elevenlabs/sound-effects/v2"),
- prompt: "Thunderclap with rain",
- duration: 5,
-});
-```
+## Files
+
+`falFiles` uploads media to fal storage, so a request can point to the file by URL. fal storage is upload-only. See [Files API](../advanced/files-api).
-## Popular Models
+## Models and options
-### Image Models
+fal has hundreds of models. These are popular picks. Any fal model id works.
+
+### Image models
| Model | Description |
|-------|-------------|
-| `fal-ai/nano-banana-pro` | Fast, high-quality image generation (4K) |
-| `fal-ai/flux-2/klein/9b` | Enhanced realism, crisp text generation |
-| `fal-ai/z-image/turbo` | Super fast 6B parameter model |
-| `xai/grok-imagine-image` | xAI highly aesthetic images with prompt enhancement |
+| `fal-ai/nano-banana-pro` | Fast, high-quality images, up to 4K |
+| `fal-ai/flux-2/klein/9b` | Realistic images, clear text |
+| `fal-ai/z-image/turbo` | A very fast 6B-parameter model |
+| `xai/grok-imagine-image` | xAI images with prompt enhancement |
-### Video Models
+### Video models
| Model | Mode | Description |
|-------|------|-------------|
-| `fal-ai/kling-video/v2.6/pro/text-to-video` | Text-to-Video | High-quality text-to-video |
-| `fal-ai/kling-video/v2.6/pro/image-to-video` | Image-to-Video | Animate images with Kling |
-| `fal-ai/veo3.1` | Text-to-Video | Google Veo text-to-video |
-| `fal-ai/veo3.1/image-to-video` | Image-to-Video | Google Veo image-to-video |
-| `xai/grok-imagine-video/text-to-video` | Text-to-Video | xAI video from text |
-| `xai/grok-imagine-video/image-to-video` | Image-to-Video | xAI animate images to video |
-| `fal-ai/ltx-2/text-to-video/fast` | Text-to-Video | Fast text-to-video |
-| `fal-ai/ltx-2/image-to-video/fast` | Image-to-Video | Fast image-to-video animation |
+| `fal-ai/kling-video/v2.6/pro/text-to-video` | Text-to-video | High-quality text-to-video |
+| `fal-ai/kling-video/v2.6/pro/image-to-video` | Image-to-video | Animate images with Kling |
+| `fal-ai/veo3.1` | Text-to-video | Google Veo text-to-video |
+| `fal-ai/veo3.1/image-to-video` | Image-to-video | Google Veo image-to-video |
+| `xai/grok-imagine-video/text-to-video` | Text-to-video | xAI video from text |
+| `xai/grok-imagine-video/image-to-video` | Image-to-video | xAI video from an image |
+| `fal-ai/ltx-2/text-to-video/fast` | Text-to-video | Fast text-to-video |
+| `fal-ai/ltx-2/image-to-video/fast` | Image-to-video | Fast image-to-video |
-### Live Models
+### Live models
-| Model | Mode | Description |
-|-------|------|-------------|
-| `minimax/h3-max/director` | Live | Steerable live stream. Use `falLiveVideo()` with `generateLiveVideo()`. |
+| Model | Description |
+|-------|-------------|
+| `minimax/h3-max/director` | A live stream that you can steer. Use `falLiveVideo()` with `generateLiveVideo()`. |
-### Text-to-Speech Models
+### Text-to-speech models
| Model | Description |
|-------|-------------|
-| `fal-ai/gemini-3.1-flash-tts` | **New** — Google's flagship TTS with 80+ languages and expressive audio tags |
-| `fal-ai/elevenlabs/tts/eleven-v3` | ElevenLabs v3 expressive multi-voice TTS |
-| `fal-ai/elevenlabs/tts/turbo-v2.5` | Low-latency ElevenLabs TTS |
-| `fal-ai/minimax/speech-2.6-hd` | MiniMax HD speech synthesis |
-| `fal-ai/minimax/speech-2.6-turbo` | MiniMax low-latency variant |
-| `fal-ai/kokoro/american-english` | Kokoro multilingual TTS — also `british-english`, `french`, `spanish`, `italian`, `japanese`, `mandarin-chinese`, `hindi`, `brazilian-portuguese` |
+| `fal-ai/gemini-3.1-flash-tts` | Google TTS with 80+ languages and style tags |
+| `fal-ai/elevenlabs/tts/eleven-v3` | ElevenLabs v3, expressive, several voices |
+| `fal-ai/elevenlabs/tts/turbo-v2.5` | ElevenLabs, low latency |
+| `fal-ai/minimax/speech-2.6-hd` | MiniMax HD speech |
+| `fal-ai/minimax/speech-2.6-turbo` | MiniMax, low latency |
+| `fal-ai/kokoro/american-english` | Kokoro. Also `british-english`, `french`, `spanish`, `italian`, `japanese`, `mandarin-chinese`, `hindi`, `brazilian-portuguese` |
| `fal-ai/inworld-tts` | Inworld TTS-1.5 Max |
-| `fal-ai/chatterbox/text-to-speech/multilingual` | Chatterbox multilingual TTS |
-| `fal-ai/dia-tts` | Dia expressive dialogue TTS |
-| `fal-ai/orpheus-tts` | Orpheus open-source TTS |
-| `fal-ai/f5-tts` | F5-TTS voice cloning |
-| `fal-ai/vibevoice/7b` | VibeVoice 7B conversational TTS |
+| `fal-ai/chatterbox/text-to-speech/multilingual` | Chatterbox, many languages |
+| `fal-ai/dia-tts` | Dia, expressive dialogue |
+| `fal-ai/orpheus-tts` | Orpheus, open source |
+| `fal-ai/f5-tts` | F5-TTS, voice cloning |
+| `fal-ai/vibevoice/7b` | VibeVoice 7B, conversation |
-### Transcription Models
+### Transcription models
| Model | Description |
|-------|-------------|
-| `fal-ai/whisper` | OpenAI Whisper on fal infra |
-| `fal-ai/wizper` | Faster-whisper variant with word-level timestamps |
-| `fal-ai/speech-to-text/turbo` | Turbo STT with diarization |
-| `fal-ai/elevenlabs/speech-to-text` | ElevenLabs STT |
+| `fal-ai/whisper` | OpenAI Whisper on fal |
+| `fal-ai/wizper` | A faster Whisper with word timestamps |
+| `fal-ai/speech-to-text/turbo` | Fast speech-to-text with speaker labels |
+| `fal-ai/elevenlabs/speech-to-text` | ElevenLabs speech-to-text |
-### Audio / Music Models
+### Music and audio models
| Model | Mode | Description |
|-------|------|-------------|
-| `fal-ai/minimax-music/v2.6` | Music | **New** — MiniMax Music 2.6, full vocal + instrumental compositions from a prompt |
+| `fal-ai/minimax-music/v2.6` | Music | MiniMax Music 2.6, vocals and instruments from a prompt |
| `fal-ai/minimax-music/v2.5` | Music | MiniMax Music 2.5 |
-| `fal-ai/minimax-music/v2` | Music | MiniMax Music v2 — supports `lyrics_prompt` |
-| `fal-ai/diffrhythm` | Music | DiffRhythm — prompt + lyrics |
-| `fal-ai/lyria2` | Music | Google Lyria 2 high-fidelity music |
-| `fal-ai/stable-audio-25/text-to-audio` | Music / Audio | Stability AI Stable Audio 2.5 |
-| `fal-ai/mmaudio-v2/text-to-audio` | Audio | MMAudio v2 text-to-audio |
-| `fal-ai/elevenlabs/sound-effects/v2` | SFX | ElevenLabs sound-effect generation |
-| `fal-ai/beatoven/sound-effect-generation` | SFX | Beatoven professional sound effects |
-| `fal-ai/thinksound` | Audio | Thinksound reasoning-based audio generation |
+| `fal-ai/minimax-music/v2` | Music | MiniMax Music v2, takes `lyrics_prompt` |
+| `fal-ai/diffrhythm` | Music | DiffRhythm, prompt and lyrics |
+| `fal-ai/lyria2` | Music | Google Lyria 2 |
+| `fal-ai/stable-audio-25/text-to-audio` | Music or audio | Stability AI Stable Audio 2.5 |
+| `fal-ai/mmaudio-v2/text-to-audio` | Audio | MMAudio v2 |
+| `fal-ai/elevenlabs/sound-effects/v2` | Sound effects | ElevenLabs sound effects |
+| `fal-ai/beatoven/sound-effect-generation` | Sound effects | Beatoven sound effects |
+| `fal-ai/thinksound` | Audio | Thinksound audio |
-> **Very new models** (e.g. `gemini-3.1-flash-tts`, `minimax-music/v2.6`, `beatoven/sound-effect-generation`) may not yet appear in `@fal-ai/client`'s type map — they still work as plain string model IDs, you just won't get autocomplete for their `modelOptions`.
+Some very new models, for example `fal-ai/gemini-3.1-flash-tts`, `fal-ai/minimax-music/v2.6`, and `fal-ai/beatoven/sound-effect-generation`, are not in the type map of `@fal-ai/client` yet. They work as plain string ids, with no autocomplete for `modelOptions`.
-## Environment Variables
-
-Create an API key at [fal.ai](https://fal.ai) and set it in your environment:
-
-```bash
-FAL_KEY=your-fal-api-key
-```
+## Advanced
-## API Reference
+### Image sizes
-### `falImage(model, config?)`
+`size` maps to `image_size`, or to `aspect_ratio` and `resolution`:
-Creates a fal.ai image adapter using the `FAL_KEY` environment variable or an explicit config.
-
-**Parameters:**
-
-- `model` - The fal.ai model ID (e.g., `"fal-ai/flux/dev"`)
-- `config.apiKey?` - Your fal.ai API key (falls back to `FAL_KEY` env var)
-- `config.proxyUrl?` - Proxy URL for client-side usage
-
-**Returns:** A `FalImageAdapter` instance for use with `generateImage()`.
-
-### `falVideo(model, config?)`
-
-Creates a fal.ai video adapter using the `FAL_KEY` environment variable or an explicit config.
-
-**Parameters:**
-
-- `model` - The fal.ai model ID (e.g., `"fal-ai/kling-video/v2.6/pro/text-to-video"`)
-- `config.apiKey?` - Your fal.ai API key (falls back to `FAL_KEY` env var)
-- `config.proxyUrl?` - Proxy URL for client-side usage
-
-**Returns:** A `FalVideoAdapter` instance for use with `generateVideo()` and `getVideoJobStatus()`.
-
-### `falLiveVideo(model, config?)`
-
-Creates a fal.ai live-video adapter for H3 Max Director. `generateLiveVideo()` returns the WMA app id on `result.model` (`fal-ai/minimax-h3-max-director`). Open that id with `wma(live.model)` through a server proxy that attaches `FAL_KEY`. Do not send `live.token` as `Key` credentials. Call `allowedFalLiveVideoProxyTarget()` in the proxy so it forwards only WMA `/ice`, `/session`, `/session/heartbeat`, and Director `/ice`.
-
-**Parameters:**
-
-- `model` - `"minimax/h3-max/director"`
-- `config.apiKey?` - Your fal.ai API key (falls back to `FAL_KEY` env var)
-
-**Returns:** A `FalLiveVideoAdapter` instance for use with `generateLiveVideo()`.
-
-See [Live Generation](../media/live-generation) for the browser connect step.
-
-### `falSpeech(model, config?)`
-
-Creates a fal.ai text-to-speech adapter.
-
-**Parameters:**
-
-- `model` - The fal.ai TTS model ID (e.g., `"fal-ai/kokoro/american-english"`)
-- `config.apiKey?` - Your fal.ai API key (falls back to `FAL_KEY` env var)
-- `config.proxyUrl?` - Proxy URL for client-side usage
-
-**Returns:** A `FalSpeechAdapter` instance for use with `generateSpeech()`. The adapter fetches the generated audio URL from fal and returns it as base64 in `result.audio`.
+| Form | `size` | Sent to fal as |
+|--------|---------|---------|
+| Named | `"landscape_16_9"` | `image_size: "landscape_16_9"` |
+| Width x height | `"1536x1024"` | `image_size: "1536x1024"` |
+| Ratio and resolution | `"16:9_4K"` | `aspect_ratio: "16:9"`, `resolution: "4K"` |
+| Ratio only | `"16:9"` | `aspect_ratio: "16:9"` |
-### `falTranscription(model, config?)`
+Named sizes change by model.
-Creates a fal.ai transcription (speech-to-text) adapter.
+### Video durations
-**Parameters:**
+`duration` gets its type for each model from the `EndpointTypeMap` of `@fal-ai/client`. Popular models also have `availableDurations()` and `snapDuration()`, for a UI slider:
-- `model` - The fal.ai STT model ID (e.g., `"fal-ai/whisper"`)
-- `config.apiKey?` - Your fal.ai API key (falls back to `FAL_KEY` env var)
-- `config.proxyUrl?` - Proxy URL for client-side usage
+| Model | `duration` type | `availableDurations()` |
+| --- | --- | --- |
+| `fal-ai/kling-video/v1.6/{standard,pro}/text-to-video`, `fal-ai/kling-video/v2.6/pro/{text,image}-to-video` | `"5"` or `"10"` | discrete |
+| `fal-ai/kling-video/v3/pro/{text,image}-to-video` | `"3"` to `"15"` | discrete |
+| `fal-ai/pika/v2.2/text-to-video` | `"5"` or `"10"` | discrete |
+| `fal-ai/ltx-2.3/{text,image}-to-video` (and `/fast`) | `"6"`, `"8"`, or `"10"` | discrete |
+| `fal-ai/luma-dream-machine/ray-2` | `"5s"` or `"9s"` | discrete |
+| `fal-ai/veo3.1`, `fal-ai/veo3.1/fast` (and `/image-to-video`), `fal-ai/veo3` (and `/image-to-video`) | `"4s"`, `"6s"`, or `"8s"` | discrete |
+| `fal-ai/wan-25-preview/text-to-video` | `"2"` to `"15"` | discrete |
+| `fal-ai/minimax/video-01` | not accepted | `{ kind: "none" }` |
+| `fal-ai/hunyuan-video-v1.5/text-to-video` | not accepted (uses `num_frames`) | `{ kind: "none" }` |
-**Returns:** A `FalTranscriptionAdapter` instance for use with `generateTranscription()`.
+```typescript
+import { generateVideo } from "@tanstack/ai";
+import { falVideo } from "@tanstack/ai-fal";
-### `falAudio(model, config?)`
+const adapter = falVideo("fal-ai/veo3.1");
+adapter.availableDurations(); // { kind: "discrete", values: ["4s", "6s", "8s"] }
+adapter.snapDuration(7); // "6s"
-Creates a fal.ai audio generation adapter (music and sound effects).
+await generateVideo({
+ adapter,
+ prompt: "A timelapse of a city skyline at dusk",
+ threadId: "thread-1",
+ duration: adapter.snapDuration(7),
+});
+```
-**Parameters:**
+Other models still get the `duration` type from the SDK when the endpoint has the field. For these models, `availableDurations()` returns `{ kind: "none" }`.
-- `model` - The fal.ai audio model ID (e.g., `"fal-ai/diffrhythm"`, `"fal-ai/minimax-music/v2"`)
-- `config.apiKey?` - Your fal.ai API key (falls back to `FAL_KEY` env var)
-- `config.proxyUrl?` - Proxy URL for client-side usage
+### Configuration
-**Returns:** A `FalAudioAdapter` instance for use with `generateAudio()`. The result contains a URL at `result.audio.url`.
+Each factory takes an optional `FalClientConfig`:
-### `getFalApiKeyFromEnv()`
+- `apiKey`: your fal key. If you do not set a config, the adapter reads `FAL_KEY`.
+- `proxyUrl`: a proxy URL on your server, for calls from the browser.
+- `fetch`: a custom `fetch` for the fal requests. The adapter wraps it to read the billed units.
-Reads the `FAL_KEY` environment variable. Throws if not set.
+```typescript
+import { falImage } from "@tanstack/ai-fal";
-**Returns:** The API key string.
+const adapter = falImage("fal-ai/nano-banana-pro", {
+ apiKey: process.env.FAL_KEY!,
+ proxyUrl: "https://your-server.com/api/fal/proxy",
+});
+```
-### `configureFalClient(config?)`
+### Live video proxy
-Configures the underlying `@fal-ai/client`. Called automatically by adapter constructors. Uses `proxyUrl` if provided, otherwise sets credentials from the API key.
+`generateLiveVideo()` returns the WMA app id on `result.model` (`fal-ai/minimax-h3-max-director`). In the browser:
-## Limitations
+- Open that id with `wma(live.model)` through a proxy on your server. The proxy attaches `FAL_KEY`.
+- Do not send `live.token` as `Key` credentials.
+- In the proxy, call `allowedFalLiveVideoProxyTarget()`. It lets the proxy forward only to WMA `/ice`, `/session`, `/session/heartbeat`, and Director `/ice`.
-- **No text/chat support** — Use OpenAI, Anthropic, Gemini, or another text adapter for `chat()`
-- **No tools support** — Tool definitions are not applicable to media generation
-- **No summarization** — Use a text adapter for `summarize()`
-- **Video is experimental** — The video generation API may change in future releases
+### API reference
-## Next Steps
+Each factory takes `model` first and an optional `config` (`apiKey`, `proxyUrl`, `fetch`). Without `apiKey`, it reads `FAL_KEY`.
-- [Getting Started](../getting-started/quick-start) - Learn the basics
-- [Other Adapters](./openai) - Explore other providers
+| Factory | Returns | Use it with |
+| --- | --- | --- |
+| `falImage(model, config?)` | `FalImageAdapter` | `generateImage()` |
+| `falVideo(model, config?)` | `FalVideoAdapter` | `generateVideo()`, `getVideoJobStatus()` |
+| `falLiveVideo(model, config?)` / `createFalLiveVideo(model, apiKey, config?)` | `FalLiveVideoAdapter` | `generateLiveVideo()` |
+| `falSpeech(model, config?)` | `FalSpeechAdapter` | `generateSpeech()`. The audio comes back as base64 in `result.audio`. |
+| `falTranscription(model, config?)` | `FalTranscriptionAdapter` | `generateTranscription()` |
+| `falAudio(model, config?)` | `FalAudioAdapter` | `generateAudio()`. The result has a URL at `result.audio.url`. |
+| `falFiles(config?)` | `FalFilesAdapter` | The [Files API](../advanced/files-api) |
+
+Helpers:
+
+- `getFalApiKeyFromEnv()`: reads `FAL_KEY`. Throws if it is not set.
+- `configureFalClient(config?)`: configures `@fal-ai/client`. The adapter constructors call it for you. It sets the credentials from the API key, and the proxy when you set `proxyUrl`.
diff --git a/docs/adapters/gemini-advanced.md b/docs/adapters/gemini-advanced.md
new file mode 100644
index 0000000000..7edc6d5f3f
--- /dev/null
+++ b/docs/adapters/gemini-advanced.md
@@ -0,0 +1,663 @@
+---
+title: "Google Gemini: Advanced"
+id: gemini-advanced
+order: 3
+description: "Advanced Gemini setup in TanStack AI: proxies, the experimental Interactions API, image sizes and edits, video understanding options, two-voice speech, every provider tool, and the API reference."
+keywords:
+ - tanstack ai
+ - gemini
+ - interactions api
+ - nano banana
+ - imagen
+ - provider tools
+ - adapter
+---
+
+This page has the details of the Gemini adapter. To get a first chat working, start at [Google Gemini](./gemini).
+
+## Configuration
+
+```typescript
+import { createGeminiChat, type GeminiTextConfig } from "@tanstack/ai-gemini";
+
+const config: Omit{error.message}
} + {result?.images.map((image, index) => ( +The agent is inspecting the repo.
- if (!final) return null - return{final.summary}
+ if (isLoading) returnThe agent is inspecting the repo.
; + if (!final) return null; + return{final.summary}
; } ``` -Full walkthrough, including the client: [Harness Agents](../structured-outputs/harnesses). +- Grok Build has no schema flag. The adapter adds the JSON Schema to the prompt, on both protocols. +- The adapter parses the last assistant text. Tool activity still streams. +- The object arrives as `structured-output.complete`. + +Do you only need JSON from a prompt, with no repo? Use `@tanstack/ai-grok`. For the full walkthrough, see [Harness Agents](../structured-outputs/harnesses). + +## Models and options + +### Models + +| Model id | Notes | +| --- | --- | +| `grok-build` | The short alias. With a grok.com browser login, the CLI lists the model under this name. | +| `grok-build-0.1` | The full id. The CLI lists the model under this name when it signs in with `XAI_API_KEY`. | +| `composer-2.5` | Also runs through the Grok Build harness. | + +The adapter changes `grok-build` to `grok-build-0.1` for the CLI, so the same code works with both ways to sign in. The package exports the list as `GROK_BUILD_MODELS`. The adapter also takes any other xAI model id as a string. + +### Adapter options + +Pass these as the second argument of `grokBuildText`: + +| Option | What it does | +| --- | --- | +| `cwd` | The working directory inside the sandbox. Default `/workspace`. | +| `authMode` | `'api-key'` (default) signs in with `XAI_API_KEY`. `'host'` uses `grok login`. See [Harness Auth](../sandbox/auth). | +| `authMethodId` | An explicit ACP auth method. It wins over `authMode`. | +| `protocol` | `'acp'` (default) or `'streaming-json'`. See [Protocols and durable runs](#protocols-and-durable-runs). | +| `transport` | The ACP transport: `'auto'` (default), `'stdio'`, or `'websocket'`. | +| `acpPort` | The port of `grok agent serve` in the sandbox, for the WebSocket transport. | +| `permissionMode` | The ACP permission policy: `'default'`, `'acceptEdits'`, or `'bypassPermissions'` (default). | +| `emitDiff` | Send the `file.changed` event with the `git diff` after the run. Default `true`. | +| `grokExecutable` | The path or name of the `grok` executable in the sandbox. Default `grok`. | +| `env` | More environment variables for the `grok` process. | +| `extraArgs` | More raw CLI flags. The adapter adds them as they are. | + +### Per-call options + +Pass these in `modelOptions` on `chat()`. They override the adapter options for one call: + +- `sessionId`: resume a session. See [Resume a session](#resume-a-session). +- `maxTurns`: the maximum number of agent turns for this call. +- `cwd`, `protocol`, `transport`, `acpPort`, `permissionMode`. +- `authMode` and `authMethodId`. + +## Advanced + +### Protocols and durable runs + +The adapter talks to `grok` over one of two protocols: + +- `'acp'` (default): the Agent Client Protocol. +- `'streaming-json'`: the headless NDJSON output of the CLI. + +A durable sandbox run (you pass `runs` and `durability` to `withSandbox`) writes the agent output to a journal, so a later request can take the run over. ACP does not write a journal. So when durability is wired and you do not set `protocol`, the adapter uses `'streaming-json'`. + +Set `protocol: 'acp'` only when you want ACP on that call: + +- A new ACP run with durability wired logs a warning. +- An attach to an ACP run throws `DurableAttachNotSupportedError`. + +See [Durable Runs](../sandbox/durable-runs) and [The Run Journal](../sandbox/journal). + +### Headless NDJSON flags + +On the `'streaming-json'` path with auto-approve, the adapter adds `--always-approve --no-plan --no-auto-update`. These flags stop Plan Mode and the CLI update check from blocking a headless run. -## Limitations +### Limitations -- **Requires a sandbox.** Always run it under `withSandbox(...)`; see the - [Sandboxes overview](../sandbox/overview). -- **Server-only (Node).** The harness spawns the `grok` CLI in a sandbox. -- **The harness owns the agent loop.** TanStack's agent-loop strategies and - per-iteration middleware don't apply inside a harness turn. -- **No sampling controls.** `temperature`-style options don't exist here. -- **Cold starts.** Each call runs a full harness turn; expect higher first-token - latency than HTTP adapters. +- **A sandbox is required.** Every call needs `withSandbox(...)`. +- **The agent owns the loop.** TanStack agent-loop strategies and per-iteration middleware do not apply inside a turn. +- **No sampling options.** Options such as `temperature` do not exist here. +- **Cold starts.** Each call runs a full CLI turn, so the first token arrives later than with an HTTP adapter. +- **No `wrapFetch`.** This adapter ignores [`wrapFetch`](../advanced/middleware#change-the-http-requests-of-a-call), because the `grok` process sends the model requests. -This adapter ignores [`wrapFetch`](../advanced/middleware#change-the-http-requests-of-a-call), because the harness process sends the model requests. +For any other ACP agent, use the [ACP-Compatible Harness](./acp-compatible). diff --git a/docs/adapters/grok.md b/docs/adapters/grok.md index 07c00a9723..c66009af7f 100644 --- a/docs/adapters/grok.md +++ b/docs/adapters/grok.md @@ -2,7 +2,7 @@ title: Grok (xAI) id: grok-adapter order: 5 -description: "Use xAI Grok models with TanStack AI — Grok 4.3, Grok Build 0.1, Grok Imagine image generation, and Grok Imagine video generation via @tanstack/ai-grok." +description: "Use xAI Grok models in TanStack AI with @tanstack/ai-grok: Grok chat on the Responses API, Grok Imagine images and video, speech, transcription, and realtime voice." keywords: - tanstack ai - grok @@ -15,9 +15,9 @@ keywords: - adapter --- -The Grok text and summarization adapters provide access to xAI's Responses API for `grok-4.3` and `grok-build-0.1`, plus Grok Imagine image generation and Grok Imagine video generation. +The Grok adapter calls xAI chat models on the Responses API. It also calls Grok Imagine for images and video, plus speech, transcription, and realtime voice. -## Installation +## 1. Install @@ -32,182 +32,253 @@ octane: @tanstack/ai-grok -## Basic Usage +## 2. Set the API key -```typescript -import { chat } from "@tanstack/ai"; -import { grokText } from "@tanstack/ai-grok"; +Put the key in the environment of your server: -const stream = chat({ - adapter: grokText("grok-build-0.1"), - messages: [{ role: "user", content: "Hello!" }], -}); +```bash +XAI_API_KEY=xai-... ``` -## Basic Usage - Custom API Key +`grokText` and the other short factories read `XAI_API_KEY`. To pass the key in code, use the `create*` factory of the same adapter: ```typescript -import { chat } from "@tanstack/ai"; import { createGrokText } from "@tanstack/ai-grok"; -const adapter = createGrokText("grok-build-0.1", process.env.XAI_API_KEY!); - -const stream = chat({ - adapter, - messages: [{ role: "user", content: "Hello!" }], -}); +const adapter = createGrokText("grok-4.7", process.env.XAI_API_KEY!); ``` -## Configuration +## 3. Stream a chat from the server ```typescript -import { createGrokText, type GrokTextConfig } from "@tanstack/ai-grok"; +import { + chat, + chatParamsFromRequest, + toServerSentEventsResponse, +} from "@tanstack/ai"; +import { grokText } from "@tanstack/ai-grok"; -const config: Omit
+
-
- )
+ setInput(event.target.value)} />
+
+ );
}
```
-## Chat Completions
-
-Pass `{ api: "chat" }` when the model must talk to Chat Completions. The default is Responses.
-
-```typescript
-import { chat } from "@tanstack/ai"
-import { lovableText } from "@tanstack/ai-lovable"
-
-const stream = chat({
- adapter: lovableText("openai/gpt-5.5", { api: "chat" }),
- messages: [{ role: "user", content: "Hello" }],
-})
-```
-
-`api: "responses"` is the same as the default. `api: "chat-completions"` is the same as `api: "chat"`.
-
-Need a header on each request, or a log of each model call? The chat adapters on this page support [`wrapFetch`](../advanced/middleware#change-the-http-requests-of-a-call).
-
-## Reasoning
+For Vue, Solid, Svelte, Angular, and the other frameworks, see [Quick Start](../getting-started/quick-start#3-render-the-chat).
-Set `reasoning` on `chat()`. The types take the levels of each model:
-
-```typescript
-import { chat } from "@tanstack/ai"
-import { lovableText } from "@tanstack/ai-lovable"
-
-const stream = chat({
- adapter: lovableText("openai/gpt-5.6-sol"),
- messages: [{ role: "user", content: "Plan a database migration." }],
- reasoning: "xhigh",
-})
-```
-
-- Responses (the default) sends `reasoning.effort`. Chat Completions sends `reasoning_effort`.
-- Neither takes `budgetTokens`.
-- The OpenAI and Gemini models take the same levels as on [OpenAI](./openai#reasoning) and [Gemini](./gemini#reasoning).
-
-## Summarize
-
-```typescript
-import { summarize } from "@tanstack/ai"
-import { lovableSummarize } from "@tanstack/ai-lovable"
-
-const result = await summarize({
- adapter: lovableSummarize("google/gemini-3.7-flash"),
- text: "The Fender Stratocaster is a versatile electric guitar.",
-})
-```
+You now have a streaming chat on Lovable. To use an OpenAI model, change the id, for example to `openai/gpt-5.6-sol`.
## Images
-Generate a new image, or pass image parts to edit one. OpenAI image models also accept a mask part (`metadata.role === "mask"`).
+Make a new image from a prompt:
```typescript
-import { generateImage } from "@tanstack/ai"
-import { lovableImage } from "@tanstack/ai-lovable"
+import { generateImage } from "@tanstack/ai";
+import { lovableImage } from "@tanstack/ai-lovable";
const result = await generateImage({
adapter: lovableImage("openai/gpt-image-2"),
prompt: "a red guitar on a wooden bench",
-})
+ threadId: "thread-1",
+});
```
+To edit an image, pass image parts. OpenAI image models also take a mask part (`metadata.role === "mask"`). See [Image Generation](../media/image-generation).
+
## Video
-Video jobs are async. Create a job, poll status, then fetch the MP4 URL. Clips last 4, 6, or 8 seconds. You can write `4`, `"4"`, or `"4s"` (same forms for 6 and 8).
+> [!NOTE]
+> Video generation is an [experimental feature](../media/video-generation).
-1080p and 4K clips are always 8 seconds. `"8s"` counts as 8 seconds. 4K works only on `google/veo-3.1-fast` and `google/veo-3.1`.
+Video runs as a job. `generateVideo()` starts the job and returns a `jobId`. Poll the job until it has an MP4 URL:
```typescript
-import { generateVideo } from "@tanstack/ai"
-import { lovableVideo } from "@tanstack/ai-lovable"
+import { generateVideo, getVideoJobStatus } from "@tanstack/ai";
+import { lovableVideo } from "@tanstack/ai-lovable";
+
+const adapter = lovableVideo("google/veo-3.1-lite");
const { jobId } = await generateVideo({
- adapter: lovableVideo("google/veo-3.1-lite"),
+ adapter,
prompt: "a red guitar on a wooden bench, slow camera push-in",
+ threadId: "thread-1",
duration: "4s",
size: "1280x720",
-})
-```
+});
-Then poll `getVideoJobStatus` with that `jobId`. When the job is complete, the result includes the MP4 URL.
+let status = await getVideoJobStatus({ adapter, jobId });
+while (status.status === "pending" || status.status === "processing") {
+ await new Promise((resolve) => setTimeout(resolve, 5000));
+ status = await getVideoJobStatus({ adapter, jobId });
+}
+
+console.log(status.status === "completed" ? status.url : status.error);
+```
-Pass one image part in `prompt` to animate a still frame.
+- A clip is 4, 6, or 8 seconds. You can write `4`, `"4"`, or `"4s"`. The same forms work for 6 and 8.
+- 1080p and 4K clips are always 8 seconds.
+- 4K works only on `google/veo-3.1-fast` and `google/veo-3.1`.
+- To animate a still frame, pass one image part in `prompt`.
## Embeddings
```typescript
-import { embed } from "@tanstack/ai"
-import { lovableEmbedding } from "@tanstack/ai-lovable"
+import { embed } from "@tanstack/ai";
+import { lovableEmbedding } from "@tanstack/ai-lovable";
const result = await embed({
adapter: lovableEmbedding("google/gemini-embedding-2"),
input: "a red guitar",
-})
+});
-console.log(result.embeddings[0]?.vector)
+console.log(result.embeddings[0]?.vector);
```
## Speech
```typescript
-import { generateSpeech } from "@tanstack/ai"
-import { lovableSpeech } from "@tanstack/ai-lovable"
+import { generateSpeech } from "@tanstack/ai";
+import { lovableSpeech } from "@tanstack/ai-lovable";
const result = await generateSpeech({
adapter: lovableSpeech("openai/gpt-4o-mini-tts"),
text: "Welcome to the guitar shop.",
+ threadId: "thread-1",
voice: "nova",
-})
+});
```
+See [Text-to-Speech](../media/text-to-speech).
+
## Transcription
+`audio` takes a `File`, a `Blob`, an `ArrayBuffer`, a base64 string, or a data URL:
+
```typescript
-import { generateTranscription } from "@tanstack/ai"
-import { lovableTranscription } from "@tanstack/ai-lovable"
+import { generateTranscription } from "@tanstack/ai";
+import { lovableTranscription } from "@tanstack/ai-lovable";
-const audio = await fetch("/voice-note.mp3").then((response) => response.blob())
+const audio = await fetch("https://example.com/voice-note.mp3").then((response) =>
+ response.blob(),
+);
const result = await generateTranscription({
adapter: lovableTranscription("openai/gpt-4o-mini-transcribe"),
audio,
language: "en",
-})
+ threadId: "thread-1",
+});
-console.log(result.text)
+console.log(result.text);
```
-## Bring Your Own Key
+See [Transcription](../media/transcription).
-Users can paste a Lovable project key in the browser. Import `lovableByok` from `@tanstack/ai-lovable/byok`, not from the package main entry.
+## Summarization
```typescript
-import { createLovableText } from "@tanstack/ai-lovable"
-import { lovableByok } from "@tanstack/ai-lovable/byok"
-import { byokMissing, getByokKey } from "@tanstack/ai/byok/server"
+import { summarize } from "@tanstack/ai";
+import { lovableSummarize } from "@tanstack/ai-lovable";
-export async function POST(request: Request) {
- const apiKey = getByokKey(request, lovableByok)
- if (!apiKey) return byokMissing(lovableByok)
+const result = await summarize({
+ adapter: lovableSummarize("google/gemini-3.7-flash"),
+ text: "The Fender Stratocaster is a versatile electric guitar.",
+ threadId: "thread-1",
+});
+```
- const adapter = createLovableText("google/gemini-3.7-flash", apiKey)
- // ...
-}
+## Models and options
+
+### Reasoning
+
+Set `reasoning` on `chat()`. The types take the levels of each model:
+
+```typescript
+import { chat } from "@tanstack/ai";
+import { lovableText } from "@tanstack/ai-lovable";
+
+const stream = chat({
+ adapter: lovableText("openai/gpt-5.6-sol"),
+ messages: [{ role: "user", content: "Plan a database migration." }],
+ threadId: "thread-1",
+ reasoning: "xhigh",
+});
```
-See [Bring Your Own Key](../advanced/byok) for the client store and a save UI.
+- Responses (the default) sends `reasoning.effort`. Chat Completions sends `reasoning_effort`.
+- Neither takes `budgetTokens`.
+- The OpenAI and Gemini models take the same levels as on [OpenAI](./openai#reasoning) and [Gemini](./gemini#reasoning).
-## Models
+### Models
Pass a model id from the curated lists. TypeScript rejects an id that is not on a list.
-Chat:
+| Activity | Default model | List |
+| --- | --- | --- |
+| Chat | `google/gemini-3.7-flash` | `LOVABLE_CHAT_MODELS` |
+| Images | `openai/gpt-image-2` | `LOVABLE_IMAGE_MODELS` |
+| Video | `google/veo-3.1-lite` | `LOVABLE_VIDEO_MODELS` |
+| Embeddings | `google/gemini-embedding-2` | `LOVABLE_EMBEDDING_MODELS` |
+| Speech | `openai/gpt-4o-mini-tts` | `LOVABLE_TTS_MODELS` |
+| Transcription | `openai/gpt-4o-mini-transcribe` | `LOVABLE_TRANSCRIPTION_MODELS` |
+
+For chat, `openai/gpt-5.5` is a fast OpenAI choice.
-- Default: `google/gemini-3.7-flash`
-- Fast OpenAI: `openai/gpt-5.5`
+## Advanced
-Image, video, embeddings, and speech:
+### Chat Completions
+
+The default text adapter uses the OpenAI Responses API. If a model must use Chat Completions, pass `{ api: "chat" }`:
+
+```typescript
+import { chat } from "@tanstack/ai";
+import { lovableText } from "@tanstack/ai-lovable";
+
+const stream = chat({
+ adapter: lovableText("openai/gpt-5.5", { api: "chat" }),
+ messages: [{ role: "user", content: "Hello" }],
+ threadId: "thread-1",
+});
+```
+
+- `api: "responses"` is the same as the default.
+- `api: "chat-completions"` is the same as `api: "chat"`.
+
+Need a header on each request, or a log of each model call? The chat adapters on this page support [`wrapFetch`](../advanced/middleware#change-the-http-requests-of-a-call).
+
+### Bring Your Own Key
+
+Users can paste a Lovable project key in the browser. Import `lovableByok` from `@tanstack/ai-lovable/byok`. The package main entry does not export it.
+
+```typescript
+import {
+ chat,
+ chatParamsFromRequest,
+ toServerSentEventsResponse,
+} from "@tanstack/ai";
+import { byokMissing, getByokKey } from "@tanstack/ai/byok/server";
+import { createLovableText } from "@tanstack/ai-lovable";
+import { lovableByok } from "@tanstack/ai-lovable/byok";
+
+export async function POST(request: Request) {
+ const apiKey = getByokKey(request, lovableByok);
+ if (!apiKey) return byokMissing(lovableByok);
+
+ const { messages, threadId, runId } = await chatParamsFromRequest(request);
+
+ const stream = chat({
+ adapter: createLovableText("google/gemini-3.7-flash", apiKey),
+ messages,
+ threadId,
+ runId,
+ });
+
+ return toServerSentEventsResponse(stream);
+}
+```
+
+See [Bring Your Own Key](../advanced/byok) for the client store and a save UI.
-- Image default: `openai/gpt-image-2`
-- Video default: `google/veo-3.1-lite`
-- Embedding default: `google/gemini-embedding-2`
-- Speech default: `openai/gpt-4o-mini-tts`
-- Transcription default: `openai/gpt-4o-mini-transcribe`
+### Requests
-The curated lists are `LOVABLE_CHAT_MODELS`, `LOVABLE_IMAGE_MODELS`, `LOVABLE_VIDEO_MODELS`, `LOVABLE_EMBEDDING_MODELS`, `LOVABLE_TTS_MODELS`, and `LOVABLE_TRANSCRIPTION_MODELS`.
+- Calls go to `https://ai.gateway.lovable.dev/v1`.
+- The adapter sends `Authorization: Bearer`, `Lovable-API-Key`, and `X-Lovable-AIG-SDK: tanstack-ai`.
-## Errors
+### Errors
-- `429 Too Many Requests`: the workspace hit its request-per-minute limit.
-- `402 Payment Required`: the workspace is out of credits.
+- `429 Too Many Requests`: the workspace hit its limit of requests per minute.
+- `402 Payment Required`: the workspace has no credits left.
See [Lovable AI features](https://docs.lovable.dev/features/ai) for models, credits, and rate limits.
diff --git a/docs/adapters/mistral.md b/docs/adapters/mistral.md
index 87f87057a9..088ee8c7f0 100644
--- a/docs/adapters/mistral.md
+++ b/docs/adapters/mistral.md
@@ -2,7 +2,7 @@
title: Mistral
id: mistral-adapter
order: 7
-description: "Use Mistral models with TanStack AI — Mistral Large, Mistral Medium, Pixtral vision models, Magistral reasoning models, and Codestral via @tanstack/ai-mistral."
+description: "Use Mistral models in TanStack AI with @tanstack/ai-mistral: Mistral Large and Medium, Pixtral vision, Magistral reasoning, Codestral, and Mistral embeddings."
keywords:
- tanstack ai
- mistral
@@ -14,9 +14,9 @@ keywords:
- llm
---
-The Mistral adapter provides access to Mistral's chat models, including Mistral Large, the multimodal Pixtral family, the Magistral reasoning models, and the Codestral code-specialized model.
+The Mistral adapter calls Mistral Large, Pixtral, Magistral, and Codestral. It also creates embeddings.
-## Installation
+## 1. Install
@@ -31,172 +31,89 @@ octane: @tanstack/ai-mistral
-## Basic Usage
+## 2. Set the API key
-```typescript
-import { chat } from "@tanstack/ai";
-import { mistralText } from "@tanstack/ai-mistral";
+Get a key from the [Mistral Console](https://console.mistral.ai/). Put it in the environment of your server:
-const stream = chat({
- adapter: mistralText("mistral-large-latest"),
- messages: [{ role: "user", content: "Hello!" }],
-});
+```bash
+MISTRAL_API_KEY=...
```
-## Basic Usage - Custom API Key
+`mistralText` and `mistralEmbedding` read `MISTRAL_API_KEY`. To pass the key in code, use `createMistralText`:
```typescript
-import { chat } from "@tanstack/ai";
import { createMistralText } from "@tanstack/ai-mistral";
-const adapter = createMistralText(
- "mistral-large-latest",
- process.env.MISTRAL_API_KEY!,
-);
-
-const stream = chat({
- adapter,
- messages: [{ role: "user", content: "Hello!" }],
-});
+const adapter = createMistralText("mistral-large-latest", process.env.MISTRAL_API_KEY!);
```
-## Configuration
+## 3. Stream a chat from the server
```typescript
import {
- createMistralText,
- type MistralTextConfig,
-} from "@tanstack/ai-mistral";
-
-const config: Omit{error}
; + if (!signIn) returnFinishing sign-in...
; + return ( + + ); +} +``` + +The save needs a click, because passkey storage shows a browser prompt. The refresh token goes into a second keyring slot. No send attaches that slot, so the refresh token stays in the browser. + +### 3. Refresh the token + +The access token lasts one hour. Call `refreshChatGptSignIn` on a timer: + +```tsx +import { useEffect } from "react"; +import { refreshChatGptSignIn } from "@tanstack/ai-openai/siwc"; +import { byok } from "./byok"; + +export function useChatGptRefresh() { + useEffect(() => { + const refresh = () => { + refreshChatGptSignIn(byok).catch(console.error); + }; + refresh(); + const timer = setInterval(refresh, 60_000); + return () => clearInterval(timer); + }, []); +} +``` + +The call does nothing until the token has less than five minutes left. If OpenAI rejects the refresh token, the call clears the sign-in. Then the user must sign in again. + +### 4. Read the token on the relay + +The relay code is the same as for a pasted key. Add `store: false`, because the ChatGPT route requires it. + +```typescript +import { + chat, + chatParamsFromRequest, + toServerSentEventsResponse, +} from "@tanstack/ai"; +import { createOpenaiChat } from "@tanstack/ai-openai"; +import { openaiByok } from "@tanstack/ai-openai/byok"; +import { byokMissing, getByokKey } from "@tanstack/ai/byok/server"; + +export async function POST(request: Request) { + const params = await chatParamsFromRequest(request); + const apiKey = getByokKey(request, openaiByok); + if (!apiKey) return byokMissing(openaiByok); + + const stream = chat({ + adapter: createOpenaiChat("gpt-6-astra", apiKey), + messages: params.messages, + threadId: params.threadId, + runId: params.runId, + modelOptions: { store: false }, + }); + return toServerSentEventsResponse(stream); +} +``` + +`store: false` also works with an API key. A ChatGPT token does not start with `sk-`. If you must send different options, use that test. + +The ChatGPT route has these limits: + +- Use a model that the ChatGPT plan of the user includes. +- Do not send `temperature`, `top_p`, `max_output_tokens`, `metadata`, or `previous_response_id`. +- Hosted tools do not work. This includes image generation, file search, code interpreter, computer use, and hosted MCP. +- Audio input and transcription do not work. + +Click **Continue with ChatGPT**, approve, save, then send a message. The relay calls OpenAI on the ChatGPT plan of the user. + +The `ts-react-chat` example has this flow in its key dialog. + +## Configuration + +```typescript +import { createOpenaiChat, type OpenAITextConfig } from "@tanstack/ai-openai"; + +const config: Omit{error}
; - if (!signIn) returnFinishing sign-in...
; - return ( - - ); -} -``` +See [Text-to-Speech](../media/text-to-speech). -The save needs a click because passkey storage shows a browser prompt. The refresh token goes into a second keyring slot. No send attaches that slot, so the refresh token stays in the browser. +## Transcription -### 3. Refresh the token +```typescript +import { generateTranscription } from "@tanstack/ai"; +import { openaiTranscription } from "@tanstack/ai-openai"; +import { audioFile } from "./audio"; -The access token lasts one hour. Call `refreshChatGptSignIn` on a timer: +const result = await generateTranscription({ + adapter: openaiTranscription("whisper-1"), + audio: audioFile, // File object or base64 string + language: "en", + threadId: "thread-1", +}); -```tsx -import { useEffect } from "react"; -import { refreshChatGptSignIn } from "@tanstack/ai-openai/siwc"; -import { byok } from "./byok"; - -export function useChatGptRefresh() { - useEffect(() => { - const refresh = () => { - refreshChatGptSignIn(byok).catch(console.error); - }; - refresh(); - const timer = setInterval(refresh, 60_000); - return () => clearInterval(timer); - }, []); -} +console.log(result.text); ``` -The call does nothing until the token has less than five minutes left. If OpenAI rejects the refresh token, the call clears the sign-in. Then the user must sign in again. +For speaker labels, use `gpt-4o-transcribe-diarize`. See [Speaker diarization](./openai-advanced#speaker-diarization). -### 4. Read the token on the relay - -The relay code is the same as for a pasted key. Add `store: false`, because the ChatGPT route requires it. +## Embeddings ```typescript -import { - chat, - chatParamsFromRequest, - toServerSentEventsResponse, -} from "@tanstack/ai"; -import { createOpenaiChat } from "@tanstack/ai-openai"; -import { openaiByok } from "@tanstack/ai-openai/byok"; -import { byokMissing, getByokKey } from "@tanstack/ai/byok/server"; +import { embed } from "@tanstack/ai"; +import { openaiEmbedding } from "@tanstack/ai-openai"; -export async function POST(request: Request) { - const params = await chatParamsFromRequest(request); - const apiKey = getByokKey(request, openaiByok); - if (!apiKey) return byokMissing(openaiByok); +const result = await embed({ + adapter: openaiEmbedding("text-embedding-3-small"), + input: ["a red guitar", "a blue drum kit"], +}); - const stream = chat({ - adapter: createOpenaiChat("gpt-6-astra", apiKey), - messages: params.messages, - threadId: params.threadId, - runId: params.runId, - modelOptions: { store: false }, - }); - return toServerSentEventsResponse(stream); -} +console.log(result.embeddings[0]?.vector); +console.log(result.usage?.promptTokens); ``` -`store: false` also works with an API key. A ChatGPT token does not start with `sk-`, so use that test if you must send different options. - -The ChatGPT route has these limits: +Both models (`text-embedding-3-small` and `text-embedding-3-large`) take the top-level `dimensions` option to make the vectors shorter: -- Use a model that the user's ChatGPT plan includes. -- Do not send `temperature`, `top_p`, `max_output_tokens`, `metadata`, or `previous_response_id`. -- Hosted tools do not work. This includes image generation, file search, code interpreter, computer use, and hosted MCP. -- Audio input and transcription do not work. +```typescript +import { embed } from "@tanstack/ai"; +import { openaiEmbedding } from "@tanstack/ai-openai"; -Click **Continue with ChatGPT**, approve, save, then send a message. The relay calls OpenAI on the user's ChatGPT plan. +const result = await embed({ + adapter: openaiEmbedding("text-embedding-3-large"), + input: "a red guitar", + dimensions: 1024, +}); +``` -The `ts-react-chat` example has this flow in its key dialog. +See the [Embeddings guide](../embeddings.md). -## Configuration +## Summarization ```typescript -import { createOpenaiChat, type OpenAITextConfig } from "@tanstack/ai-openai"; +import { summarize } from "@tanstack/ai"; +import { openaiSummarize } from "@tanstack/ai-openai"; -const config: OmitThe agent is inspecting the repo.
- if (!final) return null - return{final.summary}
+ if (isLoading) returnThe agent is inspecting the repo.
; + if (!final) return null; + return{final.summary}
; } ``` -Full walkthrough, including the client: [Harness Agents](../structured-outputs/harnesses). +- OpenCode has no schema flag. The adapter adds the JSON Schema to the prompt. +- The adapter parses the last assistant text as JSON. It removes markdown fences first. +- Tool activity still streams. The object arrives as `structured-output.complete`. + +Do you only need JSON from a prompt, with no repo? Use a model adapter such as `@tanstack/ai-openai`. For the full walkthrough, see [Harness Agents](../structured-outputs/harnesses). + +## Models and options + +### Models + +| Model | Reasoning levels | What `reasoning` sets | +| --- | --- | --- | +| `anthropic/claude-opus-4-5` | `low`, `medium`, `high` | `thinking` with a token budget | +| `anthropic/claude-sonnet-4-5`, `opencode/claude-sonnet-4-5` | `off`, `minimal`, `low`, `medium`, `high` | `thinking` with a token budget | +| `openai/gpt-5.2` | `off`, `low`, `medium`, `high`, `xhigh` | `reasoningEffort` | +| `openai/gpt-5.1-codex`, `opencode/gpt-5.1-codex` | `low`, `medium`, `high` | `reasoningEffort` | +| `google/gemini-3-pro-preview` | none | none | + +The package exports this list as `OPENCODE_MODELS`. The adapter also takes any other `provider/model` id that your OpenCode providers support. + +### Reasoning + +Set `reasoning` on `chat()`. The adapter writes it into the `options` of the model in the OpenCode server config: + +```ts +import { chat } from "@tanstack/ai"; +import { opencodeText } from "@tanstack/ai-opencode"; +import { withSandbox } from "@tanstack/ai-sandbox"; +import { sandbox } from "./sandbox"; + +const stream = chat({ + adapter: opencodeText("anthropic/claude-sonnet-4-5"), + messages: [{ role: "user", content: "Refactor the auth module" }], + threadId: "thread-1", + reasoning: { level: "high", budgetTokens: 8000 }, + middleware: [withSandbox(sandbox)], +}); +``` + +- Claude models get `thinking` with a token budget. `off` turns thinking off. +- The other models get `reasoningEffort`, plus `reasoningSummary: "auto"` when the summary is on. + +See [Reasoning](../chat/reasoning). + +### Adapter options + +Pass these as the second argument of `opencodeText`: + +| Option | What it does | +| --- | --- | +| `directory` | The working directory inside the sandbox. Default `/workspace`. | +| `port` | The port of `opencode serve` inside the sandbox. Default `4096`. On Docker, publish it with `publishPorts`. | +| `hostname` | The hostname that the server binds in the sandbox. Default `0.0.0.0`. | +| `permissionMode` | `'default'` (default), `'acceptEdits'`, or `'bypassPermissions'`. See [Permissions](#permissions). | +| `onPermissionRequest` | Your own permission handler. It replaces the default policy. | + +### Per-call options + +Pass these in `modelOptions` on `chat()`: + +- `sessionId`: resume a session. See [Resume a session](#resume-a-session). +- `permissionMode`: the permission mode for this call. +- `directory`: the working directory for this call. + +## Advanced + +### Ports on other providers + +The host connects to `opencode serve` over HTTP, so the sandbox must expose the port: + +- Docker: pass `publishPorts: [4096]` to `dockerSandbox()`. +- Vercel: expose the port with the `ports` option of the provider. +- Local-process and Daytona: nothing to do. + +### No run journal + +OpenCode does not write NDJSON to stdout, so this adapter writes no run journal, also on a durable run. See [Harnesses](../sandbox/harnesses#harness-output-can-go-to-a-journal). + +### Limitations -## Limitations +- **A sandbox is required.** Every call needs `withSandbox(...)`. +- **The agent owns the loop.** TanStack agent-loop strategies and per-iteration middleware do not apply inside a turn. +- **No sampling options.** Options such as `temperature` do not exist here. +- **Sessions stay in the sandbox.** Resume needs the same sandbox. +- **Cold starts.** Each call starts the OpenCode server in the sandbox and stops it after the turn. So the first token arrives later than with an HTTP adapter. +- **No `wrapFetch`.** This adapter ignores [`wrapFetch`](../advanced/middleware#change-the-http-requests-of-a-call), because the OpenCode server sends the model requests. -- **Server-only (Node).** The adapter spawns or attaches to an `opencode serve` process. -- **The harness owns the agent loop.** TanStack's agent-loop strategies and per-iteration middleware don't apply inside a harness turn. -- **No sampling controls.** `temperature`-style options don't exist here. -- **Sessions are server-local.** Resume requires hitting the same server instance (or a shared `baseUrl`). -- **Cold starts.** Spawning a server per turn adds first-token latency; point the adapter at a long-lived `baseUrl` to avoid it. +### Example apps -This adapter ignores [`wrapFetch`](../advanced/middleware#change-the-http-requests-of-a-call), because the harness process sends the model requests. +- [`examples/sandbox-cloudflare`](https://github.com/TanStack/ai/tree/main/examples/sandbox-cloudflare): a coding agent at the edge, with session resume, the tool timeline, and tool bridging. +- [`examples/sandbox-web`](https://github.com/TanStack/ai/tree/main/examples/sandbox-web): Claude Code on Docker, with durable runs that continue after a page refresh. To use OpenCode there, change the adapter in `src/sandbox-agent.ts`. diff --git a/docs/adapters/openrouter-advanced.md b/docs/adapters/openrouter-advanced.md new file mode 100644 index 0000000000..1342012d9a --- /dev/null +++ b/docs/adapters/openrouter-advanced.md @@ -0,0 +1,653 @@ +--- +title: "OpenRouter: Advanced" +id: openrouter-advanced +description: "Advanced OpenRouter setup in TanStack AI: retries, the Responses adapter, provider routing, prompt caching, tools with structured output, cost tracking, video, Sign in with OpenRouter, and the API reference." +keywords: + - tanstack ai + - openrouter + - provider routing + - retries + - prompt caching + - cost tracking + - sign in with openrouter + - adapter +--- + +This page has the details of the OpenRouter adapter. To get a first chat working, start at [OpenRouter](./openrouter). + +## Configuration + +```typescript +import { createOpenRouterText } from "@tanstack/ai-openrouter"; + +const adapter = createOpenRouterText( + "openai/gpt-6.1-sol", + process.env.OPENROUTER_API_KEY!, + { + serverURL: "https://openrouter.ai/api/v1", // Optional + httpReferer: "https://your-app.com", // Optional, for rankings + appTitle: "Your App Name", // Optional, for rankings + }, +); +``` + +Need a header on each request, or a log of each model call? Both text adapters support [`wrapFetch`](../advanced/middleware#change-the-http-requests-of-a-call). If you set an `httpClient`, each wrapped request still goes through it. + +## Retry rate limits + +A busy upstream provider can reply with HTTP 429. By default, the adapter retries only 5XX errors. To retry rate limits too, add `"429"` to `retryCodes`: + +```typescript +import { createOpenRouterText } from "@tanstack/ai-openrouter"; + +const adapter = createOpenRouterText( + "openai/gpt-6.1-sol", + process.env.OPENROUTER_API_KEY!, + { + retryConfig: { + strategy: "backoff", + backoff: { + initialInterval: 500, + maxInterval: 60000, + exponent: 1.5, + maxElapsedTime: 120000, + }, + retryConnectionErrors: true, + }, + retryCodes: ["429", "5XX"], + }, +); +``` + +- `retryCodes`: the HTTP status codes to retry. `"5XX"` matches every 5xx code. +- `retryConfig`: how long to wait between tries. If the response has a `Retry-After` header, the SDK waits that long. + +Now a 429 waits, then retries, and the chat continues with the next response. + +## Retry an error in the stream + +Most provider errors come as an HTTP status, and the SDK retries them as shown above. But OpenRouter can send status 200 first. Then a provider error comes as an error chunk in the stream, and the SDK cannot retry it. + +To retry that error, pass `retry` to `chat()`: + +```typescript +import { chat } from "@tanstack/ai"; +import { createOpenRouterText } from "@tanstack/ai-openrouter"; + +const adapter = createOpenRouterText( + "openai/gpt-6.1-sol", + process.env.OPENROUTER_API_KEY!, + { retryCodes: ["429", "5XX"] }, +); + +const stream = chat({ + adapter, + messages: [{ role: "user", content: "Hello!" }], + threadId: "thread-1", + retry: { maxRetries: 3 }, +}); +``` + +When `chat()` retries: + +- Only for an error code in `retryCodes`. The default is 5XX only, so add `"429"` to retry a rate limit. +- Only when the call streamed no output yet. An error after part of the answer reaches you as a `RUN_ERROR`. +- Not when `retryConfig` has `strategy: "none"`. +- Only with the Chat Completions adapter (`openRouterText`). The Responses adapter does not do this. + +The wait is the `initialInterval` of `retryConfig` (default 500 ms). It is the same for each try and does not grow. See [Let `chat()` retry for you](../chat/stream-events#let-chat-retry-for-you). + +## Chat Completions vs Responses (beta) + +OpenRouter has two OpenAI-compatible wire formats. The package has one adapter for each: + +| Adapter | Endpoint | Status | When to use | +| --- | --- | --- | --- | +| `openRouterText` | `/v1/chat/completions` | Stable | The default for almost everything. The most models and tools. | +| `openRouterResponsesText` | `/v1/responses` | Beta | The OpenAI Responses request and response shape, with more multi-turn state on OpenAI-style models. | + +The wire format is how your app talks to OpenRouter. It does not limit which provider answers. Both adapters work with any model that OpenRouter has (`anthropic/...`, `google/...`, `meta-llama/...`, and more). + +```typescript +import { chat } from "@tanstack/ai"; +import { openRouterResponsesText } from "@tanstack/ai-openrouter"; + +const stream = chat({ + adapter: openRouterResponsesText("anthropic/claude-sonnet-5.5"), + messages: [{ role: "user", content: "Hello!" }], + threadId: "thread-1", +}); +``` + +Limits of the Responses adapter while it is in beta: + +- Function tools work. The OpenRouter server tools (web search, web fetch) do not. For those, use `openRouterText`. +- The Chat Completions endpoint covers more providers and features. If you are not sure, use `openRouterText`. + +## Provider preferences + +`modelOptions.provider` sets which upstream providers can serve the request, and in which order. The adapter sends it to OpenRouter as it is. This sample pins two providers and turns off fallbacks: + +```typescript +import { chat } from "@tanstack/ai"; +import { openRouterText } from "@tanstack/ai-openrouter"; + +const stream = chat({ + adapter: openRouterText("meta-llama/llama-4-maverick"), + messages: [{ role: "user", content: "Hello!" }], + threadId: "thread-1", + modelOptions: { + provider: { + order: ["groq", "together"], // provider slugs, in priority order + allowFallbacks: false, // fail instead of routing elsewhere + }, + }, +}); +``` + +Provider slugs are the lowercase ids on the model pages of OpenRouter, for example `"anthropic"`, `"openai"`, or `"groq"`. + +Fields that limit which providers can serve the request: + +- `only` and `ignore`: an allow list or a deny list of provider slugs. OpenRouter merges them with the provider settings of your account for this request. +- `requireParameters`: send the request only to providers that support every parameter in it. Without it, OpenRouter sends each provider only the parameters it supports, and drops the rest with no error. So a `responseFormat` or a sampling option can be ignored. +- `quantizations`: only providers that serve the model at the given quantization levels, for example `"fp8"` or `"int4"`. +- `dataCollection`: `"deny"` uses only providers that do not collect user data. +- `zdr`: `true` uses only zero-data-retention endpoints. + +Fields that set the order and the price: + +- `sort`: `"price"`, `"throughput"`, or `"latency"`, or an object with `by` and `partition`. It applies when `order` is not set. It turns off load balancing. +- `maxPrice`: the highest price you accept. It takes `prompt` and `completion` in USD for each million tokens. It also takes `request` (price for each request) and `image` (price for each image) where a provider has them. OpenRouter skips providers above the limit. + +> [!NOTE] +> If you stream with `requireParameters: true`, also set `modelOptions.streamOptions: { includeUsage: false }`. Otherwise the adapter sends `stream_options`, and OpenRouter returns a 404 for providers that do not support that parameter. + +## Model variants + +`variant` adds the `:variant` suffix of OpenRouter to the model id. The adapter does not send it in the request body: + +```typescript +import { chat } from "@tanstack/ai"; +import { openRouterText } from "@tanstack/ai-openrouter"; + +const stream = chat({ + adapter: openRouterText("deepseek/deepseek-v4-pro"), + messages: [{ role: "user", content: "Hello!" }], + threadId: "thread-1", + modelOptions: { + variant: "free", // requests deepseek/deepseek-v4-pro:free + }, +}); +``` + +The values are `"free"`, `"nitro"`, `"online"`, `"exacto"`, `"extended"`, and `"thinking"`. The OpenRouter catalog sets which variants a model has. + +## Plugins + +`plugins` turns on the request plugins of OpenRouter. Each entry has an `id`, plus the options of that plugin: + +```typescript +import { chat } from "@tanstack/ai"; +import { openRouterText } from "@tanstack/ai-openrouter"; + +const stream = chat({ + adapter: openRouterText("openai/gpt-6-astra"), + messages: [{ role: "user", content: "Hello!" }], + threadId: "thread-1", + modelOptions: { + plugins: [{ id: "web", maxResults: 5 }], + }, +}); +``` + +Plugin ids include `web`, `file-parser`, `response-healing`, `moderation`, and `auto-router`. For the options of each plugin, see the OpenRouter documentation. Web fetch is a server tool (`openrouter:web_fetch`), not a plugin. For web search and fetch as tools that the model can call, see [Provider tools](#provider-tools). + +## Session and metadata + +The adapter sends these fields to OpenRouter with no change: + +- `user`: a unique id for the user. +- `sessionId`: the sticky routing key. Requests with the same `sessionId` go to the same provider, which gives more prompt-cache hits. OpenRouter also groups them for observability. +- `metadata`: up to 16 key-value pairs. + +## Prompt caching + +By default, `chat()` sets `sessionId` to the cache key, which is the `threadId` that you pass. So one conversation stays on one provider and reads its cache. + +- `anthropic/*` models also get cache markers on the system message, the last tool, and the last message. +- `promptCache: 'long'` asks for the 1-hour cache. +- `promptCache: 'none'` sends nothing. + +Your own `modelOptions.sessionId` wins. If you add a cache marker of your own, the adapter adds no automatic markers to that request. See [Prompt Caching](../advanced/prompt-caching). + +## JSON mode + +Some OpenRouter models reject strict JSON schema. For these models, `chat({ outputSchema })` sends `json_object` and adds your schema to the system prompt. + +To send JSON mode to any other model, set `modelOptions.responseFormat`: + +```typescript +import { chat } from "@tanstack/ai"; +import { openRouterText } from "@tanstack/ai-openrouter"; +import { z } from "zod"; + +const person = await chat({ + adapter: openRouterText("openai/gpt-5.6-sol"), + messages: [{ role: "user", content: "Return Ada Lovelace, age 36." }], + threadId: "thread-1", + modelOptions: { responseFormat: { type: "json_object" } }, + outputSchema: z.object({ name: z.string(), age: z.number() }), +}); +``` + +Read [When strict JSON schema is rejected](../structured-outputs/overview.md#when-strict-json-schema-is-rejected). + +## Tools and structured output in one call + +You can pass both `tools` and `outputSchema` on one `chat()` call. For some upstream models, OpenRouter returns the typed object in that same streaming request. Then the engine does not make a second call to get the final object. + +The rules: + +- This happens only when every model that can get the request is in `OPENROUTER_COMBINED_TOOLS_AND_SCHEMA_MODELS`. +- Each model sync makes this set from the OpenRouter catalog. It holds every chat model whose `supported_parameters` include `structured_outputs`, `tools`, and `tool_choice`. Examples are Claude 4.5 and later, Gemini 2.5 and later, GPT-4o and later, Grok 4, DeepSeek V3 and later, and Llama 3.1 and later. +- A model that OpenRouter does not flag, such as `anthropic/claude-opus-4.1`, uses two calls. +- If a fallback in `modelOptions.models` is not in the set, OpenRouter uses two calls. +- Routing suffixes such as `:nitro` do not change the result. + +Import the set from `@tanstack/ai-openrouter/model-meta` to check a model before you send: + +```typescript +import { OPENROUTER_COMBINED_TOOLS_AND_SCHEMA_MODELS } from "@tanstack/ai-openrouter/model-meta"; + +console.log(OPENROUTER_COMBINED_TOOLS_AND_SCHEMA_MODELS.has("openai/gpt-5.5")); // true +``` + +Both text adapters attach the schema on this path. The server with the Chat Completions adapter: + +```typescript +import { chat, toServerSentEventsResponse, toolDefinition } from "@tanstack/ai"; +import { openRouterText } from "@tanstack/ai-openrouter"; +import { z } from "zod"; + +const getWeather = toolDefinition({ + name: "get_weather", + description: "Get the current weather", + inputSchema: z.object({ location: z.string() }), +}).server(async ({ location }) => { + return { temperature: 72, conditions: "sunny", location }; +}); + +const AnswerSchema = z.object({ + summary: z.string(), + location: z.string(), +}); + +export async function POST(request: Request) { + const { messages, threadId } = await request.json(); + + const stream = chat({ + adapter: openRouterText("openai/gpt-5.5"), + messages, + threadId, + tools: [getWeather], + outputSchema: AnswerSchema, + stream: true, + }); + + return toServerSentEventsResponse(stream); +} +``` + +The server with the Responses adapter has the same `tools` and `outputSchema`: + +```typescript +import { chat, toServerSentEventsResponse, toolDefinition } from "@tanstack/ai"; +import { openRouterResponsesText } from "@tanstack/ai-openrouter"; +import { z } from "zod"; + +const getWeather = toolDefinition({ + name: "get_weather", + description: "Get the current weather", + inputSchema: z.object({ location: z.string() }), +}).server(async ({ location }) => { + return { temperature: 72, conditions: "sunny", location }; +}); + +const AnswerSchema = z.object({ + summary: z.string(), + location: z.string(), +}); + +export async function POST(request: Request) { + const { messages, threadId } = await request.json(); + + const stream = chat({ + adapter: openRouterResponsesText("openai/gpt-5.5"), + messages, + threadId, + tools: [getWeather], + outputSchema: AnswerSchema, + stream: true, + }); + + return toServerSentEventsResponse(stream); +} +``` + +The client does not change. `useChat({ outputSchema })` still reads `partial` and `final`: + +```tsx +import { useChat, fetchServerSentEvents } from "@tanstack/ai-react"; +import { z } from "zod"; + +const AnswerSchema = z.object({ + summary: z.string(), + location: z.string(), +}); + +export function useAnswer() { + const { sendMessage, partial, final } = useChat({ + connection: fetchServerSentEvents("/api/chat"), + threadId: "support-chat", + outputSchema: AnswerSchema, + }); + + return { sendMessage, partial, final }; +} +``` + +See [Structured Outputs with tools](../structured-outputs/with-tools) for the event order, and [Middleware](../advanced/middleware) for how the `structuredOutput` phase works on this path. + +To try this in a browser, run `examples/ts-react-chat` and open `/generations/openrouter-combined`. The page shows the tool call, the typed object, and the call counts of the adapter. `structuredOutputStream` must stay at 0. + +## Provider tools + +OpenRouter runs web search and web fetch for any chat model. Import the tools from `@tanstack/ai-openrouter/tools`. For how provider tools compare across adapters, see [Provider Tools](../tools/provider-tools.md). + +### `webSearchTool` + +`webSearchTool` adds web search to any OpenRouter chat model. It takes the `WebSearchConfig` of OpenRouter: + +- `engine`: `auto`, `native`, `exa`, `firecrawl`, `parallel`, or `perplexity`. +- `maxResults` and `maxTotalResults`: the number of results. +- `allowedDomains` and `excludedDomains`: which sites can be in the results. +- `searchContextSize` and `userLocation`: more control of the search. + +```typescript +import { chat } from "@tanstack/ai"; +import { openRouterText } from "@tanstack/ai-openrouter"; +import { webSearchTool } from "@tanstack/ai-openrouter/tools"; + +const stream = chat({ + adapter: openRouterText("openai/gpt-6.1-sol"), + messages: [{ role: "user", content: "What's new in AI this week?" }], + threadId: "thread-1", + tools: [ + webSearchTool({ + engine: "exa", + maxResults: 5, + allowedDomains: ["arxiv.org", "openai.com"], + }), + ], +}); +``` + +Supported models: all OpenRouter chat models. See [Provider Tools](../tools/provider-tools.md#which-models-support-which-tools). + +### `webFetchTool` + +`webFetchTool` lets any OpenRouter chat model read the full content of a URL that the model picks. It takes the `WebFetchServerToolConfig` of OpenRouter: + +- `engine`: `auto` (the default), `native`, `openrouter`, `exa`, `firecrawl`, or `parallel`. +- `maxContentTokens`: how much page content the model gets. +- `maxUses`: how many fetches the model can make in one request. +- `allowedDomains` and `blockedDomains`: which URLs the model can fetch. + +```typescript +import { chat } from "@tanstack/ai"; +import { openRouterText } from "@tanstack/ai-openrouter"; +import { webFetchTool } from "@tanstack/ai-openrouter/tools"; + +const stream = chat({ + adapter: openRouterText("openai/gpt-6.1-sol"), + messages: [ + { role: "user", content: "Summarize https://example.com/article" }, + ], + threadId: "thread-1", + tools: [ + webFetchTool({ + engine: "openrouter", + maxContentTokens: 4000, + allowedDomains: ["example.com"], + }), + ], +}); +``` + +> [!NOTE] +> The `native` engine uses the fetch of the upstream provider, for example the Anthropic `web_fetch` on Claude models. Native fetch is not the same on each provider, so it can ignore `allowedDomains` and `blockedDomains`. For the same behavior on all models, use `openrouter`, `exa`, or `firecrawl`. + +Supported models: all OpenRouter chat models. See [Provider Tools](../tools/provider-tools.md#which-models-support-which-tools). + +## Cost tracking + +OpenRouter reports the real cost of each request in the streamed response. The adapter puts it on the last `RUN_FINISHED` event: + +- `usage.cost`: the cost of the request. +- `usage.costDetails`: the breakdown of that cost from OpenRouter. + +TanStack AI does not calculate this cost from token counts. It comes from OpenRouter, so it includes routing, fallback providers, BYOK, and cached-token prices. For the meaning and units of the fields, see [Usage Accounting](https://openrouter.ai/docs/use-cases/usage-accounting) in the OpenRouter docs. + +```typescript ignore +import { chat, type RunFinishedEvent, type StreamChunk } from "@tanstack/ai"; +import { openRouterText } from "@tanstack/ai-openrouter"; + +function isRunFinished(chunk: StreamChunk): chunk is RunFinishedEvent { + return "finishReason" in chunk; +} + +for await (const chunk of chat({ + adapter: openRouterText("openai/gpt-6.1-sol"), + messages: [{ role: "user", content: "Hello!" }], + threadId: "thread-1", +})) { + if (isRunFinished(chunk)) { + console.log("cost:", chunk.usage?.cost); + console.log("breakdown:", chunk.usage?.costDetails); + } +} +``` + +- Middleware gets the same `usage`, with `cost` and `costDetails`, in the `onUsage` and `onFinish` hooks. +- If OpenRouter reports no cost, the fields are not there, and the stream completes as usual. +- Both `openRouterText` and `openRouterResponsesText` fill in the cost when OpenRouter returns it. + +How usage works when a call fails: + +- The `structuredOutputStream()` method of `openRouterText` puts the usage that it got on its last `RUN_ERROR` event. This covers parse failures, cut-off responses, empty responses, and SDK errors. +- When `chat()` uses this path, middleware `onChunk` gets the error event, also when the promise rejects. +- The `onUsage` hook reports successful calls. Each adapter call reports usage on one last event. +- A reported `completionTokensDetails.reasoningTokens` value of `0` stays zero. A missing count stays missing. +- An error before any usage arrives has no `usage` field. If the consumer stops reading the stream, it cannot get later events. + +## Video generation + +`openRouterVideo` uses the async video API of OpenRouter (`POST /api/v1/videos`). You get Seedance, Veo 3.1, Wan, Kling, and more with your one OpenRouter key. Video generation is experimental. + +Like every TanStack AI video adapter, it creates a job, then you poll the job. OpenRouter returns the finished video as a stream, not a URL. Persistence stores the stream and sets `url`, so pass the same options to both calls. + +The server: + +```typescript +import { generateVideo, getVideoJobStatus } from "@tanstack/ai"; +import { openRouterVideo } from "@tanstack/ai-openrouter"; +import { withGenerationPersistence } from "@tanstack/ai-persistence"; +import { persistence } from "./persistence"; + +const adapter = openRouterVideo("bytedance/seedance-2.5"); + +const hosting = { + threadId: "videos", + middleware: [ + withGenerationPersistence(persistence, { + artifactUrl: (ref) => `/api/artifacts/${ref.artifactId}`, + }), + ], +}; + +const { jobId } = await generateVideo({ + adapter, + ...hosting, + prompt: [ + { type: "text", content: "Animate this product shot, slow push-in" }, + { + type: "image", + source: { type: "url", value: "https://your-cdn.com/product.png" }, + metadata: { role: "start_frame" }, + }, + ], + size: "1280x720", + // `duration` is typed per model. Snap raw seconds with adapter.snapDuration(), + // or list the valid values with adapter.availableDurations(). + duration: 8, +}); + +let status = await getVideoJobStatus({ adapter, jobId, ...hosting }); +while (status.status !== "completed" && status.status !== "failed") { + await new Promise((r) => setTimeout(r, 5000)); + status = await getVideoJobStatus({ adapter, jobId, ...hosting }); +} + +// status.url is your artifact route. +// status.usage?.cost is the real cost that OpenRouter billed. +console.log(status.url, status.usage?.cost); +``` + +The client tracks the job with `useGenerateVideo`: + +```tsx +import { useGenerateVideo, fetchServerSentEvents } from "@tanstack/ai-react"; + +export function VideoResult() { + const { result } = useGenerateVideo({ + connection: fetchServerSentEvents("/api/generate/video"), + threadId: "video-studio", + }); + + return result?.url ? : null; +} +``` + +The types check sizes, durations, and the options of each model (`resolution`, `aspectRatio`, `generateAudio`, `seed`, and more) against the video model metadata of OpenRouter. See [Video Generation](../media/video-generation.md) for the full lifecycle, streaming mode, and the image-to-video role table. + +## Sign in with OpenRouter (BYOK) + +OpenRouter can make an API key that the user owns, with [OAuth PKCE](https://openrouter.ai/docs/guides/overview/auth/oauth). Import the helpers from `@tanstack/ai-openrouter/pkce`. The key is stored under `openrouterByok.id` (`openrouter`) and sent as `x-byok-openrouter`. + +On the client, start the login. When the user comes back, save the key into `defineByok`: + +```tsx +import { useEffect } from "react"; +import { defineByok, defaultByokStorage } from "@tanstack/ai-client/byok"; +import { + completeOpenRouterPkceIntoByok, + startOpenRouterPkceLogin, +} from "@tanstack/ai-openrouter/pkce"; + +const byok = defineByok({ + storage: defaultByokStorage(), +}); + +export function OpenRouterSignIn() { + useEffect(() => { + void completeOpenRouterPkceIntoByok(byok); + }, []); + + return ( + + ); +} +``` + +On the server, read the key with the same slug that the PKCE helper wrote: + +```typescript +import { + chat, + chatParamsFromRequest, + toServerSentEventsResponse, +} from "@tanstack/ai"; +import { byokMissing, getByokKey } from "@tanstack/ai/byok/server"; +import { createOpenRouterText } from "@tanstack/ai-openrouter"; +import { openrouterByok } from "@tanstack/ai-openrouter/byok"; + +export async function POST(request: Request) { + const params = await chatParamsFromRequest(request); + const apiKey = getByokKey(request, openrouterByok); + if (!apiKey) return byokMissing(openrouterByok); + + const stream = chat({ + adapter: createOpenRouterText("openai/gpt-6.1-sol", apiKey), + messages: params.messages, + threadId: params.threadId, + runId: params.runId, + }); + return toServerSentEventsResponse(stream); +} +``` + +See [Bring Your Own Key](../advanced/byok) for the keyring and passkey storage. + +### Sign in from a CLI + +A CLI or a desktop app has no web page for the redirect, so the user must copy a key by hand. `openrouterSignIn()` signs the user in with the browser and gives your app the key: + +```typescript +import { chat } from "@tanstack/ai"; +import { createOpenRouterText } from "@tanstack/ai-openrouter"; +import { openrouterSignIn } from "@tanstack/ai-openrouter/pkce"; + +const signIn = openrouterSignIn(); +const apiKey = await signIn({ + open: (url) => console.log(`Open this URL to sign in: ${url}`), +}); + +const stream = chat({ + adapter: createOpenRouterText("openai/gpt-6.1-sol", apiKey), + messages: [{ role: "user", content: "Hello!" }], + threadId: "thread-1", +}); +``` + +1. `signIn` starts a one-time listener on `127.0.0.1`, on a random port. +2. It calls `open` with the OpenRouter sign-in URL. Open that URL in the browser. +3. The user signs in and approves. The browser shows `You are signed in to OpenRouter. You can close this tab.` +4. `signIn` resolves with the API key, and the listener closes. + +The sign-in uses PKCE and a random state. It stops after 10 minutes. To change the time, pass `openrouterSignIn({ timeoutMs })`. To cancel it, pass `signal` to `signIn`. + +The listener runs on the computer of your app. So the sign-in works only when your app and the browser are on the same computer. In a hosted app, use the browser flow above. + +## API reference + +Each factory pair has the same shape. The short factory reads `OPENROUTER_API_KEY` from the environment. The `create*` factory takes an explicit API key. Both take `model` as the first argument. + +| Factories | Creates | +| --- | --- | +| `openRouterText(model, config?)` / `createOpenRouterText(model, apiKey, config?)` | Chat adapter (Chat Completions) | +| `openRouterResponsesText(model, config?)` / `createOpenRouterResponsesText(model, apiKey, config?)` | Chat adapter (Responses, beta) | +| `openRouterSummarize(model, config?)` / `createOpenRouterSummarize(model, apiKey, config?)` | Summarization adapter | +| `openRouterImage(model, config?)` / `createOpenRouterImage(model, apiKey, config?)` | Image adapter | +| `openRouterVideo(model, config?)` / `createOpenRouterVideo(model, apiKey, config?)` | Video adapter | +| `openRouterRerank(model, config?)` / `createOpenRouterRerank(model, apiKey, config?)` | Rerank adapter | +| `openRouterDecider(model, config?)` / `createOpenRouterDecider(model, apiKey, config?)` | Evaluate adapter | + +The rerank adapter forwards the optional `httpReferer` and `appTitle` config fields as attribution headers, the same as the chat adapter. diff --git a/docs/adapters/openrouter.md b/docs/adapters/openrouter.md index 1d1a4491bd..9367065d6d 100644 --- a/docs/adapters/openrouter.md +++ b/docs/adapters/openrouter.md @@ -1,7 +1,7 @@ --- title: OpenRouter Adapter id: openrouter-adapter -description: "Access 300+ LLMs from OpenAI, Anthropic, Google, Meta, Mistral, and more through a single API with OpenRouter in TanStack AI." +description: "Use 300+ models from OpenAI, Anthropic, Google, Meta, Mistral, and more in TanStack AI with one OpenRouter API key." keywords: - tanstack ai - openrouter @@ -14,9 +14,9 @@ keywords: - jev --- -OpenRouter is TanStack AI's first official AI partner and the recommended starting point for most projects. It provides access to 300+ models from OpenAI, Anthropic, Google, Meta, Mistral, and many more — all through a single API key and unified interface. +`openRouterText` calls 300+ models from OpenAI, Anthropic, Google, Meta, Mistral, and others with one API key. -## Installation +## 1. Install @@ -31,591 +31,225 @@ octane: @tanstack/ai-openrouter -## Basic Usage +## 2. Set the API key -```typescript -import { chat } from "@tanstack/ai"; -import { openRouterText } from "@tanstack/ai-openrouter"; - -const stream = chat({ - adapter: openRouterText("openai/gpt-5"), - messages: [{ role: "user", content: "Hello!" }], -}); -``` - -## Configuration - -```typescript -import { createOpenRouterText } from "@tanstack/ai-openrouter"; - -const adapter = createOpenRouterText( - "openai/gpt-5", - process.env.OPENROUTER_API_KEY!, - { - serverURL: "https://openrouter.ai/api/v1", // Optional - httpReferer: "https://your-app.com", // Optional, for rankings - appTitle: "Your App Name", // Optional, for rankings - }, -); -``` - -Need a header on each request, or a log of each model call? Both text adapters support [`wrapFetch`](../advanced/middleware#change-the-http-requests-of-a-call). If you set an `httpClient`, each wrapped request still goes through it. - -### Retry rate limits - -A busy upstream provider can reply with HTTP 429. The adapter retries only 5XX errors by default. Add `"429"` to `retryCodes` to retry rate limits too: - -```typescript -import { createOpenRouterText } from "@tanstack/ai-openrouter"; +Put the key in the environment of your server: -const adapter = createOpenRouterText( - "openai/gpt-5", - process.env.OPENROUTER_API_KEY!, - { - retryConfig: { - strategy: "backoff", - backoff: { - initialInterval: 500, - maxInterval: 60000, - exponent: 1.5, - maxElapsedTime: 120000, - }, - retryConnectionErrors: true, - }, - retryCodes: ["429", "5XX"], - }, -); +```bash +OPENROUTER_API_KEY=sk-or-... ``` -- `retryCodes`: the HTTP status codes to retry. `"5XX"` matches every 5xx code. -- `retryConfig`: how long to wait between tries. When the response has a `Retry-After` header, the SDK waits that long. - -Now a 429 waits, then retries, and the chat continues with the next response. - -### Retry an error in the stream - -Most provider errors come as an HTTP status, and the SDK retries them as shown above. But OpenRouter can send status 200 first. Then a provider error comes as an error chunk in the stream, and the SDK cannot retry it. - -To retry that error, pass `retry` to `chat()`: +`openRouterText` and the other short factories read `OPENROUTER_API_KEY`. To pass the key in code, use the `create*` factory of the same adapter: ```typescript -import { chat } from "@tanstack/ai"; import { createOpenRouterText } from "@tanstack/ai-openrouter"; -const adapter = createOpenRouterText( - "openai/gpt-5.5", - process.env.OPENROUTER_API_KEY!, - { retryCodes: ["429", "5XX"] }, -); - -const stream = chat({ - adapter, - messages: [{ role: "user", content: "Hello!" }], - retry: { maxRetries: 3 }, -}); -``` - -When `chat()` retries: - -- Only for an error code in `retryCodes`. The default is 5XX only, so add `"429"` to retry a rate limit. -- Only when the call streamed no output yet. An error after part of the answer reaches you as a `RUN_ERROR`. -- Not when `retryConfig` has `strategy: "none"`. -- Only with the Chat Completions adapter (`openRouterText`). The Responses adapter does not do this. - -The wait is the `initialInterval` of `retryConfig` (default 500 ms). It is the same for each try and does not grow. See [Let `chat()` retry for you](../chat/stream-events#let-chat-retry-for-you). - -## Available Models - -OpenRouter provides access to 300+ models from various providers. Models use the format `provider/model-name`: - -```text -model: "openai/gpt-5.1" -model: "anthropic/claude-sonnet-4.5" -model: "google/gemini-3.1-pro-preview" -model: "meta-llama/llama-4-maverick" -model: "deepseek/deepseek-v3.2" -``` - -See the full list at [openrouter.ai/models](https://openrouter.ai/models). - -`adapter.inputModalities` lists the input kinds of the selected model. It is `undefined` for a model that this package does not know. See [Check what a model accepts](../advanced/extend-adapter#check-what-a-model-accepts). - -## Example: Chat Completion - -```typescript -import { chat, toServerSentEventsResponse } from "@tanstack/ai"; -import { openRouterText } from "@tanstack/ai-openrouter"; - -export async function POST(request: Request) { - const { messages } = await request.json(); - - const stream = chat({ - adapter: openRouterText("openai/gpt-5"), - messages, - }); - - return toServerSentEventsResponse(stream); -} -``` - -## Example: With Tools - -```typescript -import { chat, toServerSentEventsResponse, toolDefinition } from "@tanstack/ai"; -import { openRouterText } from "@tanstack/ai-openrouter"; -import { z } from "zod"; - -const getWeatherDef = toolDefinition({ - name: "get_weather", - description: "Get the current weather", - inputSchema: z.object({ - location: z.string(), - }), -}); - -const getWeather = getWeatherDef.server(async ({ location }) => { - return { temperature: 72, conditions: "sunny" }; -}); - -export async function POST(request: Request) { - const { messages } = await request.json(); - - const stream = chat({ - adapter: openRouterText("openai/gpt-5"), - messages, - tools: [getWeather], - }); - - return toServerSentEventsResponse(stream); -} +const adapter = createOpenRouterText("openai/gpt-6.1-sol", process.env.OPENROUTER_API_KEY!); ``` -## JSON mode - -Some OpenRouter models reject strict JSON schema. For these models, `chat({ outputSchema })` sends `json_object` and adds your schema to the system prompt. - -To send JSON mode to any other model, set `modelOptions.responseFormat`: +## 3. Stream a chat from the server ```typescript -import { chat } from "@tanstack/ai"; -import { openRouterText } from "@tanstack/ai-openrouter"; -import { z } from "zod"; - -const person = await chat({ - adapter: openRouterText("openai/gpt-5.6-sol"), - messages: [{ role: "user", content: "Return Ada Lovelace, age 36." }], - modelOptions: { responseFormat: { type: "json_object" } }, - outputSchema: z.object({ name: z.string(), age: z.number() }), -}); -``` - -Read [When strict JSON schema is rejected](../structured-outputs/overview.md#when-strict-json-schema-is-rejected). - -## Tools and structured output together - -You can pass both `tools` and `outputSchema` on one `chat()` call. For some -upstream models OpenRouter can return the typed object in that same streaming -request, so the engine does not make a second finalization call. - -That happens only when **every** model that can receive the request is in -`OPENROUTER_COMBINED_TOOLS_AND_SCHEMA_MODELS`. The set is generated from -OpenRouter's catalog on every model sync: every chat model whose -`supported_parameters` include `structured_outputs`, `tools` and `tool_choice` -(Claude 4.5+, Gemini 2.5+, GPT-4o+, Grok 4, DeepSeek V3+, Llama 3.1+, and so -on). Models OpenRouter does not flag, such as `anthropic/claude-opus-4.1`, stay -on the legacy two-call path. - -If any fallback in `modelOptions.models` is outside that set, OpenRouter keeps -the two-call path. Routing suffixes such as `:nitro` do not change the gate. - -Import the set from `@tanstack/ai-openrouter/model-meta` if you need to check a -model before you send: - -```typescript -import { OPENROUTER_COMBINED_TOOLS_AND_SCHEMA_MODELS } from "@tanstack/ai-openrouter/model-meta"; - -OPENROUTER_COMBINED_TOOLS_AND_SCHEMA_MODELS.has("openai/gpt-5.5"); -``` - -Chat Completions (`openRouterText`) and Responses (`openRouterResponsesText`) -both attach the schema on this path. The client does not change: `useChat({ -outputSchema })` still reads `partial` and `final`. - -Server (Chat Completions): - -```typescript -import { chat, toServerSentEventsResponse, toolDefinition } from "@tanstack/ai"; +import { + chat, + chatParamsFromRequest, + toServerSentEventsResponse, +} from "@tanstack/ai"; import { openRouterText } from "@tanstack/ai-openrouter"; -import { z } from "zod"; - -const getWeather = toolDefinition({ - name: "get_weather", - description: "Get the current weather", - inputSchema: z.object({ location: z.string() }), -}).server(async ({ location }) => { - return { temperature: 72, conditions: "sunny", location }; -}); - -const AnswerSchema = z.object({ - summary: z.string(), - location: z.string(), -}); export async function POST(request: Request) { - const { messages } = await request.json(); + const { messages, threadId, runId } = await chatParamsFromRequest(request); const stream = chat({ - adapter: openRouterText("openai/gpt-5.5"), + adapter: openRouterText("openai/gpt-6.1-sol"), messages, - tools: [getWeather], - outputSchema: AnswerSchema, - stream: true, + threadId, + runId, }); return toServerSentEventsResponse(stream); } ``` -Server (Responses). Same `tools` and `outputSchema` as the Chat Completions -example, with `openRouterResponsesText`: +To use a different model, change the id. Ids have the shape `provider/model-name`, for example `anthropic/claude-sonnet-5.5` or `google/gemini-3.1-pro-preview`. -```typescript -import { chat, toServerSentEventsResponse, toolDefinition } from "@tanstack/ai"; -import { openRouterResponsesText } from "@tanstack/ai-openrouter"; -import { z } from "zod"; - -const getWeather = toolDefinition({ - name: "get_weather", - description: "Get the current weather", - inputSchema: z.object({ location: z.string() }), -}).server(async ({ location }) => { - return { temperature: 72, conditions: "sunny", location }; -}); +## 4. Render the chat -const AnswerSchema = z.object({ - summary: z.string(), - location: z.string(), -}); - -export async function POST(request: Request) { - const { messages } = await request.json(); - - const stream = chat({ - adapter: openRouterResponsesText("openai/gpt-5.5"), - messages, - tools: [getWeather], - outputSchema: AnswerSchema, - stream: true, - }); - - return toServerSentEventsResponse(stream); -} -``` - -Client: +The client is the same for every provider. Point `useChat` at your route: ```tsx +import { useState } from "react"; import { useChat, fetchServerSentEvents } from "@tanstack/ai-react"; -import { z } from "zod"; - -const AnswerSchema = z.object({ - summary: z.string(), - location: z.string(), -}); -const { sendMessage, partial, final } = useChat({ - connection: fetchServerSentEvents("/api/chat"), - outputSchema: AnswerSchema, -}); -``` - -See [Structured Outputs with tools](../structured-outputs/with-tools) for the -event order, and [Middleware](../advanced/middleware) for how -`structuredOutput` phase behaves on this path. - -To try this in a browser, run `examples/ts-react-chat` and open -`/generations/openrouter-combined`. The page shows the tool call, the typed -object, and the adapter call counts. `structuredOutputStream` must stay at 0. - -## Environment Variables - -Set your API key in environment variables: - -```bash -OPENROUTER_API_KEY=sk-or-... -``` - -## Model Routing - -OpenRouter can automatically route requests to the best available provider: - -```typescript -import { chat, toServerSentEventsResponse } from "@tanstack/ai"; -import { openRouterText } from "@tanstack/ai-openrouter"; - -export async function POST(request: Request) { - const { messages } = await request.json(); - - const stream = chat({ - adapter: openRouterText("openrouter/auto"), - messages, - modelOptions: { - models: [ - "openai/gpt-5.5", - "anthropic/claude-sonnet-4.5", - "google/gemini-3.1-pro-preview", - ], - }, +export function Chat() { + const [input, setInput] = useState(""); + const { messages, sendMessage } = useChat({ + connection: fetchServerSentEvents("/api/chat"), + threadId: "support-chat", }); - return toServerSentEventsResponse(stream); + return ( + + ); } ``` -`models` is the fallback list: OpenRouter tries them in order when the primary model is unavailable. The rest of OpenRouter's routing surface is also exposed through `modelOptions` and forwarded to the request as-is. The option names are the camelCase ones from the OpenRouter SDK. - -### Provider preferences - -`provider` controls which upstream providers may serve the request and in what order. Pin one provider and disable fallbacks: +For Vue, Solid, Svelte, Angular, and the other frameworks, see [Quick Start](../getting-started/quick-start#3-render-the-chat). -```typescript -import { chat } from "@tanstack/ai"; -import { openRouterText } from "@tanstack/ai-openrouter"; - -const stream = chat({ - adapter: openRouterText("meta-llama/llama-4-maverick"), - messages: [{ role: "user", content: "Hello!" }], - modelOptions: { - provider: { - order: ["groq", "together"], // provider slugs, in priority order - allowFallbacks: false, // fail instead of routing elsewhere - }, - }, -}); -``` +You now have a streaming chat on OpenRouter. [Tools](../tools/tools) and [Structured Outputs](../structured-outputs/overview) work the same as with every adapter. -Provider slugs are the lowercase ids shown on OpenRouter's model pages (`"anthropic"`, `"openai"`, `"groq"`, …). Other fields on `provider`: +## Model routing -- `only` / `ignore`: allow-list or deny-list of provider slugs, merged with your account-wide provider settings for this request. -- `requireParameters`: only route to providers that support every parameter in the request. Without it, OpenRouter sends each provider only the parameters it supports and silently drops the rest, so a `responseFormat` or a sampling option can be ignored without an error. When you stream with `requireParameters: true`, also set `modelOptions.streamOptions: { includeUsage: false }`. Without it, the adapter sends `stream_options`, and OpenRouter returns a 404 for providers that do not support that parameter. -- `sort`: `"price"`, `"throughput"`, or `"latency"`, or an object with `by` and `partition`, applied when `order` is not set. Setting it disables load balancing. -- `maxPrice`: the highest pricing you accept. It takes `prompt` and `completion` in USD per million tokens, plus `request` (per-request pricing) and `image` (per image) where a provider offers them. Providers above the limit are skipped. -- `quantizations`: restrict to providers serving the model at given quantization levels (e.g. `"fp8"`, `"int4"`). -- `dataCollection`: `"deny"` to use only providers that do not collect user data. -- `zdr`: `true` to restrict routing to zero-data-retention endpoints. - -### Model variants - -`variant` appends OpenRouter's `:variant` suffix to the model id. It is adapter metadata only and is never sent in the request body: +A model can be down, or slow on one provider. Give OpenRouter a fallback list in `models`. When the primary model is not available, OpenRouter tries the models in order: ```typescript import { chat } from "@tanstack/ai"; import { openRouterText } from "@tanstack/ai-openrouter"; const stream = chat({ - adapter: openRouterText("deepseek/deepseek-v4-pro"), + adapter: openRouterText("openrouter/auto"), messages: [{ role: "user", content: "Hello!" }], + threadId: "thread-1", modelOptions: { - variant: "free", // requests deepseek/deepseek-v4-pro:free + models: [ + "openai/gpt-6.1-sol", + "anthropic/claude-sonnet-5.5", + "google/gemini-3.1-pro-preview", + ], }, }); ``` -Accepted values are `"free"`, `"nitro"`, `"online"`, `"exacto"`, `"extended"`, and `"thinking"`. Which of them a model actually offers is up to OpenRouter's catalog. - -### Plugins - -`plugins` enables OpenRouter's request-level plugins. Each entry is identified by `id`, with plugin-specific options alongside: +To pick which upstream providers serve a model, set `modelOptions.provider`: ```typescript import { chat } from "@tanstack/ai"; import { openRouterText } from "@tanstack/ai-openrouter"; const stream = chat({ - adapter: openRouterText("openai/gpt-6-astra"), + adapter: openRouterText("meta-llama/llama-4-maverick"), messages: [{ role: "user", content: "Hello!" }], + threadId: "thread-1", modelOptions: { - plugins: [{ id: "web", maxResults: 5 }], + provider: { + order: ["groq", "together"], // provider slugs, in priority order + allowFallbacks: false, // fail instead of routing elsewhere + }, }, }); ``` -Plugin ids include `web`, `file-parser`, `response-healing`, `moderation`, and `auto-router`; see the OpenRouter documentation for each plugin's options. Web fetching is not a plugin but a server tool (`openrouter:web_fetch`); for web search and fetch as _tools_ the model can call, see [Provider Tools](#provider-tools) below. +For all the `provider` fields, model variants such as `:free`, and plugins, see [OpenRouter: Advanced](./openrouter-advanced#provider-preferences). -### Reasoning +## Provider tools -Set `reasoning` on `chat()`. OpenRouter gets it as `reasoning.effort`: +OpenRouter runs web search and web fetch for any chat model. Import the tools from `@tanstack/ai-openrouter/tools`: ```typescript import { chat } from "@tanstack/ai"; import { openRouterText } from "@tanstack/ai-openrouter"; +import { webSearchTool } from "@tanstack/ai-openrouter/tools"; const stream = chat({ - adapter: openRouterText("anthropic/claude-opus-5.5"), - messages: [{ role: "user", content: "Hello!" }], - reasoning: "high", + adapter: openRouterText("openai/gpt-6.1-sol"), + messages: [{ role: "user", content: "What's new in AI this week?" }], + threadId: "thread-1", + tools: [ + webSearchTool({ + engine: "exa", + maxResults: 5, + allowedDomains: ["arxiv.org", "openai.com"], + }), + ], }); ``` -- `off` sends `effort: "none"`. -- `openRouterText` takes no `budgetTokens`. `openRouterResponsesText` sends `budgetTokens` as `reasoning.maxTokens`, and adds `summary: "auto"` so the thinking text streams. -- The types take the levels of each model. Your editor shows them when you type `reasoning`. - -### Session and metadata - -`user` (a unique user identifier), `sessionId`, and `metadata` (up to 16 key-value pairs) are forwarded unchanged. `sessionId` doubles as OpenRouter's sticky routing key: requests that share it are routed to the same provider to maximize prompt-cache hits, and grouped together for observability. - -### Prompt caching +- `webSearchTool`: search the web. +- `webFetchTool`: read the full page at a URL that the model picks. -`chat()` sets `sessionId` to the cache key by default, which is the `threadId` that you pass. So one conversation stays on one provider and reads its cache. `anthropic/*` models also get cache markers on the system message, the last tool, and the last message. `promptCache: 'long'` asks for the 1-hour cache, and `promptCache: 'none'` sends nothing. See [Prompt Caching](../advanced/prompt-caching). +Both work on all OpenRouter chat models. For every option of each tool, see [OpenRouter: Advanced](./openrouter-advanced#provider-tools). For how provider tools compare across adapters, see [Provider Tools](../tools/provider-tools.md#which-models-support-which-tools). -Your own `modelOptions.sessionId` wins. A cache marker of your own turns the automatic markers off for that request. +## Tools and structured output together -## Model Options +You can pass `tools` and `outputSchema` on one `chat()` call. On some models, OpenRouter returns the typed object in the same streaming request. Then no second call is necessary. -OpenRouter supports various provider-specific options. Sampling parameters live here too — `temperature`, `topP`, and `maxCompletionTokens` (OpenRouter's token-limit key for the chat adapter) — rather than as root-level props on `chat()`: +This happens only when every model that can get the request is in `OPENROUTER_COMBINED_TOOLS_AND_SCHEMA_MODELS`. To check a model before you send: ```typescript -import { chat, toServerSentEventsResponse } from "@tanstack/ai"; -import { openRouterText } from "@tanstack/ai-openrouter"; - -export async function POST(request: Request) { - const { messages } = await request.json(); - - const stream = chat({ - adapter: openRouterText("anthropic/claude-sonnet-4.5"), - messages, - modelOptions: { - temperature: 0.7, - topP: 0.9, - maxCompletionTokens: 1024, - }, - }); +import { OPENROUTER_COMBINED_TOOLS_AND_SCHEMA_MODELS } from "@tanstack/ai-openrouter/model-meta"; - return toServerSentEventsResponse(stream); -} +console.log(OPENROUTER_COMBINED_TOOLS_AND_SCHEMA_MODELS.has("openai/gpt-6.1-sol")); // true ``` -> If you previously passed `temperature` / `topP` / `maxTokens` at the root of `chat()`, see [Moving Sampling Options into modelOptions](../migration/sampling-options-to-model-options). - -## Chat Completions vs Responses (beta) - -OpenRouter exposes two OpenAI-compatible wire formats, and the adapter -package ships one of each: - -| Adapter | Endpoint | Status | When to use | -| ------------------------- | ---------------------- | ------ | ----------------------------------------------------------------------------------------- | -| `openRouterText` | `/v1/chat/completions` | Stable | Default for almost everything. Broadest model + tool support. | -| `openRouterResponsesText` | `/v1/responses` | Beta | OpenAI Responses-shaped request/response; richer multi-turn state on OpenAI-style models. | +For the full server and client code and the rules of the set, see [OpenRouter: Advanced](./openrouter-advanced#tools-and-structured-output-in-one-call). -Both adapters route to any underlying model OpenRouter supports -(`anthropic/...`, `google/...`, `meta-llama/...`, etc.) — the wire format -describes how your client talks to OpenRouter, not which provider answers. -`/v1/responses` is OpenAI's newer API surface; OpenRouter implements it so -clients that prefer that wire format can use it across the same 300+ -model catalogue. +## Images ```typescript -import { chat } from "@tanstack/ai"; -import { openRouterResponsesText } from "@tanstack/ai-openrouter"; +import { generateImage } from "@tanstack/ai"; +import { openRouterImage } from "@tanstack/ai-openrouter"; -const stream = chat({ - adapter: openRouterResponsesText("anthropic/claude-sonnet-4.5"), - messages: [{ role: "user", content: "Hello!" }], +const result = await generateImage({ + adapter: openRouterImage("google/gemini-nano-banana-2.1"), + prompt: "A watercolor lighthouse at dusk", + threadId: "thread-1", + size: "1344x768", // sent as image_config.aspect_ratio ("16:9") + modelOptions: { + image_size: "2K", // resolution (Gemini models) + }, }); -``` -Caveats while the Responses adapter is in beta: - -- Function tools are supported; OpenRouter's branded server-tools (web - search, file search, …) are not yet wired through this path — use - `openRouterText` if you need those. -- If in doubt, prefer `openRouterText`. The Chat Completions endpoint has - broader provider coverage and feature parity today. - -## Cost Tracking - -OpenRouter reports the actual cost of each request inline on the streamed -response. When present, the adapter forwards it on the terminal `RUN_FINISHED` -event under `usage.cost`, with OpenRouter's per-request breakdown under -`usage.costDetails`. This is the cost OpenRouter itself reports for the -request — it is **not** computed locally from token counts, so it already -accounts for routing, fallback providers, BYOK, and cached-token pricing. See -OpenRouter's [Usage Accounting](https://openrouter.ai/docs/use-cases/usage-accounting) -docs for the meaning and units of these fields. - -```typescript ignore -import { chat, type RunFinishedEvent, type StreamChunk } from "@tanstack/ai"; -import { openRouterText } from "@tanstack/ai-openrouter"; - -function isRunFinished(chunk: StreamChunk): chunk is RunFinishedEvent { - return "finishReason" in chunk; -} - -for await (const chunk of chat({ - adapter: openRouterText("openai/gpt-5"), - messages: [{ role: "user", content: "Hello!" }], -})) { - if (isRunFinished(chunk)) { - console.log("cost:", chunk.usage?.cost); - console.log("breakdown:", chunk.usage?.costDetails); - } -} +console.log(result.images); ``` -The same `usage` (including `cost` / `costDetails`) is passed to middleware via -the `onUsage` and `onFinish` hooks. When OpenRouter does not report a cost, the -fields are simply absent and the stream completes normally. Both -`openRouterText` and `openRouterResponsesText` populate cost when OpenRouter -returns it. +- You get one image for each request. `numberOfImages` more than 1 throws an error. +- `size` takes one of ten `WIDTHxHEIGHT` values. Other values throw an error that lists the valid sizes. +- The prompt can mix text and image parts, for image-to-image work. `modelOptions.strength` sets how much the input image controls the result, on models that support it. -The `openRouterText` adapter's `structuredOutputStream()` method includes -received usage on its terminal `RUN_ERROR` event. This covers parse failures, -truncated responses, empty responses, and SDK errors. When `chat()` uses this -path, middleware `onChunk` receives the error event even if the promise rejects. -The `onUsage` hook reports successful calls. Each adapter call reports usage on -one terminal event. +For a React and TanStack Start walkthrough with `useGenerateImage`, see [Generate Image](../tutorials/generate-image). See also [Image Generation](../media/image-generation). -A reported `completionTokensDetails.reasoningTokens` value of `0` stays zero. -A missing count stays absent. Errors before any usage arrives have no `usage` -field. If the consumer stops reading the stream, it cannot receive later events. +## Video + +`openRouterVideo` makes videos with Seedance, Veo 3.1, Wan, Kling, and more, through the async video API of OpenRouter (`POST /api/v1/videos`). Video generation is experimental. For the full server and client code, see [OpenRouter: Advanced](./openrouter-advanced#video-generation). ## Reranking -OpenRouter exposes rerank models through its unified `/v1/rerank` endpoint -(served via the `@openrouter/sdk` SDK). Any rerank model OpenRouter offers works -by passing its slug — for example `cohere/rerank-v3.5`, `cohere/rerank-4-fast`, -`cohere/rerank-4-pro`, or `nvidia/llama-nemotron-rerank-vl-1b-v2`. Use -`openRouterRerank` with the `rerank()` activity to reorder candidate documents -by relevance to a query: +`openRouterRerank` puts documents in order of how well they match a query: ```typescript import { rerank } from "@tanstack/ai"; import { openRouterRerank } from "@tanstack/ai-openrouter"; const { rerankedDocuments } = await rerank({ - adapter: openRouterRerank("cohere/rerank-v3.5"), + adapter: openRouterRerank("cohere/rerank-4-pro"), query: "talk about rain", documents: ["sunny day at the beach", "rainy afternoon in the city"], topN: 2, }); -console.log(rerankedDocuments[0]); // 'rainy afternoon in the city' +console.log(rerankedDocuments[0]); // "rainy afternoon in the city" ``` -`openRouterRerank` reads `OPENROUTER_API_KEY` from the environment; pass a key -explicitly with `createOpenRouterRerank("cohere/rerank-v3.5", "sk-or-...")`. The -optional `httpReferer` / `appTitle` config fields are forwarded as OpenRouter -attribution headers, just like the chat adapter. - -See the [Reranking guide](../rerank/rerank) for object documents, RAG -pipelines, options, and the result shape. +Any rerank model on OpenRouter works. Known ids: `cohere/rerank-v3.5`, `cohere/rerank-4-fast`, `cohere/rerank-4-pro`, `nvidia/llama-nemotron-rerank-vl-1b-v2`. See the [Reranking guide](../rerank/rerank). ## Evaluate -OpenRouter exposes TypeSafe Jev through `POST /api/alpha/decisions`. -Use `openRouterDecider` with `decide()` to ask typed questions about a -shared `state`: +Sometimes you need an answer that your code can branch on, not chat text. Use `openRouterDecider` with `decide()` to ask typed questions about one shared `state`. The adapter calls TypeSafe Jev through `POST /api/alpha/decisions`: ```typescript import { decide, choice, score, boolean } from "@tanstack/ai"; @@ -654,299 +288,110 @@ console.log(result.queue.confidence); console.log(result.meta.usage); ``` -To bill per request, read `result.meta.usage.cost`. To look the request up later with `GET /api/v1/generation?id=`, read `result.meta.id`. `result.meta.provider` names the upstream provider that served the request. - -`openRouterDecider` reads `OPENROUTER_API_KEY` from the environment. Pass a -key explicitly with `createOpenRouterDecider("~typesafe/jev-latest", "sk-or-...")`. - -Known slugs: - -- `~typesafe/jev-latest` -- `typesafe/jev-1.13` -- `typesafe/jev-1.13.0` - -See the [Evaluate guide](../evaluate/evaluate) for question helpers, the result -shape, abort, and middleware. - -## Image Generation - -For a React + Start walkthrough with `useGenerateImage`, open [Generate Image](../tutorials/generate-image). - -`openRouterImage` routes image generation through OpenRouter's -chat-completions surface (`modalities: ['image']`). Multimodal prompts are -supported — text and image parts are forwarded in order for -image-conditioned generation: - -```typescript -import { generateImage } from "@tanstack/ai"; -import { openRouterImage } from "@tanstack/ai-openrouter"; - -const result = await generateImage({ - adapter: openRouterImage("google/gemini-2.5-flash-image"), - prompt: "A watercolor lighthouse at dusk", - size: "1344x768", // mapped to image_config.aspect_ratio ('16:9') - modelOptions: { - image_size: "2K", // resolution (Gemini models) - strength: 0.35, // image-to-image influence, i2i-capable models only - }, -}); -``` - -Notes: - -- The pathway returns **exactly one image per request** — `numberOfImages > 1` - throws instead of silently under-delivering. Make multiple requests if you - need multiple candidates. -- `size` must be one of the ten supported `WIDTHxHEIGHT` values (it is - converted to `image_config.aspect_ratio`); anything else throws with the - supported list. +- `result.meta.usage.cost`: the cost of the request, to bill per request. +- `result.meta.id`: the id to look the request up later with `GET /api/v1/generation?id=`. +- `result.meta.provider`: the upstream provider that served the request. -## Video Generation (Experimental) +Known ids: `~typesafe/jev-latest`, `typesafe/jev-1.13`, `typesafe/jev-1.13.0`. See the [Evaluate guide](../evaluate/evaluate). -`openRouterVideo` targets OpenRouter's dedicated **async video API** -(`POST /api/v1/videos`) — Seedance, Veo 3.1, Wan, Kling, and Sora 2 Pro -through your one OpenRouter key. It follows the jobs/polling architecture -shared by all TanStack AI video adapters: +## Summarization ```typescript -// Server: create the job, then poll -import { generateVideo, getVideoJobStatus } from "@tanstack/ai"; -import { openRouterVideo } from "@tanstack/ai-openrouter"; -import { withGenerationPersistence } from "@tanstack/ai-persistence"; -import { persistence } from "./persistence"; - -const adapter = openRouterVideo("bytedance/seedance-2.0"); - -// OpenRouter returns the finished video as a stream, not a URL. Persistence -// stores the stream and sets `url`. Pass the same options to both calls. -const hosting = { - threadId: "videos", - middleware: [ - withGenerationPersistence(persistence, { - artifactUrl: (ref) => `/api/artifacts/${ref.artifactId}`, - }), - ], -}; - -const { jobId } = await generateVideo({ - adapter, - ...hosting, - prompt: [ - { type: "text", content: "Animate this product shot, slow push-in" }, - { - type: "image", - source: { type: "url", value: "https://your-cdn.com/product.png" }, - metadata: { role: "start_frame" }, - }, - ], - size: "1280x720", - // `duration` is typed per model from the published metadata; coerce raw - // seconds with adapter.snapDuration() or enumerate via adapter.availableDurations(). - duration: 8, +import { summarize } from "@tanstack/ai"; +import { openRouterSummarize } from "@tanstack/ai-openrouter"; + +const result = await summarize({ + adapter: openRouterSummarize("google/gemini-3.8-flash"), + text: "Your long text to summarize...", + threadId: "thread-1", + maxLength: 100, + style: "concise", // "concise" | "bullet-points" | "paragraph" }); -let status = await getVideoJobStatus({ adapter, jobId, ...hosting }); -while (status.status !== "completed" && status.status !== "failed") { - await new Promise((r) => setTimeout(r, 5000)); - status = await getVideoJobStatus({ adapter, jobId, ...hosting }); -} -// status.url is your artifact route. status.usage?.cost is the real billed -// cost that the gateway reports. +console.log(result.summary); ``` -```tsx -// Client: track the job with the useGenerateVideo hook -import { useGenerateVideo, fetchServerSentEvents } from "@tanstack/ai-react"; +## Cost tracking -const { generate, result, videoStatus, isLoading } = useGenerateVideo({ - connection: fetchServerSentEvents("/api/generate/video"), -}); -// result?.url renders directly: -``` - -Sizes, durations, and per-model options (`resolution`, `aspectRatio`, -`generateAudio`, `seed`, …) are typed and validated per model from -OpenRouter's video model metadata. See -[Video Generation](../media/video-generation.md) for the full lifecycle, -streaming mode, and the image-to-video role-mapping table. - -## Next Steps - -- [Getting Started](../getting-started/quick-start) - Learn the basics -- [Tools Guide](../tools/tools) - Learn about tools -- [Reranking](../rerank/rerank) - Reorder documents by relevance -- [Evaluate](../evaluate/evaluate) - Ask typed questions about shared state +OpenRouter reports the real cost of each request. The adapter puts it on the last `RUN_FINISHED` event as `usage.cost`, with the breakdown in `usage.costDetails`. This cost includes routing, fallback providers, BYOK, and cached-token prices. See [Cost tracking](./openrouter-advanced#cost-tracking) for a sample. -## Provider Tools - -> **Migrated from `createWebSearchTool`?** This factory was renamed to -> `webSearchTool` and moved to the `/tools` subpath in this release. -> See [Migration Guide §6](../migration/migration.md#6-provider-tools-moved-to-tools-subpath) -> for the exact before/after. +## Sign in with OpenRouter (BYOK) -OpenRouter's gateway exposes web search via a plugin that works across -any proxied chat model. Import it from `@tanstack/ai-openrouter/tools`. +Your users can sign in with their own OpenRouter account. Then the requests use their key and their credit. The helpers are in `@tanstack/ai-openrouter/pkce`. For the browser and the CLI flows, see [OpenRouter: Advanced](./openrouter-advanced#sign-in-with-openrouter-byok). -> For the full concept, a comparison matrix, and type-gating details, see -> [Provider Tools](../tools/provider-tools.md). +## Models and options -### `webSearchTool` +### Model options -Adds web search capability to any OpenRouter-proxied chat model. The factory -accepts OpenRouter's `WebSearchConfig` directly — pick the `engine` -(`auto`, `native`, `exa`, `firecrawl`, or `parallel`), cap results with -`maxResults` / `maxTotalResults`, restrict which sites can appear in results -with `allowedDomains` / `excludedDomains`, and optionally pass -`searchContextSize` or `userLocation` for finer control. +Provider options go in `modelOptions`. This includes sampling options such as `temperature` and `maxCompletionTokens` (the token limit of the chat adapter). The type only allows the options that the selected model supports. For example, Claude Sonnet 5.5 does not take `topP`, so `topP` is a type error for that model: ```typescript import { chat } from "@tanstack/ai"; import { openRouterText } from "@tanstack/ai-openrouter"; -import { webSearchTool } from "@tanstack/ai-openrouter/tools"; const stream = chat({ - adapter: openRouterText("openai/gpt-5"), - messages: [{ role: "user", content: "What's new in AI this week?" }], - tools: [ - webSearchTool({ - engine: "exa", - maxResults: 5, - allowedDomains: ["arxiv.org", "openai.com"], - }), - ], + adapter: openRouterText("anthropic/claude-sonnet-5.5"), + messages: [{ role: "user", content: "Hello!" }], + threadId: "thread-1", + modelOptions: { + temperature: 0.7, + maxCompletionTokens: 1024, + }, }); ``` -**Supported models:** all OpenRouter chat models. See [Provider Tools](../tools/provider-tools.md#which-models-support-which-tools). +The option names are the camelCase names of the OpenRouter SDK. Sampling options do not go at the root of `chat()`. See [Moving Sampling Options into modelOptions](../migration/sampling-options-to-model-options). -### `webFetchTool` - -Lets any OpenRouter-proxied chat model fetch the full contents of a URL the -model chooses, instead of running a search. The factory accepts OpenRouter's -`WebFetchServerToolConfig` directly — pick the fetch `engine` (`auto` — the -default, `native`, `openrouter`, `exa`, or `firecrawl`), cap how much page -content the model receives with `maxContentTokens`, cap how many fetches the -model can make per request with `maxUses`, and restrict which URLs the model -can fetch with `allowedDomains` / `blockedDomains`. +### Reasoning -> The `native` engine routes to the underlying provider's own fetch (for -> example, Anthropic's `web_fetch` on Claude models). Native fetch -> capabilities vary, so `allowedDomains` and `blockedDomains` may be -> ignored. Use `openrouter`, `exa`, or `firecrawl` if you need consistent -> behaviour across models. +Set `reasoning` on `chat()`. OpenRouter gets it as `reasoning.effort`: ```typescript import { chat } from "@tanstack/ai"; import { openRouterText } from "@tanstack/ai-openrouter"; -import { webFetchTool } from "@tanstack/ai-openrouter/tools"; const stream = chat({ - adapter: openRouterText("openai/gpt-5"), - messages: [ - { role: "user", content: "Summarize https://example.com/article" }, - ], - tools: [ - webFetchTool({ - engine: "openrouter", - maxContentTokens: 4000, - allowedDomains: ["example.com"], - }), - ], + adapter: openRouterText("anthropic/claude-opus-5.5"), + messages: [{ role: "user", content: "Hello!" }], + threadId: "thread-1", + reasoning: "high", }); ``` -**Supported models:** all OpenRouter chat models. See [Provider Tools](../tools/provider-tools.md#which-models-support-which-tools). - -## Sign in with OpenRouter (BYOK) - -OpenRouter can mint a user-owned API key via [OAuth PKCE](https://openrouter.ai/docs/guides/overview/auth/oauth). Import the helpers from `@tanstack/ai-openrouter/pkce`. The key is stored under `openrouterByok.id` (`openrouter`) and sent as `x-byok-openrouter`. - -Client: start login, then on return save the key into `defineByok`. +- `off` sends `effort: "none"`. +- `openRouterText` takes no `budgetTokens`. `openRouterResponsesText` sends `budgetTokens` as `reasoning.maxTokens`, and adds `summary: "auto"` so that the thinking text streams. +- The types know the levels of each model. Your editor shows them when you type `reasoning`. -```tsx -import { useEffect } from "react"; -import { defineByok, defaultByokStorage } from "@tanstack/ai-client/byok"; -import { - completeOpenRouterPkceIntoByok, - startOpenRouterPkceLogin, -} from "@tanstack/ai-openrouter/pkce"; +See [Reasoning](../chat/reasoning). -const byok = defineByok({ - storage: defaultByokStorage(), -}); +### Models -export function OpenRouterSignIn() { - useEffect(() => { - void completeOpenRouterPkceIntoByok(byok); - }, []); +Model ids have the shape `provider/model-name`: - return ( - - ); -} -``` - -Server: read the same slug the PKCE helper wrote. - -```typescript -import { - chat, - chatParamsFromRequest, - toServerSentEventsResponse, -} from "@tanstack/ai"; -import { byokMissing, getByokKey } from "@tanstack/ai/byok/server"; -import { createOpenRouterText } from "@tanstack/ai-openrouter"; -import { openrouterByok } from "@tanstack/ai-openrouter/byok"; - -export async function POST(request: Request) { - const params = await chatParamsFromRequest(request); - const apiKey = getByokKey(request, openrouterByok); - if (!apiKey) return byokMissing(openrouterByok); - - const stream = chat({ - adapter: createOpenRouterText("openai/gpt-5.5", apiKey), - messages: params.messages, - threadId: params.threadId, - runId: params.runId, - }); - return toServerSentEventsResponse(stream); -} +```text +openai/gpt-6.1-sol +anthropic/claude-sonnet-5.5 +google/gemini-3.1-pro-preview +meta-llama/llama-4-maverick +deepseek/deepseek-v4-pro ``` -See [Bring Your Own Key](../advanced/byok) for the keyring and passkey storage. - -### Sign in from a CLI - -A CLI or a desktop app has no web page for the redirect, so the user would have to copy a key by hand. `openrouterSignIn()` signs the user in with the browser and gives your app the key: +See the full list at [openrouter.ai/models](https://openrouter.ai/models). The package exports the known ids: -```typescript -import { chat } from "@tanstack/ai"; -import { createOpenRouterText } from "@tanstack/ai-openrouter"; -import { openrouterSignIn } from "@tanstack/ai-openrouter/pkce"; - -const signIn = openrouterSignIn(); -const apiKey = await signIn({ - open: (url) => console.log(`Open this URL to sign in: ${url}`), -}); +- `@tanstack/ai-openrouter/model-meta`: `OPENROUTER_CHAT_MODELS`, `OPENROUTER_IMAGE_MODELS`, `OPENROUTER_VIDEO_MODELS`. +- `@tanstack/ai-openrouter`: `OPENROUTER_RERANK_MODELS`, `OPENROUTER_EVALUATE_MODELS`. -const stream = chat({ - adapter: createOpenRouterText("openai/gpt-6.1-sol", apiKey), - messages: [{ role: "user", content: "Hello!" }], -}); -``` +`adapter.inputModalities` lists the input kinds of the selected model. It is `undefined` for a model that this package does not know. See [Check what a model accepts](../advanced/extend-adapter#check-what-a-model-accepts). -1. `signIn` starts a one-time listener on `127.0.0.1`, on a random port. -2. It calls `open` with the OpenRouter sign-in URL. Open that URL in the browser. -3. The user signs in and approves. The browser shows `You are signed in to OpenRouter. You can close this tab.` -4. `signIn` resolves with the API key, and the listener closes. +## Advanced -The sign-in uses PKCE and a random state. It stops after 10 minutes. To change the time, pass `openrouterSignIn({ timeoutMs })`. To cancel it, pass `signal` to `signIn`. +These topics are on [OpenRouter: Advanced](./openrouter-advanced): -The listener runs on the computer of your app. So the sign-in works only when your app and the browser are on the same computer. In a hosted app, use the browser flow above. +- Configuration: base URL, app attribution headers, and `wrapFetch`. +- Retries for rate limits, and for an error in the stream. +- The Responses adapter (`openRouterResponsesText`, beta). +- Provider preferences, model variants, plugins, and session metadata. +- Prompt caching and JSON mode. +- Tools and structured output in one call, with the full code. +- `webFetchTool`, cost tracking, video generation, and Sign in with OpenRouter. +- The API reference. diff --git a/docs/adapters/perplexity.md b/docs/adapters/perplexity.md index 9e54299597..e5ad3ac480 100644 --- a/docs/adapters/perplexity.md +++ b/docs/adapters/perplexity.md @@ -2,7 +2,7 @@ title: Perplexity Search id: perplexity-adapter order: 10 -description: "Ground TanStack AI agents on the live web with the Perplexity Search API via @tanstack/ai-perplexity." +description: "Give a TanStack AI agent live web results with the Perplexity Search API and @tanstack/ai-perplexity, or chat with Sonar." keywords: - tanstack ai - perplexity @@ -11,129 +11,212 @@ keywords: - grounding --- -`@tanstack/ai-perplexity` is a **Search API** package. It wraps `POST https://api.perplexity.ai/search` as a TanStack AI tool (and a low-level HTTP client) so an agent can fetch ranked web results for citation and grounding. +`@tanstack/ai-perplexity` exposes the Perplexity Search API as a tool. Pair it with a chat adapter that calls tools, such as `openaiText` or `anthropicText`. -It does **not** ship a TanStack text adapter. Pair the search tool with a function-calling adapter such as `openaiText` or `anthropicText`. Sonar `chat()` still goes through [`openaiCompatible`](./openai-compatible.md) — Sonar already searches the web and does not accept custom tools. +Want Sonar, the Perplexity chat model, instead? See [Chat with Sonar](#chat-with-sonar). -## Installation +## 1. Install + +This page uses OpenAI for the chat. -react: @tanstack/ai @tanstack/ai-openai @tanstack/ai-perplexity -vue: @tanstack/ai @tanstack/ai-openai @tanstack/ai-perplexity -solid: @tanstack/ai @tanstack/ai-openai @tanstack/ai-perplexity -svelte: @tanstack/ai @tanstack/ai-openai @tanstack/ai-perplexity -preact: @tanstack/ai @tanstack/ai-openai @tanstack/ai-perplexity -angular: @tanstack/ai @tanstack/ai-openai @tanstack/ai-perplexity -vanilla: @tanstack/ai @tanstack/ai-openai @tanstack/ai-perplexity -octane: @tanstack/ai @tanstack/ai-openai @tanstack/ai-perplexity +react: @tanstack/ai @tanstack/ai-react @tanstack/ai-openai @tanstack/ai-perplexity +vue: @tanstack/ai @tanstack/ai-vue @tanstack/ai-openai @tanstack/ai-perplexity +solid: @tanstack/ai @tanstack/ai-solid @tanstack/ai-openai @tanstack/ai-perplexity +svelte: @tanstack/ai @tanstack/ai-svelte @tanstack/ai-openai @tanstack/ai-perplexity +preact: @tanstack/ai @tanstack/ai-preact @tanstack/ai-openai @tanstack/ai-perplexity +angular: @tanstack/ai @tanstack/ai-angular @tanstack/ai-openai @tanstack/ai-perplexity +vanilla: @tanstack/ai @tanstack/ai-client @tanstack/ai-openai @tanstack/ai-perplexity +octane: @tanstack/ai @tanstack/ai-octane @tanstack/ai-openai @tanstack/ai-perplexity octane -Set your API key (get one at