Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,11 @@ ones).
which the Discovery document lacks. `gemini-video.ts` builds one endpoint
per video model at sync time: Google's Interactions OpenAPI request with
`model` pinned and `response_format.duration` an enum from the model
page's `Output video 3s-15s` limit. The endpoint id is the model id and
page's `Output video 3s-15s` limit. A page that states resolution, fps
and a token rate instead (#697, Genie 4) is still video on the poll, and
the spec sync names it as a real-time stream with no fixed-duration
schema. A page that fails to fetch or parse rejects the video fetch, so
the sync keeps the stored video endpoints. The endpoint id is the model id and
the path is shared, so `GET /v1/openapi/gemini` merges them into one
operation whose body is an `anyOf` of components tagged
`x-modelschemas-model` (`mergeModelBodies`). The Models API lists these
Expand Down
367 changes: 367 additions & 0 deletions src/server/providers/fixtures/gemini-genie-4.md.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,367 @@
> ## AI Agent Context
> - Full documentation index: https://aistudio.google.com/llms.txt
> - Gemini Docs MCP server (consistent live info): https://gemini-api-docs-mcp.dev
> - Coding agent skills & setup guide: https://aistudio.google.com/docs/coding-agents.md

# Genie 4

Genie 4 (`models/genie-4`) is a real-time interactive world generation model on
the [Interactions API](/docs/interactions-overview). Seed a world from a text
prompt, a 16:9 reference image, or both, then explore and steer the live 24 fps
video and 48 kHz stereo audio stream in real time using keyboard movement
controls (`WASD` and arrow keys) and mid-session text events.

For offline, fixed-duration clip generation, multi-turn video editing, and
upsampling up to 4K, use
[Gemini Omni 1.2 Flash](/docs/models/gemini-omni-flash) instead.

{% button href="https://aistudio.google.com/live?model=genie-4" variant="ais" %}Try in Google AI Studio{% /button %}

## Documentation {% id="documentation" %}

* [Interactive world generation with Genie 4](/docs/genie): End-to-end setup
with `@google/genai` (WebRTC and WebSocket) and `google-genai` (async
WebSocket), ephemeral session tokens, event protocol, and media stream
decoding.
* [Genie 4 prompt and controls guide](/docs/genie-prompting): Tested prompt
patterns for causal object rules, custom key-bound action verbs, 16:9
landscape painting seeds, broadband Foley audio, and Google AI Studio Live
playground controls.

## Quickstart {% id="quickstart" %}

Genie 4 sessions require a [paid-tier Gemini API key](/docs/pricing#genie-4)
(`~$0.10/sec` of streamed output) and run over a live bidirectional connection
on `v1beta`. Always call `complete()` when ending a session so server-side
generation and token billing stop immediately.

{% tabs %}
{% tab label=JavaScript (WebRTC) %}
```javascript
import { GoogleGenAI } from '@google/genai';

// 1. Server-side: create the interaction and mint a single-session auth token
const serverAi = new GoogleGenAI({
apiKey: process.env.GEMINI_API_KEY,
apiVersion: 'v1beta',
});
const interaction = await serverAi.interactions.create({
model: 'models/genie-4',
api_version: 'v1beta',
});
const { auth_token } = await serverAi.interactions.authTokens.create(
interaction.id,
{ api_version: 'v1beta' }
);

// 2. Browser: connect over WebRTC and attach a <video autoplay playsinline> element
const clientAi = new GoogleGenAI({ apiKey: auth_token, apiVersion: 'v1beta' });
const session = await clientAi.interactions.connect(
interaction.id,
{ transport: 'webrtc', api_version: 'v1beta' },
{ timeout_ms: 30000 }
);
session.transport.setVideoElement(document.querySelector('#world-video'));

// 3. Seed the interactive world
await session.send({
event_type: 'world_setup',
world_prompt: {
text: 'Immersive first-person POV standing on a cobblestone canal bridge inside a vibrant, thick-impasto post-impressionist oil painting of a starry riverside village at twilight. Swirling cobalt-blue sky currents and glowing golden cafe lanterns shimmer with living, tactile brushstroke textures that ripple gently in the breeze, with a wooden rowboat moored at the stone steps ahead. Audio: Gentle water lapping against the stone steps, leaves rustling in a soft evening breeze, footsteps on cobblestone.',
},
});

// 4. Steer movement with native WASD / arrow keys, then end billing cleanly
await session.send({
event_type: 'user_action',
action: 'key_down',
input: 'key_w',
});
setTimeout(async () => {
await session.send({
event_type: 'user_action',
action: 'key_up',
input: 'key_w',
});
await session.complete();
}, 5000);
```

{% /tab %}

{% tab label=Python (WebSocket) %}
```python
import asyncio
import base64
from google import genai

WORLD_PROMPT = (
"Immersive first-person POV standing on a cobblestone canal bridge inside "
"a vibrant, thick-impasto post-impressionist oil painting of a starry "
"riverside village at twilight. Swirling cobalt-blue sky currents and "
"glowing golden cafe lanterns shimmer with living, tactile brushstroke "
"textures that ripple gently in the breeze, with a wooden rowboat moored "
"at the stone steps ahead. Audio: Gentle water lapping against the stone "
"steps, leaves rustling in a soft evening breeze, footsteps on cobblestone."
)


async def main():
client = genai.Client(http_options={"api_version": "v1beta"}) # Reads GEMINI_API_KEY
interaction = await client.aio.interactions.create(
model="models/genie-4", api_version="v1beta"
)

async with client.aio.interactions.connect(
interaction.id, api_version="v1beta"
) as session:
await session.send(
{
"event_type": "world_setup",
"world_prompt": {"text": WORLD_PROMPT},
}
)

frames = 0
async for event in session.receive():
if (
event.event_type == "step.delta"
and getattr(event.delta, "type", None) == "video"
):
frames += 1
if frames == 1:
with open("frame_001.jpg", "wb") as f:
f.write(base64.b64decode(event.delta.data))
await session.send(
{
"event_type": "user_action",
"action": "key_down",
"input": "key_w",
}
)
if frames == 72: # ~3 seconds at 24 fps
await session.complete()
elif event.event_type == "interaction.completed":
break


asyncio.run(main())
```

{% /tab %}

{% /tabs %}

## genie-4 {% id="genie-4" %}

{% table variant="model" %}
* Property
* Description

---

* {% section class="model-property" %}
{% icon name="id_card" /%}Model code

{% /section %}

* **Gemini API**
`models/genie-4` (also accepts `genie-4`)

---

* {% section class="model-property" %}
{% icon name="save" /%}Supported data types

{% /section %}

* **Inputs** Text (`world_setup` and mid-session `input` steering), 16:9 Reference Image (`world_setup`), Keyboard and camera controls (`user_action`)

**Output** Streaming Video (`16:9`, `24 fps`) with synchronized `48 kHz` stereo Audio


---

* {% section class="model-property" %}
{% icon name="token_auto" /%}Stream & session limits

{% /section %}

* **Output video** `1280x720` (`720p`, `16:9`, `24 fps`), `240` video output tokens per frame (`5,760` tokens/sec)

**Max session duration** Up to `120 seconds` per session (visual coherence and object permanence are strongest during the first `45-90 seconds`)

**Output audio** `48 kHz` stereo (`16-bit` PCM over WebSocket; Opus over WebRTC), bundled into video output tokens

**Reference image** Single `16:9` image in `world_setup` (keep compressed `JPEG` under `48 KB` on WebRTC so the base64 JSON event stays within the `64 KB` SCTP data channel limit)

**Serving regions** United States, Western Europe, and Japan (`< 150 ms` RTT in North America, Western/Central Europe, and East Asia; higher latency outside these regions delays steering and can reduce frame rate)


---

* {% section class="model-property" %}
{% icon name="handyman" /%}Capabilities

{% /section %}

* {% section class="three-column-grid" %}
{% section %}
[**Interactive world generation**](/docs/genie)

{% badge type="supported" /%}

{% /section %}

{% section %}
[**Audio generation**](/docs/genie)

{% badge label=Supported (bundled Foley & speech) type="supported" /%}

{% /section %}

{% section %}
[**Code execution**](/docs/code-execution)

{% badge type="not-supported" /%}

{% /section %}

{% section %}
[**Context caching**](/docs/caching)

{% badge type="not-supported" /%}

{% /section %}

{% section %}
[**File search**](/docs/file-search)

{% badge type="not-supported" /%}

{% /section %}

{% section %}
[**Function calling**](/docs/function-calling)

{% badge type="not-supported" /%}

{% /section %}

{% section %}
[**Grounding with Google Maps**](/docs/maps-grounding)

{% badge type="not-supported" /%}

{% /section %}

{% section %}
[**Grounding with Google Search**](/docs/google-search)

{% badge type="not-supported" /%}

{% /section %}

{% section %}
[**Image generation**](/docs/image-generation)

{% badge type="not-supported" /%}

{% /section %}

{% section %}
[**Live API**](/docs/live-api)

{% badge label="Uses Interactions API" type="not-supported" /%}

{% /section %}

{% section %}
[**Structured outputs**](/docs/structured-output)

{% badge type="not-supported" /%}

{% /section %}

{% section %}
[**URL context**](/docs/url-context)

{% badge type="not-supported" /%}

{% /section %}

{% /section %}


---

* {% section class="model-property" %}
{% icon name="speed" /%}Consumption options

{% /section %}

* {% section class="three-column-grid" %}
{% section %}
[**Batch API**](/docs/batch-api)

{% badge type="not-supported" /%}

{% /section %}

{% section %}
[**Flex inference**](/docs/flex-inference)

{% badge type="not-supported" /%}

{% /section %}

{% section %}
[**Priority inference**](/docs/priority-inference)

{% badge type="not-supported" /%}

{% /section %}

{% /section %}


---

* {% section class="model-property" %}
{% icon name="123" /%}Versions

{% /section %}

* Read the [model version patterns](/docs/models#model-versions) for more details.

* Stable: `models/genie-4`


---

* {% section class="model-property" %}
{% icon name="calendar_month" /%}Latest update

{% /section %}

* November 2026

---

* {% section class="model-property" %}
{% icon name="paid" /%}Pricing

{% /section %}

* [Pricing](/docs/pricing#genie-4)

---

* {% section class="model-property" %}
{% icon name="speed" /%}Rate limits

{% /section %}

* [Rate limits](/docs/rate-limits#genie-4)
{% /table %}

Loading
Loading