fix(google): drain realtime audio after tool calls#6285
Conversation
|
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| gen.function_ch.close() | ||
| self._schedule_generation_done_after_audio_drain(gen) |
There was a problem hiding this comment.
🚩 Behavioral change: text channel stays open during audio drain period
Previously, _mark_current_generation_done was called immediately on tool call, closing text_ch along with everything else. Now text_ch remains open during the audio drain window (up to 5 seconds). Any output transcription or text content arriving during this window will be delivered via push_text (livekit-plugins/livekit-plugins-google/livekit/plugins/google/realtime/realtime_api.py:1306) to the still-open text channel. This is likely desirable (capturing trailing transcription), but it's a semantic change that downstream consumers may not expect — text content could arrive after function calls have been signaled as complete.
Was this helpful? React with 👍 or 👎 to provide feedback.
| async def _mark_generation_done_after_audio_drain(self, gen: _ResponseGeneration) -> None: | ||
| started_at = time.monotonic() | ||
| last_audio_frame_at = gen._last_audio_frame_at | ||
|
|
||
| while True: | ||
| if gen._done: | ||
| return | ||
|
|
||
| elapsed = time.monotonic() - started_at | ||
| if elapsed >= TOOL_CALL_AUDIO_DRAIN_TIMEOUT: | ||
| break | ||
|
|
||
| await asyncio.sleep( | ||
| min(TOOL_CALL_AUDIO_DRAIN_QUIESCENCE, TOOL_CALL_AUDIO_DRAIN_TIMEOUT - elapsed) | ||
| ) | ||
|
|
||
| current_last_audio_frame_at = gen._last_audio_frame_at | ||
| if current_last_audio_frame_at == last_audio_frame_at: | ||
| break | ||
| last_audio_frame_at = current_last_audio_frame_at | ||
|
|
||
| self._mark_generation_done(gen) |
There was a problem hiding this comment.
🚩 Drain quiescence timer starts even when no audio has been received
When a tool call arrives for a generation that has no audio (_last_audio_frame_at is None), _schedule_generation_done_after_audio_drain still creates a drain task. The drain task sleeps for TOOL_CALL_AUDIO_DRAIN_QUIESCENCE (0.5s default), then breaks because None == None. This adds an unnecessary 0.5s delay before finalizing the generation. For text-only modalities or tool-call-only generations (like gen2 in a chained tool-call scenario), this delay is wasted. Consider checking gen._last_audio_frame_at is not None before scheduling the drain, or immediately finalizing if no audio was ever received.
Was this helpful? React with 👍 or 👎 to provide feedback.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Anrahya
force-pushed
the
fix/google-realtime-tool-audio-drain
branch
from
ce7b84b to
067f98a
Compare
2 participants
Closes #5742
Summary
audio -> toolCall -> trailing audioorderingWhy
Gemini Live exposes
toolCallas a distinct server message fromserverContent.turnComplete. The API reference definesturnCompleteas the model-turn completion signal, whiletoolCallasks the client to execute function calls and return responses. The Live API capabilities guide also notes that Gemini 3.1 Live server events can carry multiple content parts and clients should process all parts.Closing the generation at
toolCallcan therefore closeaudio_chbefore trailing Gemini audio chunks are delivered. This makesRunContext.wait_for_playout()resolve too early for custom end-call tools that wait for playout before tearing down the room.Tests
uv run pytest --plugin google tests/test_plugin_google_llm.pyuv run ruff check livekit-plugins/livekit-plugins-google/livekit/plugins/google/realtime/realtime_api.py tests/test_plugin_google_llm.pyuv run ruff format --check livekit-plugins/livekit-plugins-google/livekit/plugins/google/realtime/realtime_api.py tests/test_plugin_google_llm.pyuv run python scripts/check_types.py -- --no-sync