fix: handle google/gemini-3.5-flash thought token leak in inference - #7045
fix: handle google/gemini-3.5-flash thought token leak in inference#7045HoneyTyagii wants to merge 1 commit into
Conversation
|
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
There was a problem hiding this comment.
Devin Review found 2 potential issues.
1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)
|
|
||
| _MODEL_THINK_TAGS = { | ||
| "google/gemma-4-31b-it": ("<|channel>thought", "<channel|>"), | ||
| "google/gemini-3.5-flash": ("thought", ""), |
There was a problem hiding this comment.
🔴 Unpackaged fix never reaches users
The new _MODEL_THINK_TAGS entry sits outside the source tree selected by the workspace build. Installed agents retain the thought-token leak.
Prompt for agents
Move the Gemini model-tag configuration and the empty-end-tag filter change into the actual livekit-agents package under livekit-agents/livekit/agents/inference/llm.py and livekit-agents/livekit/agents/llm/utils.py. Remove the accidental top-level livekit/agents copies. Add a unit test that imports the packaged inference module and verifies the Gemini stream behavior so an out-of-package edit cannot pass unnoticed.
Was this helpful? React with 👍 or 👎 to provide feedback.
|
|
||
| _MODEL_THINK_TAGS = { | ||
| "google/gemma-4-31b-it": ("<|channel>thought", "<channel|>"), | ||
| "google/gemini-3.5-flash": ("thought", ""), |
There was a problem hiding this comment.
🟡 Ordinary replies lose “thought”
For Gemini 3.5 Flash, _MODEL_THINK_TAGS treats every occurrence of “thought” as a control token. Valid replies are silently altered before speech.
Prompt for agents
Restrict recognition of Gemini's bare thought marker to the exact protocol position or framing in which the gateway emits it, rather than searching all streamed response text. Preserve state across chunks so a split initial marker is still recognized, but never remove later natural-language occurrences. Add tests for marker framing, arbitrary chunk splits, and ordinary replies containing “thought”.
Was this helpful? React with 👍 or 👎 to provide feedback.
Add _MODEL_THINK_TAGS entry for google/gemini-3.5-flash with empty end tag to match the observed behavior where the thought token has no closing delimiter. The strip_thinking_tokens function already handles empty end tags correctly by treating the remainder as visible content after the start tag is found. Fixes issue where thought tokens were leaked into delta.content and spoken to callers via TTS/transcription. Closes livekit#6942
c4b18ef to
973f83b
Compare
STiFLeR7
left a comment
STiFLeR7
left a comment
There was a problem hiding this comment.
Re-checked this against the leak samples in #6942 and I think there's a residual gap in strip_thinking_tokens's end_tag == "" branch:
idx = state._buffer.find(state.start_tag)
if idx >= 0:
visible.append(state._buffer[:idx])
state._buffer = state._buffer[idx + len(state.start_tag) :]
if state.end_tag == "":
# For empty end tag, we do not look for an end tag: the rest is visible.
visible.append(state._buffer)
state._buffer = ""
breakThis strips only the literal 4-character marker "thought" and appends everything after it as visible — it doesn't consume the separator artifact that follows in the real leak shapes #6942 sampled:
thought[]I see you're calling from the number we have on file...
thought() I can help you with that...
thought [] Muddy waters...
After this filter, those become []I see you're calling..., () I can help you with that..., [] Muddy waters... — the "thought" word is gone, but the []/()/leading-space artifact immediately after it is still spoken/shown, since it's unconditionally appended along with the real content.
The issue body's own proposed app-side workaround already accounts for this shape:
_THOUGHT_PREFIX = re.compile(r"^thought\s*(?:\[[^\]]*\]|\([^)]*\))?\s*(?![a-z])")i.e. it strips "thought" plus an optional trailing [...]/(...) and whitespace, not just the bare word. Might be worth folding an equivalent pattern into the end_tag == "" branch here — e.g. matching start_tag plus an optional \s*(?:\[[^\]]*\]|\([^)]*\))?\s* immediately after it — so the artifact doesn't leak through even when this PR's fix is applied. (The 8/27 cases with a non-empty thought body and no separator at all are correctly called out in the issue as unfixable without a real delimiter — this is specifically about the empty-body cases where a separator artifact is present and could be stripped.)
|
closing since it's fixed in gateway |
Add _MODEL_THINK_TAGS entry for google/gemini-3.5-flash with empty end tag to match the observed behavior where the thought token has no closing delimiter. The strip_thinking_tokens function already handles empty end tags correctly by treating the remainder as visible content after the start tag is found.
Fixes issue where thought tokens were leaked into delta.content and spoken to callers via TTS/transcription.
Closes #6942