fix(llm): tolerate list-shaped delta content in strip_thinking_tokens#6324
fix(llm): tolerate list-shaped delta content in strip_thinking_tokens#6324KSerProject wants to merge 1 commit into
Conversation
OpenAI-compatible providers (e.g. Mistral) may stream delta.content as
a list of typed content parts. strip_thinking_tokens assumed str and
crashed with AttributeError ('list' object has no attribute 'find'),
which surfaces as a retryable APIConnectionError mid-stream and makes a
FallbackAdapter silently switch the whole session to its fallback leg.
Flatten text parts before the tag scan.
|
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| if isinstance(content, list): | ||
| # OpenAI-compatible providers (e.g. Mistral) may stream delta.content as a | ||
| # list of typed content parts instead of a plain string. | ||
| content = ( | ||
| "".join(part.get("text", "") for part in content if isinstance(part, dict)) | ||
| or None | ||
| ) |
There was a problem hiding this comment.
🚩 Joining list parts may increase likelihood of thinking token leakage
The thinking token state machine in strip_thinking_tokens processes only one tag transition per call — it finds either <think> or </think> but not both. When content was always a plain string from streaming, each chunk was typically small enough that both tags rarely appeared in the same chunk. With the new list joining (livekit-agents/livekit/agents/llm/utils.py:616), multiple content parts are concatenated into a single string, making it more likely that both <think> and </think> appear in one call. In that case, only the first tag is processed: if <think>reasoning</think>actual arrives as a single joined string, the function sets the thinking flag and returns "reasoning</think>actual", leaking the model's internal reasoning to the user. This is a pre-existing design limitation, but the list-joining makes it a more plausible scenario.
Was this helpful? React with 👍 or 👎 to provide feedback.
2 participants
OpenAI-compatible providers (e.g. Mistral) may stream
delta.contentas a list of typed content parts instead of a plain string.strip_thinking_tokensassumedstrand crashed withAttributeError, which surfaces as a retryableAPIConnectionErrormid-stream and makes aFallbackAdaptersilently switch the whole session to its fallback leg.This flattens the text parts before the tag scan; plain strings and
Noneare untouched.Fixes #6323