Skip to content

fix(soniox): keep references to the prewarm and self-close tasks - #7051

Open
Rehansanjay wants to merge 2 commits into
livekit:mainfrom
Rehansanjay:fix/soniox-tts-task-references
Open

fix(soniox): keep references to the prewarm and self-close tasks#7051
Rehansanjay wants to merge 2 commits into
livekit:mainfrom
Rehansanjay:fix/soniox-tts-task-references

Conversation

@Rehansanjay

Copy link
Copy Markdown
Contributor

The bug

Five asyncio.create_task calls in soniox/tts.py discard the task they create. The event loop only holds a weak reference to a bare task, so it can be garbage collected before it finishes.

_Connection self-close (four sites)

The connection closes itself from mark_non_current, unregister_stream, and the finally of both _send_loop and _recv_loop:

asyncio.create_task(self.aclose(), name="soniox-tts-conn-drain-close")

If that task is collected, aclose() never runs and the WebSocket stays open.

Worth noting the class already gets this right for its own loops β€” _send_task, _recv_task and _keepalive_task are all stored on the instance. The close tasks were the exception.

TTS.prewarm β€” an ordering problem too

async def _task() -> None:
    await self._current_connection(timeout=20.0)

asyncio.create_task(_task(), name="soniox-tts-prewarm")

_current_connection opens a new _Connection when there is none, and aclose() sets __current_connection = None after closing. So a prewarm still in flight when the TTS is closed reconnects afterwards, leaving a live WebSocket that nothing owns and nothing will close.

The fix

The four self-close sites go through a small helper that keeps the task and skips scheduling a second close while one is still running:

def _schedule_close(self, name: str) -> None:
    if self._close_task is not None and not self._close_task.done():
        return
    self._close_task = asyncio.create_task(self.aclose(), name=name)

prewarm stores its task, and aclose() cancels it before tearing the connection down β€” the same shape as openai/tts.py, and what utils.ConnectionPool.prewarm() does for the plugins that use it.

No behaviour change on the success path.

Checks

ruff check and ruff format --check pass on the changed file.


Found with a small AST pass looking for create_task calls whose result is discarded β€” the same sweep that turned up #7050.

Five `asyncio.create_task` calls in this module threw the task away. The
event loop only holds a weak reference to a bare task, so it can be
garbage collected before it finishes.

`_Connection` self-closes from four places β€” `mark_non_current`,
`unregister_stream`, and the `finally` of both the send and recv loops.
Each one discarded the task that closes the WebSocket, so a collected
task means the socket stays open. The class already stores its send,
recv and keepalive tasks; these were the exception. They now go through
`_schedule_close`, which also skips scheduling a second close while one
is still in flight.

`TTS.prewarm` had the ordering problem as well: its task calls
`_current_connection`, which opens a new connection when there is none,
and `aclose()` sets `__current_connection` to None. A prewarm in flight
when the TTS closed would reconnect afterwards and leave a live WebSocket
that nothing owns. It is now cancelled at the top of `aclose()`.
@Rehansanjay
Rehansanjay requested a review from a team as a code owner August 31, 2026 06:31
devin-ai-integration[bot]

This comment was marked as resolved.

Two follow-ups from the review.

`prewarm()` overwrote `_prewarm_task` unconditionally, so calling it twice
dropped the only reference to the first task while it was still running.
`aclose()` cancels just the latest one, which left the earlier attempt
free to open a WebSocket after shutdown β€” the leak this PR is meant to
close. It now starts a new task only when there is no live one, which is
how `utils.ConnectionPool.prewarm` guards the same case.

The prewarm failure log also interpolated the exception into the message
body, where it cannot be redacted. It now goes through `exc_info`, as the
send and recv loops in this file already do.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 new potential issue.

Devin Review

await self._current_connection(timeout=20.0)
except Exception as e:
logger.debug(f"Soniox TTS prewarm failed: {e}")
logger.debug("Soniox TTS prewarm failed", exc_info=e)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟨 Prewarm failures expose exception details

A failed prewarm logs the complete exception traceback through exc_info. Provider errors can expose credentials or customer data in unredactable log text.

Devin Review

Was this helpful? React with πŸ‘ or πŸ‘Ž to provide feedback.

@Rehansanjay

Copy link
Copy Markdown
Contributor Author

Both review points addressed in a1569b1.

Repeated prewarm leaks a connection β€” correct, and it undercut the point of the PR. prewarm() replaced _prewarm_task unconditionally, so a second call dropped the only reference to a task that was still running, and aclose() cancels just the latest one. It now starts a task only when there is no live one, which is how utils.ConnectionPool.prewarm guards the same case.

Prewarm errors bypass log redaction β€” moved to exc_info, matching the send and recv loops in this same file.

@Rehansanjay

Copy link
Copy Markdown
Contributor Author

On the remaining open thread β€” the exc_info one β€” I do not think it is actionable, and I would rather say so than quietly churn.

The earlier review on this series flagged the opposite problem: the prewarm failure interpolated the exception into the message body, where redaction cannot reach it. I moved it to exc_info, which is what the send and recv loops in this same file already do. The new finding says exc_info is itself the problem.

Taken literally that leaves nowhere to put the exception, and it would put this call site out of step with the rest of the codebase: exc_info= appears at 99 call sites across livekit-agents and livekit-plugins today. Dropping the exception entirely would mean a prewarm or a recycle can fail and leave an operator with a log line and no cause.

If the project does want a stricter rule for provider exceptions β€” a redacting formatter, or an lk.pii. structured field rather than exc_info β€” I am happy to follow it here and it would probably be worth applying repo-wide rather than to these two lines. Left as-is for now pending your view.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant