feat(speechify): streaming TTS with word-level timestamps via official SDK#6327
feat(speechify): streaming TTS with word-level timestamps via official SDK#6327luke-speechify wants to merge 5 commits into
Conversation
|
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| async def _synthesize() -> None: | ||
| offset = 0.0 | ||
| async for ev in sent_stream: | ||
| if not (text := ev.token.strip()): | ||
| continue | ||
| response = await self._tts._client.audio.speech( | ||
| **_request_kwargs(text, self._opts), | ||
| request_options={"timeout_in_seconds": int(self._conn_options.timeout)}, | ||
| ) | ||
| audio = base64.b64decode(response.audio_data) | ||
| timed = _timed_transcript(response.speech_marks, offset) | ||
| if timed: | ||
| output_emitter.push_timed_transcript(timed) | ||
| output_emitter.push(audio) | ||
| offset += len(audio) / (2 * SAMPLE_RATE * NUM_CHANNELS) | ||
|
|
||
| output_emitter.end_segment() |
There was a problem hiding this comment.
🟡 Streaming synthesis never emits performance metrics because the timer is never started
The streaming synthesis path never records its start time (_mark_started() is never called in SynthesizeStream._run at livekit-plugins/livekit-plugins-speechify/livekit/plugins/speechify/tts.py:280), so the metrics-collection callback always exits early.
Impact: Operators get no time-to-first-byte or duration metrics for Speechify streaming TTS, making performance monitoring impossible.
Metrics guard in the base class silently discards all data
The base class SynthesizeStream._metrics_monitor_task (livekit-agents/livekit/agents/tts/tts.py:573-577) checks if not self._started_time before emitting any metrics. _started_time is initialized to 0 (livekit-agents/livekit/agents/tts/tts.py:439), and the only way to set it to a non-zero value is by calling self._mark_started() (livekit-agents/livekit/agents/tts/tts.py:561-564). Other TTS plugins (e.g., Cartesia) call self._mark_started() before sending data to the provider. The Speechify SynthesizeStream._run never calls it, so _started_time remains 0, not 0 evaluates to True, and _emit_metrics returns without emitting anything.
| async def _synthesize() -> None: | |
| offset = 0.0 | |
| async for ev in sent_stream: | |
| if not (text := ev.token.strip()): | |
| continue | |
| response = await self._tts._client.audio.speech( | |
| **_request_kwargs(text, self._opts), | |
| request_options={"timeout_in_seconds": int(self._conn_options.timeout)}, | |
| ) | |
| audio = base64.b64decode(response.audio_data) | |
| timed = _timed_transcript(response.speech_marks, offset) | |
| if timed: | |
| output_emitter.push_timed_transcript(timed) | |
| output_emitter.push(audio) | |
| offset += len(audio) / (2 * SAMPLE_RATE * NUM_CHANNELS) | |
| output_emitter.end_segment() | |
| async def _synthesize() -> None: | |
| offset = 0.0 | |
| async for ev in sent_stream: | |
| if not (text := ev.token.strip()): | |
| continue | |
| self._mark_started() | |
| response = await self._tts._client.audio.speech( | |
| **_request_kwargs(text, self._opts), | |
| request_options={"timeout_in_seconds": int(self._conn_options.timeout)}, | |
| ) | |
| audio = base64.b64decode(response.audio_data) | |
| timed = _timed_transcript(response.speech_marks, offset) | |
| if timed: | |
| output_emitter.push_timed_transcript(timed) | |
| output_emitter.push(audio) | |
| offset += len(audio) / (2 * SAMPLE_RATE * NUM_CHANNELS) | |
| output_emitter.end_segment() |
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Good catch — fixed in 1745975. SynthesizeStream._run now calls self._mark_started() before the first synthesis request, matching the Cartesia plugin, so streaming TTS metrics are emitted.
| from .models import TTSModels | ||
| from .tts import DEFAULT_VOICE_ID, TTS, ChunkedStream, SynthesizeStream | ||
| from .version import __version__ | ||
|
|
||
| __all__ = [ | ||
| "TTS", | ||
| "Voice", | ||
| "TTSEncoding", | ||
| "ChunkedStream", | ||
| "SynthesizeStream", | ||
| "TTSModels", | ||
| "DEFAULT_VOICE_ID", | ||
| "__version__", | ||
| ] |
There was a problem hiding this comment.
🔍 Breaking public API: Voice class and TTSEncoding removed from exports
The old __init__.py exported Voice and TTSEncoding in __all__. The new version removes both and replaces them with ChunkedStream and SynthesizeStream. Any downstream code doing from livekit.plugins.speechify import Voice or from livekit.plugins.speechify import TTSEncoding will break at import time. The Voice dataclass (which held voice metadata like id, display_name, gender, models, locale) and the list_voices() method on TTS are also removed entirely, so there's no migration path for users who relied on voice enumeration. This may be intentional given the SDK migration, but should be documented in release notes.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Restored list_voices() on the TTS class (now backed by the official SDK's voices.list()), and re-exported GetVoice from the package, so voice enumeration has a migration path. The old TTSEncoding is intentionally dropped — synthesis is now fixed to 24 kHz PCM (the SDK's stream accept-set no longer exposes container/rate selection); noted for release notes.
| def _raise_from(e: Exception) -> None: | ||
| if isinstance(e, ApiError): | ||
| raise APIStatusError( | ||
| message=str(e.body) if e.body is not None else "Speechify API error", | ||
| status_code=e.status_code or -1, | ||
| request_id=None, | ||
| body=None, | ||
| ) from None | ||
| if isinstance(e, asyncio.TimeoutError): | ||
| raise APITimeoutError() from None | ||
| raise APIConnectionError() from e |
There was a problem hiding this comment.
🔍 Error wrapping catches non-API exceptions and misclassifies them
The except Exception as e: _raise_from(e) pattern at lines 249-250 and 304-305 catches every exception type. If a non-API exception occurs (e.g., base64.binascii.Error from malformed audio_data, AttributeError from an unexpected response shape, or RuntimeError from the emitter), _raise_from falls through to its default branch and wraps it as APIConnectionError. This misclassifies the error — a decode failure isn't a connection problem — and the APIConnectionError has retryable=True by default (livekit-agents/livekit/agents/_exceptions.py:106), so the framework will retry a request that will deterministically fail again. Other plugins in the repo (e.g., Deepgram, Cartesia) use more targeted exception handling. Consider adding an early check like if isinstance(e, APIError): raise before the wrapping logic.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Fixed. _raise_from now re-raises existing APIError subclasses unchanged (if isinstance(e, APIError): raise e) before the fallback, so decode/attribute errors are no longer misclassified as retryable APIConnectionError. Matches the targeted handling in the other plugins.
…l SDK Upgrade the Speechify plugin to stream audio with aligned word-level timestamps. Synthesis now uses the official speechify-api SDK and chunks streamed input into sequential /audio/speech calls, emitting audio and timed transcripts per sentence (streaming + aligned_transcript capabilities). Route maintenance to the Speechify-owned repository.
luke-speechify
force-pushed
the
lo/speechify-streaming-tts
branch
from
1745975 to
c008b99
Compare
2 participants
Summary
Upgrades the
livekit-plugins-speechifyTTS plugin to stream audio with word-level timestamps, using Speechify's officialspeechify-apiSDK.Previously the plugin was non-streaming (
streaming=False) and used rawaiohttpagainst/audio/stream(binary audio only, no timing metadata). Speechify's word-level speech marks are only returned by the batch/audio/speechendpoint, so this implementation chunks streamed input into per-sentence sequential/audio/speechcalls, emitting audio and aligned word timestamps as each sentence completes — giving near-streaming time-to-first-audio andaligned_transcriptsupport.Changes
TTSCapabilities(streaming=True, aligned_transcript=True)SynthesizeStream: sentence-chunked sequential synthesis with cumulative word-mark time offsetsChunkedStream: one-shot synthesis with word marksspeechify-apivendor SDK instead of hand-rolled HTTPlivekit-plugins-gnaniprecedent):Repository/Issuespoint at Speechify's maintenance repoMaintenance & distribution
This plugin is maintained by Speechify and distributed by LiveKit as part of
livekit/agents. Speechify keeps a mirror of the plugin source at Speechify-AI/livekit-plugin-python for maintenance and issue triage, and proposes changes upstream here. The canonical, user-installable package remainslivekit-plugins-speechify, published by LiveKit — there is no separate Speechify-published package.Verification
ruff check+ruff format --checkcleanmypycleantests/test_tts.pyround-tripHappy to adjust to match maintainer preferences.