Skip to content

feat(soniox): populate SpeechData.words with per-word timing and confidence#6292

Draft
FurlanLuka wants to merge 1 commit into
livekit:mainfrom
FurlanLuka:soniox-timed-words
Draft

feat(soniox): populate SpeechData.words with per-word timing and confidence#6292
FurlanLuka wants to merge 1 commit into
livekit:mainfrom
FurlanLuka:soniox-timed-words

Conversation

@FurlanLuka

Copy link
Copy Markdown

What

The Soniox websocket API returns per-token confidence, start_ms and end_ms, but the plugin currently keeps only the per-transcript mean confidence β€” the per-token metadata is dropped. This PR groups consecutive sub-word tokens ("walk" + "ing") into words and exposes them as TimedString entries on SpeechData.words, following the Deepgram plugin's precedent (offset applied to times, start_time_offset attribute set).

Word confidence = mean of its tokens' confidences; word timing spans first token start_ms β†’ last token end_ms.

Why

Word-level ASR confidence enables downstream consumers to detect likely mistranscriptions. Our production use case (language-learning voice agent): the agent issues grammar corrections against phrases the learner never said β€” ~16% of corrections traced to ASR artifacts. A plausible-in-context mishear ("walking" β†’ "working") is invisible to text-only downstream logic but shows up as a per-word confidence dip. The scalar per-transcript mean (added recently) is too coarse: one garbled word barely moves the mean of a long utterance.

Word-boundary rules

  • Words split at whitespace (Soniox marks word starts with a leading space in token text) β€” correct for 57 of the 60 documented supported languages.
  • The three unspaced scripts in Soniox's catalog β€” Han (zh), Kana (ja), Thai (th) β€” never produce that whitespace, so token boundaries are used as word boundaries there (Soniox emits roughly word/character-granularity tokens for these scripts).
  • Punctuation tokens attach to the preceding word, matching Deepgram's behavior.

Open question for Soniox folks: is a leading space in token["text"] a guaranteed word-start marker for spaced languages? If the server could emit an explicit boundary flag, the script-range heuristic here could be deleted.

Notes

  • Purely additive β€” text, scalar confidence and timing fields are untouched; words was previously always None for this plugin.
  • In translation mode, words follows the same side as text/confidence (the target side).
  • On INTERIM_TRANSCRIPT/PREFLIGHT_TRANSCRIPT (merged_speech_data), a word straddling the final/non-final seam may appear split until the endpoint β€” same instability interim text already has.
  • speaker_id per word is first-token-wins, consistent with the transcript-level field.

Testing

8 new tests in tests/test_plugin_soniox_stt.py (existing fake-WebSocket harness): sub-word grouping + confidence mean, start_time_offset application, Han/Thai token-boundary words, NOT_GIVEN fallbacks when metadata absent, reset() clearing, merged final+non-final words, and end-to-end FINAL_TRANSCRIPT assertions. ruff check, ruff format and scripts/check_types.py (mypy) all pass.

πŸ€– Generated with Claude Code

…idence

Soniox streams sub-word tokens that each carry confidence, start_ms and
end_ms, but the plugin previously kept only the per-transcript mean
confidence and dropped the per-token metadata. Group consecutive tokens
into words at whitespace boundaries (or at token boundaries for the
unspaced scripts among Soniox's supported languages: Han, Kana and Thai)
and expose them as TimedString entries on SpeechData.words, with each
word's confidence being the mean of its tokens' confidences.
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants