Repository navigation
Voice: make the transcription language configurable (currently hard-coded to "auto") #5226
Description
Describe the feature or problem you'd like to solve
Voice dictation (/voice) works with the multilingual Nemotron streaming model, but the voice worker always starts
the live transcription session with automatic language detection. For non-English speakers this means the first
sentence of every dictation is recognised noticeably worse.
Environment: Copilot CLI 1.0.101 (the same code is in 1.0.96–1.0.100), foundry-local-sdk 2.1.0, model
nemotron-3.5-asr-streaming-0.6b-generic-cpu:3, macOS arm64 (Apple Silicon).
voice-engine.worker.js → openStreamingSession() sets the language before starting the session (minified):
let r = t.createAudioClient().createLiveTranscriptionSession();
r.settings.language = "auto";
try { await r.start() } catch (i) { … }The model is prompt-conditioned on a language id (lang_id). With "auto" it commits to a language only at the end
of the first sentence (it emits a <de-DE> tag there), so the first sentence is decoded without the language prior.
Measured with a small Node script that makes the same SDK calls as the worker (createLiveTranscriptionSession,
settings.language, start, append in real-time-paced 100 ms chunks of 16 kHz mono s16le) on German TTS samples,
each with 1 s of leading silence so that the whole first sentence reaches the model. Reference: "Bitte prüfe, warum
der Build in der Pipeline fehlschlägt. Schau dir dazu die Logs im Kubernetes-Cluster an …". First sentence of the
result:
settings.language |
First sentence |
|---|---|
"auto" |
Bitte prüfe, warum der Bult in der Pipeline fehlt. |
"de-DE" or "de" |
Bitte prüfe, warum der Bult in der Pipeline fehlschlägt. |
"en-US" |
German speech is dropped |
| unset or numeric | no text at all |
The difference is reproducible; the rest of the text is identical once the model has detected the language.
Proposed solution
Make the language configurable: a setting (e.g. voice.language, default "auto") and/or /voice language de-DE,
passed to settings.language. Validate the value against the tags the model supports, since invalid values silently
produce no text and a wrong language drops speech.
Additional context
- Related: multilingual voice input #3658 (multilingual voice input). This request is narrower: the bundled model already handles other
languages; only the language hint is hard-coded. - A user-level workaround exists: patching the minified worker in the CLI package cache, replacing
X.settings.language="auto"withX.settings.language="de-DE". It works, but has to be re-applied after every CLI
update.
Activity