Repository navigation
SN-2b: speech through Gecko Suite behind AI_BACKEND - #215
Merged
Merged
Conversation
Read-aloud, narrate, POST /speech/tts (its provider pin becomes the suite model ref), the video renderer's narrator, dictation uploads and the recording-transcription job call the suite's /ai/speech/* in suite mode. The renderer learns its model and voice from the first chunk and sends them back, so a narration never changes voice. Notes records no usage and keeps no audio cache in this mode. Live transcription and voice mode are unchanged. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V1geD2ELWgL2sDpxYbx3Aw
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Second piece of SN-2 (image generation is SN-2c). With
AI_BACKEND=suite, Notes' speech goes to the suite's/api/v1/ai/speech/*.localis unchanged.Text to speech (
synthesize_tts_bytes, so read-aloud,/speech/narrate,/speech/tts, and the video renderer all follow):speech.ttsrouting picks the model; voices the model does not offer are replaced by its default (as before). The suite caches and bills, so Notes keeps no audio cache and writes no usage in this mode.X-Speech-Model,X-Speech-Voice) and sends them back for every later chunk. A model that fails fails the render; there is no fallback to another engine, which is what made Notes' "auto" change narrator partway through.POST /speech/ttswithproviderkeeps its contract for other apps:X-Speech-Provideris now the suite model ref (opaque), sent back asprovider.provideris therefore a plain string in the request model; the local path still rejects anything other thanauto/deepgram/falwith a 422.Speech to text (the suite's Deepgram):
POST /speech/transcribe(dictation upload) returns{"text"}as before.Not changed: live (streaming) transcription and voice mode (
stt_stream,flux_streamwebsockets) still use Notes' own Deepgram key, since the suite has no streaming STT. Image generation is SN-2c.Test plan
pytest -qinbackend/: 1374 passed, 1 skipped (Python 3.13 locally; CI is 3.11). 9 new tests intests/test_suite_proxy.pythrough the real routes with a fake suite: forwarding and credentials, no local usage, pin round trip, a failing pinned model never switching engine, chunked narration, the local 422, dictation upload, the suite's refusal sentence, the renderer's pinning, and_run_jobend to end with a fake ffmpeg and no fal key.Deployment
Nothing changes unless
AI_BACKEND=suiteis set (from #213).speech.ttsto a model with thettscapability (the fal.ai/Deepgram voices list I prepared can be loaded into a provider), and add a Deepgram provider for transcription. Without them, read-aloud and transcription answer with the suite's "not configured" sentence.ttsunder appnotes), render a narrated video, and transcribe a recording.AI_BACKEND=local,docker compose up -d.Generated by Claude Code