Skip to content

SN-2b: speech through Gecko Suite behind AI_BACKEND - #215

Merged
davior merged 1 commit into
mainfrom
claude/beautiful-wright-og4xur
Nov 4, 2026
Merged

davior merged 1 commit into
mainfrom
claude/beautiful-wright-og4xur

Conversation

@davior

@davior davior commented Nov 4, 2026

Copy link
Copy Markdown
Owner

Summary

Second piece of SN-2 (image generation is SN-2c). With AI_BACKEND=suite, Notes' speech goes to the suite's /api/v1/ai/speech/*. local is unchanged.

Text to speech (synthesize_tts_bytes, so read-aloud, /speech/narrate, /speech/tts, and the video renderer all follow):

  • The suite's speech.tts routing picks the model; voices the model does not offer are replaced by its default (as before). The suite caches and bills, so Notes keeps no audio cache and writes no usage in this mode.
  • One narration, one voice. The renderer sends no pin on its first chunk, learns the model and voice the suite resolved (X-Speech-Model, X-Speech-Voice) and sends them back for every later chunk. A model that fails fails the render; there is no fallback to another engine, which is what made Notes' "auto" change narrator partway through.
  • POST /speech/tts with provider keeps its contract for other apps: X-Speech-Provider is now the suite model ref (opaque), sent back as provider. provider is therefore a plain string in the request model; the local path still rejects anything other than auto/deepgram/fal with a 422.

Speech to text (the suite's Deepgram):

  • POST /speech/transcribe (dictation upload) returns {"text"} as before.
  • The recording-transcription job sends its extracted wav to the suite and saves the text, with no fal key needed. Stages and cancellation are unchanged.

Not changed: live (streaming) transcription and voice mode (stt_stream, flux_stream websockets) still use Notes' own Deepgram key, since the suite has no streaming STT. Image generation is SN-2c.

Test plan

  • pytest -q in backend/: 1374 passed, 1 skipped (Python 3.13 locally; CI is 3.11). 9 new tests in tests/test_suite_proxy.py through the real routes with a fake suite: forwarding and credentials, no local usage, pin round trip, a failing pinned model never switching engine, chunked narration, the local 422, dictation upload, the suite's refusal sentence, the renderer's pinning, and _run_job end to end with a fake ffmpeg and no fal key.
  • Not run against the real suite, Deepgram or fal.

Deployment

Nothing changes unless AI_BACKEND=suite is set (from #213).

  • Before enabling: in the suite, route speech.tts to a model with the tts capability (the fal.ai/Deepgram voices list I prepared can be loaded into a provider), and add a Deepgram provider for transcription. Without them, read-aloud and transcription answer with the suite's "not configured" sentence.
  • Behaviour change to expect: the suite does not fall back from Deepgram to fal, so a Notes user who relied on "auto" fallback gets an error instead of a different voice. Voice and engine choice is now the suite's routing, not Settings → Speech.
  • Dictation and transcription use Deepgram only in this mode (Notes used fal Wizper unless set to Deepgram). The suite caps one upload at 300 MB (about 2.6 hours of the 16 kHz mono wav the job sends).
  • Configuration: none new. Services rebuilt: backend only. Migrations, Caddy: none.
  • Verify: in suite mode, use read-aloud on a note (audio plays; suite Usage shows tts under app notes), render a narrated video, and transcribe a recording.
  • Roll back: AI_BACKEND=local, docker compose up -d.

Generated by Claude Code

Read-aloud, narrate, POST /speech/tts (its provider pin becomes the suite model ref),
the video renderer's narrator, dictation uploads and the recording-transcription job
call the suite's /ai/speech/* in suite mode. The renderer learns its model and voice
from the first chunk and sends them back, so a narration never changes voice. Notes
records no usage and keeps no audio cache in this mode. Live transcription and voice
mode are unchanged.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1geD2ELWgL2sDpxYbx3Aw
@davior
davior marked this pull request as ready for review November 4, 2026 09:13
@davior
davior merged commit 46f8e2d into main Nov 4, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants