Repository navigation
feat(qwen3-local): third offline engine — Qwen3-TTS via qwen3-tts.cpp, with zero-shot cloning - #49
Merged
Merged
Conversation
…, with zero-shot cloning Local GGML inference (predict-woo/qwen3-tts.cpp, MIT; models Apache-2.0) behind the 'qwen3-local' feature. The C++ library is built by the user (scripts/build-qwen3-local.sh) and located via QWEN3_TTS_LIB — opt-in like speech-dispatcher, nothing vendored. Feature parity with the other engines: - Voices: default + all ten Qwen3 languages with proper bcp47/iso639-3/display codes; language voice strings map to the codec language token ids (pinned from upstream main.cpp). - Zero-shot cloning: reference WAV path as a voice string, or an emb:<base64> ECAPA speaker-embedding handle from the qwen3-local cloner (cloning feature) — embedding extracted once, reused per utterance without re-encoding. Fully offline: no upload, consent, or quota. CloningMode::ZeroShot. - SSML-in: <speak> input is stripped (tags never spoken); volume is real PCM gain; rate/pitch have no pipeline control (documented); word boundaries are duration-scaled estimates. - check_credentials reports model load state. Safety/infra: - Process-wide mutex around every C++ call — upstream corrupts the heap with concurrent engine instances (observed: 'corrupted double-linked list' under parallel test creation). - FFI declarations macro-wrapped + cbindgen whitelist: the external library's types stay out of include/tts_wrapper.h. - sherpa registry test skips floravox-* types (registry 1.1.0). - cloud-core feature uses dep: namespaced deps (required once qwen3-local also references base64). Tests: 3 ignored live tests (synthesis+estimates, embedding clone roundtrip, language/SSML/volume) — all passing against a real build; CI feature set + fmt + clippy + doc green. Live-verified end-to-end with a Personal Voice export and a 2-minute child-speech reference (cloud A/B recorded in the project notes).
…cfg, pure helpers, docs - CPP_LOCK now also held in check_credentials + Debug (was claimed 'every entry point' but two is_loaded calls ran unlocked). - Word boundaries computed from the SSML-stripped text (tag fragments were appearing as 'words'); unknown offsets now -1 per contract. - SSML strip moved to speak() — both the C++ input and the boundary estimates share one plain-text source. - tests/qwen3_local_live.rs compiles without the cloning feature (the clone test is per-fn gated); fixes the --all-targets build break. - Pure helpers (language table, embedding codec, PCM gain) split into always-compiled qwen3_local_support with 4 unit-test groups — CI coverage without the C++ library. - README: engines table row (was silently missing) + the dedicated feature section + engine count 28. - Volume clamp 0–4 (matches sherpa); display_names dropped from the feature (ICU not needed); temp file collision → pid+nanos. All three live tests green (68s); CI set + fmt + triple clippy clean.
- Temp file: pid + nanos (concurrent clone_voice was racing on one name — silent wrong-voice embedding or a deleted reference). - EMBEDDING_SIZE pub(crate) — it leaked into the public C header via cbindgen's include semantics (include is additive, not a whitelist; constants bypass it). Header verified clean after regen. - Embedding codec fns + round-trip test gated on any(qwen3-local, cloud) — they now run in CI without the library. - Mangled CPP_LOCK doc fragment removed from decode_embedding; dead double-strip in synthesize_to_pcm16 removed (speak strips once).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds offline Qwen3-TTS as the third local engine (after sherpa-onnx and floravox) — and the only local engine with voice cloning. Backed by predict-woo/qwen3-tts.cpp (GGML; code MIT, models Apache-2.0), behind the
qwen3-localfeature.Build story: the C++ library is NOT vendored —
scripts/build-qwen3-local.shclones/builds it,QWEN3_TTS_LIBpoints the build at it (opt-in precedent: speech-dispatcher/libclang). Models are user-converted GGUFs (modelsDircredential).Feature parity (the bar this PR was held to)
"fr"-style language ids → codec language tokens; reference WAV path;emb:handlesemb:handle reused per utterance (cloner under--features qwen3-local,cloning); fully offlineSafety findings baked in
corrupted double-linked list, reproduced) → process-wideCPP_LOCKaround every C++ call.include/tts_wrapper.hvia macro-wrapped declarations + a cbindgen whitelist of our real ABI (defends against future leaks too; C-harness CI validates).Testing
emb:→ speech); language voice + SSML strip + volume — all passing (~67 s).-D warnings(both feature sets), doc — green.Also in here
floravox-*types (registry 1.1.0 drift — pre-existing failure this PR fixes).cloud-corenamespaced deps fix (dep:base64collision).Known gaps (deliberate, documented)
Accent comes from the model prior (x-vector-only upstream); no streaming; no timestamps; CPU ~0.1× realtime. The upstream-PR plan (in-context cloning + attention-tap timings) is recorded in the project's uncommitted plan notes.