Repository navigation
OmniVoice: make the audio tokenizer codebook weight type configurable - #1129
Open
kawshikbuet17 wants to merge 4 commits into
Open
kawshikbuet17 wants to merge 4 commits into
kawshikbuet17 wants to merge 4 commits into
Conversation
OmniVoice's tokenizer.json declares an NFC normalizer, which Hugging Face tokenizers (and so Python OmniVoice) apply before BPE. audio.cpp encoded the raw text, so non-NFC input (common in Bengali: precomposed য় ড় ঢ়, or ো/ৌ typed as two code points) produced different token IDs than Python. Normalize in OmniVoiceTextTokenizer::encode() with the vendored utf8proc, the same way Drama3 and Plapre do, and link the omnivoice model with utf8proc_vendor. This covers reference text, target text, style prompt and the segments around nonverbal tags. Duration estimation still uses the raw text, as in Python.
c6dd5dd pinned the codebooks, codebook score matrix, semantic feature projection and fc2 to f32 so runtime q8_0 tokenizer loading works. Before that, they followed omnivoice.audio_tokenizer_weight_type. Add omnivoice.audio_tokenizer_codebook_weight_type (native, f32, f16, bf16) for these four tensors. The default stays f32, so outputs do not change unless the option is set. Quantized types are rejected.
Owner
|
@kawshikbuet17 Two minor issues:
|
…n to the model spec codebook_weight_type=native keeps the packaged type, so on the published Q8_0 GGUF it loaded the feature projection and fc2 as q8_0 and aborted. Check the stored type up front and fail with a clear error unless it is f32, f16, or bf16. Add audio_tokenizer_codebook_weight_type to model_specs/omnivoice.json.
Contributor
Author
|
@0xShug0 Thanks, both fixed in cb9683a.
Checked with
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
c6dd5dd pinned four audio tokenizer tensors to f32 so runtime q8_0 tokenizer loading works:
the codebooks, the codebook score matrix, the semantic feature projection and fc2. Before that
commit they followed
omnivoice.audio_tokenizer_weight_type, so f16 users got them in f16.This adds
omnivoice.audio_tokenizer_codebook_weight_type(native,f32,f16,bf16) forthose four tensors, so the pre-c6dd5dd8 behaviour can be selected when needed (for example, to
reproduce outputs from older builds).
f32, so nothing changes unless the option is set.runtime q8_0 tokenizer loading.
docs/models/omnivoice.md.Checked with
audiocpp_clion CUDA (seed 42, f16 generator and tokenizer):=f32: bit-identical WAV=f16: runs, different WAV=q8_0: rejected with the new error message