Skip to content

fix: prepend BOS token for ugm vocabulary models that request it - #689

Merged
giladgd merged 2 commits into
withcatai:masterfrom
tomwangcn:fix-ugm-bos-prepend
Nov 18, 2026
Merged

giladgd merged 2 commits into
withcatai:masterfrom
tomwangcn:fix-ugm-bos-prepend

Conversation

@tomwangcn

@tomwangcn tomwangcn commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes #683.

For models with a ugm vocabulary (e.g. BAAI/bge-m3), resolveBeginningTokenToPrepend() returned null unconditionally, so the BOS token was never prepended, even when the GGUF sets tokenizer.ggml.add_bos_token = true and model.tokens.shouldPrependBosToken is true. Since bge-m3 uses CLS pooling, the first content token was pooled instead of <s>, shifting embeddings away from the reference.

llama.cpp prepends BOS for UGM when add_bos is set (llama_vocab::impl::tokenize, case LLAMA_VOCAB_TYPE_UGM in llama-vocab.cpp: if (add_special && add_bos) output.push_back(special_bos_id);), and add_bos itself comes from the GGUF tokenizer.ggml.add_bos_token metadata (defaulting to false for UGM). The addon's shouldPrependBosToken() already exposes exactly that flag via llama_vocab_get_add_bos().

Changes

  • src/utils/tokenizerUtils.ts: for ugm vocabularies, return tokens.bos when tokens.shouldPrependBosToken is true, and null otherwise. Models that don't request BOS (e.g. T5, whose GGUFs have add_bos_token = false) behave exactly as before. The ranking and completion paths share this helper, so they now match llama.cpp too.
  • test/standalone/utils/tokenizerUtils.test.ts: new standalone tests covering beginning/end token resolution for ugm (with and without add_bos_token), spm, bpe, wpm, and rwkv.

Verification

  • vitest run ./test/standalone/utils - 68 tests pass (12 new)
  • tsc --noEmit --project tsconfig.json - clean
  • eslint on the changed files - clean
  • End-to-end against a real bge-m3 model file (gpustack/bge-m3-GGUF bge-m3-Q8_0.gguf, a Hugging Face mirror of BAAI/bge-m3; metadata read from the file: tokenizer.ggml.model = t5 (UGM), tokenizer.ggml.add_bos_token = true, tokenizer.ggml.bos_token_id = 0, bert.pooling_type = CLS):
    • resolveBeginningTokenToPrepend() returns null on master and 0 (<s>, the model's BOS) with this fix, so embedding inputs now start with <s>, matching the GGUF metadata and llama.cpp's own UGM path.
    • Measured 1024-dim embeddings of 5 varied inputs (English, Chinese, a single token, a longer paragraph) differ between master and this fix: cosine similarity 0.533-0.997, with short inputs shifting the most. This shows the missing BOS materially changes model outputs; I did not run a retrieval benchmark and did not compare against the native llama.cpp CLI binary.
  • The reference tokenizer of BAAI/bge-m3 (tokenizer_config.json) is XLMRobertaTokenizer with bos_token = <s>, which prepends <s> by default - consistent with the GGUF's add_bos_token = true.

`resolveBeginningTokenToPrepend` returned `null` for `ugm` vocabulary
models unconditionally, so UGM models with `add_bos_token = true` in
their GGUF metadata (such as bge-m3) never got the BOS token prepended
to embedding, ranking and completion inputs. This diverged from
`llama.cpp`, which prepends BOS for UGM when `add_bos` is set, and
broke CLS-pooled embeddings, where the first content token was pooled
instead of `<s>`.

Respect `tokens.shouldPrependBosToken` for `ugm` models to match the
`llama.cpp` behavior. Models that don't request BOS (such as T5) are
unaffected.

Closes: withcatai#683

@giladgd giladgd left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR!

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove this test file, the tested function is too simple to be worth having a test

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed in 7484f18, thanks.

The tested function is too simple to be worth having a test.
@giladgd
giladgd merged commit 83875ac into withcatai:master Nov 18, 2026
20 of 21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: getEmbeddingFor() never prepends BOS for UGM-vocab models, even when the GGUF sets add_bos_token = true (breaks bge-m3 CLS pooling)

2 participants