Skip to content

Document that BpeOptions.ByteLevel requires a PreTokenizer - #7716

Open
zpczpc wants to merge 1 commit into
dotnet:mainfrom
zpczpc:patch-1
Open

zpczpc wants to merge 1 commit into
dotnet:mainfrom
zpczpc:patch-1

Conversation

@zpczpc

@zpczpc zpczpc commented Sep 14, 2026

Copy link
Copy Markdown

Documents the requirement reported in #7715.

BpeOptions.ByteLevel only documented the byte <-> unicode mapping (for example Space -> 'Ġ').
It did not state that a pre-tokenizer is required. Without one, byte-level encoding maps the space
character to 'Ġ' but does not keep the whitespace attached to the following token, so spaces and
newlines are dropped during encoding and cannot be recovered by decoding - with no exception and
no warning.

This adds a <remarks> block to the ByteLevel property that:

  • states that PreTokenizer must be set when ByteLevel is true;
  • describes the failure mode (silent whitespace loss, not recoverable on decode);
  • points at RegexPreTokenizer built from the GPT-2 pattern as a configuration that round-trips losslessly.

Measured against HuggingFace tokenizers as the reference, a properly configured pre-tokenizer makes
encoding match token-for-token and decoding exact, while without one the lossless round-trip rate drops
to 1/5 on a small mixed-language sample.

This is a documentation-only change: no code and no public API surface change.

The documentation for `BpeOptions.ByteLevel` only described the byte <-> unicode mapping
(for example `Space -> 'Ġ'`) and did not mention that a pre-tokenizer is required.

Without a pre-tokenizer, byte-level encoding maps the space character to `'Ġ'` but does not
keep the whitespace attached to the following token, so spaces and newlines are dropped
during encoding and cannot be recovered by decoding.

This adds a `<remarks>` block stating the requirement and points at `RegexPreTokenizer`
with the GPT-2 pattern as the configuration that keeps the round trip lossless.

Relates to dotnet#7715.
@codecov

codecov Bot commented Sep 14, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 69.87%. Comparing base (e4dff01) to head (53885de).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #7716      +/-   ##
==========================================
- Coverage   69.88%   69.87%   -0.01%     
==========================================
  Files        1487     1487              
  Lines      276249   276249              
  Branches    28291    28291              
==========================================
- Hits       193064   193042      -22     
- Misses      75696    75712      +16     
- Partials     7489     7495       +6     
Flag Coverage Δ
Debug 69.87% <ø> (-0.01%) ⬇️
production 64.06% <ø> (-0.02%) ⬇️
test 89.82% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/Microsoft.ML.Tokenizers/Model/BpeOptions.cs 68.42% <ø> (ø)

... and 5 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant