Conversation
The documentation for `BpeOptions.ByteLevel` only described the byte <-> unicode mapping (for example `Space -> 'Ġ'`) and did not mention that a pre-tokenizer is required. Without a pre-tokenizer, byte-level encoding maps the space character to `'Ġ'` but does not keep the whitespace attached to the following token, so spaces and newlines are dropped during encoding and cannot be recovered by decoding. This adds a `<remarks>` block stating the requirement and points at `RegexPreTokenizer` with the GPT-2 pattern as the configuration that keeps the round trip lossless. Relates to dotnet#7715.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #7716 +/- ##
==========================================
- Coverage 69.88% 69.87% -0.01%
==========================================
Files 1487 1487
Lines 276249 276249
Branches 28291 28291
==========================================
- Hits 193064 193042 -22
- Misses 75696 75712 +16
- Partials 7489 7495 +6
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Documents the requirement reported in #7715.
BpeOptions.ByteLevelonly documented the byte <-> unicode mapping (for exampleSpace -> 'Ġ').It did not state that a pre-tokenizer is required. Without one, byte-level encoding maps the space
character to
'Ġ'but does not keep the whitespace attached to the following token, so spaces andnewlines are dropped during encoding and cannot be recovered by decoding - with no exception and
no warning.
This adds a
<remarks>block to theByteLevelproperty that:PreTokenizermust be set whenByteLevelistrue;RegexPreTokenizerbuilt from the GPT-2 pattern as a configuration that round-trips losslessly.Measured against HuggingFace
tokenizersas the reference, a properly configured pre-tokenizer makesencoding match token-for-token and decoding exact, while without one the lossless round-trip rate drops
to 1/5 on a small mixed-language sample.
This is a documentation-only change: no code and no public API surface change.