SentencePieceTokenizer: charsmap entries that map to an empty string are not applied, so control characters become <unk> #7908

System Information:

• OS & Version: Debian 13 (trixie), x64
• ML.NET Version: Microsoft.ML.Tokenizers 3.0.0
• .NET Version: .NET SDK 10.0.404

Describe the bug
The precompiled nmt_nfkc charsmap used by SentencePiece Unigram models maps 30 control characters to an empty string, i.e. the normalizer deletes them. SentencePieceTokenizer does not apply these entries: the character survives normalization and is encoded as <unk>. As a result, token IDs differ from both Hugging Face tokenizers and SentencePiece C++ on any text that contains these characters.

The affected code points are U+0001–U+0008, U+000B, U+000E–U+001F, U+007F, U+008F and U+009F.

To Reproduce
Model: intfloat/multilingual-e5-small, revision 614241f622f53c4eeff9890bdc4f31cfecc418b3, file onnx/tokenizer.json.

using Microsoft.ML.Tokenizers;

using var stream = File.OpenRead("tokenizer.json");
var tokenizer = SentencePieceTokenizer.CreateFromTokenizerJson(
    stream, addBeginningOfSentence: false, addEndOfSentence: false);

Console.WriteLine(string.Join(", ", tokenizer.EncodeToIds("a\u0001b")));
Console.WriteLine(string.Join(", ", tokenizer.EncodeToIds("hello\u007Fworld")));

Expected behavior
Charsmap entries whose replacement is an empty string should delete the matched input, as SentencePiece and Hugging Face do.

Additional context
We ran every Unicode scalar value (1,112,063 code points) through the normalizer and compared the result with SentencePiece's own normalization. 1,112,033 match; all 30 differences are the code points listed above. Removing these characters before calling the tokenizer brings the result to 1,112,063 of 1,112,063. The behavior is the same whether the tokenizer is created from tokenizer.json or from the .model protobuf.

Workaround: strip the 30 code points before encoding.

A related tie-breaking difference is reported in #7909.