[ThomasK33]

Why

The operator's goal for this library: "this library wasn't updated in a long while. It needs to get the new tokenizer for opus and for a few other models ... Please take it from here on autopilot and get this landed."

This PR adds exact offline token counts for two providers whose counts are wrong today.

• Gemini counts use o200k_base × 1.08. Google's Gemma 4 report says Gemma 4 uses the Gemini tokenizer, and Google's own google-genai SDK counts gemini-3.1 and gemini-3.5 with Gemma 4.
• Llama 4 Scout maps to the old Claude vocabulary, with 0% accuracy at 500 tokens in the README.

This PR adds encodings and the engine that runs them. It does not change models.json. A later PR maps models to encodings, using the table below.

What

• ai-tokenizer/encoding/llama4 (200,000 tokens). It runs on the existing Tokenizer.
• ai-tokenizer/encoding/gemma4 (262,144 tokens, 514,906 merges) and a new engine in src/sentencepiece.ts. Gemma 4 has no regex pre-split, and it merges by merge-list index. The existing engine cannot run that. The new engine uses a heap and a linked symbol list, so it takes O(n log n) time on a long run with no space.
• Tokenizer takes the Codec shape from PR feat(encoding): add claude_v3 and claude_v4_7 encodings ported from ctok #10. The gemma4 module exports kind = "sentencepiece", name and a codec that builds the engine on first use. Tokenizer delegates encode and decode to it, and instanceof Tokenizer holds. The tokenizer.ts hunks equal PR feat(encoding): add claude_v3 and claude_v4_7 encodings ported from ctok #10's except one change: Codec.encode also takes allowedSpecial and disallowedSpecial, so gemma4 keeps Tokenizer's special-token rules. gemma4 and llama4 are both in src/encoding/index.ts, so new Tokenizer(encoding[model.encoding]) typechecks.
• scripts/generate-hf-encodings.ts writes both modules from a tokenizer.json pinned to a Hugging Face revision and a SHA-256. The raw files stay out of git. The script asserts every property the runtime relies on.
• fixtures/ and src/hf-encodings.test.ts compare both encodings with Hugging Face tokenizers on 1,680 samples. scripts/compare-hf.ts runs the same comparison on any files. scripts/measure-gateway.ts runs the online check.

Revisions: Gemma 4 is google/gemma-4-E4B-it at ee0ef602 (ungated). Llama 4 is the unsloth/Llama-4-Scout-17B-16E-Instruct mirror at afd8e498. The tokenizer.json for Maverick has the same SHA-256.

Evidence

Exactness against Hugging Face tokenizers 0.23.3, token id by token id, on this head (PR #8 is merged):

Corpus	gemma4	llama4
Wikipedia, 99 articles, 55 languages, plus slices (8.7M chars)	0 mismatches	0
Code and docs: Python, Go, Rust, JS/TS, C headers, /usr/share/doc (152M chars)	0	0 (before PR #8: 2 files with U+FEFF)
Fuzz: 100,000 small-alphabet and 30,000 random Unicode strings (11M chars)	0	0
Long runs with no space: _, space, ab, 日, 😀, base64, minified JSON (14.8M chars, up to 3M per file)	0	skipped (issue #6)
Contraction folding: X' + every code point, with 7 left contexts and suffixes (11.1M strings), and 8,554 words + 'ſ	n/a	0

The committed test runs 1,680 samples: gemma4 is 1680/1680 exact and llama4 is 1668/1668 exact with 12 long runs skipped (issue #6, PR #15). I mutated the engine and the pattern to check that the test can fail. Dropping the space mark, the stale-entry check or the tie rule fails gemma4 samples. Using JS \s, dropping /, or dropping the U+017F fold from the llama4 pattern fails llama4 samples.

Online check through https://ai-gateway.vercel.sh/v1/chat/completions, max_tokens 16, 11 samples per model (English, Chinese, Japanese, Arabic, Hindi, Russian, German, TypeScript, Python, JSON, emoji). Content tokens are prompt_tokens(sample) - prompt_tokens("a") + 1. The ratio is content tokens divided by the offline count.

Model	Encoding	Ratio, all 11 samples	o200k_base total ratio
google/gemini-2.5-flash, 2.5-pro, 2.5-flash-lite	gemma4	1.0000 (min = max)	1.0176
google/gemini-3-flash, 3.1-pro-preview, 3.1-flash-lite	gemma4	1.0000	1.0176
google/gemini-3.5-flash, 3.5-flash-lite	gemma4	1.0000	1.0176
google/gemini-3.6-flash, 3.7-flash, 3.8-flash	gemma4	1.0000	1.0176
meta/llama-4-scout	llama4	1.0000	0.9597
meta/llama-4-maverick	llama4	not measurable (see Risks)	n/a
google/gemma-4-31b-it (control)	gemma4	0.9999 (min 0.9993)	1.0175
google/gemini-4-argon	not gemma4	0.9844 (min 0.8108, max 1.1051)	1.0018

• Provider cleanup is not modeled. A second run of 14 whitespace probes found three behaviors. Gemini 2.5 (flash, pro, flash-lite) matches on all 14. Gemini 3 to 3.8 drops each \r, drops leading newlines, and counts a literal U+2581 as 3 byte tokens. Llama 4 Scout trims leading and trailing whitespace of message content. The README states this. The mapping PR can apply the cleanup per model before it counts.
• Gemini 2.x uses the same counts as 3.x. Gemma 3 and Gemma 4 share all 514,906 merges and differ only in 19 control pieces and <unusedN> ids. My engine gives equal counts for both on all fixture samples and on 99 articles (2,588,778 tokens each). One gemma4 encoding covers gemini-2.5-*, gemini-3-* and gemini-3.1 to 3.8. The old o200k_base × 1.08 is off by +6.1% on this set.
• gemini-4-argon is a different tokenizer. Keep it on o200k_base for now.
• Gemini adds no message framing. A lone "a" costs 1 token. Llama 4 Scout adds 10 tokens (a lone "a" costs 11).
• Spend: usage.cost summed to about $0.17 over all runs.

Size

Module	Raw	gzip
src/encoding/llama4.ts	8.24 MB	3.25 MB
src/encoding/gemma4.ts	10.07 MB	3.86 MB
src/encoding/o200k_base.ts (for scale)	8.14 MB	3.24 MB

npm pack on this head: tarball 27.0 MB, unpacked 69.2 MB, 107 files (main before PR #10 and this PR: 12.5 MB tarball). dist/index.d.ts is 68 KB and dist/sdk.d.ts is 70 KB. merges and special_tokens carry a type annotation, so tsdown does not copy their literals into the .d.ts files. An install of the tarball runs both encodings through new Tokenizer(...) under Node ESM and CJS, and tsc accepts the README pattern new Tokenizer(encoding[model.encoding]).

Gates on the pushed head: bun run format --check, bun run typecheck, bun test (618 pass, 0 fail, including the 429 Claude tests), bun run build --dts=false and bun run build.

Risks

• One Codec interface. PR feat(encoding): add claude_v3 and claude_v4_7 encodings ported from ctok #10 is merged. This head merges main, and Codec.encode keeps the two optional special-token parameters. The ctok codecs from PR feat(encoding): add claude_v3 and claude_v4_7 encodings ported from ctok #10 stay assignable to it. src/encoding/index.ts and package.json list all of the Claude encodings plus gemma4 and llama4.
• Extended special tokens. Like PR feat(encoding): add claude_v3 and claude_v4_7 encodings ported from ctok #10, new Tokenizer(gemma4, { "<x>": id }) throws gemma4 does not support special tokens. The engine class itself decodes extended tokens.
• llama4 inherits issue Quadratic time in bytePairMerge: one long run of repeated characters blocks for minutes #6. A long run of one character takes quadratic time in the default engine. The test skips the 12 long samples. Remove the skip after PR fix: O(n log n) byte pair merge and one shared Tokenizer declaration #15 lands.
• o200k_base has the same fold gap. The reference regex (?i:'s) folds U+017F, and the repo's '[sS] does not. o200k_base gives 3 tokens where tiktoken gives 2 for " I'ſ" (also for " I'ſe" and " I'ſl"). cl100k_base shows no difference over the same scan. I did not change those encodings in this PR.
• Maverick was not measured online. The gateway routes it only to Bedrock, which refuses calls from this sandbox's region with HTTP 400. Its tokenizer.json matches Scout's byte for byte.
• Package size grows. The tarball holds ESM and CJS copies of each module.
• Unicode data. The regex runs on the JavaScript engine's Unicode tables, and Hugging Face uses Oniguruma's. They can differ on code points added in the newest Unicode versions. I did not test those.
• Special-token text. Llama 4 Scout reads Llama special-token text in message content as the special token (3 content tokens for <|begin_of_text|>hi<|end_of_text|>, where count gives 13 and encode(text, "all", []) gives 3). Gemini 3.5 does not read <bos> or <eos> as special tokens. count treats such text as ordinary text, as PR fix: count special-token text as ordinary text #9 does. The README says how to match Scout.

---

Generated with xum • Model: anthropic:claude-sonnet-5-5 • Thinking: xhigh

[ThomasK33]

@codex review

[ThomasK33]

@codex review

[ThomasK33]

@codex review

[ThomasK33]

Review record for head ce60dee8c159be1305fc583ddb9864437a798361.

• Reviews: 4 rounds. Codex does not respond on this repository, so the operator waived it here. Two independent review lanes (Claude Opus and gpt-6.1-sol) replaced it, and an independent verifier re-ran each round. In the final round both lanes found no issues and the verifier passed.
• A clean-context advisor recommends "ready with tracked follow-ups".
• Tests on this head: typecheck ok, bun test 189 pass / 0 fail, build ok, CI green.
  • Fixture: gemma4 1680/1680 and llama4 1668/1668 exact, with 12 long samples skipped until fix: O(n log n) byte pair merge and one shared Tokenizer declaration #15 lands.
  • The verifier's own Hugging Face comparison: gemma4 has 0 mismatches. llama4 mismatches only on Unicode 17 code points.
  • Gateway check: Gemini 2.5 to 3.8 and Llama 4 Scout give the offline count on every clean sample.
• Follow-ups. Later PRs in this series fix them:
  1. Make encoding data lazy. Importing the ai-tokenizer/encoding barrel now costs about 68 MB more heap than main. This must land before the next npm release.
  2. Re-enable the 12 llama4 long samples after fix: O(n log n) byte pair merge and one shared Tokenizer declaration #15.
  3. Model the Gemini 3.x and Llama 4 provider-side text normalization per model when models.json maps models to these encodings. gemini-4-argon stays on o200k_base.

---

Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: high