[ThomasK33]

Why

This library wasn't updated in a long while. It needs the new tokenizer for Opus and for a few other models. Anthropic switched tokenizers at Claude Opus 4.7. The current claude encoding (the Claude 2 BPE times 1.10) is off by -51% to +10% on Opus 4.7 and later, and by -32% to +10% on Opus 4.5 and 4.6.

ctok (MIT, Python, by Sander Land) rebuilt both Claude vocabularies from count_tokens probes and reproduces the API. This PR ports ctok to TypeScript as offline encodings, one per counting behavior, so every current Claude model has an exact encoding.

What

Four encodings, exported like the others (ai-tokenizer/encoding/<name>). Each is named after the first model that counts that way. The last three share one vocabulary module.

Encoding	Models	Frame	What differs
claude_v3	Claude 3 through Opus 4.6, Sonnet 4.5/4.6, Haiku 4.5	7	ctok v3 vocabulary
claude_v4_7	Opus 4.7	11	new vocabulary, frame ends in a word boundary, newline tail n + 2 (ctok 4.7)
claude_v4_8	Opus 4.8, Sonnet 5, Fable 5	6	no word boundary at message start, newline tail n
claude_v5	Opus 5, Opus 5.5, Sonnet 5.5, Haiku 5.5, Fable 5.1	6 (Opus 5), 8 (others)	as claude_v4_8, after dropping trailing ASCII whitespace

• count(text) and encode(text).length are the content tokens of one user message. The frame is left to the model config, which a later PR updates. models.json does not change here. The old claude encoding stays as it is.
• claude_v5 trims trailing space, tab, CR, LF, VT and FF from the raw text before counting. A trailing control character or NBSP stops the trim. The API applies this at the end of each user message and at the end of the system prompt, never in assistant messages or tool_result content. It joins a message's text blocks with no separator and trims only the last block, unless that block is only whitespace (["a ", "b ", "c "] costs what ["a ", "b ", "c"] costs, and ["a", " "] costs what "a " costs on claude_v4_8). The README states the rule. The sdk PR applies it per message.
• Tokenizer accepts a CodecEncoding ({ kind, name, codec }, kind = "ctok") next to the BPE Encoding, and hands encode and decode to the codec. src/tokenizer.ts changes only at the constructor head and the first line of encode and decode. The engine is in src/claude/.
• Ids are synthetic (Anthropic does not publish ids): piece index, then single bytes, then byte prefixes. decode returns the normalized text. It is lossy and throws RangeError on an id outside the vocabulary. count("") is 0, because the API rejects empty content.
• The data is generated from ctok commit 8c43ffd3c7be467622a540f025516e49ef78d8b1 (ctok 1.3.0) by scripts/generate-claude-encodings.ts, with ctok's Unicode 15.1 answers per BMP codepoint, so counts do not depend on the JS engine's Unicode version. NFC runs on the engine between codepoints unassigned in Unicode 15.1. Runs of combining marks are sorted by combining class first, because the engines' NFC is quadratic on a long unsorted run.
• MIT notice for ctok appended to LICENSE. README section "Claude encodings" with the table, semantics and credit.

Two deliberate differences from ctok, both measured on the API. First, on the Opus 4.8 and later frames, a run of n newlines at the end of the content tiles as n newlines, not n + 2 (284 of 284 probes; ctok's rule matches 236). Second, claude_v5 trims trailing whitespace, which ctok does not model.

Type changes

• Model["encoding"] is keyof typeof encodings, so it widens from 4 names to 8: claude_v3, claude_v4_7, claude_v4_8 and claude_v5 join the existing ones. Code that switches over model.encoding exhaustively gets four new cases. This PR keeps the widening on purpose, because new Tokenizer(encoding[model.encoding]) must accept every encoding that a later models.json can name.
• The Tokenizer constructor parameter widens from Encoding to Encoding | CodecEncoding. New exported types: Codec and CodecEncoding. Encoding itself is unchanged.

Evidence

• API parity, in CI (bun test): claude count_tokens fixture: 392/392 exact. The fixture has 98 texts measured on claude-opus-4-6, claude-opus-4-7, claude-opus-4-8 and claude-opus-5-5. It covers the 9 audit samples, all-caps, contractions, whitespace runs, leading and trailing whitespace, trailing newline runs, mixed scripts, emoji, Devanagari, Arabic, code with tabs, JSON, numbers, URLs, a single character, control and private-use characters, a space followed by VT or FF, and more. Removing VT and FF from the claude_v5 trim fails three of these cases. Content = input_tokens minus the frame. scripts/generate-claude-fixture.ts measures the frame as count_tokens("a") - 1 and fails unless it equals the expected frame (7, 11, 6, 8). The API rejects empty and whitespace-only content, so count("") is asserted as 0 instead.

• A separate 295-text corpus (from the root's verifier), measured on all 14 models, one line per model:

  • claude-opus-4-5-20251101 (claude_v3): 295/295, claude-sonnet-4-5-20250929 (claude_v3): 295/295, claude-haiku-4-5-20251001 (claude_v3): 295/295, claude-opus-4-6 (claude_v3): 295/295, claude-sonnet-4-6 (claude_v3): 295/295
  • claude-opus-4-7 (claude_v4_7): 295/295
  • claude-opus-4-8 (claude_v4_8): 295/295, claude-sonnet-5 (claude_v4_8): 295/295, claude-fable-5 (claude_v4_8): 295/295
  • claude-opus-5 (claude_v5): 295/295, claude-opus-5-5 (claude_v5): 295/295, claude-sonnet-5-5 (claude_v5): 295/295, claude-haiku-5-5 (claude_v5): 295/295, claude-fable-5-1 (claude_v5): 295/295
  • all models: 4130/4130 exact

• Parity with Python ctok, dev only (bun scripts/compare-claude-ctok.ts --ctok <ctok clone>, ctok at the pinned commit through uv on Python 3.13, with the two divergences applied to the reference):

  • claude_v3 vs ctok: 7603/7603, claude_v4_7 vs ctok: 7603/7603, claude_v4_8 vs ctok: 7603/7603, claude_v5 vs ctok: 7603/7603
  • ctok's own recorded API counts (UDHR, Rosetta Code, MultiPL-E): claude_v3 vs claude-opus-4-5: 2514/2514, claude_v4_7 vs claude-opus-4-7: 2514/2514, claude_v5 vs claude-opus-5: 2514/2514
  • Two adversarial reviewers also compared 1,667,440 and 301,424 more inputs on Bun and on Node 22 (Unicode 17) and found 0 count mismatches. They found one defect, the quadratic NFC, which this PR fixes.

• Performance (bun bench/claude.ts, load average 88 on 96 cores, median of 3, ms at 1x and 2x size):

  Input	claude_v3	claude_v4_7
  1M chars of prose	172 / 332	141 / 279
  "a" x 100k	8 / 14	7 / 12
  " " x 100k	17 / 31	17 / 34
  "\n" x 100k	113 / 264	28 / 57
  "!" x 100k	8 / 16	8 / 16
  "日" x 100k	15 / 28	14 / 25
  100k out-of-order combining marks	26 / 54	25 / 43

  Every 2x/1x ratio is between 1.73 and 2.33, so the time is linear. claude_v4_8 and claude_v5 run the same engine on the same vocabulary. The first count builds the trie once: 65 ms (v3 vocabulary) and 10 ms (shared vocabulary).

• Size (bun run build, ESM, raw / gzip -9 bytes): claude_v3 683,715 / 140,622. Shared vocabulary of the other three 174,733 / 44,168. Each of their entry chunks is about 600 / 380. Shared engine and Unicode tables 85,919 / 26,136. The old claude encoding is 2,347,960 / 904,855.

• Dogfood with the repo's consumer harness (.agents/skills/verify-ai-tokenizer/verify.sh launch, then drive entry-points, packed tarball in a scratch project): ESM 30/30 and CJS 22/22 checks pass, including the 20 checks for the four new subpaths. The two types-* steps fail on this branch and on main alike: dist/index.d.ts and dist/sdk.d.ts each declare their own Tokenizer class with private fields, so the README's count({ tokenizer }) does not type-check against the published types. That bug predates this PR.

• Gates on the pushed head: npx prettier --check "src/**/*.{ts,tsx,json}" (only the pre-existing src/models.json warning), bun run typecheck, bun test (594 pass), bun run build --dts=false, bun run build.

Risks

• ctok reconstructs counts, not Anthropic's token boundaries. Inputs outside the measured sets carry no guarantee.
• Some Latin letters added in Unicode 16 (for example U+A7CC, U+A7CD, U+A7DA) count a few tokens off on every encoding. ctok has the same gap.
• A single space at the end of the text, right after one of the ideographic tone marks U+302A to U+302F, counts 1 token too many on claude_v3, claude_v4_7 and claude_v4_8 ("x \u302e ": API 5, this PR 6 on Opus 4.6 and Opus 4.8). Python ctok also says 6. claude_v5 is right.
• A JS engine with Unicode data older than 15.1 can count text with newer characters differently. Ids change when the data is regenerated.
• The encodings count one text. The sdk must join a user message's text blocks before counting, and must not apply claude_v5 to assistant text, where it would undercount trailing whitespace by about 1 token per message.
• A sibling PR (feat(encoding): add Gemma 4 (Gemini) and Llama 4 encodings #16, Gemma 4) also adds a kind branch at the top of the Tokenizer constructor. Whichever merges second resolves that conflict.
• bun run format --check expands to prettier --write ... --check, so it always passes and rewrites src/models.json on main. This PR does not change that.

Follow-ups

• Map each Claude model in models.json to its encoding and frame (the table above).
• Report the Opus 4.8+ newline-tail and the Opus 5 trailing-whitespace measurements, and the Unicode 16 gap, to ctok upstream.
• Mention in the changelog that Claude token ids are synthetic and change when the data is regenerated.
• Fix the duplicated Tokenizer declaration in the published .d.ts files (pre-existing).

---

Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: xhigh

[ThomasK33]

Review record for head fbab6e3ecdb6d0d88b3be3b47fccf791bc896a17.

@kylecarbs, your approval is on 1c58418, an early single-encoding draft. Since then the PR has split into four encodings, one per Claude tokenizer behavior. It also adds the kind + codec shape and a public Codec / CodecEncoding type. Please re-review this head.

• Reviews: 3 rounds plus a docs-only round. Codex does not respond on this repository, so the operator waived it here. Two independent review lanes (Claude Opus and gpt-6.1-sol) replaced it, and an independent verifier ran each round. A clean-context advisor recommends "ready with tracked follow-ups".

• Exactness against count_tokens:

  • A verifier corpus of 295 texts gave 4130/4130 exact over 14 Claude models.
  • A holdout of 735 texts gave 10290/10290.
  • The committed fixture gives 392/392.
  • Parity with Python ctok is 7603/7603 per encoding.

• Tests on this head: bun test 594 pass / 0 fail, build ok, CI green.

• Known limits, documented in the README:

  1. A final space after U+302A to U+302F counts 1 too many on v3, v4_7 and v4_8.
  2. Unicode 16 letters U+A7CC, U+A7CD and U+A7DA count about 1 off.

  ctok has both gaps.

• Follow-ups. Later PRs in this series do them:

  1. Map Claude models to these encodings in models.json.
  2. Apply the claude_v5 trim only at the end of a user message in count().
  3. Reconcile the Codec.encode signature with feat(encoding): add Gemma 4 (Gemini) and Llama 4 encodings #16.

---

Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: high