[ThomasK33]

Why

The operator's goal: "this library wasn't updated in a long while. It needs to get the new tokenizer for opus and for a few other models ... Please take it from here on autopilot and get this landed."

This PR fixes two open bugs in the library first, so the new tokenizer work lands on a base that works.

• Fixes Quadratic time in bytePairMerge: one long run of repeated characters blocks for minutes #6. One long run of one character (underscores, spaces, dots) is a single regex piece, and bytePairMerge took O(n²) time on it. A 1.1M-character OCR-like document took 172 s to count here. One production service went down four times from this.
• Fixes Typescript error while counting tokens #1. Passing a Tokenizer from ai-tokenizer to count() from ai-tokenizer/sdk failed to typecheck ("Types have separate declarations of a private property 'name'").

What

1. Merge in O(n log n) (src/tokenizer.ts). The parts of a piece form a linked list. A min-heap of adjacent pairs, ordered by (rank, position), picks the next merge. This is the same rule as before: lowest rank first, leftmost pair on ties. Stale heap entries are skipped when popped. I measured the old short-piece scan against the heap on 61.8k real pieces that need a merge. The heap was not slower, so there is no second code path.
2. One shared declaration (tsdown.config.ts). index and sdk were two separate builds, so each bundled its own Tokenizer class and its own .d.ts. TypeScript treats classes with private members as nominal types, so the two classes were not compatible. Both entry points are now one build, and the bundler emits the class and its declaration once, in a shared chunk. No new dependency, no package.json change, no new export.
3. Regression checks.
   • src/byte-pair-merge.test.ts: time bounds for the issue's inputs, a differential test against a copy of the old algorithm, and long-run inputs against tiktoken.
   • scripts/check-package-types.ts plus two CI steps: pack the package, install the tarball in a fresh project, and typecheck the README example under bundler, node16 (ESM and CJS) and node10 resolution.

The commits are in failing-first order: the timing test (red), the fix, the package-types check (red), the fix.

Credit: the diagnosis in #2 (each entry point bundles its own Tokenizer, so share one) is what I built on. That PR does it by externalising a tokenizer module, upgrading tsdown and adding a ./tokenizer export. A single multi-entry build gives the same result without those. I did not reuse its code or touch that PR.

I did not edit src/sdk.ts or src/models.json. The Tokenizer constructor and public API are unchanged.

Evidence

Issue #6, before and after. Node 22 on the built dist, fresh tokenizer per row, a shared host with load average 70 to 130, so expect ±30% noise. Token counts are identical before and after in every row.

input	chars	before (ms)	after (ms)	tokens
underscores	6,250	49.4	13.9	100
underscores	12,500	166.5	17.1	197
underscores	25,000	611.3	29.5	392
underscores	50,000	1,610.9	35.4	782
underscores	100,000	7,050.2	80.3	1,563
underscores	1,000,000	not run (the issue measured more than 9.5 min)	1,014.7	15,625
spaces	50,000	1,406.3	48.4	392
ab x 25,000	50,000	2,025.2	29.7	12,500
dots	50,000	1,823.7	41.6	782
OCR-like document (bun)	1,128,002	172,007	1,002 to 1,055	14,821

I measured with a small script (not committed) that calls count() on each input. The sweep of 23 repeated-character shapes x 4 encodings at 100k and 200k characters (92 rows) shows linear growth: the slowest 200k-character row takes 255 ms.

Identity with the old algorithm. bun test src/byte-pair-merge.test.ts prints these count lines:

differential: encodings=4 pieces=38428 identical=38428
end to end: inputs=420 identical=420

The differential test runs the old merge (a copy in the test, with an independent rank lookup built from the encoding data) and the new merge on 35,428 pieces for o200k_base, cl100k_base, p50k_base and claude. The pieces are random bytes (mostly invalid UTF-8), one repeated byte at 300 lengths, two and three letter alphabets, vocabulary tokens joined and cut at random, random code points with 2, 3 and 4 byte UTF-8, and every distinct regex piece of the repo's own files and some Chinese, Japanese, Arabic, Russian and emoji text. The end to end test compares Tokenizer#encode with tiktoken on 10 repeated characters x 14 lengths for three encodings. I also broke the new merge four ways (ties go right, no stale-entry check, no update of the left neighbour, a wrong final lookup). The differential test failed on all four.

Normal text is not slower. bun cold-ab.ts: same process, merge cache size 0 so almost every piece that needs a merge pays for it, old and new Tokenizer interleaved, CPU time, 15 rounds, median.

text	tokens (both)	old (ms)	new (ms)	new / old
1 MB of Markdown	272,987	121	118	0.98
1 MB of TS and JS	252,110	74	72	0.97
207k characters of mixed scripts	99,000	221	183	0.83

On 61.8k real pieces that need a merge: old scan min 302.6 ms / median 382.9 ms, heap min 275.9 ms / median 332.5 ms. Only 50 of the 309k pieces that need a merge are 128 bytes or longer.

bun run bench (runs bench/merge-cache.ts) over 3 alternating rounds: old 15,832 / 25,615 / 18,779 ms, new 13,214 / 19,320 / 16,846 ms. bun run bench/versus.ts shows no consistent difference between old and new, within the host noise (rounds differ by up to 2x). The cache hides the merge in that benchmark.

Issue #1. Before: bun scripts/check-package-types.ts on the old build prints FAIL for all four resolutions with TS2322 ... Types have separate declarations of a private property 'name', and package types: 0 of 4 resolutions typecheck. After: PASS for bundler, node16 (ESM), node16 (CJS) and node10, and package types: 4 of 4 resolutions typecheck. dist/index.d.ts and dist/sdk.d.ts now both import Tokenizer from one shared index-<hash>.d.ts. A Node smoke test counts the same total (14) through the ESM and CJS builds.

Gates on head 1125a60: bun run format --check, bun run typecheck, bun test (614 pass, 0 fail), bun run build --dts=false, bun run build, bun scripts/check-package-types.ts.

Risks

• Memory. The merge allocates about 36 bytes per byte of one piece (a few typed arrays). A 1.1M-character piece uses about 40 MB for the length of the call. Ordinary pieces are tiny.
• Output file names. dist now has hashed shared chunks (tokenizer-<hash>.js, index-<hash>.d.ts). Consumers import only the package entry points, so nothing changes for them.
• CI time. The two new CI steps add a type-declaration build and an npm install of typescript and ai (about 30 s).
• Not in this PR: bun run format --check runs prettier --write, rewrites src/models.json and cannot fail. bun run bench runs only bench/merge-cache.ts. Both are pre-existing.
• Known limit, not a regression: the ESM and CJS builds each have their own declaration chunk. A project that imports ai-tokenizer as ESM and requires ai-tokenizer/sdk as CJS can still see the TS2322 error. The same split exists on main.

Review record

• Codex: silent, then waived by the operator for this repo. I requested @codex review on 2eafae8 at 2026-11-03T11:25:28Z and got no response in over 60 minutes. No other PR in this repo has a Codex response either. No review round was used.
• Independent advisory review (a fresh agent that saw only the goal, the diff and the test results): ready with tracked follow-ups, no blocker. It reproduced the tests, caught three new mutants of its own, and checked that the exports of index and sdk are unchanged.
• Follow-ups from that review, done in a93d793 and 9bca535: scripts/check-package-types.ts now fails on any error TS line outside other packages' node_modules (with a bad module option in one tsconfig it prints FAIL node10 and 3 of 4, and the old build config still gives 0 of 4). The merge heap throws a RangeError instead of dropping a write if it ever overflows.
• After PR fix: match tiktoken on special tokens, BOM, whitespace and inherited names #8 and fix: count special-token text as ordinary text #9 landed, I merged main (ab45414) and removed the U+FEFF skip. The differential test now compares every piece.
• After PR feat(encoding): add claude_v3 and claude_v4_7 encodings ported from ctok #10 landed, I merged main (6dec619) with no conflicts. Its kind + codec dispatch in the Tokenizer constructor, encode and decode is untouched, and the claude_v3, claude_v4_7, claude_v4_8 and claude_v5 encodings do not call bytePairMerge. src/claude.test.ts passes (claude count_tokens fixture: 392/392 exact). The packed-types check now also passes a claude_v5 tokenizer to count(), and a Node smoke test gives the same tokens from the ESM and CJS builds for all four.

---

Generated with xum • Model: anthropic:claude-sonnet-5-5 • Thinking: xhigh

[ThomasK33]

@codex review

[ThomasK33]

Review record for head 0d28d1b7465b7ece2f7188824fa27b178f504923.

• Reviews: Codex does not respond on this repository, so the operator waived it here. Two independent review lanes (Claude Opus and gpt-6.1-sol) found no issues at 9bca535. An independent verifier passed 2eafae8, 9bca535 and this head. This head is 9bca535 plus a clean merge of main and the removal of a BOM test skip that main made unnecessary.
• A clean-context advisor recommends "ready with tracked follow-ups".
• Tests on this head: typecheck ok, bun test 185 pass / 0 fail, build ok, CI green. Against main's tokenizer, 69,516 texts (48.8M chars, all 4 encodings) give identical output. Against tiktoken, 49,734 comparisons are identical. The repo's verify-ai-tokenizer skill passes 168 of 168 checks. The packed-tarball consumer matrix fails 16 of 16 on main (issue Typescript error while counting tokens #1) and passes 16 of 16 here.
• Timings (CPU, loaded host): 50k _ takes 1118 ms on main and 51 ms here. 100k _ takes 14997 ms on main and 85 ms here. 480k of prose takes 7.78 ms on main and 7.52 ms here.
• Follow-ups. A later PR in this series fixes them:
  1. Make the CI format step able to fail. format --check runs prettier --write today.
  2. Shrink the default differential corpus. bun test peak memory went from about 0.55 GB to 1.8 GB.

Fixes #6. Fixes #1.

---

Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: high