[ThomasK33]

Why

This library has not been updated in a long while, and this PR brings its AI SDK support up to date. ai-tokenizer still declares peerDependencies: { ai: ^5.0.0 }, but ai is at 7.0.162 (v6 shipped 2025-12-22, v7 2026-06-25). The goal is to support v5, v6 and v7. Today count() from ai-tokenizer/sdk reads tool schemas by walking zod 4 internals (_def), so it counts 0 schema tokens for zod 3, jsonSchema(), MCP and Standard Schema tools, and it ignores v7's new part shapes.

What

• Tool schemas. src/sdk.ts reads each tool's input schema through the AI SDK's public asSchema(), then walks the resulting JSON Schema (properties, descriptions, enums, nested objects, arrays of objects) with the same ModelTokens fields as before. zod 3, zod 4, jsonSchema() and (v6+) Standard Schema tools all count. The walker reads through .optional()/.nullable()/.default() wrappers (anyOf with null) and follows local JSON pointer $refs (#, #/definitions/X, #/$defs/X). It counts each definition once per tool (a definition inside a cycle is counted with its back edge cut), so the walk is bounded by the size of the schema. Keywords next to a $ref, such as enum and description, count with the target's (a sibling keyword that is undefined is ignored, and sibling properties are added to the target's and win on a clash). The JSON Schema read from each tool input schema is remembered (a WeakMap on the input schema), so counting the same tools again does not convert them again.
• Message parts. Every image-like and file-like part, in every AI SDK version (image, file, v5 media, v6 image-*/file-*, v7 tagged file), goes through one function, countMediaTokens(). The 85 (image) and 100 (file) constants are unchanged, and a file part counts 100 whatever its media type, as on main. The later image-counting change replaces that one function. A v7 file with inline text data counts as text.
• Tool results count as on main: JSON.stringify(output), wrapper included. A content output still prices its images and files with countMediaTokens() instead of serialising their base64.
• Reasoning is skipped, as on main. reasoning and reasoning-file parts add 0.
• count() stays synchronous and never throws because of a tool definition. A schema it cannot read synchronously (a Promise, a thunk that returns one, a Standard Schema vendor without JSON Schema conversion, a throwing v7 description function) counts as 0 and is listed in the new result.tools.unresolved. I chose this over throw (an estimator in a hot path must not fail on a tool the SDK accepts) and over countAsync() (two ways to do one task). A schema the AI SDK converts to nothing (arktype on ai v5) is listed the same way.
• Package. The peer range is ai ^5.0.0 || ^6.0.0 || ^7.0.0. Dev dependencies are ai ^7.0.162 and an explicit zod ^4.6.7 (zod was only a transitive peer before).
• CI. scripts/test-sdk-matrix.sh installs one ai + zod pairing, type-checks and tests the sdk files, builds dist, and loads the CJS and ESM builds (scripts/smoke-sdk-dist.mjs). CI runs the 8 cells the script lists (ai 5/6/7 x zod 3.25.76/4, plus Node 20 for ai 5 and 6).
• Runtime requirement. sdk.ts now imports asSchema from ai at runtime, so ai-tokenizer/sdk needs ai installed (it needed only its types before). ai stays an optional peer because the root ai-tokenizer entry does not use it. tsdown externalises it, so there is no config change.
• Not touched: src/tokenizer.ts, src/models.json, tsdown.config.ts.

Evidence

Gates on head 0c2d44a: bun run typecheck ok, bun test 730 pass / 1 skip / 0 fail (the skip is the Standard Schema test for the other AI SDK major), bun run build --dts=false ok, bun run build (dts) ok, bun run format --check exit 0.

Accuracy does not move from main on this repo's own workload. I replayed the message loop of scripts/generate-accuracy.ts (same tool, same tool-call/tool-result messages, seeded randomness, sonnet-4.5 config) at 3 sizes with 5 seeds each, and took the truth from Anthropic's free count_tokens endpoint on the request the AI SDK builds (@ai-sdk/anthropic, claude-sonnet-4-5). main is origin/main and PR is this head:

target 500: n=5 tool-results avg 0.0 | truth avg 856 | main acc 98.50% signed -1.50% | PR acc 98.50% signed -1.50%
target 5000: n=5 tool-results avg 3.0 | truth avg 4603 | main acc 98.65% signed -1.35% | PR acc 98.65% signed -1.35%
target 50000: n=5 tool-results avg 28.0 | truth avg 45195 | main acc 99.62% signed -0.38% | PR acc 99.62% signed -0.38%

PR and main give the same total on all 15 conversations. Counting only the payload of a tool result, without its wrapper, would lower every result by 5 to 11 tokens (I measured that against count_tokens on an earlier version of this PR), so tool results keep the wrapper.

Reasoning. Anthropic's count_tokens gives the same total with and without the signed thinking block of an earlier assistant turn: 194 with, 194 without. Main and this PR both give 145 for that conversation. A v7 file part with inline text data (data: { type: "text" }) is sent as a document: the API says 275, main gave 113 and this PR gives 272.

This branch merges origin/main as of PR #10 (the ctok Claude encodings and codec dispatch), and src/sdk.ts keeps calling tokenizer.count() as main does.

Matrix, one line per cell (scripts/test-sdk-matrix.sh --all, run locally on Node 22.23.4, so the two Node 20 cells print the same lines as their Node 22 twins and are shown once. "tests" is sdk.test.ts + sdk.compat.test.ts. cjs and esm are the totals of one fixed count() call through each build):

CELL ai@5.0.285 zod@3.25.76 typecheck=ok tests: 157 pass  1 skip  0 fail | smoke node=v22.23.4 ai=5.0.285 zod=3.25.76 cjs=172 esm=172
CELL ai@5.0.285 zod@4.6.7 typecheck=ok tests: 157 pass  1 skip  0 fail | smoke node=v22.23.4 ai=5.0.285 zod=4.6.7 cjs=172 esm=172
CELL ai@6.0.322 zod@3.25.76 typecheck=ok tests: 157 pass  1 skip  0 fail | smoke node=v22.23.4 ai=6.0.322 zod=3.25.76 cjs=172 esm=172
CELL ai@6.0.322 zod@4.6.7 typecheck=ok tests: 157 pass  1 skip  0 fail | smoke node=v22.23.4 ai=6.0.322 zod=4.6.7 cjs=172 esm=172
CELL ai@7.0.162 zod@3.25.76 typecheck=ok tests: 157 pass  1 skip  0 fail | smoke node=v22.23.4 ai=7.0.162 zod=3.25.76 cjs=172 esm=172
CELL ai@7.0.162 zod@4.6.7 typecheck=ok tests: 157 pass  1 skip  0 fail | smoke node=v22.23.4 ai=7.0.162 zod=4.6.7 cjs=172 esm=172

CI runs the two Node 20 cells as well. I ran the same smoke by hand under Node 20.20.2 with ai 5, 6 and 7, and all three load the CJS build (cjs=172 esm=172). I did not test older Node versions. v7 itself declares engines.node >= 22.

zod 4 totals are the same as before wherever no wrapper hides the enum, object or description. I ran the old walker (git show 107814b:src/sdk.ts) and the new code over 41 zod 4 tool shapes with a synthetic ModelTokens whose fields all differ. 26 are identical (flat, described, enums, nested objects, arrays of objects/strings/enums, strict/passthrough, z.record/z.union/z.literal, unicode keys, a description on the outer .default()/.optional()/.nullable(), and schemas registered with .meta({ id }), which reach the walker as a $ref). src/sdk.compat.test.ts pins 21 of them for both zod 3 and zod 4 and compares the .meta({ id }) shapes with their inline form. The tool in scripts/generate-accuracy.ts gives identical tool totals old vs new for all 59 models with a claude or o200k encoding, and the 18 expected totals in sdk.test.ts are unchanged.

15 shapes change on purpose. The old walker read the wrapper, not the inner type, so it dropped the description, enum or nested properties behind .optional(), .nullable(), .default() or .readonly(). The provider sees them, so they count now. One more shape drops: TypeScript numeric enums lost their reverse-mapping labels.

shape (inputSchema tokens)	old	new
.describe("x").optional()	14	35
.describe("x").nullable()	14	35
z.enum([a,b]).optional()	14	39
z.enum([a,b]).optional().describe("U")	34	59
z.enum([a,b]).nullable()	14	39
z.enum([a,b]).readonly()	14	39
z.object({x}).optional()	14	77
z.object({x}).nullable()	14	57
z.object({x}).default({})	14	57
nullable object with descriptions	14	97
z.array(z.object({x})).optional()	14	59
z.array(z.object({x})).nullable()	14	59
nullable + nullish mix	32	52
realistic tool with optional fields	110	196
TS numeric enum enum N {A, B}	41	39

I checked these shapes against Anthropic's count_tokens (claude-sonnet-4-5-20250929, two runs of 13 and 8 schemas, 1 tool each, description: "d". I took them before the $ref and memo changes. I repeated the 41-shape old-vs-new run above on this head and it gives the same 26 identical and 15 changed shapes). Columns are the tool tokens by the API, the new prediction and the old prediction, with the error in brackets. Every required-only shape is identical old/new and sits +20 above the API: that is a constant of the current sonnet-4.5 config, not of the walker. Raw errors hide the old under-count behind that constant. Measured against it, the 13 wrapped shapes are within -22..+9 with the new walker and within -51..-2 with the old one:

flat                     actual  576  new  596 (+20)  old  596 (+20)
nested                   actual  666  new  685 (+19)  old  685 (+19)
arrayOfObjects           actual  612  new  623 (+11)  old  623 (+11)
optionalEnum             actual  553  new  581 (+28)  old  570 (+17)
optionalObject           actual  576  new  602 (+26)  old  570 (-6)
optionalArrayOfObjects   actual  580  new  597 (+17)  old  570 (-10)
defaultObject            actual  578  new  596 (+18)  old  570 (-8)
mixedRealistic           actual  644  new  673 (+29)  old  623 (-21)
nullableObject           actual  589  new  596 (+7)   old  570 (-19)
nullableObjectInnerDesc  actual  601  new  608 (+7)   old  570 (-31)
nullableArrayOfObjects   actual  599  new  597 (-2)   old  570 (-29)
realUnion (not walked)   actual  621  new  576 (-45)  old  576 (-45)

Shared and recursive definitions are bounded. Each definition is walked once per tool. These cases are tests in src/sdk.compat.test.ts, and each asserts under 500 ms. Times on the previous head (without counting cycles once) and on this head:

schema	previous head	this head
23 definitions, each referencing the next one twice	24.9 s	33 ms (the whole test, schema construction included)
10 definitions, each referencing all the others	21.4 s	about 1 ms
20-level diamond whose bottom points back at the root	7.9 s	about 1 ms

The last two timings come from the same schemas run through count() directly (diamond depth 20 and clique of 10).

Risks

1. Totals change for wrapped zod shapes (table above). They rise toward the provider's count. Anyone who calibrated against the old under-count will see higher tool totals for tools with optional or nullable fields.
2. ai-tokenizer/sdk now needs ai installed at runtime. The root entry is unaffected. A CJS app on a Node without require() of ESM that uses v7 cannot require("ai") v7 itself either.
3. A v5 user with a non-zod Standard Schema tool gets an unresolved entry: v5's asSchema() only converts zod. v6 and v7 convert any vendor that exposes JSON Schema.
4. Not walked, same as before: real unions (anyOf/oneOf with 2+ non-null variants), allOf, $refs that point outside the schema or use an anchor (#name), and $ref alias chains longer than one hop. The last row of the API table shows the cost of a real union (-45).
5. A definition used from several places counts once per use, as the old walker counted inline copies. The provider receives a shared definition once, so a shape used twice is over-counted. I measured a 4-property .meta({ id }) object against count_tokens (claude-sonnet-4-5-20250929): used once, +3 (old and new); used twice, +113 (old and new); used four times, +333 (old and new). Counting each referenced definition once would fix that, and it would change totals for repeated .meta({ id }) shapes from main. I kept the main totals.
6. Tool results keep the wrapper tokens. They stand in for the tool_result block overhead, which ModelTokens has no field for. A later PR can calibrate a per-model field and drop the wrapper.
7. CountResultMessageContent is unchanged and CountResultTools gains the field unresolved. A file part still counts 100 whatever its media type ({ type: "file", data, mediaType: "image/png" } on openai/gpt-5 is 110 on main and 110 here), and an image part counts 85. A v7 file with an image media type is therefore not counted as an image yet.
8. Tools the AI SDK cannot convert now count 0 and are listed. zod 4's own conversion throws for z.date(), z.bigint(), z.set(), z.map() and similar, so the SDK cannot send such a tool either. The old walker counted their properties. Provider tools (type: "provider" or "provider-defined") count their name and description but not their schema, because the provider ignores it.
9. v7 callable descriptions are called with { context: {} }. A throwing function lands in unresolved.
10. count() converts each tool input schema once per input schema object, then reads the remembered JSON Schema. With 20 zod 4 tools and the same tool objects on every call I measured 0.35 ms per call on main and 0.07 ms now (2000 calls, Bun 1.2.15). A call with tool objects built fresh each time converts them each time (0.67 ms for the same 20 tools in an earlier run).
11. bun run format --check runs prettier --write and rewrites src/models.json locally. I reverted it every time and did not commit it.

---

Generated with xum • Model: anthropic:claude-sonnet-5-5 • Thinking: xhigh

[ThomasK33]

@codex review

[ThomasK33]

@codex review

[ThomasK33]

Review record for head 3e7ceec58ec51237f72cf9f1d1968d7961998303.

• Reviews: 6 rounds. Codex does not respond on this repository, so the operator waived it here. Two independent review lanes (Claude Opus and gpt-6.1-sol) replaced it. An independent verifier re-ran every round.
• Final round: both lanes clean, verifier pass. A clean-context advisor recommends "ready with tracked follow-ups".
• Tests on this head: typecheck ok, bun test 301 pass / 1 skip / 0 fail, build ok, CI 10 of 10 (8-cell ai × zod × node matrix). A count_tokens replay of the repo's accuracy workload equals main on 15 of 15 conversations. 30,000 random v5-shaped message lists give the same totals as main.
• Follow-ups (the next PR that changes src/sdk.ts fixes them; I will open a tracking issue when this merges):
  1. A nullable unwrap drops sibling keywords such as enum.
  2. A $ref with sibling properties bypasses the memo, which is quadratic on large schemas.
  3. ai-tokenizer/sdk imports asSchema from ai at runtime while ai is an optional peer.
  4. The schema cache returns stale counts after in-place schema edits.

---

Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: high