feat(encoding): add Gemma 4 (Gemini) and Llama 4 encodings

README.md

@@ -88,8 +88,25 @@ const tokenizer = new Tokenizer(o200k_base);
 const total = tokenizer.count("some text input");
 ```
 
+Gemini models use the Gemma 4 tokenizer, and Llama 4 has its own vocabulary. Both encodings give the same token ids as the open tokenizers (checked against Hugging Face `tokenizers`). The provider APIs clean up message text before they tokenize it, and the encodings do not model that. Measured through the Vercel AI Gateway on 2026-11-03:
+
+- Gemini 2.5 (`flash`, `pro`, `flash-lite`): no difference on 14 probes.
+- Gemini 3 and newer (`3-flash`, `3.1-*`, `3.5-*`, `3.6-flash` to `3.8-flash`): the API drops every `\r` and every leading newline, and it counts a literal U+2581 as 3 byte tokens. `count` reads 1 token too many for each `\r` and 1 too many for a leading newline run. For a text with one U+2581 it reads 3 too few.
+- Llama 4 Scout: the API trims leading and trailing whitespace of the message content. `count` reads 1 token too many for a whitespace run at the start, and 1 for one at the end. The API also reads Llama special-token text such as `<|eot|>` as the special token, so `count` reads too many tokens for it. Use `encode(text, "all", []).length` to match. Gemini does not read `<bos>` or `<eos>` in the text as special tokens.
+
+Text without those characters matched the API on all 11 samples per model. Apply the cleanup yourself first if you need an exact count for such text.
+
+```ts
+import { Tokenizer } from "ai-tokenizer";
+import * as llama4 from "ai-tokenizer/encoding/llama4";
+import * as gemma4 from "ai-tokenizer/encoding/gemma4";
+
+new Tokenizer(llama4).count("some text"); // Llama 4 Scout and Maverick
+new Tokenizer(gemma4).count("some text"); // Gemini
+```
+
 > [!WARNING]
-> Import encodings selectively based on what you need. Each encoding is 2-8MB uncompressed.
+> Import encodings selectively based on what you need. Each encoding is 2-10MB uncompressed.
 
 ## Claude encodings
 

package.json

@@ -38,7 +38,9 @@
       "encoding/claude_v3": ["./dist/encoding/claude_v3.d.ts"],
       "encoding/claude_v4_7": ["./dist/encoding/claude_v4_7.d.ts"],
       "encoding/claude_v4_8": ["./dist/encoding/claude_v4_8.d.ts"],
-      "encoding/claude_v5": ["./dist/encoding/claude_v5.d.ts"]
+      "encoding/claude_v5": ["./dist/encoding/claude_v5.d.ts"],
+      "encoding/gemma4": ["./dist/encoding/gemma4.d.ts"],
+      "encoding/llama4": ["./dist/encoding/llama4.d.ts"]
     }
   },
   "exports": {
@@ -151,6 +153,26 @@
         "types": "./dist/encoding/claude_v5.d.cts",
         "default": "./dist/encoding/claude_v5.cjs"
       }
+    },
+    "./encoding/gemma4": {
+      "import": {
+        "types": "./dist/encoding/gemma4.d.ts",
+        "default": "./dist/encoding/gemma4.js"
+      },
+      "require": {
+        "types": "./dist/encoding/gemma4.d.cts",
+        "default": "./dist/encoding/gemma4.cjs"
+      }
+    },
+    "./encoding/llama4": {
+      "import": {
+        "types": "./dist/encoding/llama4.d.ts",
+        "default": "./dist/encoding/llama4.js"
+      },
+      "require": {
+        "types": "./dist/encoding/llama4.d.cts",
+        "default": "./dist/encoding/llama4.cjs"
+      }
     }
   },
   "description": "A faster than tiktoken tokenizer with first-class support for Vercel's AI SDK.",