feat: DocumentSplitter — add split_by="token" mode using tiktoken

[rautaditya2606]

Related Issues

closes #12528

Proposed Changes

DocumentSplitter supports 7 split_by modes but not token-based splitting. Users who need
simple token-aware chunking are forced to use RecursiveDocumentSplitter even when they don't
need recursion. This PR adds split_by="token" to DocumentSplitter using tiktoken, aligning
it with the existing split_unit="token" support in RecursiveDocumentSplitter.

What changed:

• split_by="token" added to the Literal type and valid_split_by list
• New tokenizer_encoding: str = "o200k_base" param (keyword-only, matches RecursiveDocumentSplitter default)
• Lazy tiktoken import via LazyImport — same pattern as RecursiveDocumentSplitter
• __init__: eager tiktoken_imports.check() if token mode; sets self._tiktoken_tokenizer = None
• warm_up(): loads tiktoken encoder if token mode — mirrors sentence splitter pattern exactly
• run(): auto-triggers warm_up() for token mode — same lazy-init pattern as sentence mode
• _split_by_token(): encode → slice by [i : i + split_length] with step = split_length - split_overlap → decode; tracks cur_page and cur_start_idx; delegates to _create_docs_from_splits
• to_dict(): serialises tokenizer_encoding

Before:

# No token-aware option — forced to use RecursiveDocumentSplitter
splitter = RecursiveDocumentSplitter(split_length=256, split_overlap=32, split_unit="token")

After:

splitter = DocumentSplitter(split_by="token", split_length=256, split_overlap=32)
result = splitter.run(documents=[Document(content="...")])

How did you test it?

Added TestSplittingByToken class (7 @pytest.mark.integration tests) to
test/components/preprocessors/test_document_splitter.py:

• test_basic_chunking — chunks produced, no chunk exceeds split_length tokens
• test_metadata_set — source_id, split_id, split_idx_start, page_number all set correctly
• test_overlap_produces_shared_text — overlap produces correct index overlap between consecutive chunks
• test_page_tracking — page_number advances correctly across \f boundaries
• test_empty_document_skipped — empty content skipped with correct log message
• test_custom_encoding — tokenizer_encoding="cl100k_base" works correctly
• test_serialization_roundtrip — to_dict / from_dict round-trip preserves all params

All 56 existing unit tests and 7 new integration tests pass. mypy clean.

hatch run test:unit test/components/preprocessors/test_document_splitter.py
hatch run test:integration test/components/preprocessors/test_document_splitter.py
hatch run test:types

Notes for the reviewer

• _split_by_token is self-contained and does not share code with RecursiveDocumentSplitter —
  the recursive splitter's tokenizer logic is deeply intertwined with its overlap and chunk-length
  helpers. A future refactor can extract shared utilities if needed.
• split_threshold is not applied in token mode — consistent with how it is silently irrelevant
  for sentence and function modes.
• The existing split_overlap >= split_length guard at _init_checks L153–154 covers token mode
  automatically — no additional guard needed.

Checklist

• I have read the contributors guidelines and the code of conduct.
• I have updated the related issue with new insights and changes.
• I have added unit tests and updated the docstrings.
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test: and added ! in case the PR includes breaking changes.
• I have documented my code.
• I have added a release note file, following the contributors guidelines.
• I have run pre-commit hooks and fixed any issue.

[vercel]

@rautaditya2606 is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[github-actions]

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
haystack/components/preprocessors
document_splitter.py					130-131, 187, 210, 236, 272-295
Project Total

This report was generated by python-coverage-comment-action