fix: deep-copy document metadata in PythonCodeSplitter - #12468
Open
bharadwaj-pendyala wants to merge 1 commit into
Open
fix: deep-copy document metadata in PythonCodeSplitter#12468bharadwaj-pendyala wants to merge 1 commit into
bharadwaj-pendyala wants to merge 1 commit into
Conversation
PythonCodeSplitter shallow-copied the parent document's metadata, so every chunk shared its nested objects with the other chunks and with the caller's Document. The oversized-function secondary split shared one base_meta the same way. Deep-copy the filtered mapping in one call, which also keeps two metadata keys pointing at one object aliased within a chunk, matching DocumentSplitter.
|
@bharadwaj-pendyala is attempting to deploy a commit to the deepset Team on Vercel. A member of the Team first needs to authorize it. |
Contributor
Coverage report
Click to see where and how coverage changed
This report was generated by python-coverage-comment-action |
||||||||||||||||||||||||
bharadwaj-pendyala
marked this pull request as ready for review
August 29, 2026 02:16
bharadwaj-pendyala
requested review from
bogdankostic
and removed request for
a team
August 29, 2026 02:16
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related Issues
None.
Proposed Changes:
PythonCodeSplittershallow-copied parent metadata: appending tochunks[0].meta["tags"]onmainat 75ee514 changed every chunk'stagsand the inputDocumenttoo.DocumentSplitterdoesn't._build_chunk_metanow deep-copies filtered parent metadata, matchingDocumentSplitter. The oversized-function secondary split also gave every piece a shallowdict()of one sharedbase_meta; each piece now builds its own.The copy preserves
meta["a"] is meta["b"]within a chunk, asDocumentSplitterdoes.How did you test it?
Three unit tests in
test_python_code_splitter.py, named after equivalents in the CSV, Markdown and hierarchical splitter test files. The twotest_nested_metadata_is_not_shared_*tests fail without the fix and pass with it.test_metadata_keys_sharing_one_value_still_share_it_within_a_chunkpreserves the aliasing contract.hatchisn't available in my environment, so I ran pytest, ruff, mypy, and the pre-commit hooks directly. All passed. The only failures acrosstest/components/preprocessors/are the 31 NLTK-download tests that already fail on a clean checkout here.Notes for the reviewer
Deep-copying per chunk isn't free: 200 chunks against a 5,000-entry meta takes about 4.3s, but
DocumentSplitteralready pays exactly that. Metadata holding somethingdeepcopycan't handle raisesTypeError, as every other splitter already does.This PR was fully generated with an AI assistant. I have reviewed the changes and run the tests and checks above.
Checklist