Repository navigation
refactor(index): pass resolved bucket mode to index manifests - #494
Merged
SteNicholas merged 3 commits intoNov 3, 2026
Merged
Conversation
SteNicholas
reviewed
Nov 2, 2026
Member
There was a problem hiding this comment.
@fastio, thanks for contribution. I left some comments for this pull request. PTAL.
Member
|
Hi, @fastio I saw the #493 issue—thank you for your contribution and the submission, and also thanks to @SteNicholas for the review. Aside from the review mentioned above, I don’t have any other concerns. Once the fix is complete, please ping us so we can merge. |
Contributor
Author
|
@SteNicholas @lxy-9602 All comments are addressed in eb2b040, PTAL. |
SteNicholas
approved these changes
Nov 3, 2026
SteNicholas
left a comment
Member
There was a problem hiding this comment.
LGTM. Thanks for contribution.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Linked issue: part of #493
IndexManifestFileandIndexManifestFileHandlerreceived the configuredbucketoption andtreated
-1asBUCKET_UNAWAREwhen choosing how to combine deletion vector index entries. Aprimary-key table with dynamic buckets also has
bucket = -1, but keeps one deletion vector indexper (partition, bucket), so it must use the bucketed combiner, as Java does for
BucketMode.HASH_DYNAMIC.This change passes the resolved
BucketMode(ResolveBucketMode(options.GetBucket(), schema))everywhere an
IndexManifestFileis created, and only uses the global deletion vector combinerfor
BUCKET_UNAWARE. That combiner merges entries by index file name and uses data file namesonly to reject a data file covered by more than one deletion vector.
There is no behavior change for tables that can be written today: dynamic bucket primary-key
tables are still rejected by the writer. This is the first step of #493; batch reads of those
tables follow in a separate PR.
Tests
IndexManifestFileHandlerTest: existing cases now pass an explicitBucketMode; addedDynamicBucketDvUsesBucketedCombiner, which goes throughIndexManifestFile::Create()andWriteIndexFiles()so the stored bucket mode is exercised end to end. It checks that a newdeletion vector index of a
HASH_DYNAMICtable replaces the previous one of the same(partition, bucket) and leaves other buckets untouched, while a separately created
BUCKET_UNAWAREmanifest rejects the same delta withTrying to add a second deletion vector for data file data-0.orc.IndexFileHandlerTestresolves the bucket mode withResolveBucketMode()from the tableschema, the same way production callers do.
CommitScannerTest,ExpireSnapshotsTest,FileSystemWriteRestoreTestand
OrphanFilesCleanerTest.Verified on Ubuntu 24.04 (aarch64), GCC 13, Debug: the affected suites pass (40 tests) and the
full
paimon-core-testpasses (2536 tests).API and Format
No public API change (only internal headers under
src/). No storage format or protocol change.Documentation
No.
Generative AI tooling
Generated-by: Claude Code (Claude Opus 5.5)