Repository navigation
[python] Batch temporal alignment metadata and native matching - #10846
Open
XiaoHongbo-Hope wants to merge 4 commits into
Open
XiaoHongbo-Hope wants to merge 4 commits into
XiaoHongbo-Hope wants to merge 4 commits into
Conversation
XiaoHongbo-Hope
marked this pull request as ready for review
October 28, 2026 14:01
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Temporal alignment currently splits upstream batches without filling the requested batch size and materializes anchor keys as Python dictionaries. Coalesce metadata up to the requested size, pass Arrow tables directly, and reuse group encodings across aligned sources. No custom metadata container is introduced.
Use the optional Rust batch matcher for int64 temporal keys. As-of result mapping and scalar floating-point interpolation remain columnar; unsupported key types and older native packages retain the existing Python path. Integer payload interpolation retains exact-ratio arithmetic. Batch defaults, snapshots, authorization and projection semantics are unchanged.
This is separate from #10843, which optimizes row-ID fetching. Native kernel: apache/paimon-rust#1238. Prototype benchmark timings are not claimed for this narrower implementation.
Validation
Native CI dependency
Temporarily pin Native CI to paimon-rust#1238 commit
927a0248163d4d230ceed8f33c281a3c538781e3. Verify the installed commit and requiretemporal_matchbefore testing. The differential test fails rather than skips whenPYPAIMON_TEST_NATIVE_READ=1; normal old-package compatibility keeps its fallback.Local native-read temporal suite: 84 passed, 155 subtests passed, 439 native reads exercised. Old-package suite: 83 passed, 1 native-only skip, 155 subtests passed. Workflow YAML, provenance-check syntax, Flake8 and diff checks pass.
Merge paimon-rust#1238 first, restore the Native CI dependency to main and rerun Native CI before merging this PR.