Skip to content

[python] Batch temporal alignment metadata and native matching - #10846

Open
XiaoHongbo-Hope wants to merge 4 commits into
apache:masterfrom
XiaoHongbo-Hope:codex/temporal-batch-native
Open

XiaoHongbo-Hope wants to merge 4 commits into
apache:masterfrom
XiaoHongbo-Hope:codex/temporal-batch-native

Conversation

@XiaoHongbo-Hope

@XiaoHongbo-Hope XiaoHongbo-Hope commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Temporal alignment currently splits upstream batches without filling the requested batch size and materializes anchor keys as Python dictionaries. Coalesce metadata up to the requested size, pass Arrow tables directly, and reuse group encodings across aligned sources. No custom metadata container is introduced.

Use the optional Rust batch matcher for int64 temporal keys. As-of result mapping and scalar floating-point interpolation remain columnar; unsupported key types and older native packages retain the existing Python path. Integer payload interpolation retains exact-ratio arithmetic. Batch defaults, snapshots, authorization and projection semantics are unchanged.

This is separate from #10843, which optimizes row-ID fetching. Native kernel: apache/paimon-rust#1238. Prototype benchmark timings are not claimed for this narrower implementation.

Validation

  • Compatibility path: 82 tests and 155 subtests pass; the native-only differential test requires the new extension.
  • Added checks for bounded coalescing, ordering, empty input batches, timestamp precision and old-package compatibility.
  • Full Rust extension plus integration suite: 101 passed, 155 subtests passed.
  • Full extension release build passes; all 300,000 output rows checked across 500 synthetic episodes (19 batches).
  • Flake8 with repository configuration and git diff --check pass.

Native CI dependency

Temporarily pin Native CI to paimon-rust#1238 commit 927a0248163d4d230ceed8f33c281a3c538781e3. Verify the installed commit and require temporal_match before testing. The differential test fails rather than skips when PYPAIMON_TEST_NATIVE_READ=1; normal old-package compatibility keeps its fallback.

Local native-read temporal suite: 84 passed, 155 subtests passed, 439 native reads exercised. Old-package suite: 83 passed, 1 native-only skip, 155 subtests passed. Workflow YAML, provenance-check syntax, Flake8 and diff checks pass.

Merge paimon-rust#1238 first, restore the Native CI dependency to main and rerun Native CI before merging this PR.

@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as ready for review October 28, 2026 14:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant