Skip to content

feat(python): add batch temporal matching kernels - #1238

Open
XiaoHongbo-Hope wants to merge 1 commit into
apache:mainfrom
XiaoHongbo-Hope:codex/temporal-batch-kernels
Open

XiaoHongbo-Hope wants to merge 1 commit into
apache:mainfrom
XiaoHongbo-Hope:codex/temporal-batch-kernels

Conversation

@XiaoHongbo-Hope

@XiaoHongbo-Hope XiaoHongbo-Hope commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Add an Arrow batch temporal matching kernel for PyPaimon alignment. It returns matching positions for grouped as-of joins, interpolation brackets and windows, avoiding one Python binary search per anchor row.

The kernel accepts sorted, non-null int64 keys and validated group bounds, releases the GIL while matching, and uses i128 arithmetic for timestamp limits. Duplicate selection, nearest ties, exact-match exclusion, tolerance and window closure follow PyPaimon's existing semantics. It performs no table reads or writes.

Validation

  • Three standalone Rust kernel boundary tests pass.
  • The full Rust extension plus PyPaimon integration tests: 101 passed, 155 subtests passed. Covers random differential cases, sliced arrays, empty/missing groups, duplicates, closure, int64 limits and invalid inputs.
  • Full release extension build passes with Rust 1.94.0.
  • Complete-output integration validation: all 300,000 rows checked across 500 synthetic episodes.

Consumer integration: apache/paimon#10846.

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The 18 temporal tests passed against an isolated build of the unchanged temporal binding. I found one Arrow input-handling issue; the inline comment includes a reproducer and a suggested fix.

"Expected a non-null int64 Arrow array",
));
}
Ok(Int64Array::from(data))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Align foreign buffers before constructing Int64Array

ArrayData::from_pyarrow_bound() preserves the foreign buffer address, but Int64Array::from(data) requires 8-byte alignment and panics when it is missing. PyArrow permits valid, sorted, non-null int64 arrays backed by unaligned buffers, including zero-copy NumPy inputs. I reproduced a PanicException with the unchanged temporal binding using Arrow 58.4.0:

import struct
import pyarrow as pa
from pypaimon_rust import temporal_match

raw = b'x' + struct.pack('<3q', 1, 3, 5)
times = pa.Array.from_buffers(
    pa.int64(), 3, [None, pa.py_buffer(raw).slice(1)])
times.validate(full=True)  # Valid, sorted, non-null int64 input
temporal_match(
    times, pa.array([1], type=pa.int64()),
    pa.array([0], type=pa.int64()),
    pa.array([3], type=pa.int64()), 'nearest')

This should return first=[0], second=[-1], but instead raises PanicException: Memory pointer from external source ... is not aligned with the specified scalar type.

Please call data.align_buffers() before constructing the typed array, as the existing arrow_input::array_from_pyarrow helper already does, and add a regression test for unaligned input. The same helper imports all four input arrays, so this affects more than just times.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants