Repository navigation
feat(python): add batch temporal matching kernels - #1238
XiaoHongbo-Hope wants to merge 1 commit into
Conversation
JingsongLi
left a comment
There was a problem hiding this comment.
The 18 temporal tests passed against an isolated build of the unchanged temporal binding. I found one Arrow input-handling issue; the inline comment includes a reproducer and a suggested fix.
| "Expected a non-null int64 Arrow array", | ||
| )); | ||
| } | ||
| Ok(Int64Array::from(data)) |
There was a problem hiding this comment.
[P2] Align foreign buffers before constructing Int64Array
ArrayData::from_pyarrow_bound() preserves the foreign buffer address, but Int64Array::from(data) requires 8-byte alignment and panics when it is missing. PyArrow permits valid, sorted, non-null int64 arrays backed by unaligned buffers, including zero-copy NumPy inputs. I reproduced a PanicException with the unchanged temporal binding using Arrow 58.4.0:
import struct
import pyarrow as pa
from pypaimon_rust import temporal_match
raw = b'x' + struct.pack('<3q', 1, 3, 5)
times = pa.Array.from_buffers(
pa.int64(), 3, [None, pa.py_buffer(raw).slice(1)])
times.validate(full=True) # Valid, sorted, non-null int64 input
temporal_match(
times, pa.array([1], type=pa.int64()),
pa.array([0], type=pa.int64()),
pa.array([3], type=pa.int64()), 'nearest')This should return first=[0], second=[-1], but instead raises PanicException: Memory pointer from external source ... is not aligned with the specified scalar type.
Please call data.align_buffers() before constructing the typed array, as the existing arrow_input::array_from_pyarrow helper already does, and add a regression test for unaligned input. The same helper imports all four input arrays, so this affects more than just times.
Purpose
Add an Arrow batch temporal matching kernel for PyPaimon alignment. It returns matching positions for grouped as-of joins, interpolation brackets and windows, avoiding one Python binary search per anchor row.
The kernel accepts sorted, non-null int64 keys and validated group bounds, releases the GIL while matching, and uses i128 arithmetic for timestamp limits. Duplicate selection, nearest ties, exact-match exclusion, tolerance and window closure follow PyPaimon's existing semantics. It performs no table reads or writes.
Validation
Consumer integration: apache/paimon#10846.