Repository navigation
Conversation
Initialize Arrow compute from TableRead::Create so predicate pushdown and other compute-backed read paths work for downstream users that build Arrow compute as a static library.
There was a problem hiding this comment.
@wgtmac, thanks for contribution. Please create issue for this bugfix. Meanwhile, I left one comment for build compatibility issue with the bundled Arrow version.
| #include <vector> | ||
|
|
||
| #include "arrow/c/bridge.h" | ||
| #include "arrow/compute/initialize.h" |
There was a problem hiding this comment.
Guard initialization for the bundled Arrow version: The standard build pins Arrow 17.0.0 in third_party/versions.txt, which provides neither arrow/compute/initialize.h nor arrow::compute::Initialize(). cmake_modules/arrow.diff does not backport them, so this unconditional include breaks compilation with the repository's bundled dependency. Please guard both the include and the initialization call for Arrow versions that provide this API. Arrow 17 already registers the compute kernels when constructing its global function registry, with ARROW_COMPUTE=ON as configured here.
There was a problem hiding this comment.
Thanks for the feedback! I just found this issue when I tried to link paimon-cpp to a newer version of Arrow C++. Perhaps it is better for us to internally initialize the Arrow compute kernels before calling any paimon function. :)
Purpose
Linked issue: N/A
Initialize Arrow Compute from
TableRead::Create()so kernels used by predicate pushdown and other compute-backed paths are registered for downstream applications that link Arrow Compute statically.Without this initialization, the global Arrow function registry only contains the core kernels. Predicate binding then fails at runtime with errors such as:
arrow::compute::Initialize()is idempotent. Calling it from the shared table-read entry point covers Parquet row-group predicate pushdown, Arrow-backed predicate evaluation, and future read paths without requiring every embedding application to initialize Arrow separately.Tests
TableRead::Create()and this code path.API and Format
No public API, storage format, or protocol changes.
Documentation
No documentation changes are required. The initialization is internal to Paimon's table-read creation path.
Generative AI tooling
Generated-by: OpenAI Codex 0.162.1