Skip to content

fix(read): initialize Arrow compute kernels - #487

Open
wgtmac wants to merge 1 commit into
apache:mainfrom
wgtmac:codex/init-arrow-compute
Open

wgtmac wants to merge 1 commit into
apache:mainfrom
wgtmac:codex/init-arrow-compute

Conversation

@wgtmac

@wgtmac wgtmac commented Nov 2, 2026

Copy link
Copy Markdown
Member

Purpose

Linked issue: N/A

Initialize Arrow Compute from TableRead::Create() so kernels used by predicate pushdown and other compute-backed paths are registered for downstream applications that link Arrow Compute statically.

Without this initialization, the global Arrow function registry only contains the core kernels. Predicate binding then fails at runtime with errors such as:

Key error: No function registered with name: equal

arrow::compute::Initialize() is idempotent. Calling it from the shared table-read entry point covers Parquet row-group predicate pushdown, Arrow-backed predicate evaluation, and future read paths without requiring every embedding application to initialize Arrow separately.

Tests

  • Compiled the changed translation unit with the downstream project's C++23/libc++ flags. Compilation passed.
  • Ran the repository's pre-commit hooks on the changed file; clang-format, codespell, and cpplint passed.
  • No focused unit test was added because the failure depends on a process-global registry that has not been initialized; existing reader and predicate pushdown tests exercise TableRead::Create() and this code path.

API and Format

No public API, storage format, or protocol changes.

Documentation

No documentation changes are required. The initialization is internal to Paimon's table-read creation path.

Generative AI tooling

Generated-by: OpenAI Codex 0.162.1

Initialize Arrow compute from TableRead::Create so predicate pushdown and other compute-backed read paths work for downstream users that build Arrow compute as a static library.

@SteNicholas SteNicholas left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wgtmac, thanks for contribution. Please create issue for this bugfix. Meanwhile, I left one comment for build compatibility issue with the bundled Arrow version.

#include <vector>

#include "arrow/c/bridge.h"
#include "arrow/compute/initialize.h"

@SteNicholas SteNicholas Nov 2, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Guard initialization for the bundled Arrow version: The standard build pins Arrow 17.0.0 in third_party/versions.txt, which provides neither arrow/compute/initialize.h nor arrow::compute::Initialize(). cmake_modules/arrow.diff does not backport them, so this unconditional include breaks compilation with the repository's bundled dependency. Please guard both the include and the initialization call for Arrow versions that provide this API. Arrow 17 already registers the compute kernels when constructing its global function registry, with ARROW_COMPUTE=ON as configured here.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the feedback! I just found this issue when I tried to link paimon-cpp to a newer version of Arrow C++. Perhaps it is better for us to internally initialize the Arrow compute kernels before calling any paimon function. :)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants