Repository navigation
Support PyArrow 26 in the Python SDK - #40927
Anurag-S1ngh wants to merge 1 commit into
Conversation
Raise the pyarrow upper bound to <27.0.0 and add the 26 tox environment and Gradle task so the dependency postcommit covers it. PyArrow 26 requires NumPy 2, so that environment installs NumPy 2 while the older environments stay on 1.26.4.
There was a problem hiding this comment.
📋 Review Summary
This PR adds support for pyarrow 26 by raising the upper bound to <27.0.0 in sdks/python/setup.py and adding the py311-pyarrow-26 test environment in sdks/python/tox.ini and sdks/python/test-suites/tox/py311/build.gradle. Because pyarrow 26 dropped support for numpy 1, the tox configuration appropriately pins numpy>=2.0.0,<2.5.0 for pyarrow 26 while preserving numpy==1.26.4 for earlier pyarrow versions.
🔍 General Feedback
- The changes across
setup.py,tox.ini, andbuild.gradleare clean, well-aligned, and follow the established patterns used for previous dependency updates (such as the recent pandas 2.3 / NumPy 2 split). - Consider incrementing
.github/trigger_files/beam_PostCommit_Python_Dependency.jsonif you would like to trigger thePostCommit Python Dependencyworkflow directly on this PR to verify the newtestPy311pyarrow-26Gradle task in CI before merging.
💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.
|
Assigning reviewers: R: @tvalentyn for label python. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
Summary
Adds PyArrow 26 to the supported range of the Python SDK and to the dependency postcommit matrix.
PyArrow 26 requires NumPy 2 at runtime, so the new test environment installs NumPy 2 while the existing PyArrow 17 to 25 environments keep using NumPy 1.26.4.
Changes
sdks/python/setup.py: raise thepyarrowupper bound from<26.0.0to<27.0.0sdks/python/tox.ini: add26to thepy311-pyarrow-...environment, add26: pyarrow>=26,<27, and make NumPy conditional so 17 to 25 stay on 1.26.4 while 26 usesnumpy>=2.0.0,<2.5.0sdks/python/test-suites/tox/py311/build.gradle: addtestPy311pyarrow-26and wire it intotestandpostCommitPyDepTesting
Installed the SDK from source with Python 3.11,
pyarrow==26.0.0andnumpy==2.4.6:pytest apache_beam/io/parquetio_test.py -m uses_pyarrow: 35 passed, 1 skipped, 5 deselectedpytest apache_beam/typehints/arrow_type_compatibility_test.py apache_beam/typehints/schemas_test.py: 821 passed, 5 skippedtox c -e py311-pyarrow-26 -k depsresolvespyarrow>=26,<27andnumpy>=2.0.0,<2.5.0tox c -e py311-pyarrow-25 -k depsstill resolvespyarrow>=25,<26andnumpy==1.26.4The new full matrix entry runs as
testPy311pyarrow-26for reviewers who want the completeuses_pyarrowsuite.fixes #40919