Repository navigation
Conversation
|
Synced with latest main (13 commits, including the pyarrow 24→25 upgrade). No conflicts. Verified end-to-end against a REST catalog: with |
9b4d96a to
06bf26d
Compare
06bf26d to
d8b8052
Compare
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that's incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
|
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. |
Related: #3847
Rationale for this change
PyArrow supports page-index writing, but PyIceberg did not expose it through table properties. Readers such as ClickHouse can use these indexes to avoid decoding nonmatching pages for selective queries.
What changes?
Adds an opt-in
write.parquet.page-index-enabledtable property. When enabled, PyIceberg passeswrite_page_index=Trueto PyArrow'sParquetWriter, producing Parquet column and offset indexes for page-level predicate pruning.The default is
false, preserving existing output and file-size behavior.Are these changes tested?
PYTHONPATH=. uv run pytest tests/io/test_pyarrow.py -k parquet_page_index_writer_property -q— passedruff checkon changed Python files — passedgit diff --check— passedAre there any user-facing changes?
Yes. A new opt-in
write.parquet.page-index-enabledtable property (defaultfalse), documented inmkdocs/docs/configuration.md. Tables with the property enabled write Parquet column and offset indexes. Changelog label requested.Tooling note: developed with assistance from DS v4 Pro. I reviewed and verified the changes.