Skip to content

Add optional Parquet page index writes - #3829

Open
adonm wants to merge 1 commit into
apache:mainfrom
adonm:feat/parquet-page-index
Open

Add optional Parquet page index writes#3829
adonm wants to merge 1 commit into
apache:mainfrom
adonm:feat/parquet-page-index

Conversation

@adonm

@adonm adonm commented Aug 21, 2026

Copy link
Copy Markdown

What changes?

Adds an opt-in write.parquet.page-index-enabled table property. When enabled, PyIceberg passes write_page_index=True to PyArrow's ParquetWriter, producing Parquet column and offset indexes for page-level predicate pruning.

The default is false, preserving existing output and file-size behavior.

Why?

PyArrow supports page-index writing, but PyIceberg did not expose it through table properties. Readers such as ClickHouse can use these indexes to avoid decoding nonmatching pages for selective queries.

Testing

  • Added a direct unit test for default/opt-in writer kwargs
  • Extended the existing Parquet writer-property integration parameterization
  • PYTHONPATH=. uv run pytest tests/io/test_pyarrow.py -k parquet_page_index_writer_property -q — passed
  • ruff check on changed Python files — passed
  • git diff --check — passed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant