Skip to content

Enable page-index pruning for Iceberg scans - #65

Open
terbed wants to merge 2 commits into
apache:mainfrom
quantscious:feat/page-index-pruning
Open

terbed wants to merge 2 commits into
apache:mainfrom
quantscious:feat/page-index-pruning

Conversation

@terbed

@terbed terbed commented Oct 11, 2026 •

Copy link
Copy Markdown

Fixes #62

Forward datafusion.execution.parquet.enable_page_index to Iceberg's
scan builder. Iceberg scans now follow the session setting, including
DataFusion's enabled default.

Add regression coverage for reduced Parquet reads with identical query
results, explicit disabling, files without page indexes, and nullable
decimal columns with projection. Tests cover both static and
catalog-backed table providers.

Update the Iceberg dependency to include apache/iceberg-rust#3314, which prevents incorrect row elimination for decimal predicates when page-index pruning is enabled. Adjust existing tests for the updated dependency’s API changes.

Comment thread Cargo.toml Outdated
Comment thread crates/datafusion/src/test_utils/harness.rs Outdated
Comment thread crates/datafusion/src/test_utils/harness.rs Outdated
Comment thread crates/datafusion/src/test_utils/harness.rs Outdated
Comment thread crates/datafusion/src/write/project.rs
@terbed
terbed force-pushed the feat/page-index-pruning branch from d89efcb to f95e1a6 Compare October 20, 2026 20:58
@terbed

terbed commented Oct 20, 2026 •

Copy link
Copy Markdown
Author

Thanks @gabotechs for taking the time to review and comment on this PR.

  • I've removed the version bump
  • Removed tests proving page index pruning reduces byte reads
  • Keeping correctness tests using the taxi dataset with and without page indexing enabled

This approach significantly simplified the pr, the main complexity is proving page index pruning's effect. That could be a separate focused PR if such test would be important/necessary.

f95e1a6

Would you recommend further changes? Or could this PR in this form be merged?

@gabotechs gabotechs left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 All good here! thanks @terbed. Will merge it as soon as CI passes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Enable page index pruning for Iceberg scans

2 participants