[alexanderbianchi]

What changes

For opt-in eager Iceberg file planning, retain selected file bytes, optional row counts, and whether a predicate or delete files can change output cardinality. Use those planned-file statistics rather than whole-snapshot totals and cap the desired distributed scan task count by available file work. Empty plans report exact zero rows; filtered or delete-bearing scans do not claim exact row counts. With eager planning off, existing snapshot-based estimation remains unchanged.

This is the file-statistics/task-sizing portion of DataDog#30 that was intentionally excluded from upstream #773. It does not introduce byte-balanced assignment, column-byte join costing, or worker routing.

Stack

Based on #773, then #786, then upstream main. Review only the base-to-head diff.

Validation

• cargo fmt --all -- --check
• cargo test -p datafusion-distributed-iceberg --features integration
• cargo clippy -p datafusion-distributed-iceberg --features integration --all-targets -- -D warnings

Tests cover manifest totals when snapshot summaries are missing, filtered/inexact statistics, selected-file counts and bounded task sizing. No performance claim. Developed with AI assistance.

[gabotechs]

👍 nice!

[gabotechs]

🤔 So we cannot byte-range split files here no?

It'd be great if somehow at some point we could use native DataFusion parquet reads for Iceberg.

[gabotechs]

This occupies a fair amount of real-state, but the only important bit for a test reviewer is:

                config.eager_file_planning = true;

Maybe we can add one more helper that says:

.iceberg_config_mut(|&mut iceberg_cfg| { ... })

Or something like:

.with_eager_file_planning()