chore: upgrade to DataFusion main + arrow/parquet 59.1.0 [WIP]

[andygrove]

Summary

WIP draft to track the upgrade to DataFusion main (currently pinned to 179b32c9b60103d9c4e6a4364f10f6286c963904, dated 2026-08-04). Bumps arrow/parquet from 58.4.0 to 59.1.0 to match what DataFusion main uses. object_store stays at 0.13.2 (DF main is still on that).

Not ready for merge, opened as a draft so CI runs against latest DF main and breaking changes are surfaced continuously. The pin is bumped periodically to keep pace with main.

Changes

• native/Cargo.toml: swap datafusion, datafusion-datasource, datafusion-physical-expr-adapter, datafusion-spark from crates.io 54.1.0 to git = "https://github.com/apache/datafusion", rev = "..."
• native/core/Cargo.toml: same treatment for the dev-dependency datafusion-functions-nested
• Bump arrow and parquet workspace pins to 59.1.0
• Adapt native code to the breaking changes surfaced by the upgrade
• Put the native Iceberg scan behind an off-by-default iceberg-scan feature (see below)

Adaptations made

Compile breakage from pinning to DataFusion main + arrow 59 has been worked through:

• GroupsAccumulator::convert_to_state is now a required trait method and supports_convert_to_state() has been removed. That gate is what previously kept skip-partial-aggregation away from Comet's accumulators, so stubbing convert_to_state with not_impl_err would now fail at runtime once the probe fires — a behavior regression, not just a compile fix. It is instead implemented for all 13 accumulators (avg, avg_decimal, correlation, covariance, hll_plus_plus, percentile, stddev, sum_decimal, sum_int ×3, variance, merge_as_partial), via a shared convert_to_state_per_row helper that accumulates each row into its own group on a freshly created accumulator, so the emitted state columns match state() by construction. avg builds its state arrays directly because it holds a non-cloneable avg_fn. MergeAsPartialGroupsAccumulator returns its input verbatim, since that input is already the inner accumulator's intermediate state.
• Drop opt_filter from GroupsAccumulator::merge_batch impls and their delegating call sites to match the upstream signature change (avg, avg_decimal, correlation, covariance, hll_plus_plus, percentile, stddev, sum_decimal, sum_int, variance, merge_as_partial).
• Migrate MutableArrayData::extend/extend_nulls to try_extend/try_extend_nulls across array_funcs and copy.rs (arrow 59 returns a Result instead of panicking).
• Rewire shuffle spill to the new DiskManager API, where create_tmp_file returns Arc<dyn SpillFile> and path() returns Option<&Path>.
• Replace deprecated TableSchema::from_file_schema/with_table_partition_cols with the TableSchema builder.
• Use FixedSizeBinaryArray::try_from (arrow 59 dropped the From impl).
• FileMetadataCache is now a type alias, so Arc<dyn FileMetadataCache> becomes Arc<FileMetadataCache>.
• UnnestOptions::preserve_nulls was replaced by a NullHandling enum. Uses the backward-compatible with_preserve_nulls setter so behavior is unchanged.
• arrow::ipc::writer::CompressionContext is deprecated in favor of IpcWriteContext.

Native Iceberg scan is feature-gated

iceberg-rust — including its main branch — is still on arrow 58, so its RecordBatch does not unify with the arrow 59 that DataFusion main requires. That single type mismatch was enough to stop the whole datafusion-comet crate from compiling, which in turn prevented CI from surfacing any other breakage as the DF pin advanced — defeating the purpose of this branch.

The native Iceberg scan is therefore behind a new iceberg-scan cargo feature, off by default:

• With the feature off, the workspace builds and clippy-checks clean.
• With the feature on, that one RecordBatch mismatch in iceberg_scan.rs is the only remaining error, so re-enabling is a one-flag change once iceberg-rust upgrades to arrow 59.
• Planning an IcebergScan operator in a build without the feature returns an explanatory error rather than failing obscurely.

iceberg/iceberg-storage-opendal are now optional dependencies. The gating is confined to the module declaration, the Iceberg-only planner helpers, the Iceberg half of the S3 credential bridge, and the three Iceberg planner tests.

This must be reverted (feature made default again) before this PR is mergeable.

Test plan

• cd native && cargo check --workspace --all-targets compiles clean
• cd native && cargo clippy --workspace --all-targets -- -D warnings clean
• make test-rust passes — except 3 pre-existing failures in datafusion-comet-jni-bridge's panic-handling tests (error_from_panic, jlong_panic_exception, jint_array_panic_exception), which fail identically on apache/main and are unrelated to this branch
• make test-jvm passes on default profile
• Re-enable the native Iceberg scan once iceberg-rust is on arrow 59
• Address any additional breakage revealed as the DataFusion pin advances

Notes for reviewers

While adapting UnnestOptions, I noticed that explode_outer maps to NullHandling::Preserve, which produces no output rows for an empty array, whereas Spark's explode_outer emits a null row for empty arrays too. DataFusion main now expresses that with NullHandling::PreserveAndExpandEmpty. This is a pre-existing gap on main rather than upgrade fallout, so behavior is left unchanged here; it likely deserves its own issue.

[comphead]

related to #4865