fix(proto): preserve Parquet source and sink state

[buraksenn]

Which issue does this PR close?

• Closes Destructure proto hooks for the Parquet source and sink #24620.

Rationale for this change

Parquet source and sink protobuf hooks accessed fields individually and silently dropped source metadata prefetch hints and sink sorting-column metadata. This changed performance characteristics and omitted metadata from files written after a plan round trip.

What changes are included in this PR?

• Exhaustively destructure ParquetSource, ParquetSink, and their protobuf messages.
• Preserve ParquetSource::metadata_size_hint with a checked integer conversion.
• Preserve ParquetSink::sorting_columns, including the distinction between None and an empty list.
• Explicitly document source/sink fields that are reconstructed or runtime-only.
• Document the generated Rust API migration in the DataFusion 56.0.0 upgrade guide.

#24930 already preserves reverse_row_groups and sort_order_for_reorder; this PR builds on that behavior rather than duplicating it.

The protobuf wire changes are additive and backward compatible.

What is the testing strategy for this PR?

Physical-plan round-trip coverage verifies:

• Metadata-size hints preserve None, zero, a normal value, and usize::MAX.
• Parquet sorting columns preserve None, an empty list, and populated lists.
• Both fields survive binary protobuf and public JSON API round trips.

Both regression tests were ablation-checked and fail when their corresponding field is omitted from serialization.

Validated with:

• cargo test -p datafusion-proto --test proto_integration --features json (262 passed)
• cargo clippy -p datafusion-datasource-parquet -p datafusion-proto --all-features --tests -- -D warnings
• cargo fmt --all -- --check
• ./ci/scripts/doc_prettier_check.sh --write --allow-dirty

Are there any user-facing changes?

Parquet scan metadata prefetch hints and sink sorting-column metadata now survive protobuf plan round trips. The protobuf wire additions are backward compatible.

Adding metadata_size_hint to the generated public ParquetScanExecNode Rust struct and sorting_columns to ParquetSink is a source-level API change for callers using exhaustive struct literals. Such callers must initialize the new fields (use None to retain previous behavior) or use ..Default::default(). This migration is documented in the DataFusion 56.0.0 upgrade guide.

[github-actions]

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion-datasource-parquet v55.1.0 (current)
       Built [  55.784s] (current)
     Parsing datafusion-datasource-parquet v55.1.0 (current)
      Parsed [   0.035s] (current)
    Building datafusion-datasource-parquet v55.1.0 (baseline)
       Built [  54.856s] (baseline)
     Parsing datafusion-datasource-parquet v55.1.0 (baseline)
      Parsed [   0.036s] (baseline)
    Checking datafusion-datasource-parquet v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.171s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [ 112.274s] datafusion-datasource-parquet
    Building datafusion-proto v55.1.0 (current)
       Built [  61.063s] (current)
     Parsing datafusion-proto v55.1.0 (current)
      Parsed [   0.018s] (current)
    Building datafusion-proto v55.1.0 (baseline)
       Built [  61.495s] (baseline)
     Parsing datafusion-proto v55.1.0 (baseline)
      Parsed [   0.019s] (baseline)
    Checking datafusion-proto v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.116s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [ 124.165s] datafusion-proto
    Building datafusion-proto-models v55.1.0 (current)
       Built [  27.069s] (current)
     Parsing datafusion-proto-models v55.1.0 (current)
      Parsed [   0.135s] (current)
    Building datafusion-proto-models v55.1.0 (baseline)
       Built [  26.785s] (baseline)
     Parsing datafusion-proto-models v55.1.0 (baseline)
      Parsed [   0.138s] (baseline)
    Checking datafusion-proto-models v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   1.808s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip

--- failure constructible_struct_adds_field: struct exhaustively constructible through public API adds field ---

Description:
A pub struct that could be exhaustively constructed with a literal using only public API has a new pub field, breaking existing exhaustive literals.
        ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/constructible_struct_adds_field.ron

Failed in:
  field ParquetScanExecNode.metadata_size_hint in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:2031
  field ParquetScanExecNode.metadata_size_hint in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:2031
  field ParquetSink.sorting_columns in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:1561
  field ParquetSink.sorting_columns in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:1561

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  57.153s] datafusion-proto-models

[codecov-commenter]

Codecov Report

❌ Patch coverage is 78.99160% with 50 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.28%. Comparing base (16ecdc9) to head (e9111a8).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/proto-models/src/generated/pbjson.rs	74.34%	20 Missing and 19 partials ⚠️
datafusion/datasource-parquet/src/sink.rs	86.53%	2 Missing and 5 partials ⚠️
datafusion/datasource-parquet/src/source.rs	88.23%	0 Missing and 4 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #25057      +/-   ##
==========================================
+ Coverage   81.97%   82.28%   +0.31%     
==========================================
  Files        1137     1137              
  Lines      429901   430108     +207     
  Branches   429901   430108     +207     
==========================================
+ Hits       352414   353928    +1514     
+ Misses      56437    54759    -1678     
- Partials    21050    21421     +371     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[kumarUjjawal]

Looks good 👍

[kumarUjjawal]

Thank you @buraksenn

[kumarUjjawal]

@buraksenn can you fix the clippy issue

[buraksenn]

@buraksenn can you fix the clippy issue

sorry missed this while fixing conflict

[kumarUjjawal]

sorry missed this while fixing conflict

There seems to be a conflict 😬