Describe the bug
Comet validates Parquet field id lookups in the physical expression adapter, which resolves each file's mapping once per file (#5654). DataFusion 55's Parquet opener skips the adapter entirely when the file's physical schema compares equal to the logical schema and there is no pushed predicate. In that case a file whose struct carries two children with the same field id is read positionally instead of failing with Spark's duplicate field id error.
Spark-written files always carry key-value metadata that arrow-rs folds into the physical schema, so they never compare equal and always reach the adapter. The gap is limited to files with no key-value metadata at all, read with field id matching enabled and no data filter.
Steps to reproduce
- Write a Parquet file without key-value metadata whose schema is
s<x: long id=1, y: long id=1> (parquet-mr with a hand-built MessageType).
- Read it with the native scan,
spark.sql.parquet.fieldId.read.enabled=true, and a requested schema identical to the file schema, with no filter.
- Spark rejects the duplicate id during schema clipping; Comet returns the rows.
Expected behavior
The duplicate field id error, as when a cast or predicate is present.
Additional context
Found in review of #5654, where the once-per-file validation was added. The short circuit is in DataFusion's opener, so the fix is either a Comet-side check before the opener decides, or an upstream option to always run the adapter.
Describe the bug
Comet validates Parquet field id lookups in the physical expression adapter, which resolves each file's mapping once per file (#5654). DataFusion 55's Parquet opener skips the adapter entirely when the file's physical schema compares equal to the logical schema and there is no pushed predicate. In that case a file whose struct carries two children with the same field id is read positionally instead of failing with Spark's duplicate field id error.
Spark-written files always carry key-value metadata that arrow-rs folds into the physical schema, so they never compare equal and always reach the adapter. The gap is limited to files with no key-value metadata at all, read with field id matching enabled and no data filter.
Steps to reproduce
s<x: long id=1, y: long id=1>(parquet-mr with a hand-built MessageType).spark.sql.parquet.fieldId.read.enabled=true, and a requested schema identical to the file schema, with no filter.Expected behavior
The duplicate field id error, as when a cast or predicate is present.
Additional context
Found in review of #5654, where the once-per-file validation was added. The short circuit is in DataFusion's opener, so the fix is either a Comet-side check before the opener decides, or an upstream option to always run the adapter.