Describe the bug
DataFusion's Parquet opener builds the physical expression adapter only when a predicate is pushed or the file schema differs from the requested schema. A file written without key-value metadata whose schema equals the requested schema is therefore read positionally, and none of the native scan's duplicate-name resolution runs for it.
For a nested struct with two sibling names that collide case-insensitively, read under Spark's default spark.sql.caseSensitive=false, Spark's matchCaseInsensitiveField raises foundDuplicateFieldInCaseInsensitiveModeError at read time. The native scan returns rows, with each requested child bound by position.
#6004 covers the case where the requested schema itself repeats a field id. It does not cover duplicate names, and Spark's analysis-time COLUMN_ALREADY_EXISTS check only covers the root columns of the output, not nested struct fields.
Steps to reproduce
Write a Parquet file with parquet-mr and no key-value metadata whose schema is s: struct<x int, X int>, then read it with the same requested schema and spark.sql.caseSensitive=false. Spark raises; the native scan reads.
Expected behavior
The native scan raises the same error as Spark, or falls back to Spark for the file.
Additional context
The same opener behaviour means any nested duplicate-name shape in such a file bypasses the resolver, not only the case-insensitive one. A check at the point where the footer is read, or a planning-time decline when the requested schema holds nested names that collide under the session's case sensitivity, are the two candidate fixes.
Describe the bug
DataFusion's Parquet opener builds the physical expression adapter only when a predicate is pushed or the file schema differs from the requested schema. A file written without key-value metadata whose schema equals the requested schema is therefore read positionally, and none of the native scan's duplicate-name resolution runs for it.
For a nested struct with two sibling names that collide case-insensitively, read under Spark's default
spark.sql.caseSensitive=false, Spark'smatchCaseInsensitiveFieldraisesfoundDuplicateFieldInCaseInsensitiveModeErrorat read time. The native scan returns rows, with each requested child bound by position.#6004 covers the case where the requested schema itself repeats a field id. It does not cover duplicate names, and Spark's analysis-time
COLUMN_ALREADY_EXISTScheck only covers the root columns of the output, not nested struct fields.Steps to reproduce
Write a Parquet file with parquet-mr and no key-value metadata whose schema is
s: struct<x int, X int>, then read it with the same requested schema andspark.sql.caseSensitive=false. Spark raises; the native scan reads.Expected behavior
The native scan raises the same error as Spark, or falls back to Spark for the file.
Additional context
The same opener behaviour means any nested duplicate-name shape in such a file bypasses the resolver, not only the case-insensitive one. A check at the point where the footer is read, or a planning-time decline when the requested schema holds nested names that collide under the session's case sensitivity, are the two candidate fixes.