Skip to content

Duplicate field ids inside a struct are not validated when the file schema equals the requested schema and no predicate is pushed #5801

Description

@dwsmith1983

Describe the bug

Comet validates Parquet field id lookups in the physical expression adapter, which resolves each file's mapping once per file (#5654). DataFusion 55's Parquet opener skips the adapter entirely when the file's physical schema compares equal to the logical schema and there is no pushed predicate. In that case a file whose struct carries two children with the same field id is read positionally instead of failing with Spark's duplicate field id error.

Spark-written files always carry key-value metadata that arrow-rs folds into the physical schema, so they never compare equal and always reach the adapter. The gap is limited to files with no key-value metadata at all, read with field id matching enabled and no data filter.

Steps to reproduce

  1. Write a Parquet file without key-value metadata whose schema is s<x: long id=1, y: long id=1> (parquet-mr with a hand-built MessageType).
  2. Read it with the native scan, spark.sql.parquet.fieldId.read.enabled=true, and a requested schema identical to the file schema, with no filter.
  3. Spark rejects the duplicate id during schema clipping; Comet returns the rows.

Expected behavior

The duplicate field id error, as when a cast or predicate is present.

Additional context

Found in review of #5654, where the once-per-file validation was added. The short circuit is in DataFusion's opener, so the fix is either a Comet-side check before the opener decides, or an upstream option to always run the adapter.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions