Skip to content

Keep files readable when they use future logical types#3669

Open
kevinjqliu wants to merge 1 commit into
apache:masterfrom
kevinjqliu:kevinjqliu/unknown-logical-type-forward-compatible
Open

Keep files readable when they use future logical types#3669
kevinjqliu wants to merge 1 commit into
apache:masterfrom
kevinjqliu:kevinjqliu/unknown-logical-type-forward-compatible

Conversation

@kevinjqliu

@kevinjqliu kevinjqliu commented Jul 16, 2026

Copy link
Copy Markdown

This is part of the effort to ensure that new logical types are forward compatible (See apache/parquet-format#599)

Description

This makes parquet-java more tolerant of Parquet files written by newer implementations.

A newer writer may use a logical type that this version of parquet-java does not know about yet. The file can still have a perfectly valid physical column type, but today the reader fails while trying to interpret the unknown logical-type metadata.

The fix is to handle that case in ParquetMetadataConverter, where file metadata is translated into parquet-java schema annotations. If the logical type cannot be understood, the converter now leaves the column as its physical type instead of failing. That is the right behavior for forward compatibility: readers should preserve the data they can understand, and only apply logical annotations when they are known.

This also keeps the existing fallback behavior intact for files that include older converted_type metadata.

Tests

Added tests cover reading an unknown logical type both with and without a converted_type fallback.

import org.junit.Test;

public class TestInterOpReadUnknownLogicalType {
private static final String REFERENCE_FILE = "unknown-logical-type.parquet";

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a test using the parquet-testing artifact

unknown-logical-type.parquet
A file containing a column annotated with a LogicalType whose identifier has been set to an abitrary high value to check the behaviour of an old reader reading a file written by a new writer containing an unsupported type (see related issue).

-- from https://github.com/apache/parquet-testing/blob/master/data/README.md

ref: https://github.com/apache/parquet-testing/blob/master/data/unknown-logical-type.parquet

@kevinjqliu
kevinjqliu force-pushed the kevinjqliu/unknown-logical-type-forward-compatible branch from 4aa0e2e to f9fba56 Compare July 16, 2026 19:14
@kevinjqliu
kevinjqliu force-pushed the kevinjqliu/unknown-logical-type-forward-compatible branch from f9fba56 to 33d2f58 Compare July 16, 2026 19:44
@kevinjqliu
kevinjqliu marked this pull request as ready for review July 17, 2026 03:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant