[weijietong]

Purpose

This PR introduces a customized Parquet footer parser to significantly improve metadata reading performance, especially for wide tables with hundreds to thousands of columns.

Background

When reading Parquet files, Paimon needs to parse the footer metadata serialized with Thrift Compact Protocol. For wide tables (hundreds to thousands of columns), footer parsing becomes a significant performance bottleneck. The official ParquetMetadataConverter materializes every column chunk's metadata including all Statistics, even when only a subset of
columns is needed.

Implementation

The selective footer parser walks the Thrift compact-protocol bytes directly with a hand-written ThriftCompactReader and:

• Selective materialization: Only materializes column chunks whose top-level column is requested. Other column chunks are still scanned (the compact protocol forces a full byte pass) but their bytes are not retained.
• Statistics skipping: Optionally skips Statistics parsing for kept columns (AI training reads never consume footer statistics since stats-based row-group filtering requires a pushdown filter).
• Schema preservation: Always materializes the complete schema (including full LogicalType union) regardless of the column subset.
• Fallback mechanism: Automatically falls back to the official parser when encountering encrypted footers, encrypted columns, or unexpected structures.

Object construction is delegated to ParquetMetadataConverter.fromParquetMetadata() to reuse the battle-tested conversion, ensuring byte-for-byte equivalence with the official reader for the retained subset.

Performance Results

Benchmark on a 232MB file with 1,570 columns and 8 row groups (1.29MB footer):

Requested Columns	Official Parser	Selective Parser	Speedup	Memory Saved
10 (0.6%)	21.11 ms	2.46 ms	8.6x	54%
50 (3.2%)	21.65 ms	2.62 ms	8.3x	53%
100 (6.4%)	21.06 ms	2.90 ms	7.3x	52%
500 (31.8%)	21.58 ms	4.50 ms	4.8x	42%
1000 (63.7%)	20.62 ms	6.26 ms	3.3x	29%
1570 (100%)	20.21 ms	8.47 ms	2.4x	14%

Even in full-parse mode (all columns requested), the selective parser achieves 2.4x speedup due to:

• Hand-written parser vs Thrift-generated code (avoiding reflection and setter overhead)
• Direct byte array access vs unbuffered TIOStreamTransport (avoiding 50,000+ byte[] allocations for Statistics)
• No String.intern() calls (avoiding global String pool synchronization)
• Single-phase parsing (avoiding Thrift tree memory)
• Avoiding ~8 wrapper objects per column chunk

Integration

The selective parser is enabled in ParquetReaderFactory when no pushdown predicates are present (since predicates require Statistics for row-group filtering). It's controlled by two configuration properties:

• paimon.parquet.footer.requested.columns: Comma-separated list of top-level column names to retain
• paimon.parquet.footer.skip.statistics: Whether to skip Statistics parsing (default: true for AI training workloads)

Tests

Added comprehensive test coverage in SelectiveFooterParserTest with 30 test cases:

Equivalence Tests

• Full parse equivalence with official NO_FILTER parse
• Selective subset parse correctness
• Dropped columns do not affect block-level info
• Skip statistics parse mode

Complex Schema Tests

• Deep nesting (3+ levels)
• Map type (key-value structures)
• Array type (list structures)

Boundary Condition Tests

• Empty file (0 rows)
• Single row group
• All columns requested
• Non-existent columns requested

Exception Path Tests

• Malformed footer (truncated)
• Malformed footer (wrong thrift type)
• Fallback signal handling

Encryption Tests

• Encrypted footer fallback verification
• Encrypted column fallback verification

Integration Tests

• ParquetReadOptions integration
• Split requested columns utility
• Many row groups (1000 rows, small row group size)
• Selective parse with nested columns

All 30 tests pass successfully, validating byte-for-byte equivalence with the official parser for the retained subset.