[wgtmac]

Rationale for this change

Draft reference snapshot — DO NOT REVIEW OR MERGE. Preserve a local exploration of Parquet compact-Thrift parsing for future investigation. This is not a complete feature implementation or a production-ready parser replacement.

The starting point was the Arrow Rust Parquet metadata article. Its reported up-to-9x results are motivation, not a reproduced C++ result.

What changes are included in this PR?

• Custom compact readers for footer metadata, page headers, column/offset indexes and bloom-filter headers.
• A borrowed direct-footer representation, retained buffer ownership, direct schema-tree construction and on-demand compatibility materialization.
• Experimental statistics-selection policies and an encoding-presence mask.
• An additive FastFileMetaData summary API that omits column metadata and schema-tree construction. Its timing is reported separately because it does less work.
• An anonymized experiment record and a standalone synthetic-footer benchmark driver. No source dataset, source identifiers, schema contents or local paths are included.

See THRIFT_CUSTOM_PARSER.md for the design directions, reproducible commands, discarded experiment, uncompleted ideas and known gaps.

On October 23, 2026, seven-sample medians for construction from an already resident footer were:

Workload	Generated API	Direct API, full statistics	Summary-only API
Synthetic: 100,000 String columns, 20 row groups, 170,453,566-byte footer	459.995 ms	285.512 ms (1.61x)	187.604 ms (2.45x)
Workload A: 536,809,024-byte file, 154,661,319-byte footer, 83,925 columns, 8 row groups	260.994 ms	163.663 ms (1.59x)	102.205 ms (2.55x)

Generation/I/O and destruction are excluded from these construction measurements; destruction and statistics-policy results are reported separately in the document. Workload A is a supplemental local observation; public reproduction uses the synthetic generator. These are exploratory measurements, not a controlled attribution of each individual optimization or an end-to-end read benchmark.

Are these changes tested?

Rerun on October 23, 2026, against baseline ca5564e007a209e251b9d4648af4bc6a53c68c4f plus this snapshot:

• Configured Release build: make -j8 passed.
• Parquet internals: 354 passed, including statistics-policy/mask tests and summary count, buffer-lifetime and invalid-buffer checks.
• Parquet reader: 154 passed, 4 skipped.
• Arrow reader/writer: 847 passed, 7 skipped; 1 disabled.
• Schema: 43 passed. File deserialization: 33 passed.
• Standalone benchmark: synthetic footer and anonymous local workload; counts checked across modes.

The configuration does not exercise every encryption/compression path. Passing these checks does not establish complete semantic compatibility. Equality, parser limits, ownership/thread-safety, policy forwarding, serialization/fallback behavior, Meson integration and portability remain incomplete or require further audit. PMR, resumable decoding, mmap/cache, SIMD and full metadata projection were discussed but were not established as completed A/B experiments.

Are there any user-facing changes?

Yes. In this experimental snapshot eligible plaintext footers use the direct path by default, ReaderProperties has experimental policies, and FastFileMetaData exposes summary-only access. PARQUET_USE_GENERATED_THRIFT=1 selects the generated footer baseline for comparison. Existing entry points remain present, but this PR makes no production API/ABI or behavioral-compatibility claim.

Was AI used for this PR?

AI generated the experimental implementation, benchmark driver, tests and documentation under human direction. AI assisted with scope/privacy checks and validation; this is not a completed human correctness review.

PR code and description written by:

• Human
• AI

Reviewed before submission by:

• Human
• AI
• Not reviewed

[github-actions]

Thanks for opening a pull request!

This pull request has been automatically converted to a draft because its title doesn't match Arrow's required format.

If this is not a minor PR, could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

After updating the title, you can mark the pull request as ready for review.

See also:

• Other pull requests
• Contribution Guidelines - Contributing Overview