Repository navigation
GH-3967: Skip the page copy for decompressors with stable output - #3968
Open
NicolaeNMV wants to merge 1 commit into
Open
NicolaeNMV wants to merge 1 commit into
NicolaeNMV wants to merge 1 commit into
Conversation
apacheGH-3150 (apache#3801) copies every decompressed page in ColumnChunkPageReadStore, because the BytesInputDecompressor contract allows a decompressor to reuse its output buffer. On the default heap path that copy is redundant for UNCOMPRESSED (NO_OP returns the page as read) and for ZSTD/LZ4_RAW (HeapBytesDecompressor already decompresses into a fresh array). Add a package-private CodecFactory.BytesDecompressor.returnsStableBytes(), false by default and overridden by NO_OP and by HeapBytesDecompressor for eagerly materialized codecs. The reader skips the copy only for those; DirectCodecFactory and third-party decompressors are still copied.
NicolaeNMV
force-pushed
the
gh3967-skip-stable-page-copy
branch
from
November 17, 2026 21:37
91eebbf to
1ef9965
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
Fixes the performance regression from #3801 described in #3967. On the default heap path the GH-3150 copy is redundant for UNCOMPRESSED and ZSTD/LZ4_RAW, and costs +28.5% / +9.6% page-read time plus one extra allocation per page.
What changes are included in this PR?
CodecFactory.BytesDecompressor.returnsStableBytes(),falseby default and overridden byNO_OPand byHeapBytesDecompressorfor ZSTD/LZ4_RAW.ColumnChunkPageReadStoreskips the copy (data pages V1/V2 and dictionary) only for those decompressors.DirectCodecFactoryand third-party decompressors are still copied, so Reading fails when using DirectCodecFactory #3150 stays fixed without changing theBytesInputDecompressorcontract.Are these changes tested?
TestColumnChunkPageReadStore: stable output is not copied, other output still is, and which decompressors opt in. The first fails on current master.TestParquetReader,TestColumnChunkPageReadStore,TestDirectCodecFactory) pass, and so does its reproduction with the reporter's file. Allocation per scan returns to the 1.18.2 baseline (benchmark numbers in Performance regression from #3801 (unreleased): heap page reads copy every decompressed page #3967).Are there any user-facing changes?
No.
Closes #3967