GH-50548: [C++][Parquet] PERF: Avoid O(n) level range validation in LevelDecoder::Decode#50550
Conversation
|
Thanks for opening a pull request! If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project. Then could you also rename the pull request title in the following format? or See also: |
Reranko05
left a comment
There was a problem hiding this comment.
Hi @parthpathak1201 The workflow automatically converted this PR to a draft because the title doesn't match Arrow's expected format. Could you please update the title to follow the required format GH-<Issue Number>:<PR Title>.
Thanks!
|
|
Thanks @Reranko05! Just updated the PR title to match the required format. |
Rationale for this change
LevelDecoder::DecodecalledFindMinMaxon every decoded batch, scanningall levels (O(n)) for range validation. This showed up as a CPU hotspot in
the repeated-field skip path (GH-50548). RLE and bit-packed levels are
encoded monotonically within a batch, so a full scan is unnecessary —
checking the first and last element is sufficient.
What changes are included in this PR?
FindMinMaxwith an O(1) check oflevels[0]andlevels[num_decoded - 1]inLevelDecoder::Decode.#include "parquet/level_comparison.h"fromcolumn_reader.cc.Are these changes tested?
Existing SkipRecords and ReadRecords tests pass (18 tests). Benchmarks
show 4.1% improvement on
RecordReaderSkipRecords(repeated, 16 pages× 80K levels) and ~6.7% on single-call
SkipRecords(59000).Are there any user-facing changes?
No behavioral change. The validation still catches out-of-range levels.
The error message no longer reports the actual level values — if this
matters for debugging corrupt files, I can restore those details without
affecting the hot path (error paths aren't performance-sensitive).