HIVE-30059: LLAP IO Cache: native parquet data cache by abstractdog · Pull Request #6793 · apache/hive

abstractdog · 2026-09-16T13:49:58Z

What changes were proposed in this pull request?

Native LLAP encoded-data path for Parquet, mirroring the existing ORC pipeline:

ParquetColumnVectorProducer → ParquetEncodedDataReader → ParquetEncodedDataConsumer, feeding VectorizedColumnReaders through a shared ParquetRowGroupDecoder (extracted from VectorizedParquetRecordReader).
Column chunks are held natively as MemoryBuffer[] in ParquetEncodedColumnBatch; ParquetCachedPageReadStore parses pages in place over those cached buffers with lazy per-page decompression — no ParquetFileReader, no copy from cache into a Parquet read buffer.
Enabled by default via hive.llap.io.parquet.native.enable=true.

Why are the changes needed?

Parquet under LLAP already caches byte-ranges via LlapCacheAwareFs, but reads still go through a stock ParquetFileReader over a virtual llapcache:// URL: cached bytes get copied into the reader's own buffers, page parsing and decompression run per-read, and the path isn't wired into LLAP's encoded-column pipeline or fragment counters. This PR brings Parquet to parity with ORC's native LLAP path: zero-copy page parsing over cached buffers, lazy decompression, full LLAP IO counter integration (METADATA_CACHE_HIT/MISS, SELECTED_ROWGROUPS, HDFS/IO time), and the shared ColumnVectorProducer/EncodedDataConsumer execution model.

Does this PR introduce any user-facing change?

Better Parquet read performance under LLAP (zero-copy page parsing over cached buffers, lazy decompression). Query summaries now report DATA_HIT / METADATA_CACHE_HIT (and their _MISS counterparts) for Parquet, so cache effectiveness is visible the same way it is for ORC. Opt-out via hive.llap.io.parquet.native.enable=false.

How was this patch tested?

TestParquetEncodedDataReader (27 tests) — plan/fetch/assemble, cache hit/miss, footer caching, encryption fallback, schema evolution.
TestParquetCacheLayout, TestParquetRangeBuffers (8 tests) — page-range math and buffer reuse.
Existing TestVectorizedParquetRecordReader unchanged (shared ParquetRowGroupDecoder).
Manual verification through miniHS2 with LLAP.

deniskuzZ · 2026-09-16T19:36:04Z

   * Vectorized reads of parquet files from columns with list or map type is only supported if the nested types are of
   * primitive type category
-   * check {@link VectorizedParquetRecordReader#checkListColumnSupport} for details on nested types under lists
+   * check {@link org.apache.hadoop.hive.ql.io.parquet.vector.ParquetRowGroupDecoder#checkListColumnSupport} for


do we need full name with package here?

deniskuzZ · 2026-09-16T19:38:51Z

        serdeCache, bufferManagerGeneric, conf, cacheMetrics, ioMetrics, tracePool, encodeExecutor) : null;
+    // Native Parquet IO is gated per query by the job conf at the dispatch sites.
+    this.parquetCvp = dataCache != null
+        ? new ParquetColumnVectorProducer(dataCache, bufferManagerOrc, conf, cacheMetrics, ioMetrics)


Should we drop orc from bufferManagerOrc ?

deniskuzZ · 2026-09-16T19:41:33Z

  private boolean checkOrcSchemaEvolution() {
    SchemaEvolution evolution = rp.getSchemaEvolution();
+    if (evolution == null) {
+      // No ORC-style schema evolution to validate (e.g. native parquet path);


why we check non-ORC schema inside checkOrcSchemaEvolution ?

deniskuzZ · 2026-09-16T19:43:01Z

+
+  /** Slices of the cached buffers covering exactly the chunk's byte region, in file order. */
+  private static List<ByteBuffer> chunkBuffers(ParquetEncodedColumnBatch batch, int pc) {
+    long start = batch.chunks[pc].getStartingPos(), end = start + batch.chunks[pc].getTotalSize();


please move env var on a new line

deniskuzZ · 2026-09-16T19:43:47Z

+    List<ByteBuffer> slices = new ArrayList<>(batch.columnBuffers[pc].length);
+    for (int i = 0; i < batch.columnBuffers[pc].length; ++i) {
+      long offset = batch.bufferOffsets[pc][i];
+      long from = Math.max(start, offset), to = Math.min(end, offset + batch.bufferLengths[pc][i]);


same, new line for to, maybe need better naming

deniskuzZ · 2026-09-16T19:47:09Z

+    super(consumer, includes.getPhysicalColumnIds().size(), ioMetrics, counters);
+    this.jobConf = jobConf;
+    this.useDecimal64ColumnVectors = HiveConf.getVar(jobConf,
+        ConfVars.HIVE_VECTORIZED_INPUT_FORMAT_SUPPORTS_ENABLED).equalsIgnoreCase("decimal_64");


do we have constant for "decimal_64" ?

Copilot

🟡 Changes recommended

Unresolved critical and moderate findings affect schema projection, cache ownership, row lineage, configuration, and fallback behavior.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds opt-in native Parquet LLAP caching with vectored reads, cached-page decoding, fallback integration, and supporting tests.

Changes:

Adds native Parquet LLAP configuration, APIs, and dispatch.
Implements cache layouts, range buffers, vectored reads, and page decoding.
Refactors shared Parquet reader logic and expands test coverage.

File summaries

File	Reviewed changes and final comments
`ql/src/test/org/apache/hadoop/hive/llap/TestParquetRangeBuffers.java`	Tests range-buffer reuse and slicing; no final comments.
`ql/src/test/org/apache/hadoop/hive/llap/TestParquetCacheLayout.java`	Tests cache layout behavior; no final comments.
`ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java`	Enables native Parquet LLAP eligibility; no final comments.
`ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java`	Critical, 1 vote: Native reads omit row-lineage columns, producing null/default lineage values.
`ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetRowGroupDecoder.java`	Shares row-group decoder construction; no final comments.
`ql/src/java/org/apache/hadoop/hive/ql/io/parquet/ParquetRecordReaderBase.java`	Reviewed as part of Parquet reader integration; no final comments.
`ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java`	Gates native Parquet wrapping; no final comments.
`ql/src/java/org/apache/hadoop/hive/llap/ParquetRangeBuffers.java`	Manages vectored-read buffers; no final comments.
`ql/src/java/org/apache/hadoop/hive/llap/ParquetCacheLayout.java`	Moderate, 1 vote: Native layout can create fallback-incompatible cache gaps and trigger an assertion instead of refetching.
`ql/src/java/org/apache/hadoop/hive/llap/LlapCacheAwareFs.java`	Supports cache-aware fallback reads; no final comments.
`llap-server/src/test/org/apache/hadoop/hive/llap/io/encoded/TestParquetEncodedDataReader.java`	Tests native reads and caching; no final comments.
`llap-server/src/test/org/apache/hadoop/hive/llap/io/api/impl/TestLlapRecordReader.java`	Tests LLAP reader behavior; no final comments.
`llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/ParquetEncodedDataReader.java`	Critical, 2 votes: Top-level field indexes are used against flattened leaf columns, misselecting nested projections. Critical, 1 vote: Partial cache insertion failure can cause cache-owned buffers to be deallocated. Nit, 1 vote: `count(*)` fetches and caches unnecessary column chunks.
`llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/ParquetEncodedColumnBatch.java`	Carries cached row-group data; no final comments.
`llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ParquetEncodedDataConsumer.java`	Moderate, 1 vote: Decode exceptions are swallowed, allowing later I/O and resource use after failure.
`llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ParquetColumnVectorProducer.java`	Creates native Parquet pipelines; no final comments.
`llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ParquetCachedPageReadStore.java`	Reads cached Parquet pages; no final comments.
`llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java`	Supports producer fallback; no final comments.
`llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java`	Moderate, 1 vote: Discards supplied `fileKey` and `tag`. Moderate, 1 vote: Can create the native producer without a metadata cache in memory mode `none`. Moderate, 1 vote: Does not honor the per-query native Parquet disable flag.
`llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java`	Integrates LLAP input handling; no final comments.
`llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIo.java`	Extends the Parquet LLAP API; no final comments.
`iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/vector/HiveVectorizedReader.java`	Updates footer-cache API usage; no final comments.
`iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java`	Reviewed for storage-handler integration; no final comments.
`common/src/java/org/apache/hadoop/hive/conf/HiveConf.java`	Adds native Parquet configuration; no final comments.

Review details

Suppressed comments (6)

llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java:503

This constructs a plain FileSplit and silently discards the fileKey and tag supplied by the public API. The native reader then derives a different key from the job configuration/filesystem in loadFooter() and computes its own tag, so callers that provide a stable file identity cannot read or populate the intended data-cache entry or preserve the administrative cache tag. Propagate these arguments through the native reader, or remove them from the API instead of ignoring them.

    FileSplit split = new FileSplit(path, offset, length, (String[]) null);
    try {
      LlapRecordReader rr = LlapRecordReader.create(conf, split, tableIncludedCols, HiveStringUtils.getHostname(),

llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java:285

dataCache is non-null in LLAP_IO_MEMORY_MODE=none because LlapIoImpl assigns a SimpleBufferManager to it while leaving fileMetadataCache null. With the native flag enabled, this condition still creates parquetCvp; ParquetEncodedDataReader.loadFooter then calls getParquetFooterBuffersFromCache, whose first operation requires a non-null metadata cache, so the LLAP input format fails instead of falling back to the normal Parquet reader. Gate this producer on the metadata cache as well (or make the native reader bypass the footer cache when it is unavailable).

    this.parquetCvp = dataCache != null
        ? new ParquetColumnVectorProducer(dataCache, bufferManagerOrc, conf, cacheMetrics, ioMetrics)
        : null;

llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java:500

The API contract says this method returns null when native Parquet IO is disabled, but parquetCvp is constructed independently of the per-query flag and this check only tests for null. A caller can therefore set hive.llap.io.parquet.native.enabled=false and still start the native reader, bypassing the configuration gate used by HiveInputFormat. Include ConfVars.LLAP_IO_PARQUET_NATIVE_ENABLED in this guard.

    if (parquetCvp == null) {
      return null;
    }

llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ParquetEncodedDataConsumer.java:195

These exceptions are reported but swallowed, so EncodedDataConsumer.consumeData returns normally and ParquetEncodedDataReader.performDataRead continues fetching and decoding later row groups before calling setDone. A corrupt page in an early group can therefore keep doing I/O and holding cache resources after the downstream reader has failed. Propagate the decode failure after recording it (or let consumeData perform the error notification) so the reader's abort path stops the split.

    } catch (IOException | RuntimeException e) {
      // parquet-mr reports decode failures as runtime ParquetDecodingException.
      LlapIoImpl.LOG.error("Parquet decodeBatch failed for rowGroup " + batch.rowGroupIx + " of " + path, e);
      downstreamConsumer.setError(e);
    } finally {

llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/ParquetEncodedDataReader.java:197

For the count(*) projection, colsToInclude can be empty while requestedSchema still contains the table fields; the shared decoder explicitly documents that all readers are null in this case. This code nevertheless derives every requested field as a projected leaf, so the native path fetches and caches every column chunk even though it only needs each row group's row count. Short-circuit the projected-leaf list when includes.getPhysicalColumnIds() is empty.

    int[] projected = projectedLeaves(requestedSchema, fileSchema);

ql/src/java/org/apache/hadoop/hive/llap/ParquetCacheLayout.java:88

This native layout is incompatible with the existing Parquet fallback cache writer. LlapCacheAwareFs.getAndValidateMissingChunks requires every missing gap to start and end on maxAlloc boundaries, while this method can split a 5 MiB chunk into 4 MiB + 1 MiB. If the 1 MiB native buffer is evicted and a later nested-type/fallback reader requests the chunk, the fallback sees a gap starting at chunkStart + 4 MiB and throws an AssertionError instead of refetching it. Both paths need to share the same layout, or the fallback must accept arbitrary cache gaps.

  public int[] bufferSizes(long length) {
    int count = 0;
    for (long left = length; left > 0; ++count) {
      left -= left < minBuffer ? left : Math.min(maxBuffer, Long.highestOneBit(left));
    }

Files reviewed: 24/24 changed files
Comments generated: 3
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Three Critical findings on the native Parquet cache path: - projectedLeaves used fileSchema.getFieldIndex to pick column chunks, but BlockMetaData.getColumns() is in flat leaf order. A file schema like 'group nested {a,b}, x' projecting 'x' would hit nested.b instead of x. Map each requested top-level primitive to its single-segment leaf in fileSchema.getColumns() instead. Add a regression test. - putColumn set part.owned only after processing all missing ranges for the column, so a mid-run throw from putFileData (e.g. its length-mismatch guard) left already-inserted cache buffers looking like raw allocations and finishFetch would allocator.deallocate cache-owned memory. Insert one range at a time and flip ownership per part. - ParquetEncodedDataReader.loadFooter builds requestedSchema without the row-lineage columns the fallback reader adds via RowLineageUtils.getRequestedSchemaWithRowLineageColumns, so with row lineage on the native path would silently emit nulls for ROW__LINEAGE__ID / LAST__UPDATED__SEQUENCE__NUMBER. Detect that case in the producer and fall back, matching the nested-projection fallback. Co-Authored-By: Claude Code <noreply@anthropic.com>

@VisibleForTesting

Fix the mechanical issues flagged on files this PR modifies. Skips the 15 findings inside verbatim extractions from VectorizedParquetRecordReader (kept behavior-preserving) and the switch/case indentation reports. Notable non-trivial fixes: - LlapIo.llapVectorizedParquetReaderForPath: 9 params -> 3, collected into a new LlapParquetReadRequest record. - LlapInputFormat.getRecordReader: extract wrapOrFallback() to bring cognitive complexity under threshold. - ParquetEncodedDataReader: extract planMissRun() from planColumnChunk; drop unused 'includes' field/param, unused maxAlloc param, and Part.miss; narrow catch(Throwable) -> catch(Exception) at four sites and preserve InterruptedException on the thread instead of masking it. - ParquetCacheLayout / ParquetEncodedDataReader: replace loop-counter mutation with while loops. - CacheChunk: drop stale @VisibleForTesting; it is part of the cache's public read surface for both ORC and Parquet. - checkstyle/suppressions.xml: suppress VisibilityModifier for ParquetEncodedColumnBatch (matches parent EncodedColumnBatch pattern). - Delete stale checkListColumnSupport from VectorizedParquetRecordReader (moved to ParquetRowGroupDecoder). Style-only elsewhere: pattern-instanceof, split multi-decls, empty-body 'why' comments, header rewraps, paren-pad, restricted-identifier rename (record() -> recordRanges()). Verified: mvn checkstyle:check on llap-server (0 violations) and the existing test suites: TestParquetEncodedDataReader (27/27), TestLlapRecordReader (2/2), TestParquetRangeBuffers (2/2). Co-Authored-By: Claude Code <noreply@anthropic.com>

aturoczy · 2026-09-16T15:04:20Z

    LLAP_IO_ENCODE_ENABLED("hive.llap.io.encode.enabled", true,
        "Whether LLAP should try to re-encode and cache data for non-ORC formats. This is used\n" +
        "on LLAP Server side to determine if the infrastructure for that is initialized."),
+    LLAP_IO_PARQUET_NATIVE_ENABLED("hive.llap.io.parquet.native.enabled", false,


Wouldn't be this enabled by default?

it could be, tpcds 10TB testing is already done, we might be confident enough to make it true

let's keep false for now, we need to retest after merge

ack, feature is going to be present in the code anyway

aturoczy · 2026-09-16T15:06:18Z

  private boolean checkOrcSchemaEvolution() {
    SchemaEvolution evolution = rp.getSchemaEvolution();
+    if (evolution == null) {
+      // No ORC-style schema evolution to validate (e.g. native parquet path);


no orc-style?

you might be looking at an old commit, the current state of PR already has an improved comment:
https://github.com/apache/hive/pull/6793/changes#diff-f00087a16c2c9fed9e0899d7a3718edad0215e33cc41fdd38895945ba45f2517R357-R361

// Only the ORC pipeline hangs an ORC SchemaEvolution off the ReadPipeline; other formats // (native Parquet today) handle column resolution themselves, so there is nothing to // validate here.

…LLAP cache Whoever puts a Parquet column chunk in the LLAP cache decides how the file is cut into cacheable pieces. Today that is the cache-aware stream under the vectorized reader; the native reader added later does the same. If the two cut a file differently, a chunk cached by one is not reusable by the other and the same bytes end up held twice under different keys. Define the layout once, before either uses it: the piece sizes, the largest range a vectored read may ask for, and the pooled buffers those ranges are read into. Pieces are powers of two, largest first, so each fills its buddy allocation exactly. The cache-aware stream cached a chunk as one buffer and left the allocator to round it up, so a 5 Mb chunk took 8 Mb; over a mix of chunk sizes that lost about half the cache. Any contiguous run of pieces decomposes into itself, so a gap left by eviction re-caches on the same boundaries. The range bound is the larger of 8 Mb -- where reads against S3 stop getting faster, and where Trino also splits -- and the allocator's maximum, since a cache buffer is always read whole. The buffer pool is per stream: the pool never evicts, so one shared across the daemon would pin every executor's peak on the heap for the daemon's life.

… its own class VectorizedParquetRecordReader built its VectorizedColumnReader array inline: walk the requested schema, match each Hive type to a Parquet column, apply the column defaults a schema-evolved file needs, and decide which nested shapes are readable at all. That logic depends only on the schemas and the Hive types, not on how the pages were obtained, but it could not be called from anywhere else. Move it to ParquetRowGroupDecoder unchanged, so a reader that gets its pages from somewhere other than a file -- an LLAP cache-backed consumer, next -- builds the same readers from the same rules rather than growing a second copy that drifts. No behaviour change: the reader now delegates to the new class and the qtests are untouched. checkListColumnSupport moves with it, so the Iceberg storage handler's reference to it follows.

Parquet is cache-only in LLAP IO: the file bytes are cached, but every task re-runs the vectorized Parquet reader over them. ORC instead caches the column chunks it decoded and hands the consumer a column-vector batch. This adds the same three pieces for Parquet -- an encoded data reader, an encoded data consumer and a column vector producer -- so a Parquet scan is served from the cache in the same shape as ORC. The reader reads the footer, prunes row groups against the search argument and fetches only the projected column chunks, filling the cache with what it read. The consumer decodes a row group from a cache-backed InputFile, so a second reader of the same chunk decodes from memory and never touches the filesystem. Rather than a second set of readers, as ORC needed, the cached pages are served through parquet's own PageReadStore, so both paths run the same decode: the row-group logic moves out of VectorizedParquetRecordReader into ParquetRowGroupDecoder unchanged, and both callers use it. Off by default behind hive.llap.io.parquet.native.enabled. A projection that reaches into a nested type falls back to the vectorized Parquet reader, as does any error in the native path.

… summary (META_HIT, META_MISS) The summary's META_HIT / META_MISS columns are wired to METADATA_CACHE_HIT and METADATA_CACHE_MISS, which OrcEncodedDataReader bumps around its file tail lookups. The Parquet path uses the same FileMetadataCache (LlapIoImpl. getParquetFooterBuffersFromCache calls getFileMetadata / putFileMetadata on it) but never bumped the counters, so a Parquet-only run showed zeroes in those columns. Bump them from the same site. LlapIoImpl cannot see QueryFragmentCounters (it lives in llap-server; LlapIo in llap-client), so the impl reports the outcome through a nullable BooleanRef out-parameter -- the same signalling pattern storage-api already uses -- and ParquetEncodedDataReader.loadFooter increments the counter it already holds. The two non-native callers (VectorizedParquetRecordReader on the vectorized fallback, HiveVectorizedReader on the Iceberg path) do not have per-fragment counters plumbed and pass null; their footer traffic continues to go unaccounted, as it did before. TestParquetEncodedDataReader reads the file twice and asserts the second read reports one hit and zero misses; only the second read is asserted because the footer cache is a @BeforeClass singleton and any earlier test in the class may have populated it. Co-Authored-By: Claude Code <noreply@anthropic.com>

…ducer

Verifies the descriptors and time counters wired into ParquetEncodedDataReader in the previous commit: SELECTED_ROWGROUPS, TOTAL_IO_TIME_NS, HDFS_TIME_NS, METADATA_CACHE_HIT/MISS, plus FILE and STRIPES descriptors visible via the QueryFragmentCounters summary string. Co-Authored-By: Claude Code <noreply@anthropic.com>

- HiveIcebergStorageHandler: import ParquetRowGroupDecoder so the javadoc {@link} can use the short name. - LlapIoImpl: rename local bufferManagerOrc -> bufferManagerData; the buffer manager is now shared with the native Parquet producer, so the Orc suffix is stale. - LlapRecordReader: rename checkOrcSchemaEvolution -> checkSchemaEvolution since the method already tolerates producers (like native Parquet) that do not expose an ORC-style SchemaEvolution. - ParquetCachedPageReadStore.chunkBuffers: split the two-per-line locals, and rename from/to -> sliceStart/sliceEnd (with chunkStart/chunkEnd and bufferStart/bufferEnd for the surrounding bounds). Co-Authored-By: Claude Code <noreply@anthropic.com>

Three Critical findings on the native Parquet cache path: - projectedLeaves used fileSchema.getFieldIndex to pick column chunks, but BlockMetaData.getColumns() is in flat leaf order. A file schema like 'group nested {a,b}, x' projecting 'x' would hit nested.b instead of x. Map each requested top-level primitive to its single-segment leaf in fileSchema.getColumns() instead. Add a regression test. - putColumn set part.owned only after processing all missing ranges for the column, so a mid-run throw from putFileData (e.g. its length-mismatch guard) left already-inserted cache buffers looking like raw allocations and finishFetch would allocator.deallocate cache-owned memory. Insert one range at a time and flip ownership per part. - ParquetEncodedDataReader.loadFooter builds requestedSchema without the row-lineage columns the fallback reader adds via RowLineageUtils.getRequestedSchemaWithRowLineageColumns, so with row lineage on the native path would silently emit nulls for ROW__LINEAGE__ID / LAST__UPDATED__SEQUENCE__NUMBER. Detect that case in the producer and fall back, matching the nested-projection fallback. Co-Authored-By: Claude Code <noreply@anthropic.com>

@VisibleForTesting

Fix the mechanical issues flagged on files this PR modifies. Skips the 15 findings inside verbatim extractions from VectorizedParquetRecordReader (kept behavior-preserving) and the switch/case indentation reports. Notable non-trivial fixes: - LlapIo.llapVectorizedParquetReaderForPath: 9 params -> 3, collected into a new LlapParquetReadRequest record. - LlapInputFormat.getRecordReader: extract wrapOrFallback() to bring cognitive complexity under threshold. - ParquetEncodedDataReader: extract planMissRun() from planColumnChunk; drop unused 'includes' field/param, unused maxAlloc param, and Part.miss; narrow catch(Throwable) -> catch(Exception) at four sites and preserve InterruptedException on the thread instead of masking it. - ParquetCacheLayout / ParquetEncodedDataReader: replace loop-counter mutation with while loops. - CacheChunk: drop stale @VisibleForTesting; it is part of the cache's public read surface for both ORC and Parquet. - checkstyle/suppressions.xml: suppress VisibilityModifier for ParquetEncodedColumnBatch (matches parent EncodedColumnBatch pattern). - Delete stale checkListColumnSupport from VectorizedParquetRecordReader (moved to ParquetRowGroupDecoder). Style-only elsewhere: pattern-instanceof, split multi-decls, empty-body 'why' comments, header rewraps, paren-pad, restricted-identifier rename (record() -> recordRanges()). Verified: mvn checkstyle:check on llap-server (0 violations) and the existing test suites: TestParquetEncodedDataReader (27/27), TestLlapRecordReader (2/2), TestParquetRangeBuffers (2/2). Co-Authored-By: Claude Code <noreply@anthropic.com>

- ORC dot-continuation lines: fix 4-space continuation indent - LlapIoImpl: split combined declarations, remove chained assignments - ParquetEncodedDataConsumer: NOSONAR on try line for S2093 - ParquetEncodedColumnBatch: make fields private, add accessors - ParquetEncodedDataReader / TestParquetEncodedDataReader: fix '{ '-after-brace whitespace, use accessors - ParquetCachedPageReadStore: use accessors - ParquetRowGroupDecoder: reduce parameter counts via TimestampConversionOptions record; replace generic RuntimeException with UnsupportedOperationException / IllegalStateException / InvalidSchemaException; pattern-match instanceof; rewrite commented-out MAP schema example; merge case labels; retire the moved TODO - TestParquetRangeBuffers: drop unused throws IOException Co-Authored-By: Claude Code <noreply@anthropic.com>

sonarqubecloud · 2026-09-18T13:54:23Z

Quality Gate passed

Issues
28 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.2% Duplication on New Code

See analysis details on SonarQube Cloud

asf-ci-hive added the tests pending label Sep 16, 2026

abstractdog changed the title ~~Parquet io cache~~ HIVE-30059: LLAP IO Cache: native parquet data cache Sep 16, 2026

deniskuzZ reviewed Sep 16, 2026

View reviewed changes

deniskuzZ requested a lite review from Copilot September 16, 2026 19:56

Copilot started reviewing on behalf of deniskuzZ September 16, 2026 19:56 View session

Copilot AI reviewed Sep 16, 2026

View reviewed changes

asf-ci-hive added tests unstable and removed tests pending labels Sep 16, 2026

abstractdog force-pushed the parquet-io-cache branch from 66f34dc to 65352ad Compare September 17, 2026 11:02

asf-ci-hive added tests pending tests passed and removed tests unstable tests pending labels Sep 17, 2026

aturoczy reviewed Sep 17, 2026

View reviewed changes

deniskuzZ and others added 9 commits September 18, 2026 09:15

unused import

471599f

clarifying comment on LlapCacheAwareFs

e367a84

cacheMetrics.incrCacheReadRequests for parity with OrcColumnVectorPro…

dafb754

…ducer

createReadPipeline doesn't throw exception

185fe73

final metrics fields in OrcColumnVectorProducer

a1b4eca

abstractdog and others added 7 commits September 18, 2026 09:15

QueryFragmentCounters parity with ORC

6b1f41c

HIVE_VECTORIZED_INPUT_FORMAT_SUPPORTS_DECIMAL_64

59ae554

hive.llap.io.parquet.native.enable=true default

2ac8454

abstractdog force-pushed the parquet-io-cache branch from 65352ad to 2ac8454 Compare September 18, 2026 07:19

abstractdog requested a review from deniskuzZ September 18, 2026 07:19

asf-ci-hive added tests pending and removed tests passed labels Sep 18, 2026

asf-ci-hive added tests unstable tests pending and removed tests pending tests unstable labels Sep 18, 2026

asf-ci-hive added tests unstable and removed tests pending labels Sep 18, 2026

Conversation

abstractdog commented Sep 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Suppressed comments (6)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud Bot commented Sep 18, 2026

Quality Gate passed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

abstractdog commented Sep 16, 2026 •

edited

Loading