HIVE-30053: Support multi-threaded CachedStore prewarm

[neatHyperTxt-meesho]

What changes were proposed in this pull request?

Adds metastore.cached.rawstore.prewarm.threads (default 1 = current single-threaded behavior). With N > 1, prewarm fans the per-table work (table + partitions + statistics + constraints) out to N worker threads, each with its own RawStore instance — mirroring how CacheUpdateMasterWork creates its private store. Workers drain the existing TablesPendingPrewarm stack, so each table is handed out exactly once and the prioritizeTableForPrewarm hot-table promotion keeps working. The memory-full early stop uses a shared flag with completePrewarm called exactly once, and the worker pool plus worker stores are torn down when prewarm finishes. The per-table logic is extracted verbatim into prewarmTable(); the single-threaded path runs through the same code.

Why are the changes needed?

Prewarm is single-threaded and latency-bound: one thread issues ~6-8 sequential queries per table. On our production deployment (Hive 4.2.0, 21k cached tables) prewarm takes ~10 minutes; with 8 threads it takes 195s vs 602s baseline (3.1x). Each worker uses one connection from the shared pool, so the config doc advises keeping the value below the pool size.

Does this PR introduce any user-facing change?

A new optional config, metastore.cached.rawstore.prewarm.threads, default 1 (no behavior change unless set).

How was this patch tested?

New unit test TestCachedStore#testPrewarmMultiThreaded runs a full prewarm with 4 threads against Derby and asserts every database, table and partition is cached. Full TestCachedStore suite passes (32/32). Benchmarked on a production deployment: 602s (1 thread) -> 195s (8 threads) for 21k tables.

[deniskuzZ]

cc @dengzhhu653, @saihemanth-cloudera

[Copilot]

🟡 Changes recommended

Parallel prewarm can lose unfinished work, mishandle event updates, and race during shared cache size-estimator initialization.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds configurable multi-threaded CachedStore prewarming for faster metastore startup while preserving single-threaded default behavior.

Changes:

• Adds configurable worker threads with dedicated RawStore instances.
• Refactors table prewarm logic and worker cleanup.
• Adds configuration and regression tests.

File summaries

File	Description
CachedStore.java	Implements parallel prewarming and synchronization changes.
TestCachedStore.java	Adds multi-threaded and monitor regression tests.
MetastoreConf.java	Defines the prewarm thread configuration.

Review details

Suppressed comments (1)

standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java:582

• Every expected table-level MetaException/NoSuchObjectException is handled inside prewarmTable, so an ExecutionException here represents an unchecked worker failure. Continuing leaves the remaining names on tblsPendingPrewarm unprocessed, but the outer loop then calls completePrewarm(..., true), advertising a complete cache. Propagate the failure or mark the run incomplete and retry rather than moving on.

              } catch (ExecutionException e) {
                LOG.warn("Prewarm worker failed for database {}, moving on", dbName, e);
              }

• Files reviewed: 3/3 changed files
• Comments generated: 3
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[sonarqubecloud]

Quality Gate passed

Issues
5 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[neatHyperTxt-meesho]

@dengzhhu653 @deniskuzZ @saihemanth-cloudera would appreciate a review when you get a chance

[dengzhhu653]

Can you separate the prewarm from this class please, and create an interface to warm up the cache?
An interface for example:

interface MetaCachePreWarm extends AutoClosable, Configurable {
   void initialize() throws MetaException;
   void preWarm() throws MetaException;
....
}

[neatHyperTxt-meesho]

Done, prewarm now lives behind a MetaCachePreWarm interface @dengzhhu653