[JingsongLi]

Purpose

Follow-up to #1253, extracted from #1251. Deduplicate lookup compaction currently selects every active file in a bucket. Use incremental sorted-run selection and fetch retained higher-level history through local Lookup Files, so small L0 updates can generate complete changelogs without rewriting the large history run.

Brief change log

Lookup Files store complete KV rows in disposable SST/IPC artifacts and reuse the actual catalog/FileIO local-cache backend, namespace, eviction and capacity. Memory, disk and shared-disk caches are supported; local-cache.dir selects disk storage. Cold reads use the existing schema-evolution decoder, and later executors reuse the generated files. Missing or corrupt blocks rebuild instead of becoming false key misses. Working memory is bounded per executor.

Key-range selection remains effective across schema versions with the same primary-key IDs, order and types, including value changes and PK renames. Valid sorted levels use binary search to select a candidate file. Hot lookups retain one bounded parsed SST index and one decoded IPC batch. Failed or cancelled builds reclaim only their unpublished generation, including in-flight backend writes, without invalidating unrelated shared-cache blocks.

Commit validation uses the existing selected-input consistency and positive-level disjoint-key-range checks, matching Java. It does not require all unselected history files to remain unchanged. Prepared-result recovery continues to validate actual file changes when optional task output-level metadata is absent. This enables incremental lookup for deduplicate without deletion vectors; other merge-engine selection policies remain unchanged.

Tests

Validation covers memory/disk/shared cache reuse and concurrent builds, eviction/corruption and failure recovery, shared download-lock collisions, update/delete/insert payloads and sequence metadata, schema rename/type promotion, safe concurrent history changes, LSM conflicts, and idempotent commits. Integration tests remove the original history object between cold and warm execution to prove the Lookup File is consumed with ordinary data caching disabled.

4,671 library unit tests and 164 related integration tests passed with --test-threads=4; six existing unit tests were ignored. This includes all 64 PK compaction tests, append/Data Evolution compaction, DV writes, incremental/streaming reads and partial-update state. Focused checks also passed all 7 lookup-file tests and 96 cache tests. Regression checks demonstrate one candidate history read instead of ten after schema changes, parsed-index reuse, cleanup after source/head failures and cancellation, and preservation of other shared-cache files, usage accounting and epoch. Temporarily disabling the corresponding fixes makes the new behavior assertions fail.

The first default-concurrency full run had one failure in the existing batch_download_locks_deduplicate_and_skip_busy_stripes test; that test passed in isolation and in the four-thread full rerun. Its implementation was not changed.

cargo fmt --all -- --check, git diff --check, and cargo clippy --locked -p paimon --lib --test pk_compact_test -- -D warnings -A clippy::nonminimal_bool passed. The Clippy allowance covers existing repository findings with the local toolchain.

Behavior was checked against Java LookupLevels, LookupUtils, LookupMergeFunction and LookupChangelogMergeFunctionWrapper at Apache Paimon commit 5be7e2dd1f.

API and Format

Public compaction APIs and table data formats are unchanged. Generated Lookup Files are private, disposable local-cache artifacts. Deduplicate lookup caching accepts lookup.cache-max-memory-size and lookup.cache-spill-compression; its storage uses local-cache.max-size, and lookup.cache-max-disk-size is rejected. No full-compaction API is added.

Documentation

Updated the compaction guide with incremental lookup scope, cache backends and budgets, rebuild behavior and commit conflicts.

[leaves12138]

LGTM