[JingsongLi]

Purpose

Support deletion-vector compaction and lookup changelog production for primary-key tables without repeatedly rewriting large historical runs. A small L0 update now looks up its previous state through local-cache, writes the merged value and changelog, and marks the previous physical row deleted while preserving the higher-level data file.

Brief change log

• Replace whole-bucket selection with Java-style universal incremental run selection and capture the untouched lookup dependencies.
• Generate INSERT, UPDATE_BEFORE, UPDATE_AFTER and DELETE changelogs with the existing merge engines and Java-compatible changelog options; first-row emits inserts for new keys.
• Store generated SST indexes and Arrow IPC value blocks in the existing local-cache backend, sharing its namespace, capacity, CRC validation and LRU across memory, disk and shared disk backends. Rebuild evicted or corrupt artifacts and publish descriptors only after completing their blocks.
• Atomically merge new historical-position DVs, retire rewritten-file vectors, preserve unrelated indexes, and validate lookup dependencies on commit retries. Keep concurrent L0 appends, historical snapshots and idempotent subset commits working.

Tests

268 related tests passed:

• 166 integration tests covering PK compaction, append and Data Evolution compaction, DV merge engines, incremental/streaming changelogs, and partial-update sequence/delete state.
• 102 unit tests covering local-cache and incremental PK planning/lookup, including shared capacity, concurrent misses, cancellation, corruption, reopening and resource limits.
• cargo fmt --all -- --check and git diff --check passed.
• cargo clippy --locked -p paimon --lib --test pk_compact_test -- -D warnings -A clippy::nonminimal_bool passed; the allowance covers the repository's existing lint findings with the local toolchain.

API and Format

DV, lookup-changelog and first-row compact executors require an actual cache attached to the catalog/FileIO (local-cache.enabled=true). local-cache.dir selects disk storage and local-cache.max-size governs the shared quota; lookup.cache-max-disk-size is rejected.

Task/result serialization now carries incremental selection, lookup DV notifications and changelog files. Public APIs expose changelog files and compaction/cache metrics. Table data and DV formats remain unchanged; local Lookup File artifacts use a disposable Rust-private format. Compaction uses incremental run selection exclusively.

Documentation

Updated the compaction guide, local-cache configuration guide and SQL feature documentation.