[JingsongLi]

Purpose

Follow-up to #1254, extracted from #1251. Enable incremental primary-key compaction for deletion-vectors.enabled=true, merge-engine=deduplicate, and changelog producers none, input and lookup. Small L0 writes can become visible to ordinary DV scans while large historical runs remain in place.

Brief change log

• Extend the existing local Lookup File cache with absolute physical row positions. Cold loading reuses KV schema evolution without merging or filtering physical rows; immutable cached bytes stay independent of snapshot DVs. Apply captured masks to selected inputs and lookup hits, then record physical deletions when historical state moves into the output.
• Reuse incremental run selection and the actual catalog/FileIO local-cache for all three producers. Calculate lookup changelogs before dropping retracts from DV data output; DELETE-only compaction can emit no data file. Bound data-file rows to the existing Rust bitmap position range.
• Publish data changes, historical DV additions and retired input ranges atomically. Validate selected masks and actual lookup-hit targets, union with the latest bitmaps on each CAS attempt, preserve unrelated ranges sharing a container, and retain HASH/other-bucket indexes. Read each index manifest once and decode/write one bucket at a time.
• Share DV index path handling and bucket reads/writes through the internal deletion_vector_index module. Primary-key and Data Evolution compaction also share commit-time manifest reading, per-bucket container preparation, index replacement and generated-file cleanup. Keep lookup-position union and stable row-ID remapping as the respective mask transformations.
• Keep commit validation based on captured snapshots and file changes. Prepared results carry physical deletion metadata; missing metadata fails first publication while already-committed results can still recover idempotently. Changelog output levels are validated from captured selection even when task output-level metadata and data outputs are absent.
• Keep one DV container per affected bucket, matching the current Java index-manifest combiner. Compare logical selected masks when containers are repacked. On a definite CAS loss, reclaim only DV files generated by that attempt; retain files when publication has an unknown outcome.
• Reject attempts to enable DV over files written without it. No full-compaction API or additional merge engines are enabled.

Tests

Six DV integration tests cover none/input/lookup across fixed and dynamic buckets, exact cross-batch physical DV positions, default scans without merge-on-read, retained history and old snapshots, DELETE-only output and before-image metadata, selected-input mask filtering/retirement, shared index ranges, latest HASH preservation, concurrent L0 writes, independent DV union, changed input/hit masks, removed targets, schema-mode overrides, serialized-result validation and retries. Consecutive updates verify that deletion masks for multiple historical files remain in one bucket container and that old snapshots stay readable. Two commit regressions inject CAS loss and a lost publication response, and distinguish unchanged masks repacked into new containers from actual selected-mask changes. All three repairs have RED/GREEN regression evidence. Data Evolution retry fault injection also verifies failed-attempt DV cleanup and retention after a lost publication response, including an intervening DELETE.

4,674 library unit tests and 193 related integration tests passed with --test-threads=4; six existing unit tests were ignored. All 70 PK compaction tests passed alongside append/Data Evolution compaction, DV writes, incremental/streaming reads and partial-update state, predicate deletes, external data paths and custom data directories. A dense-vector unit regression verifies memory accounting follows bitmap container storage rather than four bytes for every deleted row.

cargo fmt --all -- --check, git diff --check, and cargo clippy --locked -p paimon --lib --test pk_compact_test -- -D warnings -A clippy::nonminimal_bool passed. The allowance covers existing repository findings with the local toolchain.

Behavior was checked against Java LookupLevels, LookupMergeTreeCompactRewriter, LookupChangelogMergeFunctionWrapper, MergeTreeCompactManager and conflict detection at Apache Paimon commit 5be7e2dd1f.

API and Format

The compaction API is unchanged. CompactResult adds defaulted deletion_vector_updates; uncommitted DV results must retain this metadata, including an explicit empty list. CompactTask remains flat and commit resolution does not receive tasks. Lookup artifacts advance to private disposable format v2 for physical positions; table data and DV index formats use existing readers/writers.

Documentation

Updated compaction and SQL guides with DV scope, local-cache requirements, physical deletion/commit behavior, row limits and mode-migration restrictions.

[leaves12138]

LGTM