[JingsongLi]

Purpose

Follow-up to #1252, extracted from #1251. Small writes to a deduplicate primary-key table currently cause compaction to rewrite the captured bucket, including large historical files. Incremental sorted-run selection now leaves large higher-level runs in place and rewrites only the selected runs.

This PR enables the policy for merge-engine=deduplicate with changelog-producer=none/input and no deletion vectors. Other modes retain their merged behavior while their lookup/cache dependencies are split separately.

Brief change log

• Port Java UniversalCompaction run grouping, size amplification, size ratio, run-count maintenance and output-level selection. Apply normal selection before forced L0 progress, including Java's double rounding at size-ratio boundaries. Match Java's L0 ordering, including sequence and creation-time tie-breakers, and accept a compaction trigger of one.
• Capture the selected output level in serialized tasks for execution. Publication independently validates Add/Delete file changes against the captured snapshot, without passing execution tasks into commit resolution. Captured files are indexed by partition, bucket and total-bucket count once, avoiding a whole-snapshot scan per bucket.
• Preserve winning retracts and user sequences across partial-run rewrites, retain concurrent L0 appends, and keep completed legacy results recoverable by idempotent retry. Legacy tasks without output-level metadata require replanning for execution; prepared results can be published when their file changes and actual output levels match the captured policy.
• Add focused behavior and boundary regressions, including an order-sensitive aggregation example that must retain its history in the merge.

Tests

4,663 library unit tests and 159 related integration tests passed; six existing unit tests were ignored. The integration run covers all 60 PK compaction tests plus append/Data Evolution compaction, DV writes, incremental/streaming reads and partial-update state.

Eight new planner tests cover stable run grouping/order, Java threshold boundaries, normal-maintenance precedence over forced L0 progress, large-run double rounding, singleton/empty inputs, existing high levels, large-byte arithmetic and configuration bounds. Six new integration tests cover untouched history files, data and system-sequence equivalence, historical snapshots, user-sequence winners, tombstones, tampered tasks/results, incorrect physical output levels, concurrent L0 appends, task-independent publication and retry recovery (with both retained and merged history), and the aggregation scope boundary.

cargo fmt --all -- --check, git diff --check, and cargo clippy --locked -p paimon --lib --test pk_compact_test -- -D warnings -A clippy::nonminimal_bool passed. The Clippy allowance covers existing repository findings with the local toolchain.

Selection was checked against Java UniversalCompaction, Levels and CoreOptions at Apache Paimon commit 5be7e2dd1f.

API and Format

Adds a defaulted output_level field directly to serialized CompactTask objects. Table data formats are unchanged. No full-compaction option is added.

Documentation

Updated the compaction guide with the enabled modes, run-selection options, output levels and serialized-task recovery behavior.

[leaves12138]

LGTM