[JingsongLi]

Purpose

Extract non-DV lookup changelog production from #1251 into an independently mergeable change. This uses the existing primary-key compaction planner and provides the changelog foundation for subsequent deletion-vector and incremental-lookup work.

L0 inserts, updates and deletes produce logical before/after events atomically with compacted data. Incremental and streaming scans can consume the resulting COMPACT snapshot's changelog manifest.

Brief change log

• Add lookup merge wrapping and source-level metadata so the lowest positive level supplies the latest materialized history, avoiding repeated aggregation of older states.
• Generate INSERT, UPDATE_BEFORE, UPDATE_AFTER and DELETE events, with Java-compatible filtering and row deduplication. Map equality compares entries by key, including Maps nested in ROW values.
• Match Java's separate first-row wrapper: retain the earliest accepted record across all input levels, suppress events for existing keys, and preserve the first add's INSERT or UPDATE_AFTER kind for new keys.
• Always invoke raw partial-update/aggregation reducers during lookup compaction, including singleton retracts. Keep the query reader's outer singleton shortcut before history selection, and preserve ignored legacy partial-update DELETE payloads for subsequent sparse updates.
• Close both data and changelog writers before handing off prepared files, clean both sides on failure, serialize changelog results/metrics, and include logs in idempotent commit recovery.
• Reject concurrent positive-level history changes, including additions at another level, while allowing concurrent L0 appends.
• Require a real enabled catalog/FileIO local-cache for lookup compact executors.

Tests

Final local validation: 4,808 tests passed (4,655 library unit tests and 153 related integration tests); six existing tests were ignored.

Integration coverage includes PK/append/Data Evolution compaction, incremental and streaming reads, DV writes, and partial-update state. Lookup regressions cover overlapping first-row history levels, first-add row kinds, ignored legacy retracts, raw singleton aggregation versus ordinary queries, sequence-group replay, and reordered/changed/null/empty Map values directly and nested in ROW.

cargo fmt --all -- --check, git diff --check, and cargo clippy --locked -p paimon --lib --test pk_compact_test -- -D warnings -A clippy::nonminimal_bool passed. The Clippy allowance covers existing repository findings with the local toolchain.

Behavior was compared against the local Apache Paimon Java checkout at 5be7e2dd1f, including lookup/first-row wrappers, the outer query reducer wrapper, partial-update and aggregate reducers, and generated value equalisers.

API and Format

Adds CompactResult::changelog_files() and changelog counters to CompactMetrics; serialized results include a defaulted changelog-file list. Lookup compact executors require local-cache.enabled=true on the catalog/FileIO; a table-property flag alone cannot attach a cache. Table data formats are unchanged.

Documentation

Updated the compaction guide and SQL documentation for non-DV lookup changelog production, including first-row and singleton-retract behavior.

[leaves12138]

LGTM