[iceberg] Assign manifest-level row lineage for Iceberg format version 3

[vbabenkoru]

Second PR of the 3-PR stack for #8972, stacked on #9244: GitHub cannot target a fork branch as base, so this PR shows the full 13-commit diff against master — only the top 9 commits belong to this PR; the first 4 are #9244 and will disappear from the diff once it merges and this branch is rebased. Verified in both configurations: default on JDK 11 (50 tests, 2 GA-only skips) and -Piceberg-ga on JDK 17 (50 tests, no skips). The VARIANT enablement PR sits on top of this one.

Purpose

#9244 made v3 table metadata carry row lineage. The Iceberg v3 spec also requires lineage at the manifest level: first_row_id in the manifest list (field 520) and in data-file entries (field 142), with defined inheritance — an ADDED entry with null first_row_id inherits from the manifest, and the inherited value must be materialized when an entry is rewritten as EXISTING/DELETED. Without these fields, GA readers cannot expose stable _row_id values, and row IDs silently change across snapshots.

This PR completes writer-side v3 compliance with stable, synthetic row IDs:

• Manifest lists gain first_row_id (field 520); manifest entries gain first_row_id on the data file (field 142). Both are written only for format version 3; v2 file shapes are unchanged (pinned by tests).
• At manifest-list write time, every DATA manifest without an assigned first_row_id receives one from the table's next-row-id watermark, which advances by the rows that actually inherit an ID. DELETE manifests are never assigned, per spec.
• When a v3 manifest is rewritten (delete compaction, metadata compaction), entries that inherited their first_row_id get the value materialized into the rewritten file, so IDs stay stable across rewrites.
• Accounting is migration-safe: v3 manifests written before this PR contain EXISTING/DELETED entries without materialized IDs; the true number of inheriting rows is computed by reading such manifests once, so added-rows / next-row-id never double-assign a row-id range.
• Snapshot expiration now decides manifest liveness by physical path instead of value equality of the manifest-list entry. Assigning first_row_id to a carried-over manifest re-lists the same file with a different list-level field; the old equality check deleted the shared file while newer snapshots still referenced it.
• The module keeps compiling and testing against Iceberg 1.8.1 by default (so the JDK 11 CI builds it unchanged); the few assertions that need the GA (1.10+) reader API resolve it reflectively and skip themselves on 1.8.1. An opt-in iceberg-ga profile (mvn test -pl paimon-iceberg -Ppaimon-iceberg,iceberg-ga on JDK 17) runs the identical suite against Iceberg 1.11, validating compliance with the reference implementation. iceberg-core remains provided-scope; the runtime classpath still decides the deployed Iceberg version.

Known follow-up (performance, not correctness): computing the true inheriting row count re-reads manifests that were rewritten in the same commit; this can be folded into the manifest write itself later.

Tests

IcebergRowLineageCompatibilityTest grows from 5 to 17 cases, including:

• field 520/142 presence and round-trip for v3, absence for v2 (shape pin);
• assignment starts at 0 and advances across commits;
• multi-bucket (bucket = 2) assignment is stable across snapshots — IDs never shift when unrelated buckets commit;
• rewrites (delete compaction, metadata compaction) materialize inherited IDs and keep them identical before/after;
• migration from metadata-only lineage: legacy manifests get IDs exactly once, ranges stay unique, added-rows counts inherited legacy rows;
• GA reader matrix: Apache Iceberg 1.11 reads the produced tables and observes the expected first_row_id per data file;
• expiration regression: a manifest shared between a pre-assignment manifest list and its assigned successor survives snapshot expiration and all retained snapshots stay readable.

IcebergRestMetadataCommitterTest covers v3 registration and recreation against a REST catalog with value-equality checks vs the local metadata.

AI notice

The code is generated using Fable 5 (with reviews from Codex) but has been verified to run on a real cluster with Flink, Paimon, Iceberg, StarRocks and Snowflake.

[vbabenkoru]

The JDK 11 CI failures were caused by this PR bumping paimon-iceberg to Iceberg 1.11 outright: Iceberg 1.10+ ships Java-17 bytecode, and the paimon-iceberg module is built by the JDK 11 workflows, so javac could not read the Iceberg classes.

I've restructured the PR to keep CI green without losing the GA validation:

• The module compiles and tests against Iceberg 1.8.1 by default, exactly as on master, so the JDK 11 workflows build and run everything again.
• The handful of assertions that need the GA (1.10+) reader API — per-manifest / per-file firstRowId() — look the method up reflectively and skip themselves when the API is absent (2 GA-only tests skip, plus one guarded assertion block; everything else runs unchanged on 1.8.1).
• A new opt-in iceberg-ga profile (mvn test -pl paimon-iceberg -Ppaimon-iceberg,iceberg-ga on JDK 17) switches the module to Iceberg 1.11 with the Hadoop/Jetty/Jackson pins its test fixtures need, so the full suite — including the GA reader matrix — validates against the reference implementation. I've run both configurations locally: default on JDK 11 (50 tests, 2 skipped) and -Piceberg-ga on JDK 17 (all tests, nothing skipped).

Longer term, actually bumping the module's Iceberg dependency (and giving it a JDK 17 CI job) is a project-level decision — 1.8.1 predates GA row lineage, and Iceberg's active line has moved to Java 17. Happy to file that separately or adjust here, whichever the maintainers prefer.