[iceberg] Allow publishing VARIANT columns with Iceberg format version 3 - #9246
Open
vbabenkoru wants to merge 14 commits into
Open
[iceberg] Allow publishing VARIANT columns with Iceberg format version 3#9246vbabenkoru wants to merge 14 commits into
vbabenkoru wants to merge 14 commits into
Conversation
vbabenkoru
force-pushed
the
iceberg-v3-variant
branch
2 times, most recently
from
August 17, 2026 16:44
e82046a to
f2c59f2
Compare
Adds tests only, per the follow-up to the Layer 2 verification sweep (task-7-report.md gap summary): compactMetadataIfNeeded's explicit-142 passthrough branch and its v3 merge call site now have coverage; the REST recreation-consistency test compares first-row-id values against the local file-based mirror instead of only checking non-nullity; a GA-reader test resolves actual per-file first_row_id values (with the iceberg-data generics limitation verified and documented); and the v2 manifest-list shape (14 columns, no first_row_id) is now pinned by a byte-level regression test. No production code changes; all behavior matched the spec exactly.
Assigning first_row_id to a carried-over v3 manifest re-lists the same physical avro file with a different list-level field. expireManifestList compared IcebergManifestFileMeta values, so expiring the pre-assignment manifest list deleted the shared file while newer manifest lists still referenced it, breaking every subsequent Iceberg read of the retained snapshots. Decide liveness by manifest path instead, and cover the upgrade boundary with an expiration regression test.
…rg-ga profile
Iceberg 1.10+ ships Java-17 bytecode, so bumping paimon-iceberg to 1.11
outright broke the JDK 11 CI workflows that build this module. Restore
the 1.8.1 default so the module compiles and tests on JDK 11: the few
assertions that need the GA reader API (per-manifest / per-file
firstRowId) resolve it reflectively and skip themselves when absent, the
pre-GA enableRowLineage opt-in in IcebergMetadataTest is called
reflectively so the test passes on both lines, and the update logging in
the REST committer no longer names the RemoveSnapshot(s) class that was
renamed between the two lines. The full GA validation - Iceberg 1.11
with the Hadoop/Jetty/Jackson pins its REST test fixtures need - moves
behind the opt-in iceberg-ga profile (JDK 17):
mvn test -pl paimon-iceberg -Ppaimon-iceberg,iceberg-ga
Verified: default suite on JDK 11 (50 tests, 2 GA-only skips) and
-Piceberg-ga on JDK 17 (50 tests, no skips).
vbabenkoru
force-pushed
the
iceberg-v3-variant
branch
from
August 18, 2026 03:53
f2c59f2 to
a91857d
Compare
GA ManifestReader assigns inherited first_row_id only to non-DELETED entries. Materialization and manifest-level accounting advanced the watermark through DELETED entries with a null field 142, shifting every following live file relative to what readers had already inferred. Skip DELETED entries when materializing, and reserve each unassigned manifest's range with the spec-sanctioned added+existing upper bound instead of reading manifest contents in the commit path: the exact count, and the serial full-manifest scan it required, are gone; already-materialized rows now leave spec-legal id gaps. Regression test: a legacy manifest with a DELETED entry before a live one keeps the live file's inherited id equal to the manifest's first_row_id, cross-checked against the GA reader. Reported by JingsongLi in review.
Rebuilding a REST table through TableMetadata.Builder recomputes next-row-id from each snapshot's added-rows, so an id space that does not start at 0 (self-heal and rollback rebuilds) ended below the ids already assigned in manifests, and a later external REST writer could have reused them. Recreation, initial creation of a table with history, and the exists-but-empty recovery now write the REST-adjusted metadata to a file and register it, importing every high-water mark verbatim. The drop+create failure loop the empty-table recovery guarded against cannot occur here: registration has no post-create commit step. Registration also preserves the real partition spec when a partition column has field id 0, where creation used to fall back to an unpartitioned spec. Tests assert the recreated server watermark is at least the local one and that an external append through the Iceberg API allocates above it. Reported by JingsongLi in review.
The iceberg-ga profile was not exercised by CI, and running it as a single 'mvn test -am' session fails: the not-yet-shaded paimon-bundle is substituted with its unshaded constituent modules, whose direct Avro references clash with the Avro 1.12 line Iceberg 1.11 requires (the installed bundle relocates Avro and has no clash). Add a JDK 17 workflow that installs the shaded bundle first and then runs the suite, and document the two-step invocation in the profile. Reported by JingsongLi in review.
vbabenkoru
force-pushed
the
iceberg-v3-variant
branch
from
August 18, 2026 04:12
a91857d to
ce39155
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Final PR of the 3-PR stack for #8972 — closes #8972. Stacked on #9244 and #9245: only the top commit belongs to this PR; the 13 commits below it are the two PRs underneath and will disappear from the diff as they merge.
Purpose
Iceberg supports the VARIANT type from format version 3 onward, but v3 also made row lineage mandatory. Paimon therefore rejected VARIANT columns in the Iceberg compatibility layer for all format versions, because the v3 metadata it produced was not lineage-compliant and GA readers would reject the whole table (#8972).
With #9244 and #9245, Paimon's v3 writer emits complete row lineage (table metadata, manifest lists, manifest entries), so the reason to block VARIANT on v3 is gone. This PR lifts the guard for
metadata.iceberg.format-version>= 3:varianttype.The guard lift is intentionally the last commit of the stack: every v3 table that can publish VARIANT is guaranteed to have carried full row lineage from its first commit.
Tests
IcebergRowLineageCompatibilityTest#testVariantPublishableWithFormatVersion3: a VARIANT column publishes on v3, the schema maps to Icebergvariant, and lineage fields are present.IcebergRowLineageCompatibilityTest#testVariantRejectedWithFormatVersion2: v2 commits with VARIANT still fail with the explicit error.AI notice
The code is generated using Fable 5 (with reviews from Codex) but has been verified to run on a real cluster with Flink, Paimon, Iceberg, StarRocks and Snowflake.