) TBLPROPERTIES (
+ | 'row-tracking.enabled' = 'true',
+ | 'data-evolution.enabled' = 'true')
+ |""".stripMargin)
+ sql("INSERT INTO t VALUES (1, named_struct('a', 10, 'b', 'x'))")
+
+ Seq((1, 100)).toDF("id", "newa").createOrReplaceTempView("s")
+
+ sql(s"""
+ |MERGE INTO t
+ |USING s
+ |ON t.id = s.id
+ |WHEN MATCHED THEN UPDATE SET t.nest.a = s.newa
+ |""".stripMargin).collect()
+
+ // correctness still holds: nest.a updated, nest.b preserved
+ checkAnswer(sql("SELECT id, nest.a, nest.b FROM t"), Seq(Row(1, 100, "x")))
+
+ // but no sub-field incremental file is produced: the whole nest column is rewritten
+ val deltaCols = latestDeltaWriteCols("t")
+ assert(
+ !deltaCols.exists(cols => cols.contains("nest.a")),
+ s"expected no dotted (sub-field) writeCols when feature is disabled, got: $deltaCols")
+ }
+ }
+}
From 282fb9f629ed279e9f779559273e16fb16327e4a Mon Sep 17 00:00:00 2001
From: Xiangyi Zhu <82511136+zhuxiangyi@users.noreply.github.com>
Date: Mon, 17 Aug 2026 00:56:22 +0800
Subject: [PATCH 4/4] [core][spark][flink] Fix ordering and null-safety issues
in sub-field data evolution
Addresses the review on #8334. Six issues, all of which could store or read
values under the wrong field:
- Spark: the changed-leaf set kept WHEN MATCHED clause order while the output
struct was laid out in schema order, so two clauses touching sub-fields in
reverse order made writeType disagree with the physical layout and swapped
values on read. The leaf set is now canonicalized to schema order once and
reused for the output struct, writePaths and writeType.
- Spark: buildPrunedStruct built a CreateNamedStruct unconditionally, turning a
copied NULL struct into a non-null struct of NULL children. Guarded in
copyOutput, and in updateOutput only when the action sets no leaf of the
column, so an explicit SET on a previously NULL struct still materializes it.
- RowType: a nested leaf path flattened to "a.b" could collide with a literal
top-level field of the same name, which projectByPaths resolves to the wrong
field. leafPaths now rejects such a write instead of encoding it ambiguously.
- RowType: coversFully compared field presence but not order, so a
complete-but-reordered struct collapsed to the bare column name and lost its
physical layout. It now compares field ids positionally and recurses.
- Flink: isFullyCompatibleStruct validated a whole-struct assignment by field
name only while the write is positional, accepting a reordered or wider source
that would be stored under the wrong names. It now requires matching arity and
per-position names.
- Flink: parseCommaSeparatedKeyValues collapses the SET list into a map, so a
duplicate target lost the earlier entry before the duplicate check ran.
Duplicates are now detected beforehand, normalizing qualified and unqualified
forms of the same target.
Also documents the mixed-version barrier on data-evolution.nested-field.enabled:
every reader, writer, compactor and maintenance job must be upgraded before the
option is enabled, and downgrading is unsafe once a file whose write columns
contain a nested sub-field path has been committed.
Both ordering and null regressions are covered by end-to-end MERGE INTO tests
using the clause shapes that actually expose them; a single clause is normalized
into schema order by Spark's own assignment alignment, and an unmatched row keeps
its parent-struct nullness from the base file, so neither reproduces the bugs.
Adds coverage for compaction over sub-field files and for adding a nested
sub-field after such files exist. Updates a DataTypesTest assertion that had
encoded the old order-insensitive coversFully behaviour.
---
docs/generated/core_configuration.html | 2 +-
.../java/org/apache/paimon/CoreOptions.java | 9 +-
.../java/org/apache/paimon/types/RowType.java | 40 ++-
.../org/apache/paimon/types/RowTypeTest.java | 106 +++++++
.../apache/paimon/types/DataTypesTest.java | 9 +-
.../action/DataEvolutionMergeIntoAction.java | 50 +++-
.../NestedSubfieldMergeIntoActionITCase.java | 72 +++++
.../MergeIntoPaimonDataEvolutionTable.scala | 43 ++-
.../MergeIntoPaimonDataEvolutionTable.scala | 43 ++-
...ergeIntoPaimonDataEvolutionTableTest.scala | 51 +++-
.../sql/NestedSubfieldMergeIntoTest.scala | 275 ++++++++++++++++++
11 files changed, 666 insertions(+), 34 deletions(-)
create mode 100644 paimon-api/src/test/java/org/apache/paimon/types/RowTypeTest.java
diff --git a/docs/generated/core_configuration.html b/docs/generated/core_configuration.html
index a6d20af0a013..38abe8b7e73d 100644
--- a/docs/generated/core_configuration.html
+++ b/docs/generated/core_configuration.html
@@ -516,7 +516,7 @@
data-evolution.nested-field.enabled |
false |
Boolean |
- Whether to enable sub-field-level data evolution for nested (struct) columns. When enabled, an update that only touches some sub-fields of a nested column writes an incremental file containing just those sub-fields (aligned by row id); when disabled, the whole top-level column is rewritten. Requires data-evolution.enabled=true. |
+ Whether to enable sub-field-level data evolution for nested (struct) columns. When enabled, an update that only touches some sub-fields of a nested column writes an incremental file containing just those sub-fields (aligned by row id); when disabled, the whole top-level column is rewritten. Requires data-evolution.enabled=true. Mixed-version compatibility warning: once a file's write columns record a nested sub-field path (e.g. 'nest.a'), a reader, writer, compactor, or other maintenance job on an older version cannot reconstruct it. Every such component reading or writing this table must be upgraded before enabling this option, and downgrading the binary is unsafe once such files have been committed. |