Enable the split-operator plan and native Iceberg writes by default

[andygrove]

What is the problem the feature request solves?

Native Iceberg writes ship behind two flags that both default to false: spark.comet.write.iceberg.splitOperator.enabled (the writer/committer split from #4658) and spark.comet.write.iceberg.enabled (the iceberg-rust data-file writer from #5361, renamed from spark.comet.iceberg.write.enabled by #5306). This issue defines what has to be true before Comet's Iceberg write path becomes the default.

#6664 makes spark.comet.write.iceberg.enabled the one setting for both and defaults it to true in 1.2.0, on every Spark version; it plans the split operator by itself, and spark.comet.write.iceberg.splitOperator.enabled stays a testing setting that plans the split operator with the native writer off. 1.2.0 is targeted for late October or early November (#6550), so flipping now gives the write path nightly runs before the release. The split plan changes only the plan shape. The native writer writes the data files of each write its eligibility gate accepts; every other write falls back to iceberg-java's writer, and iceberg-java commits every write. Setting spark.comet.write.iceberg.enabled=false restores Spark's own write operator, as in Comet 1.1.0.

Describe the potential solution

Before #6664 merges

These are #6664's blockers. With the native writer on by default, each of them would reach every user who writes Iceberg tables.

Split plan:

• Iceberg write split-operator plan: CI failures when enabled by default #5259 The four CI failure buckets seen with the split plan on by default are fixed
• IcebergWriteStrategy ignores spark.comet.enabled, so the split-operator write plan survives the kill switch #6142 IcebergWriteStrategy respects spark.comet.enabled, so disabling Comet restores Spark's plan
• IcebergCommitExec rethrows the raw cause when the abort fails, where Spark wraps it in writingJobFailedError #6143 Job and commit failures surface as the same exception types as Spark's own write
• Preserve catalog transactions for split Iceberg writes on Spark 4.2 #6586 Catalog transactions are committed on Spark 4.2 (TransactionalExec)

Native writer correctness:

• Native Iceberg writes use an iceberg-rust revision missing partition-path escaping #5636, Native Iceberg writes drop Hadoop GCS configuration for HadoopFileIO + gs:// #5637, Native Iceberg write merges -0.0 and 0.0 rows into one float/double identity partition #6138, Native Iceberg write gate reads hdfs:/path as file, so the write fails natively instead of falling back #6140, Native Iceberg write panics or writes a NULL partition value for dates and timestamps beyond year 262143 #6145, docs: fix stale and missing native write details in iceberg-writes.md and iceberg.md #6147
• Native Iceberg write silently drops S3 settings it cannot honour (credentials provider, SSE, ACL, tags, remote signing) instead of falling back #6139 S3 settings the native writer cannot honour fall back instead of being dropped (PR fix: gate unsupported Iceberg S3 write settings #6441)
• Native Iceberg write fails on a V1 spec that mixes a live field with a void field whose source column was dropped #6141 A V1 spec that mixes a live field with a void field whose source column was dropped falls back, so the write fails with iceberg-java's own error (PR fix: decline native Iceberg writes through a void partition field whose source column was dropped #6677)
• Native Iceberg write over-counts NaNs for nested float fields when the input batch is already sliced #6146 and Native Iceberg writer counts NaNs under NULL structs and NULL list or map entries #6562 NaN counts for nested float fields (PRs fix: compact already-sliced batches before the native Iceberg writer counts NaNs #6238 and fix: drop values under NULL parents before the native Iceberg writer counts NaNs #6658)

Failure handling:

• Native Iceberg write: clean up task-attempt data files on task failure #5618, Remove orphaned files in comet iceberg split writers #5277, Native Iceberg Write: close cleanup ownership gap during JVM handoff #6581
• Spark errors after the first batch of a JVM input stream reach the user as CometNativeException #6234 A Spark error after the first input batch reaches the user as the exception Spark threw (PR fix: rethrow the exception a JVM input throws instead of a CometNativeException #6243)

Memory and performance:

• Account the native Iceberg writer's buffers in Comet's memory pool #5648 and Charge native write buffers to the shared off-heap memory pool #6115 The writer's buffers, including the fanout writer's per-partition buffers, are charged to Comet's memory pool (PR feat: charge native write buffers to the task's memory pool #6247)
• Native Iceberg writer keeps a dictionary page for high-cardinality columns where iceberg-java writes none #6114 The native writer drops the dictionary of a high-cardinality column the way iceberg-java does, so selective reads no longer fetch a ~2 MB dictionary page per column (PR perf: choose dictionary encoding like iceberg-java in the native Iceberg writer #6241)

Coverage and evidence:

• Support Iceberg's system functions (bucket, truncate, years/months/days/hours) as native expressions #5635 The default hash-distribution write is native end to end
• Make native Iceberg write coverage visible in the Iceberg Spark test CI jobs #6148 The Iceberg CI jobs report which writer ran each write
• Iceberg's Spark test suites run with both flags on: the split plan since feat: Optionally split the Iceberg V2 write operator into distinct writer and committer operations #4658, the native writer since test: run the Iceberg Spark tests with the native Iceberg writer enabled #5677 (2026-09-22). Iceberg 1.8, 1.9 and 1.10 run nightly, 1.11 in the merge queue. From 2026-09-16 to 2026-10-04, 19 nightly runs of 25 Iceberg jobs each had one failure, the upstream Iceberg 1.9 partial-progress flake (Flaky test: TestRewriteDataFilesAction.testParallelPartialProgressWithMaxFailedCommitsLargerThanTotalFileGroup() iceberg#12889). In the last 30 merge-queue runs, the one Iceberg job that failed stopped before any test ran.
• Comet's own suites pass with both defaults on, on every Spark profile (feat: enable native Iceberg writes by default #6664 is labelled run-all-spark-profiles)

In #6664

• spark.comet.write.iceberg.enabled moves from CATEGORY_TESTING to CATEGORY_EXEC and switches both layers, and the docs that describe the flags as off by default are updated: user-guide/latest/iceberg-writes.md, iceberg.md, operators.md, contributor-guide/iceberg-writes.md and the Iceberg write review skill
• Release note: a 1.2.0 upgrade-guide entry names the setting and how to turn it off. Iceberg write plans show IcebergCommit over IcebergWrite (or CometIcebergWrite) in place of Spark's AppendData, OverwriteByExpression, OverwritePartitionsDynamic and ReplaceData exec nodes. The split happens at physical planning, so analyzed and optimized logical plans are unchanged, and listeners that read logical plans (lineage tools) are unaffected. Code that matches Spark's physical write nodes will not find them.

Spark 3.4 gets the same defaults. No Iceberg Spark test job covers Iceberg 1.5.2, which the 3.4 profile pins, so on 3.4 the evidence is Comet's own suites.

Before 1.2.0 ships

Evidence gathered while the defaults run in CI and nightly:

• A stretch of nightly history with both defaults on and no failure that only the native writer causes (length to be agreed)
• Benchmark results (CometIcebergWriteBenchmark, Add a native Iceberg write benchmark #5647) show no regression against iceberg-java for unpartitioned, clustered, fanout and copy-on-write delete writes
• Each fallback reason in the Iceberg CI write report is either intended or tracked. On 2026-10-06, 6,501 of the 13,678 writes in the Iceberg 1.11 jobs ran natively (47.5%), before Native Iceberg writes: support format-version 3 tables #6668 (PR feat: support native Iceberg writes to format-version 3 tables #6672) admits format-version 3. Avro and ORC tables (2,458 writes) and format-version 4 (580) are intended fallbacks, so the native share is not a useful target on its own.
• Manifest metrics parity asserted per data type on all four pinned Iceberg versions (1.5.2, 1.8.1, 1.10.0, 1.11.0)
• Decide keep-or-lift for each remaining native Iceberg write eligibility restriction #5643 Every remaining eligibility restriction has a keep-or-lift decision

Not blocking the defaults

• revertToSpark erases CometIcebergWriteExec / CometNativeWriteExec because originalPlan is the node's own child #5719 revertToSpark keeps the write node (PR fix: restore Spark write execs when reverting transition-heavy stages #5957). It is reached only with spark.comet.exec.transitionRevert.enabled, which is off by default.
• Failure-injection tests for native Iceberg writes #5646 Failure-injection tests that check storage state (PR test: cover speculation and executor loss in multi-task native Iceberg writes #6601 covers speculation and executor loss)

Additional context

Part of #5649. Refreshed on 2026-10-06 to match #6664, which makes spark.comet.write.iceberg.enabled the one switch and defaults it to true in 1.2.0 on every Spark version. The 2026-10-05 version flipped the native writer one release after the split plan and left Spark 3.4 open; it had folded in the original list from 2026-09-02 and the issues the 2026-09-23 audit added (#6138 to #6148). Related: #4658, #5298, #5361.