A native fanout write keeps a data file open for every partition a task writes to, and since #6247 its buffers count against the task's memory pool. The writer cannot spill, so a task that writes to enough partitions fails with Additional allocation failed for IcebergWriteExec, where iceberg-java's fanout writer, buffering on the JVM heap, succeeds. Spark's retry takes the same path, so the job fails.
On Iceberg 1.5+ an unsorted partitioned table uses the fanout writer by default, and #6664 makes native writes the default, so this has to be fixed first. Raised in review of #6664.
A native fanout write keeps a data file open for every partition a task writes to, and since #6247 its buffers count against the task's memory pool. The writer cannot spill, so a task that writes to enough partitions fails with
Additional allocation failed for IcebergWriteExec, where iceberg-java's fanout writer, buffering on the JVM heap, succeeds. Spark's retry takes the same path, so the job fails.On Iceberg 1.5+ an unsorted partitioned table uses the fanout writer by default, and #6664 makes native writes the default, so this has to be fixed first. Raised in review of #6664.