[EPIC] Bug fixes to consider backporting to branch-1.0

[andygrove]

What is the problem the feature request solves?

branch-1.0 was cut at af534e0f on 2026-08-03, and 1.0.1 has not been tagged yet. Bug fixes keep landing on main, but nothing tracks which of them should go to branch-1.0, which are already there, and which open PRs will become candidates once they merge.

#5815 triaged the first 192 commits after the cut, up to 2026-09-09, and it remains the record of why each earlier exclusion was made. This epic carries forward #5815's bug-fix recommendations from tiers 1 and 3. It adds the 166 commits merged since then and every open bug-fix PR on main. Performance backports are out of scope; #5815's tier 2 covers those.

Describe the potential solution

Tick a box once the fix is on branch-1.0, and put the backport PR next to it. When an open PR merges, move it up to the merged list.

Candidates are ordered by severity within each group. "Clean" means the commit cherry-picks onto branch-1.0 (8407d475c) without conflicts; it does not mean it compiles. Where one fix has to follow another, the entry says so.

Merged on main, not yet on branch-1.0

Wrong results

• fix: let decimal SUM recover from an intermediate overflow like Spark #6041 A decimal SUM returns NULL, or raises ARITHMETIC_OVERFLOW under ANSI, when an intermediate sum overflows the buffer precision but the final result fits. Closes SUM(decimal) returns NULL or throws under ANSI when an intermediate sum overflows the buffer precision but the final result fits #6002. Conflicts in sum_decimal.rs and operators.scala (fewer after fix: explain ObjectHashAggregate fallback when Comet shuffle is disabled #5746). Take fix: correct two nightly test failures on Spark 3.4 and 4.2 #6156 with it: it limits the new fallback to Spark 3.5+.
• fix: revert unsafe partial aggregates after final fallback #5421 After a late shuffle fallback, a native partial aggregate can feed a Spark final aggregate: avg returns NULL, and the collect_list / collect_set buffers mismatch. Closes Unsafe native partial aggregates survive child-triggered final fallback #5419. Conflicts in CometExecRule.scala and RevertNativeForTransitionHeavyStages.scala; drop the Celeborn parts.
• fix: preserve aggregate result identity during exchange reuse #5470 Exchange reuse shares one shuffle between aggregates that differ only in their result expressions, such as COUNT(*) + 1 and COUNT(*) - 1. Conflicts in operators.scala imports and CometAggregateSuite.
• fix: preserve join and generator semantics in plan identity #5828 Exchange reuse treats semi and anti joins, null-aware NOT IN, and explode vs explode_outer as the same plan. Closes Comet join and explode operators omit joinType / outer from equals, so exchange reuse returns wrong results #5824. After fix: preserve aggregate result identity during exchange reuse #5470 only the tests conflict.
• fix: include ABFS container in object store cache key #5053 Two ABFS containers in one storage account share one cached object store, so a read can return the other container's data. Closes Object store cache key drops the ABFS container, so two containers in one storage account share a store instance #4993. Needs a manual port: it builds on URL normalization and ObjectStoreBackend, which exist only on main. Take fix: isolate object-store registration by backend and configuration #5503 first.
• fix: isolate object-store registration by backend and configuration #5503 With asymmetric fs.comet.libhdfs.schemes (for example s3 listed, s3a not), Hadoop and native object stores share a cache key, and the registry can overwrite an entry. Closes Avoid object-store cache and registry collisions across backends and configurations #5502. Conflicts in the parquet_support.rs test module only.
• fix: normalize signed zero in nested float array comparisons #5235 arrays_overlap and array_position over nested float arrays treat -0.0 and 0.0 as different. Closes Nested array comparison does not match Spark for signed zero #5191. Context conflicts only.
• fix: Nested floating-point IN membership does not match Spark for signed zero #6073 IN / InSet and nested = over arrays and structs treat -0.0 / 0.0, and different NaN encodings, as unequal. Closes Nested floating-point IN membership does not match Spark for signed zero #6019. After fix: normalize signed zero in nested float array comparisons #5235; context conflicts only.
• fix: normalize noncanonical NaN literals in comparisons #5472 A NaN literal with non-canonical bits, and static float IN / InSet hashing raw bits, drop rows that Spark keeps. After fix: Nested floating-point IN membership does not match Spark for signed zero #6073. On branch-1.0 keep withFallbackReason, because liftFallbackReasons exists only on main.
• fix: fall back when a struct repeats a Parquet field id #6004 With spark.sql.parquet.fieldId.read.enabled, a struct that repeats a field id returns rows where Spark raises. Closes Duplicate field ids inside a struct are not validated when the file schema equals the requested schema and no predicate is pushed #5801. After fix: decline structs with duplicate field names before they reach Java Arrow #5866; drop the in-memory cache hunk.
• fix: correctly rounded decimal to double/float cast matching BigDecimal.doubleValue/floatValue #5684 Decimal to double / float casts are off by one ulp for essentially every DECIMAL(38,18) value above 0.01. Clean.
• fix: preserve ANSI errors for rejected TIMESTAMP_NTZ casts #5752 Rejected TIMESTAMP_NTZ casts return NULL under ANSI, where Spark raises CAST_INVALID_INPUT. Closes CAST(string AS TIMESTAMP_NTZ) swallows ANSI errors when the year-range guard rejects #5749. Clean.
• fix: align string to timestamp parsing with Spark's segment rules #5682 String to timestamp casts give 2020-10-1 a spurious offset, accept date-only zones, and panic on Unicode fractional digits. Closes String-to-timestamp cast diverges from Spark's parseTimestampString: rejects 1-digit segments and empty fractions, accepts zone suffixes on date-only strings and 7-digit years #5674. Clean after fix: preserve ANSI errors for rejected TIMESTAMP_NTZ casts #5752.
• fix: Accept explicit positive years in timestamp casts #5858 String to timestamp casts return NULL for years written as +7528, and malformed signs bypass ANSI errors. Closes String-to-timestamp cast rejects explicit positive years accepted by Spark #5716. After fix: align string to timestamp parsing with Spark's segment rules #5682; the inline tests in string.rs conflict.
• fix: skip codegen dispatcher null short-circuit when a foldable subtree can raise #5623 The codegen dispatcher's null short-circuit swallows ANSI errors raised by a foldable subtree. Closes Codegen dispatcher single-ordinal null short-circuit swallows ANSI errors from literal subtrees #5608. Clean.
• fix: prevent silent overflow when reading Parquet TIMESTAMP_MILLIS values #5177 and fix: check nested TIMESTAMP_MILLIS overflow in unfiltered scans #5740 An overflowing TIMESTAMP_MILLIS value silently wraps in Parquet reads, at the top level and nested, where Spark throws. Conflicts in parquet_support.rs.
• fix: align time parsing and native second extraction with Spark #5738, the to_time hunk only: to_time / try_to_time reject 'T12' and '12:30:45.', which Spark accepts. Closes to_time / try_to_time: native parser rejects 'T12' and '12:30:45.' which Spark accepts #5366, which is in the 1.0.1 milestone. to_time.rs and to_time.sql apply by themselves. The rest of the PR adds native EXTRACT(SECOND FROM TIME) and TIME literals.
• feat: unix_timestamp codegen dispatch for strings, and fix pre-epoch fractional rounding #5789, the unix_timestamp.rs hunk only: native unix_timestamp floors pre-epoch fractional timestamps, returning -2 where Spark returns -1. Closes Native unix_timestamp returns incorrect results for negative fractional timestamps #5896. The hunk applies by itself. The rest of the PR is a new codegen-dispatch route.
• fix: normalize floating-point values in native collect_set #5166 On Spark 4.2 only, collect_set does not deduplicate NaN and -0.0 the way SPARK-57298 requires. Closes Spark 4.2: Comet native collect_set does not normalize NaN / -0.0 (SPARK-57298) #4966. Conflicts in the aggregates.scala imports.

Parquet field matching: backport these together or not at all

#5602 cherry-picks clean on its own, but it must not go alone. On 1.0.0, a struct with byte-identical duplicate field names panics. With #5602 applied, the same read silently returns duplicated rows (#5783), and #5786 is the fix. On top of #5602 alone, #5786 conflicts in 12 hunks of schema_adapter.rs, because it builds on #5681, which in turn builds on #5177 and #5740. The two options are the full stack, in the order below, or a hand port of #5786's duplicate check onto #5602.

• feat: support unicode case sensitive field names for reading parquet #5602 Non-ASCII field names that differ only in case silently read as NULL. Closes Support Spark-compatible Unicode case-insensitive Parquet field matching #5495, which is in the 1.0.1 milestone.
• fix: apply Spark's Parquet conversion rules to nested struct/list/map fields #5681 Nested struct, list and map Parquet conversions return NULL on overflow, accept values Spark rejects, or panic. Closes Nested (struct/list/map) Parquet schema-evolution conversions bypass Spark's type-conversion rules: silent NULLs, string parsing, and a native panic #5671. About 1,600 lines.
• fix: propagate Parquet field-name folding failures #5845 When the JVM call that case-folds a field name fails, Comet falls back to Rust's Unicode tables, which can select a different column than Spark would.
• fix: reject duplicate Parquet field names before decoding #5786 Rejects duplicate Parquet field names before decoding. Closes Native Parquet scan multiplies rows for a struct with duplicate field names #5783.

Task failures, panics, and errors Spark does not raise

• fix: make collect_list/collect_set argument coercion a normalization barrier #5159 collect_list / collect_set over a nested argument fails with "column types must match schema types" when nested-field nullability drifts. Closes collect_list/collect_set can fail with "column types must match schema types" on nested-field nullability drift #5158. Clean.
• fix: decline structs with duplicate field names before they reach Java Arrow #5866 Native shuffle accepts a struct with duplicate field names, then fails the task importing the batch into Java Arrow. With the fix, it falls back. Closes Native shuffle accepts structs with duplicate field names, then fails importing the batch back to the JVM #5605. Only the test conflicts.
• fix: let Comet memory pools overcommit on grow instead of panicking #6128 Comet memory pools panic in grow when Spark grants less than requested, for example on the sort-merge join spill path. Closes Comet memory pools panic on MemoryPool::grow instead of allowing overcommit #6127 and CometMemoryPool may release more memory than acquired memory #1733. Clean after fix: make task-shared memory pool as ref-counted RAII guard #5494 apart from docs.
• fix: support null calendar interval literals #5133 A null calendar-interval literal passes the JVM check and then fails native planning. Closes Null CalendarInterval literal throws native exception instead of evaluating to null #5058. Clean.
• fix: return NULL from rpad/lpad when the length column is NULL instead of panicking #5680 rpad / lpad panic on a NULL length. Clean.
• fix: rebase map offsets in mapsort so sliced maps do not overrun entries #5630 A sliced map fails map_sort with "Max offset exceeds length of entries". Only the test conflicts.
• fix: fall back for concat_ws with array arguments instead of failing natively #5679 concat_ws with array arguments fails natively instead of falling back. Clean.
• fix: match Spark's ANSI bound check for float/double to integral casts #5683 Float / double to integral casts raise a spurious CAST_OVERFLOW for exactly Int.MaxValue / Long.MaxValue. Clean.
• fix: use Spark type names in ANSI abs overflow errors #5357 ANSI abs overflow errors say Int64 overflow where Spark says long overflow. Clean.
• fix: propagate Arrow array copy errors #5747 copy_array panics on offset overflow instead of returning an error. Closes copy_array panics on offset overflow instead of returning an error #5706. Conflicts in copy.rs.
• fix: align Spark 4.2 Python worker configuration #5561 On Spark 4.2, accelerated mapInArrow / mapInPandas fails outright because of the Python worker configuration. Clean.
• fix: read Iceberg tables partitioned by an unknown transform #5759 Reading an Iceberg table partitioned by an unknown transform fails with Invalid Iceberg scan task. Conflicts in CometIcebergNativeScan.scala.

Leaks, hangs and memory

• fix: attach tokio runtime threads to the JVM as daemon threads #5748 The JVM hangs if an application returns from main without calling spark.stop(), because Tokio runtime threads are not daemon threads. Only the CI YAML conflicts.
• fix: make task-shared memory pool as ref-counted RAII guard #5494 A stranded task-shared memory pool entry pins TaskMemoryManager and TaskContext for the executor's lifetime. Conflicts in the CI YAML and jni_api.rs.
• fix: make CometDiskBlockWriter spill registry per-task instead of executor-global #5493 The static CometDiskBlockWriter spill registry lets one task force-spill another task's buffered bytes. Clean.
• fix: prevent memory leak after failed Arrow vector import #5539 A failed Arrow vector import leaks native buffers. Closes NativeUtil.getNextBatch leaks Arrow structs when importVector fails #5534. Only the test conflicts.
• fix: release native shuffle reservation after spill failure #5461 After a failed partition write, native shuffle leaks its reservation, its pinned buffers, and repeated-spill bytes. Clean, but its test needs metric fields from fix: report native shuffle write metrics accurately #5370 and has to be adapted.
• fix: size JVM shuffle pointer array growth from the array, not the data pages #5907 The JVM shuffle sizes pointer-array growth from the data pages instead of the array, so it over-allocates and spills early. Part of perf: shuffle reader and writer performance review (native, JVM columnar, reader, Celeborn) #5905. Clean.

Open on main: candidates once merged

Each of these bugs was confirmed present in branch-1.0's code. The PR state and the cherry-pick result are as of 2026-09-24.

Wrong results

• fix(iceberg): don't push transform residuals as their source column, fail on residual errors #6154 Native Iceberg scans push a bucket, truncate or days residual to iceberg-rust as a predicate on the source column. Combined with AND, OR or NOT, this drops rows. Closes Iceberg serde: residual reflection failures can silently drop residual filtering #5992. The same code is on branch-1.0, but the repro has only been run on main. Conflicts in CometIcebergNativeScan.scala and the CI YAML.
• fix: fall back for ANSI and TRY integer sums over sliding windows #6069 Under ANSI or TRY, an integer SUM over a sliding window frame wraps on overflow, because sliding frames use DataFusion's wrapping sum. Closes Sliding-window SUM(BIGINT) ignores ANSI and TRY overflow semantics #6043. Approved. Clean.
• fix: match Spark variance updates for large nearby values #6076 var_pop, var_samp and stddev lose precision on large, closely spaced doubles. Closes Variance and standard deviation return incorrect results for large nearby values #6044. Approved. Drop the regr.rs hunk, which is main-only. The PR fixes the update path only; with more than one input partition the merge path still diverges.
• fix: match Spark ordering in native array extrema #5403 array_min / array_max pick the wrong signed zero, or the wrong one of several equal nested elements. Closes [Bug] array_min and array_max differ from Spark on signed-zero ties #5401. Approved. Needs adapting, because it uses the main-only hasConditionalNativeDefault.
• fix: dispatch map lookups with normalized keys and nondeterministic null-guarded children #5867 The IS NOT NULL guard serializes a nondeterministic child twice in size, array_append, arrays_zip and map_from_arrays. Closes size, arrays_zip, map_from_arrays and array_append return wrong answers for a nondeterministic child #5781. Approved. Take only the null-guard half; the map-lookup dispatch half closes the enhancement Map lookups with float, collated or complex keys fall back to Spark (map_col[key], element_at) #5580.
• fix: validate map constructor row lengths and null short-circuiting #5846 When map_from_arrays gets key and value arrays whose lengths differ per row but match in total, it shifts values across rows. Approved. Conflicts in map_funcs/mod.rs and maps.scala.
• fix: refuse codegen dispatch for a TRY cast that can put a null key in a map #6178 The codegen dispatcher writes a null map key as the type's default value, so a failing key cast under TRY_CAST returns [1, 0] instead of [1, NULL]. Closes Codegen dispatcher writes a null map key as the key type's default value #6172. Only the test conflicts.
• fix: make wide date-to-timestamp casts safe #5457 A wide DATE cast to TIMESTAMP_NTZ silently wraps in release builds, and a cast to TIMESTAMP panics. Closes Date-to-timestamp casts can overflow or panic for wide dates #5456. Waiting on the author since 09-17; 43 files.
• fix: preserve Spark AVG buffer and overflow semantics #5420 An empty native AVG partial exports (NULL, 0), which erases the result when Spark merges it, and decimal overflow does not stick. Closes Incorrect AVG result from empty native partial buffers #5418. After fix: revert unsafe partial aggregates after final fallback #5421.
• fix: make Iceberg delete-file reflection failures fatal #5515 When Iceberg delete-file reflection fails, the serde defaults to position deletes with spec 0 and no equality ids, so deletes are misapplied. Closes Iceberg serde: delete-file fields fall back to wrong defaults on reflection failure #5256. Approved. Only reachable when reflection fails.
• fix: fall back to Spark for negative-scale decimal casts and arithmetic #5050 With the legacy allowNegativeScaleOfDecimal flag, rescaling a negative-scale decimal wraps silently, and decimal to double loses precision. Closes Native panic casting to negative-scale decimal when spark.sql.legacy.allowNegativeScaleOfDecimal=true #5013. Only reachable with that flag.

Errors where Spark succeeds, or success where Spark raises

• fix: reject a file without field ids at any depth whether or not id matching is on #6116 Comet checks for files without field ids only at the root, and only with fieldId.read.enabled. So some reads return rows where Spark raises, and files with ids only on nested fields are rejected where Spark reads them. Closes Field id gating differs from Spark: root-only check and no dependence on fieldId.read.enabled #5936. Approved. Overlaps the Parquet field-matching stack above; conflicts in parquet_support.rs and schema_adapter.rs.
• fix: fall back to Spark for a TRY cast of a map key that can fail #6179 (draft) A TRY_CAST between maps whose key cast can fail errors natively with "Found unmasked nulls". Closes TRY_CAST on narrowing map keys fails where Spark returns a map with a null key #5995. Stacked on fix: refuse codegen dispatch for a TRY cast that can put a null key in a map #6178.
• fix: align nested collection buffer nullability before spilling #5903 collect_list / collect_set over structs with required fields fail when the final aggregate spills. Closes Native collect_set(struct) aggregate crashes on spill emit #5239. Approved. Take it with fix: make collect_list/collect_set argument coercion a normalization barrier #5159.
• fix: reject scalar subqueries inside a codegen-dispatch kernel #5923 A scalar subquery inside a codegen-dispatched ScalaUDF fails with "Subquery … has not finished". Approved. Clean.
• fix: support CalendarIntervalType hashing #5135 hash / xxhash64 over CalendarInterval fail natively. Closes Hashing a CalendarInterval value fails with "Unsupported data type in hasher: Interval(MonthDayNano)" #5059. Approved. Clean.
• fix: defer throwing literal casts to Spark at runtime #6035 A throwing literal cast in a branch that never runs fails the query, because Comet folds it at planning time. Changes requested.
• fix: honor the S3 profile name and file and Hadoop's addressing mode for custom endpoints #5872 The S3 profile name and file settings are ignored, and path.style.access is applied inverted for custom endpoints. Closes virtual_hosted_style_request bad calculation #2802 and Support fs.s3a.auth.profile.name and fs.s3a.auth.profile.file for ProfileCredentialsProvider #4245. Approved. Adds an aws-runtime dependency.
• fix: let explicit Hadoop Azure auth outrank ambient AZURE_* environment variables #6059 Ambient AZURE_* environment variables outrank the configured Hadoop Azure auth, giving a 403 or the wrong identity. Closes Native Azure store lets ambient AZURE_* environment variables override or corrupt explicit Hadoop auth config #5542. Clean, but about 2,400 lines in azure.rs.
• fix: native ansi-errors raised as spark error #5169 Decimal ANSI errors surface as plain Arrow errors, without Spark's error class or SQLSTATE. Closes Native ANSI errors raised as Arrow errors bypass SparkError conversion (wide decimal, decimal divide, decimal-to-decimal cast) #5072. Approved.
• fix: align wide-decimal ANSI overflow value with Spark (#5211) #5276 (draft) The wide-decimal overflow message reports a different value than Spark. Closes Wide-decimal overflow reports a different value than Spark #5211. Stacked on fix: native ansi-errors raised as spark error #5169. Message text only.
• fix: surface SparkArithmeticException(DIVIDE_BY_ZERO) for divide-by-zero in dispatched ScalaUDF path #4653 A wrapped ANSI divide-by-zero surfaces as CometNativeException instead of SparkArithmeticException. Closes Native divide-by-zero in a dispatched ScalaUDF surfaces CometNativeException instead of SparkArithmeticException #4517. Stalled; clean on branch-1.0.
• fix: preserve Spark errors for Parquet timestamp overflow #5891 A Parquet TIMESTAMP_MILLIS overflow surfaces as a raw Arrow error instead of Spark's file-read error. Closes Surface Parquet TIMESTAMP_MILLIS overflow as a Spark-faithful exception instead of a raw Arrow error #5517. Only meaningful after fix: prevent silent overflow when reading Parquet TIMESTAMP_MILLIS values #5177 and fix: check nested TIMESTAMP_MILLIS overflow in unfiltered scans #5740.

Resource use, hangs and plan quality

• fix: read shuffle write buffer, spill limit and off-heap sizes in bytes #6191 Three settings are read in the wrong unit. The native shuffle write buffer is 1 byte by default. A maxTempDirectorySize with a unit silently becomes 100 GiB. A bare spark.memory.offHeap.size is read as MiB. Closes spark.comet.shuffle.native.writeBufferSize is sent to native code in MiB but used as bytes, so the default write buffer is 1 byte #6183, spark.comet.maxTempDirectorySize silently ignores values with a unit #6184 and A bare byte count for spark.memory.offHeap.size is read as MiB when sizing the Comet memory pool #6185. Approved.
• fix: park the native scan loop instead of busy-polling while waiting on native I/O #6092 The native scan loop busy-polls, pinning a core per task while it waits on I/O with a JVM-fed input in the plan. Closes Native executePlan busy-polls at 100% CPU while a plan with a JVM-fed input waits on native I/O #6091. Under review; the loop as written can lose wake-ups.
• fix: preserve current AQE logical-stage links on Comet operators #5483 CometExecRule overwrites AQE logical-stage links, so AQE loses broadcast-join conversions. Results are correct. Closes CometExecRule overwrites direct AQE LogicalQueryStage links during replanning #5482. Approved.

Needs a decision

Merged on main

• fix: decode invalid UTF-8 at the JVM to native FFI import boundary #5310 decodes invalid UTF-8 at the JVM to native FFI import (part of [EPIC] Consistent handling of invalid UTF-8 in native StringType (ingress policy) #4764). It is a soundness fix, but it is about 600 lines, changes the output for invalid bytes, and adds a validation pass per batch.
• bug: slice crashes on arrays with a non-nullable element field #5743: slice over an array with non-nullable elements, such as slice(split(...)), fails with a type mismatch. The fix is the CometSlice hunk of feat: address remaining issues for CreateArray #5766 plus the tests in test: cover slice over expression-produced non-null element arrays (#… #5839. Not yet reproduced on branch-1.0.
• Backport candidates for 1.0.x: triage of every PR merged since branch-1.0 was cut #5815's tier 3, unchanged: fix: match Spark's null short-circuiting in array_join and enable it natively #5558, fix: normalize scalar float sort and window rank keys #5469, fix: distinguish "nothing spilled" from a spill backend with no local path #5726, fix: expand object store option references, uniquify constant metadata names, drop dead parquet JNI #5653, fix: RAII for memory pool registration  #5464 and fix: preserve Arrow Field metadata across C Data exports #5552. For fix: make CometExplodeExec respect batch size #5362, DataFusion 55.1 now carries the upstream UnnestExec batch-size fix ([branch-55] fix: make UnnestExec respect datafusion.execution.batch_size (#24384) datafusion#24529). branch-1.0's DataFusion 54.1 does not, so fix: make CometExplodeExec respect batch size #5362's 1,491-line operator is still the only route on 1.0.x.

Open on main

• fix: load the Iceberg storage scheme lists from the native factory #6065 (approved): the JVM claims gcs, abfs, abfss, wasb and wasbs Iceberg files that the native factory cannot open, so those tasks fail. Closes Iceberg native scan claims schemes it cannot execute; three scheme lists disagree #5541. The PR rewrites main-only code; on branch-1.0, narrowing the scheme Set in CometScanRule would be a one-line fix.
• fix: enforce null-key rejection and mapKeyDedupPolicy in native map construction #5854 (approved): a map built with a NULL key succeeds where Spark raises, and that half is a real bug on branch-1.0. The PR relies on DataFusion 55's map_key_dedup_policy, so 1.0 would need a separate null-key check. Closes [Bug] map_from_arrays / map_from_entries do not enforce null-key rejection or spark.sql.mapKeyDedupPolicy #4680 and str_to_map does not honour mapKeyDedupPolicy=LAST_WIN #5857.
• fix: gate array distinct and union signed-zero semantics by Spark version #5750 (approved): branch-1.0's DataFusion 54.1 keeps -0.0 and 0.0 distinct in array_distinct / array_union. That matches Spark through 4.1.3 and diverges on 4.2.0, 4.0.5+ and 4.1.4+. The PR's gate falls back exactly where 1.0 is already right; 1.0 would need the opposite gate. Closes array_distinct and array_union diverge from Spark on -0.0 for Spark versions without SPARK-54918 #5701.
• fix: fall back for Parquet datetime rebasing #5048: the native scan ignores legacy Julian/Gregorian rebasing, so pre-1582 dates and timestamps read shifted. This is documented as a limitation on 1.0. The fix adds a config and reads file footers on the driver at planning time. Closes Datetime rebase: track the documented scan limitation, and spark.comet.exceptionOnDatetimeRebase is dead code #5010.
• fix: preserve skipped unbase64 rows #5533: unbase64 can fail on rows that Spark skips under LIMIT or a semi-join. The fix adds a planner policy and a config. Closes unbase64 can fail on rows skipped by LIMIT and semi/anti joins #5532.
• fix: match Spark's duplicate field and field id semantics in parquet field lookup #5654 (changes requested): nested duplicate field ids silently take the first match. It builds on main's fix: reject duplicate Parquet field names before decoding #5786 and fix: fall back when a struct repeats a Parquet field id #6004, and field-id reads are off by default.
• perf: stop holding the fair pool lock across blocking memory calls #5613 (changes requested) and fix: account fair-pool memory across sibling reservations #5847 (draft): the default fair_unified pool holds its lock across the JNI acquire, which can hang, and forces spills below the fair share. perf: stop holding the fair pool lock across blocking memory calls #5613 builds on fix: let Comet memory pools overcommit on grow instead of panicking #6128.
• fix: use Spark-compatible shuffle for wide decimal hash keys #6005 (draft): native hash partitioning of decimals above precision 18 differs from Spark's. It depends on fix: revert unsafe partial aggregates after final fallback #5421, and Support Spark-compatible native hash and xxhash64 for decimals with precision >18 #5994 tracks the full fix.
• fix: prevent libhdfs thread destructor use-after-free #5036 (draft): the libhdfs thread destructor can use freed memory and crash at JVM teardown, and hdfs-opendal is a default feature on 1.0. The fix pins hdfs-sys to a personal fork through [patch.crates-io]. Closes Sporadic SIGSEGV in macOS [scans] workflow in CI #5023.
• fix: experiments with large types for aggregated values #4791: an aggregation task fails with offset overflow once the group keys exceed 2 GiB. The fix adds a config and proto changes. Stalled. Closes Comet aggregation task crashes with offset overflow #4718.
• fix: Native S3 scan on EKS/IRSA turns a transient STS throttle into a hard 403 storm #6025: a transient STS throttle on IRSA becomes a storm of 403s. The fix adds a provider module, dependencies and a config, and its Iceberg half is main-only. Closes Native S3 scan on EKS/IRSA turns a transient STS throttle into a hard 403 storm #6024.
• fix: do not run Comet in on-heap mode without spark.comet.exec.onHeap.enabled #6195 (approved): disables Comet when its extensions are registered without off-heap memory. That is a behaviour change for a patch release. Closes Comet runs in on-heap mode without spark.comet.exec.onHeap.enabled when the session extension is registered directly #6186.
• fix: string-to-timestamp does not trim ISO control characters, and leading '+' returns null under ANSI #5172 (draft): string to timestamp casts do not trim ISO control characters. It is stacked on perf: classify string-to-timestamp shapes in one byte scan (up to 5.6x faster) #5130, and branch-1.0 lacks fix: align string to timestamp parsing with Spark's segment rules #5682. Not yet verified on 1.0. Closes fix: string-to-timestamp does not trim ISO control characters, and leading '+' returns null under ANSI #5165.
• fix: report Input column when native Iceberg scan is enabled or native shuffle is enabled #5265 (approved): the Input column reads 0 with the native Iceberg scan or native shuffle. It is metrics-only, which Backport candidates for 1.0.x: triage of every PR merged since branch-1.0 was cut #5815 excluded, but it is in the 1.0.1 milestone. Closes Reading Iceberg tables show 0.0B for the Input column on Spark's executor UI #5264.

Already on branch-1.0

• fix: normalize nested field nullability in ShuffleScanExec and ExpandExec #5138, fix: skip null slots when checking overflow in unary negation #5162, fix: surface next_day and make_date ANSI errors as Spark exceptions #5167 and fix: propagate the Spark task ClassLoader to JVM UDF calls #5282, via fix: [branch-1.0] backport four bug fixes for 1.0.1 #5321
• feat: remove native cast from boolean to decimal #5185, fix: match Spark whitespace trimming in to_time and try_to_time #5364, fix: canonicalize NaN in flat arrays_overlap float keys #5376, fix: Native shuffle fails with a 2GB task serialization OOM on jobs with many partitions #5392 and fix: support wide years in native make_date #5443, via fix: [branch-1.0] backport five bug fixes for 1.0.1 #5480, which also took chore: fix clippy warnings for Rust 1.98 #5400 to keep clippy green
• fix: keep the dictionary hash fast path off nested and reseeded buffers #5757, via fix: [branch-1.0] keep the dictionary hash fast path off nested and reseeded buffers (#5757) #5817
• fix: apply the parent struct's null mask before hashing its fields #5754, via fix: [branch-1.0] apply the parent struct's null mask before hashing its fields (#5754) #5823

The other backports since the cut are #5192 (#5261), #5315 (#5316), #5737 (#5820) and #5782 (#5833).

Not candidates

#5815 lists everything excluded before 2026-09-09. Since then:

• Fixes code that only exists on main.
  • The native Iceberg writer: fix: roll native Iceberg data files on iceberg-java's 1000-row grid #5780, fix: list a fanout Iceberg write's data files in a stable order #5810, fix: Delete completed tasks' data files when an Iceberg write job fails #5663, fix: render float and double Iceberg partition values like iceberg-java #5840, fix: fall back to Spark for native Iceberg writes to gs:// through HadoopFileIO #5935, fix: wrap Iceberg split-write failures the way Spark does when abort fails #6153 and fix: plan Iceberg writes with Spark's own operator when Comet is disabled #6151.
  • Native Iceberg scan metrics: fix(iceberg): guard native Iceberg scan driver-metric double-post, add metrics docs and tests #6085.
  • regr_*: fix: gate the regr_r2 degenerate-case swap on the Spark patch release #6042.
  • Join runtime filters: fix: preserve Parquet conversion errors during join filtering #6067.
  • The libhdfs routing regression from feat: support S3 compliant filesystems #5314: fix: decide libhdfs routing from the scheme as written #5825.
  • Open PRs: fix: bound shuffle schema cache retention and preserve eviction order #6098, feat: full native make_interval with 2.6x faster than spark and 1.1x faster than codegen dispatch #5292, fix: validate Parquet VARIANT annotation against the requested read type #5932, fix: preserve Spark evaluation for next_day and levenshtein #5972, fix: preserve array_join evaluation semantics across Spark versions #5851 and fix: honor Spark’s commit protocol in Spark 3.x native writes #4746.
• Off by default, or latent on branch-1.0.
  • fix: write cached batches to the schema width, not the batch width #6090, which only matters behind sparkToColumnar. It is also the crash follow-up to perf: bulk copy fixed-width columns in ArrowWriter #5442, so take it along if perf: bulk copy fixed-width columns in ArrowWriter #5442 from Backport candidates for 1.0.x: triage of every PR merged since branch-1.0 was cut #5815's tier 2 is backported.
  • Experimental features: fix: decode dictionary input for PyArrow UDFs #5560 (PyArrow UDFs) and feat: hook native Parquet writes into Spark's WriteFilesExec seam on Spark 4.0+ #5763 (WriteFilesExec).
  • Open PRs: fix: get_json_object returns first value for duplicate keys to match Spark #4971, fix: restore Spark write execs when reverting transition-heavy stages #5957, fix: preserve DPP filters during transition revert #6137 and fix: reverse ordering and range in NegativeExpr::get_properties #5982.
• Closes an enhancement, or only changes when Comet falls back. fix: support Utf8/LargeUtf8/Utf8View in native RLike without panicking #5215, fix: preserve map field metadata and honor target sorted flag in cast_map_to_map #5227, fix: keep Iceberg complex null checks on native scans #5732 and perf: spill every shuffle partition of a task into one file #5916, and the open fix: preserve duplicate named_struct fields in codegen dispatch #5603, fix: route length/bit_length/octet_length binary input through codegen dispatcher #5607, fix: support struct-typed scalar subquery results #5889, fix: stop gating columnar shuffle on native-serde checks it never uses #6110, feat: built-in S3 credential provider adapters for the native Parquet scan #6023, feat: add native Parquet datetime rebasing #5047, fix: reject casts involving non-default collated strings #5302 and fix: remove misleading native opt-in for dispatch-only datetime expressions #6182.
• Explain output, tooling, error class or metrics only. fix: ignore structural tags when lifting expression coverage #5471, fix: explain ObjectHashAggregate fallback when Comet shuffle is disabled #5746, fix: count each memory pool once in analyze_trace #5991 and fix: preserve Spark row index read errors #6046, and the open fix: report task input metrics after the native iterator closes and add to Spark's counters #5880 and fix: cancel background batch producers before final metrics #5505.
• A behaviour change. fix: remove the ineffective spark.executor.memoryOverhead adjustment from the driver plugin #6054.
• Blocked on an unreleased DataFusion fix. fix: preserve Spark Parquet schema conversion errors (WIP) #5958.
• Test, CI, docs, bench or dependency only. fix: preserve nulls for Boolean/Byte/Short/Integer columns in FuzzDataGenerator #5855 and the rest of the test:, ci:, docs:, chore:, bench: and deps: commits, and the open test: wait for Parquet write plan callbacks #6108, ci: retry the scalafix lint when Maven cannot resolve the plugin #6061, fix: disable conflicting IfBraceChecker #5721, chore: run CI on contributor forks #4570 and fix: keep parquet view types out of the shuffle benchmark input #5804.

Additional context

How the lists were built:

• Merged. Every fix: commit in 424c31aa7..67803a7a4, plus any other commit that closes a bug issue. Each one was classified by the labels on the issue it closes, not by its prefix. Applicability was tested with git merge-tree against branch-1.0, and every conflicting path was checked for existence there. A conflict in a file that branch-1.0 lacks means the fix is for code that only exists on main.
• Open. Every open PR against main whose title starts with fix, or that closes a bug issue: 67 PRs. For each one, the defective code was confirmed on branch-1.0 with git grep, and the PR head was test-merged onto branch-1.0.
• Not build-verified. Each backport PR needs cargo check --workspace --all-targets and ./mvnw test-compile on branch-1.0 before it goes up. A clean cherry-pick is not a compiling one. fix: release native shuffle reservation after spill failure #5461 is the known case, and the usual causes are a test module's shared use block or a proto struct literal.
• Rust version. branch-1.0 CI builds with Rust stable. If clippy goes red on code the backport did not touch, include the newest chore: fix clippy warnings commit from main.
• Dependencies. branch-1.0 is on DataFusion 54.1.0 and Arrow 58.4.0. main is on DataFusion 55.1.0 and Arrow 59.2.0.

Still open in the 1.0.1 milestone, with no fix on main: #4412 (CometHashAggregateExec and AQEPropagateEmptyRelation) and the enhancement #5015.