[EPIC] Regressions in 1.2.0 since 1.1.0 #7204

What / Why

This epic tracks functional regressions in 1.2.0 against the 1.1.0 release. They come from an audit of branch-1.2 (head 889856d) against the 1.1.0 tag. I'll keep this description updated as fixes and backports land.

A regression here is something that worked in 1.1.0 with default configs and is broken or worse in 1.2.0. That covers a wrong or different result, a new failure, a loud failure that became a silent wrong answer, or a severe slowdown on a default path. Bugs that 1.1.0 already had are not listed. Neither are bugs in features that are off by default.

Each issue below carries the regression label.

Regressions on branch-1.2

• Text broadcast build sides and string min/max apply the invalid UTF-8 decode where 1.1.0 stayed on Spark #7118: invalid UTF-8 bytes are replaced with U+FFFD in two paths that 1.1.0 kept on Spark. One is a Text-format broadcast build side, converted by feat: run broadcast joins natively when an unsupported scan is on the build side  #6786 (spark.comet.convert.broadcastBuildSide.enabled, on by default). The other is min/max over a string cast from binary, now native through feat: run min and max over string and binary columns natively #6927. The fix for the build side is fix: keep Text build sides on Spark by default and add Text conversion configs #7123, which keeps Text build sides on Spark by default. It is merged to main, and that closed Text broadcast build sides and string min/max apply the invalid UTF-8 decode where 1.1.0 stayed on Spark #7118, but it is not yet on branch-1.2, so 1.2.0 still ships the bug until it is backported. The min/max case falls under the ordering difference the compatibility guide already documents.
• RANGE window frame boundary overflow diverges from Spark for DATE and DECIMAL ORDER BY #5022: feat: native RANGE window frames with explicit offset on DATE ORDER BY #4974 made RANGE frames with an explicit offset on a DATE ORDER BY native; 1.1.0 fell back for any such frame. The overflow guard only falls back above 2,144,550,751 days. An offset whose bound overflows Date32 lets a NULL peer group into the frame. With an offset of 100000000 days, Spark gives 2,2,1,2 and 1.2.0 gives 2,4,3,2. The fix is to lower the threshold to about 92M days. The issue is on the 1.3.0 milestone and should move to 1.2.0.
• Native PartialMerge over input sorted on a prefix of its keys fails to spill on DataFusion 55.2 #6880: fix: keep one row per key in a native PartialMerge that runs short of memory #6867 runs a native PartialMerge as Single, which DataFusion 55.2 cannot spill when the input is sorted on a prefix of the grouping keys. A count(DISTINCT k) over input sorted on the group key now fails the task. In the audit it failed at every pool size from 2 MiB to 256 MiB, with Additional allocation failed for GroupedHashAggregateStream. 1.1.0 returned the right count there when keys were unique, and a wrong count when keys repeated (Native PartialMerge aggregate emits duplicate keys under memory pressure, so count(DISTINCT) over-counts #6857).
• Native string and binary min/max under-report memory in hash aggregates #6936: feat: run min and max over string and binary columns natively #6927 made string and binary min/max native, where 1.1.0 fell back. DataFusion's MinMaxBytesAccumulator under-reports its memory, so these aggregates spill late and can run out of memory. Results are not affected.
• Native ABFS scans fail every task instead of falling back when Hadoop declines the auth type #7202: fix: resolve Azure ABFS authentication through Hadoop and keep ambient credentials out #6059 resolves ABFS authentication through Hadoop on the driver. When the driver declines an auth type (Custom, UserboundSASWithOAuth, custom SAS providers, RefreshToken and UserPassword providers), the scan still plans natively and every task fails. The same happens when the driver cannot reflect into hadoop-azure, which also hits users who configure Azure only through environment variables. 1.1.0 read these natively through the ambient identity. This was found by reading the code.
• collect_set over floats beside count(DISTINCT) diverges from Spark on signed zeros and NaN #7203: feat: add native distinct-combined collect_list aggregate support #4727 made collect_set native in count(DISTINCT ...) chains. On Spark before 4.2, the documented collect_set floating-point difference reaches that shape: -0.0 and 0.0 stay as two entries, and NaNs merge into one. 1.1.0 kept the shape in Spark and matched it.

Behavior changes for the release notes

These are intended, but a user upgrading from 1.1.0 can see them.

• chore: move native write configs under spark.comet.write #6494 renamed spark.comet.parquet.write.enabled and spark.comet.iceberg.write.enabled to spark.comet.write.parquet.enabled and spark.comet.write.iceberg.enabled, with no alias. Users who opted in on 1.1.0 under the old names now silently get Spark's writers.
• fix: normalize float operands in the native comparison builder #6447: on a Parquet DOUBLE or FLOAT column with a bloom filter, d = 0.0, d <=> 0.0 and d IN (0.0, ...) against a file that holds only -0.0 now return the row. Spark's own pushdown, and 1.1.0, skip the file and return nothing. The 1.2.0 answer is the correct one, and compatibility/floating-point.md documents it.
• fix: preserve Spark AVG buffer and overflow semantics #5420: under ANSI, AVG(DECIMAL(38,0)) overflow now raises NUMERIC_VALUE_OUT_OF_RANGE where 1.1.0 returned NULL. This matches Spark.
• perf: share one decoded copy of a broadcast among the tasks of an executor #7016 shares one decoded broadcast per executor, through a new value class. CometKryoRegistrator registers it. Only a user who lists Comet's classes by hand under spark.kryo.registrationRequired needs to add it.

Audit scope

• Baseline: every fix in 1.1.0 has its original on branch-1.2. That includes the four PRs merged straight into branch-1.1: fix: [branch-1.1] fall back for rank limits over nested float keys (#6468) #6487, fix: [branch-1.1] fall back to Spark for _metadata.file_block_start and file_block_length (#6510) #6511, fix: [branch-1.1] make adaptive aggregation skipping opt-in (#6474) #6488 and fix: [branch-1.1] build with Rust 1.99, which deprecates the legacy f64 constants and fetch_update (#6507) #6526.
• Dependencies: DataFusion moved from 55.1.0 to 55.2.0 (deps: bump to datafusion 55.2.0 #6766). arrow and parquet stay at 59.3.0. iceberg-rust moved from bb1e4a4 to 1f3bc34 (fix: make the native Iceberg per-I/O timeout configurable #6649, feat: bump iceberg-rust and write float and double identity partitions natively #6851), and both ranges were reviewed.
• Phase 1: I triaged 251 production PRs from the 388 commits in 36ab57c68..branch-1.2, reading the code. 3 rated HIGH and 31 MEDIUM.
• Phase 2: every HIGH and MEDIUM PR was checked against Spark with targeted tests on the Spark 4.1 profile. So were the 14 PRs from an earlier pass on 2026-10-19: perf: store shuffle offsets as lengths with zstd #6943, perf: convert Spark's unsafe rows to Arrow straight from their memory #6730, feat: run global aggregates with no grouping keys or functions natively #6899, feat: run min and max over string and binary columns natively #6927, feat: wire native existence join support #4587, perf: optimize native CASE WHEN and IF (up to 12x faster) #6350, fix: match Spark statistical aggregate updates and merges #6076, feat: run broadcast joins natively when an unsupported scan is on the build side  #6786, perf: read only the nested fields Spark keeps in the native Iceberg scan #6725, feat: support PivotFirst aggregate for the optimized PIVOT fast path #4798, perf: reuse validity buffers and use unary kernels in native expressions #6953, perf: optimize JVM columnar-to-row conversion #5496, feat: native RANGE window frames with explicit offset on DATE ORDER BY #4974 and feat: bump iceberg-rust and write float and double identity partitions natively #6851. The PRs that showed no regression are:
  • Plans and aggregation: feat: support SortAggregateExec #4565, feat: convert OneRowRelation to Arrow by default #6666, fix: keep a streamed side sorted outside the native plan in order through outer hash joins #6835, fix: preserve Spark AVG buffer and overflow semantics #5420, perf: read the shuffle natively in operators AQE reuses from the initial plan #6547, fix: refresh stale shuffle scans when AQE prepares a stage, not while it re-plans #6646, fix: coalesce shuffle partitions under Comet unions the way Spark does #6459, fix: publish metrics on the update interval for native blocks without a JVM input #6314.
  • Floating point: fix: match Spark's float ordering in min, max, greatest and least #6457, fix: match Spark's float semantics in array_remove and sort_array #6518, perf: compare floats in Spark's order without normalizing copies of the operands #6691, feat: run array_contains on float elements natively with Spark's equa… #6599, fix: normalize float operands in the native comparison builder #6447, fix: preserve Spark nested null ordering in native sort #6968, feat: normalize floats in native array_distinct and array_union #6563.
  • Expressions: perf: classify string-to-timestamp shapes in one byte scan (up to 5.6x faster) #5130, feat: Use DataFusion date_trunc for scalar-format timestamp truncation #5956, fix: match Spark ordering in native array extrema #5403, fix: skip an array or map function's later arguments where the array or map is NULL #6716, feat: enable native replace for safe inputs #6680, feat: support AtLeastNNonNulls natively #6180, feat: support multiply_ym_interval with YearMonth interval codegen dispatch #4715.
  • Shuffle and spill: perf: bulk-append long nullable primitive arrays in SparkUnsafeArray conversion #6824, perf: write every partition of a JVM shuffle spill in one native call #7065, feat: support native wide-decimal hashing #6548, feat: add a spill compression codec config, defaulting to lz4 #6698.
  • Scans and object stores: perf: load the next Parquet file's footer and page index while the current file is read #7094, refactor: share Parquet field-id and name resolution between root and nested fields #6990, fix: read each native scan partition through the object store its files live in #6753.
  • Memory and broadcast: fix: return a native plan's memory before its Spark task ends #6261, perf: share one decoded copy of a broadcast among the tasks of an executor #7016.
  • Iceberg reads: perf: default the native Iceberg scan to two concurrent data files #7132, perf: cache Iceberg FileIO per executor instead of building one per task #6106, fix: make the native Iceberg per-I/O timeout configurable #6649.
• Phase 3: I classified the 73 open bugs filed since the 1.1 branch cut. Only Native PartialMerge over input sorted on a prefix of its keys fails to spill on DataFusion 55.2 #6880 and Native string and binary min/max under-report memory in hash aggregates #6936 are 1.2 regressions. JVM columnar shuffle fails with ArrayIndexOutOfBoundsException when spark.shuffle.checksum.enabled=false #6256 (JVM columnar shuffle with spark.shuffle.checksum.enabled=false) looks fixed on branch-1.2 by perf: write every partition of a JVM shuffle spill in one native call #7065.
• Not covered: Spark profiles other than 4.1 were mostly not run, and nothing ran against real cloud object stores or Celeborn.