[EPIC] Timezone handling bugs

[andygrove]

What / Why

I audited how Comet handles timezones, starting from #2730. The model is simple and mostly sound. Spark's TimestampType is a UTC instant, so nothing is converted at the JVM/native boundary. Comet passes the raw microseconds in both directions and labels them Timestamp(Microsecond, "UTC"). TimestampNTZType is Timestamp(Microsecond, None). The session timezone never becomes part of a value. Each timezone-aware expression carries it, and it's applied inside the native kernel, or the expression runs through the codegen dispatcher with Spark's own timeZoneId. #6337 adds a contributor guide page that describes the model in more detail.

The bugs cluster where that model breaks down:

• native expressions that emit a TimestampType value with some other label
• session timezone IDs that the native parser can't read
• timezone rules that come from a different database than the JVM's

Label drift is easy to miss. The scan and shuffle boundaries cast every column back to its declared type, so a test that only projects the result passes. It shows up when the result is compared, goes through a CASE, or feeds another native expression.

All of the new bugs below reproduce on main at 764936187, on Spark 3.5 and 4.1.

Bugs

• Native timestamp_seconds returns a TIMESTAMP_NTZ-typed array, so downstream expressions ignore the session timezone #6328 timestamp_seconds returns a TIMESTAMP_NTZ-typed array, so hour, casts and comparisons over it are wrong or fail in non-UTC sessions (critical). Fixed by fix: label native timestamp_seconds results as UTC timestamps #6341.
• Native date_trunc labels its output with the session timezone, so comparing it with another timestamp fails in Etc/UTC sessions #6330 date_trunc labels its output with the session timezone, so comparing it with another timestamp fails in Etc/UTC sessions, the default on Ubuntu and Debian images (high). Fixed by fix: keep the input's timezone label on native date_trunc results #6345.
• CASE and COALESCE over timestamps panic when the branches have different Arrow timezones #6327 CASE and COALESCE over timestamps panic when the branches have different Arrow timezones (high). Fixed by fix: relabel CASE and COALESCE timestamp branches instead of panicking #6347.
• Session timezone IDs such as GMT+8, Z and PST make native timestamp expressions fail #6329 session timezone IDs such as GMT+8, Z and PST make casts, hour and df.show() fail (high). Fixed by fix: normalize session timezone IDs before passing them to native code #6351.
• timestamp_trunc panics on DST-transition timestamps in a DST timezone #5633 timestamp_trunc panics on DST-transition timestamps in a DST timezone. Native only with allowIncompatible (high). feat: Use DataFusion date_trunc for scalar-format timestamp truncation #5956 fixed it for literal formats. fix: truncate per-row date_trunc formats with the literal-format kernel #6354 is being reworked to cover formats that come from a column.
• Native timezone rules come from chrono-tz's bundled tzdata, so results differ from Spark when the JVM's tzdata differs #6331 native timezone rules come from chrono-tz's bundled tzdata (2025b), not the JVM's (medium). fix: warn when native and JVM timezone databases differ #6353 logs a warning when the two versions differ.
• Native CSV V2 scan parses timestamps as UTC and ignores the session timezone #6332 the native CSV V2 scan ignores the session timezone (low, testing-only config). Fixed by fix: fall back from the native CSV scan for timestamps outside UTC #6349.
• days transform is evaluated in the session timezone, while hours and Iceberg use UTC #6333 days is evaluated in the session timezone, while hours and Iceberg use UTC (low). Fixed by fix: count the days partition transform in UTC #6348.

Status (2026-10-07)

Seven of the eight bugs are fixed on main. #6351 also replaced every getOrElse("UTC") fallback with CometTimeZone.nativeId, and #6347 turned the asserts in array_with_timezone into errors, so the section on #2730 below describes the code before those changes.

For #5633, #5956 replaced the literal-format kernel. It truncates in local time the way Spark does, follows LocalDate.atStartOfDay for the date levels, and handles the whole timestamp range in UTC sessions. When the format comes from a column, date_trunc still goes through the old helpers and can panic, and #6354 is being reworked on top of #5956 to cover that path.

#6686 brought the contributor guide page up to date, and #6340 points the PR review skills at it. Both are merged.

Follow-ups:

• fix: match Spark 4.2 date_trunc overflow for SECOND and MILLISECOND #6740 makes date_trunc raise long overflow for SECOND and MILLISECOND on Spark 4.2, which changed truncTimestamp to use Math.subtractExact. Until it merges, the overflow test feat: Use DataFusion date_trunc for scalar-format timestamp truncation #5956 added fails on the 4.2 profile (date_trunc SECOND/MILLISECOND wraps on Spark 4.2, where Spark raises long overflow #6739, nightly Nightly CI failed on 2026-10-07 #6744).
• Vectorize the lower-bound check in native date_trunc for fine units #6717 vectorizes the lower-bound check feat: Use DataFusion date_trunc for scalar-format timestamp truncation #5956 added for the fine units.
• There's still no systematic check that compares each native timestamp-returning expression with another timestamp, or feeds it to another native expression, in both Etc/UTC and a non-UTC session. timestamp_seconds (fix: label native timestamp_seconds results as UTC timestamps #6341) and date_trunc (fix: keep the input's timezone label on native date_trunc results #6345) got tests that compare their results, and session_timezone_ids.sql (fix: normalize session timezone IDs before passing them to native code #6351) covers the GMT+8-style IDs.

The UTC fallbacks in #2730

I instrumented every timeZoneId.getOrElse("UTC") site. Then I ran the datetime, cast, SQL-file, JSON, CSV, fuzz and expression suites. About 11,700 serde calls happened across 1,020 tests, and about 900 of them arrived without a timezone. Every one of those was a cast that Spark doesn't consider timezone-sensitive: numeric casts, Comet's own nullability-widening casts, and the cast inside IntegralDivide. None was a timezone-aware expression, which fits Spark refusing to resolve one without a timezone. So the fallback isn't a correctness bug today. The helper proposed in #6329 would replace it. The fallback can't simply be removed, though, because array_with_timezone asserts a non-empty timezone even for casts that don't use one.

Related

• Date-to-timestamp casts can overflow or panic for wide dates #5456 date-to-timestamp casts overflow or panic for wide dates (fix: make wide date-to-timestamp casts safe #5457 is open)
• Datetime rebase: track the documented scan limitation, and spark.comet.exceptionOnDatetimeRebase is dead code #5010 datetime rebasing isn't supported (feat: add native Parquet datetime rebasing #5047 and fix: fall back for Parquet datetime rebasing #5048 are open). Spark's legacy rebase is itself timezone-dependent.
• Evaluate moving from chrono to Jiff #4754 moving from chrono to jiff
• DataFusion / DataFusion-Spark functions whose Arrow return type drifts from Spark catalyst's declared type #4515 return type drift, where Native timestamp_seconds returns a TIMESTAMP_NTZ-typed array, so downstream expressions ignore the session timezone #6328 and Native date_trunc labels its output with the session timezone, so comparing it with another timestamp fails in Etc/UTC sessions #6330 are the timestamp cases with consequences inside a stage
• Audit Spark SQL configs that affect query semantics and ensure Comet honors them (or falls back) #4180 has an item for honoring spark.sql.session.timeZone everywhere

Already documented: Python Arrow UDFs see timestamps labelled UTC rather than the session timezone, and chrono-tz's DST rules end around 2100. Not in the user guide yet: spark.sql.parquet.int96TimestampConversion=true disables Comet for the session.

Fixed earlier, same class: #2720 (SparkToColumnar labelled timestamps with the session timezone), #2649 via #4761 (the date_trunc schema mismatch, whose fix introduced the label in #6330), and #5556 (the Python runner accepts Etc/UTC for UTC).

Test gaps

The SQL-file tests use UTC, America/Los_Angeles, America/New_York, Asia/Kolkata and +05:30. None of them use Etc/UTC, or the offset and short-ID forms from #6329. Most expression tests only project their result. Adding GMT+8 to the datetime files' ConfigMatrix would have caught #6329. #6328, #6330 and #6327 need more than a timezone setting: a test that compares each native timestamp-returning expression with another timestamp, or uses it in a CASE, in both a non-UTC session and Etc/UTC.