Repository navigation
Review use of hard-coded UTC assumptions #2730
Description
What is the problem the feature request solves?
This code pattern worries me. We should fall back to Spark if we do not know the timezone.
spark/src/main/spark-3.5/org/apache/comet/shims/CometExprShim.scala: .setTimezone(timeZoneId.getOrElse("UTC"))
spark/src/main/spark-4.0/org/apache/comet/shims/CometExprShim.scala: .setTimezone(timeZoneId.getOrElse("UTC"))
spark/src/main/scala/org/apache/comet/expressions/CometCast.scala: castBuilder.setTimezone(timeZoneId.getOrElse("UTC"))
spark/src/main/scala/org/apache/comet/serde/structs.scala: .setTimezone(expr.timeZoneId.getOrElse("UTC"))
spark/src/main/scala/org/apache/comet/serde/datetime.scala: val timeZone = expr.timeZoneId.getOrElse("UTC")
spark/src/main/scala/org/apache/comet/serde/datetime.scala: val timeZone = expr.timeZoneId.getOrElse("UTC")
spark/src/main/scala/org/apache/comet/serde/datetime.scala: val timeZone = expr.timeZoneId.getOrElse("UTC")
spark/src/main/scala/org/apache/comet/serde/datetime.scala: val timeZone = expr.timeZoneId.getOrElse("UTC")
Describe the potential solution
No response
Additional context
No response
Activity
andygrove commentedon Nov 7, 2025
Also
serde.rsto_arrow_datatypehas hard-codedUTC.andygrove commentedon Oct 7, 2026
All of the
timeZoneId.getOrElse("UTC")sites listed here are gone. #6351 replaced them withCometTimeZone.nativeId, which normalizes the timezone each serde passes to native code and reports the expression as unsupported when native code can't represent its timezone, so it falls back or runs in the codegen dispatcher. An expression with no timezone still gets"UTC", but Spark only leaves the timezone unset on casts that don't use it, which the audit in #6335 confirmed across the test suites.The hard-coded
UTCinto_arrow_datatypeis intended. EveryTimestampTypevalue in a native plan is labelledTimestamp(Microsecond, "UTC"), and the session timezone travels with each expression instead of with the values.docs/source/contributor-guide/timezones.mddescribes that model. Closing.