feat: support interval codegen dispatch for nested values and native shuffle

[peterxcli]

Which issue does this PR close?

Closes #4975.

Rationale for this change

YearMonthIntervalType and DayTimeIntervalType values fall back to Spark when they cross codegen boundaries as nested values or native-shuffle payloads because their Arrow vectors and getter paths are not dispatched.

What changes are included in this PR?

• Add interval handling across scalar and nested codegen getters.
• Support intervals at Comet sink boundaries and in native-shuffle payloads.
• Add SQL regressions for both interval types.
• Leave interval file scans and JVM shuffle unchanged.

How are these changes tested?

• make core
• CometSqlFileTestSuite make_ym_interval on Spark 4.1 and Spark 3.5
• CometSqlFileTestSuite make_dt_interval on Spark 4.1 and Spark 3.5
• Spotless and git diff --check

[andygrove]

The array-of-interval and top-level native-shuffle cases are nice coverage.

Since supportedSinkDataType also allows intervals nested inside StructType and MapType, would it be worth adding a struct-with-interval-field and a map-with-interval-value case too? The nested struct field readers in CometBatchKernelCodegenInput do handle the interval cases, so the paths are probably already wired. A struct and a map test would confirm those permitted paths actually round-trip through the sink and shuffle, rather than just being allowed by the predicate.

[andygrove]

Thanks for this. The interval plumbing is well scoped and the coverage looks good.

CI: The one red check (macOS [expressions]) is a transient infra flake, not a real failure. All 1167 tests passed and the build then died in the shade step resolving org.ow2.asm:asm with Unknown host repo.maven.apache.org. A re-run should clear it.

One cleanup I'd ask for before merge: in CometBatchKernelCodegenInput.scala, classOf[DurationVector] gets added twice even though it was already there in the base. It appears in the primitiveArrowClasses set (already at line 65, added again next to IntervalYearVector) and in the longCases collect (already present a few lines up, added again). The Set dedupes and the duplicate || is a no-op, so it is harmless, but it reads as a copy-paste slip. DayTimeIntervalType was already covered through those existing DurationVector entries, so I think only IntervalYearVector needs to stay and the two duplicate lines can be dropped. Was a different vector intended there?

On correctness: the mapping looks right. YearMonthIntervalType routes to getInt (Arrow Interval(YEAR_MONTH), int months) and DayTimeIntervalType to getLong (Arrow Duration(MICROSECOND), int64 micros), matching Utils.scala:161-164. Matching on _: YearMonthIntervalType / _: DayTimeIntervalType correctly covers all the start/end field variants since they share the same physical storage. The new recursive supportedSinkDataType never hands a complex type to the non-allowComplex base call, so it should be equivalent to the old check for non-interval nested types with intervals now allowed at the leaves.

Tests: glad to see the struct and map cases added. One small thing: the null interval is only exercised in the array case. Would it be worth adding a null interval at the top level, and in a struct field and map value, so the native-shuffle path sees nulls too?

Nice work.