Repository navigation
Conversation
| } | ||
|
|
||
| // Spark treats all NaNs as equal and -0.0 as equal to 0.0. | ||
| macro_rules! float_eq { |
There was a problem hiding this comment.
#2545 also adds float equality logic in spark_float_cmp. The comparison logic could be added to native-engine/datafusion-ext-commons/src and shared across both cases since the float checking is not scoped to just this function. Here the array contains check could check if spark_float_cmp returns Ordering::Equal.
There was a problem hiding this comment.
Thanks for the suggestion. I've added spark_float_cmp to datafusion-ext-commons and reused it in array_contains, with Float/Double tests for NaN, signed zero, and ordering.
| try { | ||
| exec.setTagValue(convertibleTag, true) | ||
| convert(exec) | ||
| Shims.get.setLogicalLink(convert(exec), exec) |
There was a problem hiding this comment.
Agreed. I've removed the tryConvert change and the logical-link assertion, leaving that fix to #2492. The fallback test now executes each DataFrame once while still checking both fallback modes and the expected rows～
3 participants
Which issue does this PR close?
Closes #2572
Rationale for this change
array_contains currently falls back to Spark execution.
What changes are included in this PR?
Add native array_contains support for supported atomic element types. Match Spark's null, NaN, and signed-zero semantics, and reuse scalar arguments without expanding constant arrays for each input row.
Keep unsupported element types on Spark execution and test both fallback configurations.
Are there any user-facing changes?
array_contains can run natively for supported atomic element types.
How was this patch tested?
cargo test -p datafusion-ext-functions --locked../build/mvn -Ppre -Pspark-3.5 -Pscala-2.12 -pl spark-extension-shims-spark -DskipBuildNative -Dsuites=org.apache.auron.AuronFunctionSuite test../dev/reformat.Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-6)