Support Spark's `AtLeastNNonNulls` natively

[hsiang-c]

What is the problem the feature request solves?

When I enabled Comet in our workload, I found that AtLeastNNonNulls is not supported yet so I wired it via the CometCodegenDispatch.

object CometAtLeastNNonNulls extends CometCodegenDispatch[AtLeastNNonNulls]

It turns out that doing so is slower (~ 8% more in total runtime) than falling back to Spark on AtLeastNNonNulls.

Describe the potential solution

Support AtLeastNNonNulls with native Rust counterpart.

Additional context

No response

[rich7420]

@hsiang-c hi , could I try this?

[hsiang-c]

@rich7420 Please feel free to take it

[rich7420]

@hsiang-c I have a native prototype that is faster than codegen dispatch in local benchmarks, but some na.drop(...).count() cases still favor Spark fallback. Could you share a representative query, schema, threshold, and approximate NULL/NaN rates so I can test your workload?

[rich7420]

take

[rich7420]

Progress update: the native AtLeastNNonNulls implementation is being tracked in #6180, which remains a draft.

Following review feedback, I’m separating the general filter output projection optimization into its own PR. I’m also preparing a separate IF optimization to address an end-to-end performance regression observed while validating the native execution path.

The combined changes are currently undergoing CI validation in my fork. Once validated, I’ll submit the standalone optimizations and update #6180 with the review follow-ups.

A possible contribution of the reusable functionality to datafusion-spark is being tracked separately as follow-up work.