feat: route translate and to_csv through codegen dispatch by default#5032
Open
andygrove wants to merge 1 commit into
Open
feat: route translate and to_csv through codegen dispatch by default#5032andygrove wants to merge 1 commit into
andygrove wants to merge 1 commit into
Conversation
`translate` (StringTranslate) and `to_csv` (StructsToCsv) have native implementations that are incompatible with Spark by default, so today the whole enclosing operator falls back to Spark unless allowIncompatible is set. Route them through the JVM codegen dispatcher by default instead, using the same NativeOptInAvailable pattern as `to_json` and `split`: the dispatcher runs Spark's own generated code (bit-exact) while keeping the operator on the native pipeline, and the faster native path stays available via allowIncompatible. Update the translate SQL test (previously asserting fallback) to verify native dispatch, and add a to_csv test.
andygrove
force-pushed
the
feat/incompat-codegen-dispatch
branch
from
July 24, 2026 23:01
7cd0e22 to
3abb614
Compare
andygrove
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #.
Rationale for this change
translate(StringTranslate) andto_csv(StructsToCsv) have native implementations that are incompatible with Spark by default, so today the whole enclosing operator falls back to Spark unless allowIncompatible is set. Routing them through the JVM codegen dispatcher instead keeps the operator on the native pipeline while producing bit-exact Spark results, and preserves the faster native path as an opt-in.What changes are included in this PR?
translate(StringTranslate) andto_csv(StructsToCsv) through the JVM codegen dispatcher by default, using the sameNativeOptInAvailablepattern asto_jsonandsplit. The dispatcher runs Spark's own generated code (bit-exact) while keeping the operator on the native pipeline; the native path stays available viaallowIncompatible.to_csv.How are these changes tested?
string_translateSQL test (previously asserting fallback) to verify native codegen dispatch.to_csvSQL file test.