Skip to content

fix(spark): preserve compilation with newer toPrettySQL signatures - #1419

Open
alexandrefimov wants to merge 1 commit into
substrait-io:mainfrom
alexandrefimov:spark-prettysql-compatibility
Open

alexandrefimov wants to merge 1 commit into
substrait-io:mainfrom
alexandrefimov:spark-prettysql-compatibility

Conversation

@alexandrefimov

Copy link
Copy Markdown
Contributor

Spark 4.1 and 4.2 add an optional argument to toPrettySQL. Passing the method directly to map no longer matches the expected unary function. Use an explicit lambda so the default argument is applied.

Validated with Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 using the runtime-profile harness: 108 schema cases in both ANSI modes, plus decimal and overflow controls.

Spark 4.1 and 4.2 add an optional argument to toPrettySQL. Passing
its method reference to map no longer matches the expected unary
function. Call it through a lambda so the default argument is applied.
@coderabbitai

coderabbitai Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Repository: substrait-io/substrait-java/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: f3d3a35f-387d-4435-8ef5-b1f34eb01657
📥 Commits

Reviewing files that changed from the base of the PR and between e7f2c55 and b47e1f7.

📒 Files selected for processing: 1
  • spark/src/main/scala/io/substrait/spark/logical/ToLogicalPlan.scala

Included review availability: This review used your included allowance. Your plan provides up to 2 included reviews per hour; 1 remain after this review.

📝 Summary

Summary by CodeRabbit

  • Refactor
    • Updated the handling of fallback names for partition-scan projections. This is an internal implementation adjustment; it does not change the names users see or otherwise alter partition-scan behavior. No new features or user-facing behavior changes are included in this update.

The partition-scan projection literal-name fallback now calls toPrettySQL through an explicit lambda. The fallback behavior is otherwise unchanged.

Changes: Partition-scan literal naming

Layer / File(s) Summary
Literal-name fallback
spark/src/main/scala/io/substrait/spark/logical/ToLogicalPlan.scala
The fallback calls toPrettySQL through an explicit lambda. The fallback behavior is otherwise unchanged.

Priority: ⬇️ Low

Estimated code review effort: 1 (Trivial) | ~2 minutes

Change: Bug fix

Suggested reviewers: nielspardon

Merge Risk: ⚪ Minimal · up to b47e1

The change is limited to how the literal-name fallback invokes Spark’s formatter; no actionable merge risk is identified.

Pre-merge checks: Passed 5
Check name Status Explanation
Title check Passed The title is concise, uses valid Conventional Commit syntax, and clearly identifies the Spark compatibility fix for newer toPrettySQL signatures.
Description check Passed The description explains the cause, the implementation, and the validation performed across supported Spark versions and test cases.
Docstring Coverage Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0 files. (1 skipped: 1 unsupported.)

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant