[SPARK-58193][SQL] Classify Databricks syntax errors by message#57333
Closed
alekjarmov wants to merge 4 commits into
Closed
[SPARK-58193][SQL] Classify Databricks syntax errors by message#57333alekjarmov wants to merge 4 commits into
alekjarmov wants to merge 4 commits into
Conversation
Recognize remote Databricks parse errors when the JDBC driver reports the SYNTAX_ERROR condition only in its message.
Normalize JDBC error messages before detecting the remote SYNTAX_ERROR condition while retaining the SQLState documentation.
cloud-fan
reviewed
Jul 17, 2026
cloud-fan
left a comment
cloud-fan
left a comment
Contributor
There was a problem hiding this comment.
0 blocking, 2 non-blocking, 0 nits.
The change is correct and consistent with the MsSqlServerDialect precedent (message-substring classification for a vendor without a reliable syntax-error SQLState); the SQLState-42 path is retained, so the check only broadens. Two non-blocking test suggestions below.
Suggestions (2)
- JDBCTableCatalogSuite.scala:232: pure dialect unit test placed here instead of
JDBCSuitewhere the otherDatabricksDialectunit tests live — see inline - JDBCTableCatalogSuite.scala:233: only the returns-true path is asserted; a negative case would guard the best-effort substring match — see inline
Move dialect coverage to JDBCSuite and verify that unrelated errors are not classified as syntax errors.
cloud-fan
approved these changes
Jul 19, 2026
cloud-fan
left a comment
cloud-fan
left a comment
Contributor
There was a problem hiding this comment.
2 addressed, 0 remaining, 0 new. (0 newly introduced, 0 late catches.)
0 blocking, 0 non-blocking, 0 nits.
Verification
Static trace: JDBCRDD consults this hook during schema resolution and query execution, while DatabricksDialect preserves the existing SQLState path and adds only the explicit vendor marker fallback. Tests were not run as part of this review.
PR description suggestions
- Update the user-facing-change section from
No: matching this driver error now produces Spark'sJDBC_EXTERNAL_ENGINE_SYNTAX_ERRORclassification during schema resolution or query execution.
HyukjinKwon
approved these changes
Jul 19, 2026
cloud-fan
pushed a commit
that referenced
this pull request
Jul 20, 2026
### What changes were proposed in this pull request? Recognize remote Databricks parse errors when the JDBC driver reports the SYNTAX_ERROR condition only in its message. Optimise `isSyntaxErrorBestEffort` for Databricks to catch syntax error even if SQLState is missing. ### Why are the changes needed? To improve the error classification. ### Does this PR introduce _any_ user-facing change? Update the user-facing-change section from No: matching this driver error now produces Spark's JDBC_EXTERNAL_ENGINE_SYNTAX_ERROR classification during schema resolution or query execution. ### How was this patch tested? Added a unit-test and checked a case on a Databricks workspace ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude 4.7 Closes #57333 from alekjarmov/alekjarmov/classify-databricks-syntax-errors. Authored-by: Alek Jarmov <alek.jarmov@databricks.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com> (cherry picked from commit 4b38f9c) Signed-off-by: Wenchen Fan <wenchen@databricks.com>
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Recognize remote Databricks parse errors when the JDBC driver reports the SYNTAX_ERROR condition only in its message.
Optimise
isSyntaxErrorBestEffortfor Databricks to catch syntax error even if SQLState is missing.Why are the changes needed?
To improve the error classification.
Does this PR introduce any user-facing change?
Update the user-facing-change section from No: matching this driver error now produces Spark's JDBC_EXTERNAL_ENGINE_SYNTAX_ERROR classification during schema resolution or query execution.
How was this patch tested?
Added a unit-test and checked a case on a Databricks workspace
Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude 4.7