[SPARK-58226][SQL] Honor persistentCatalogFirst for DDL on two-part name resolution#57381
Open
jiashenC wants to merge 1 commit into
Open
[SPARK-58226][SQL] Honor persistentCatalogFirst for DDL on two-part name resolution#57381jiashenC wants to merge 1 commit into
jiashenC wants to merge 1 commit into
Conversation
jiashenC
force-pushed
the
ddl-multi-part-resolution
branch
from
July 20, 2026 16:30
20933bc to
b8297e0
Compare
…ession.<name> The DDL / misc-command resolution path (`Analyzer.lookupTableOrView`, used by DESCRIBE TABLE, ALTER TABLE, etc.) always resolved a local temp view before the persistent relation for a two-part `session.<name>` identifier, ignoring `spark.sql.legacy.persistentCatalogFirst`. The SELECT/DML path (`RelationResolution.resolveRelation`) already honors that config for the same name shape, so the two paths diverged: the same `session.<name>` could resolve to different relations in a query vs. a DDL command. This makes `lookupTableOrView` mirror `resolveRelation`: for a two-part `session.<name>`, resolution order follows `SQLConf.prioritizeSystemCatalog` (the inverse of `PERSISTENT_CATALOG_FIRST`). All other name shapes and the default (`prioritizeSystemCatalog = true`) keep their existing behavior. Adds SECTION 15 to `RelationQualificationSuite`, the DDL companion to the existing SELECT-path SECTION 13.
jiashenC
force-pushed
the
ddl-multi-part-resolution
branch
from
July 20, 2026 17:01
b8297e0 to
84aa25c
Compare
jiashenC
changed the title
jiashenC
changed the title
uros-b
approved these changes
Jul 21, 2026
uros-b
left a comment
uros-b
left a comment
Member
There was a problem hiding this comment.
Thank you @jiashenC! LGTM, adding @cloud-fan @srielau @zhengruifeng - PTAL
cloud-fan
reviewed
Jul 22, 2026
cloud-fan
left a comment
cloud-fan
left a comment
Contributor
There was a problem hiding this comment.
1 blocking, 0 non-blocking, 0 nits.
One blocking correctness issue: the DDL path can still perform persistent lookup for a missing three-part system.session.<name>, unlike the temp-only SELECT/DML path.
Correctness (1)
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:1333: Three-part
system.session.<name>is not temp-only in the DDL path when the temp view is missing. -- see inline
Verification
Traced Analyzer.lookupTableOrView candidate evaluation against RelationResolution.resolveRelation and checked every added test branch. No tests were run.
PR description suggestions
- Update the user-facing change section: the PR intentionally changes which relation two-part DDL identifiers resolve to when persistentCatalogFirst is enabled.
| !conf.prioritizeSystemCatalog) { | ||
| persistentCandidate.orElse(tempViewCandidate) | ||
| } else { | ||
| tempViewCandidate.orElse(persistentCandidate) |
Contributor
There was a problem hiding this comment.
Please keep system.session.<name> temp-only before taking this fallback. Three-part names also reach this else, so a missing temp view evaluates persistentCandidate, unlike SELECT/DML's early return from resolveTempView. The new test always creates the temp view and cannot catch that miss path; please add a no-temp case as well.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
The DDL / misc-command resolution path (
Analyzer.lookupTableOrView, used by DESCRIBE TABLE, ALTER TABLE, etc.) always resolved a local temp view before the persistent relation for a two-partsession.<name>identifier, ignoringspark.sql.legacy.persistentCatalogFirst. Whereas, the SELECT/DML path (RelationResolution.resolveRelation) already honors that config for the same name shape from #53630, so the two paths diverged: the samesession.<name>could resolve to different relations in a query vs. a DDL command.Why are the changes needed?
To resolve the resolution divergence and ambiguity between DDL and SELECT.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Add test to check for DDL commands like
DESCRIBE TABLEandALTER TABLEresolve to the correct table entity.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude-Opus4.8