[SPARK-57109][SQL] Fix exchange reuse failure when a VIEW is referenced in both UNION ALL branches via a CTE

[kush2439p]

What changes were proposed in this pull request?

Fix a performance regression where ReusedExchangeExec fails to match the shuffle of the underlying table when a persisted VIEW is referenced in both branches of a UNION ALL through a CTE, causing the table to be scanned and shuffled twice.

The fix is confined entirely to the canonicalization layer (QueryPlan.doCanonicalize()). Inside mapExpressions, a new case is added before the generic case a: Alias branch:

• Detect a no-op rename alias — Alias(attr: Attribute, name) where
  ame == attr.name AND �.metadata == attr.metadata. This is precisely the condition RemoveRedundantAliases itself uses to decide an alias is truly redundant.
• When matched, canonicalize the wrapping alias away and normalize the bare �ttr exactly as the existing bare-AttributeReference branches do:
  • If �ttr.exprId is not in �llAttributesSeq (the node's children's output), assign it the next sequential id (id += 1; attr.withExprId(ExprId(id)).canonicalized).
  • Otherwise normalize via QueryPlan.normalizeExpressions(attr, allAttributesSeq).

No changes are made to RemoveRedundantAliases, DeduplicateRelations, or InlineCTE.

Why are the changes needed?

Root cause chain (5 steps):

1. VIEW construction (SPARK-34269) wraps each output column in a schema-compat Alias(UpCast(col, field.dataType), field.name)(explicitMetadata = Some(field.metadata)) — a genuine rename since #scan != #catalog.

2. InlineCTE: both UNION branches share identical ExprIds, making Union.duplicateResolved = false.

3. DeduplicateRelations resolves duplicates by wrapping the non-first branch in an outer Project(Alias(attr, attr.name)) and rewriting ExprIds via
   ewInstance()/
   ewriteAttrs, turning the view's inner schema-compat alias into a no-op Alias(a#X AS a#X).

4. RemoveRedundantAliases (SPARK-39887 Union first-child protection): protects branch 1's attribute set from alias removal, strips the no-op alias from branch 2. Result: branch 1 keeps the Alias node, branch 2 has a bare AttributeReference — structurally asymmetric though semantically identical. This protection is required for correctness and is not touched.

5. Canonicalization asymmetry: doCanonicalize() assigns a sequential id to the top-level Alias (branch 1) but uses ordinal-position normalization for the bare AttributeReference (branch 2). The two branches get different canonical forms, so ReuseExchangeAndSubquery (and AQE's stage cache) cannot match the upstream Exchange, and 1 is shuffled twice.

Does this PR introduce any user-facing change?

Yes — this is a performance fix. Queries matching the pattern (a persisted VIEW referenced in both branches of a UNION ALL via a CTE) will now correctly reuse the shuffle/exchange across both branches, avoiding redundant table scans and shuffles.

How was this patch tested?

Two tests were added:

1. Unit test in sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/plans/QueryPlanSuite.scala:

   • SPARK-57109: no-op rename Alias canonicalizes the same as a bare attribute
   • Constructs two Project branches over the same LocalRelation: one wraps the attribute in a no-op rename Alias, the other uses the bare AttributeReference. Asserts �liasedBranch.sameResult(bareBranch) is rue.

2. End-to-end regression test in sql/core/src/test/scala/org/apache/spark/sql/execution/ReuseExchangeAndSubquerySuite.scala:

   • SPARK-57109: Exchange reused when a VIEW is referenced via a CTE in both UNION branches
   • Uses the exact repro SQL from the issue (creates 1, 2, view �, runs the CTE + UNION ALL query).
   • Asserts that plan.collectWithSubqueries { case re: ReusedExchangeExec => re }.nonEmpty.

Closes #57109

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Google Antigravity (Gemini 2.5 Pro)

[kush2439p]

I already went through your issue and this is the pr i have made please review this.