Describe the bug
Spark's DynamicPartitionPruningV1SuiteAEOn test "SPARK-34637: DPP side broadcast query stage is created firstly" fails in the Spark 3.4 sql_core-1 shard:
hasReuse was false SubqueryBroadcast dynamicpruning#66615, 0, [store_id#66614], [id=#157769]
...
should have been reused in
AdaptiveSparkPlan isFinalPlan=true
+- == Final Plan ==
*(4) BroadcastHashJoin [store_id#65490], [store_id#66614], Inner, BuildLeft, false
The initial plan broadcasts the right side, which the DPP subquery reuses. The final plan broadcasts the left side instead, so the subquery's broadcast is never reused.
It fails on #4565 (run 37212552551) and on #5841 (run 37222066341). #5841 changes only CI, and both runs merged with main after #6547. It passed on #6415's run on 2026-09-29 (main at e3d52fe). Spark 3.4 SQL tests run only on pull requests that carry run-spark-3.4-tests, so neither the merge queue nor the nightly catches it.
Cause
#6547 patches the stale Scan leaf of a native operator that AQE reuses over a shuffle stage (the final aggregate of a two-phase aggregate). It does this in CometExecRule while that rule runs as AQE's query stage preparation rule, so during each re-plan. The re-planned plan then differs from the current one, and AQE adopts it. After that, AQE cannot link the broadcast stage it later builds over the reused operator back into its logical plan, because that operator's logical node now sits inside a LogicalQueryStage. The join's build side is no longer pinned.
Once the probe side's shuffle stage materializes, the next re-plan picks the smaller side using runtime statistics. On Spark 3.4 the AQE DPP probe side stays in Spark, and its shuffle reports a smaller size than Comet's, so the join flips to BuildLeft.
Steps to reproduce
Run the test against Spark 3.4 with Comet enabled. It passes with spark.comet.shuffle.directRead.enabled=false, which keeps #6547's refresh from firing, and fails with the default.
Expected behavior
AQE re-plans should not change because of Comet's shuffle read refresh, and the DPP broadcast should be reused as it was before #6547.
Describe the bug
Spark's
DynamicPartitionPruningV1SuiteAEOntest "SPARK-34637: DPP side broadcast query stage is created firstly" fails in the Spark 3.4sql_core-1shard:The initial plan broadcasts the right side, which the DPP subquery reuses. The final plan broadcasts the left side instead, so the subquery's broadcast is never reused.
It fails on #4565 (run 37212552551) and on #5841 (run 37222066341). #5841 changes only CI, and both runs merged with
mainafter #6547. It passed on #6415's run on 2026-09-29 (mainat e3d52fe). Spark 3.4 SQL tests run only on pull requests that carryrun-spark-3.4-tests, so neither the merge queue nor the nightly catches it.Cause
#6547 patches the stale
Scanleaf of a native operator that AQE reuses over a shuffle stage (the final aggregate of a two-phase aggregate). It does this inCometExecRulewhile that rule runs as AQE's query stage preparation rule, so during each re-plan. The re-planned plan then differs from the current one, and AQE adopts it. After that, AQE cannot link the broadcast stage it later builds over the reused operator back into its logical plan, because that operator's logical node now sits inside aLogicalQueryStage. The join's build side is no longer pinned.Once the probe side's shuffle stage materializes, the next re-plan picks the smaller side using runtime statistics. On Spark 3.4 the AQE DPP probe side stays in Spark, and its shuffle reports a smaller size than Comet's, so the join flips to
BuildLeft.Steps to reproduce
Run the test against Spark 3.4 with Comet enabled. It passes with
spark.comet.shuffle.directRead.enabled=false, which keeps #6547's refresh from firing, and fails with the default.Expected behavior
AQE re-plans should not change because of Comet's shuffle read refresh, and the DPP broadcast should be reused as it was before #6547.