fix: count ReusedSubquery and CometSubqueryBroadcast correctly in extended explain - #5206
Conversation
…ed explain The extended explain summary reported operator coverage and transition count but said nothing about expressions, so there was no way to see how much of a plan's expression evaluation ran in native DataFusion kernels versus Spark's own generated code inside the codegen dispatcher. Track both paths on the expression tree with a new `NATIVE_EXPRS` tag, the counterpart of the existing `CODEGEN_DISPATCH_EXPRS`, and roll them up per operator onto the converted Comet plan node alongside the info messages. The summary line now ends with, for example: Comet accelerated 14 expressions (14 native, 1 codegen dispatch). Counts are distinct expression names per plan. A name can appear in both buckets when the same function is lowered natively for one set of arguments and dispatched for another, so the total is the union rather than the sum. Structural nodes (attribute references, literals, aliases, bound references) are excluded; they appear in nearly every tree and would swamp the names that matter, and excluding literals also keeps the dispatcher's closure-serialized payload out of the native count. Add `ExtendedExplainInfo.getNativeExpressions` and `getCodegenDispatchExpressions` for programmatic access to the two name lists. Dispatch tagging is no longer gated on `spark.comet.explain.codegen.enabled`; that config now controls only whether the `[COMET-INFO: JVM codegen dispatcher: ...]` segment is rendered, so the counts are available by default. Enable that config in `CometPlanStabilitySuite` so the TPC-DS goldens record which expressions took the dispatcher path, not just how many. Golden files regenerated for Spark 3.4, 3.5, 4.0, 4.1 and 4.2; the only changes are summary lines and `COMET-INFO` segments.
Quality pass over the previous commit, no behavior change:
- Collapse the three byte-identical "append a name to a Set-valued tag" bodies
(`withInfo`, `withCodegenDispatchExpr`, `withNativeExpr`) onto one private
`appendTagValue`, with a bulk `appendTagValues` for lifting a whole name set.
- Add `CometExplainInfo.collectTagValues` and use it from both roll-up sites in
`CometExecRule` (including the pre-existing `EXTENSION_INFO` line) and from
`liftCoverageTags`, replacing three spellings of the same union.
- `rollUpInfoMessages` now writes the coverage tags through `appendTagValues`
rather than raw `setTagValue`, so there is one way to write these tags.
- `liftCoverageTags` walks the promoted tree once instead of twice, and drops
the identity `collect { case e: Expression => e }`.
- Classify native-vs-dispatched from a dedicated `DISPATCHED_SELF` marker
instead of sniffing `CODEGEN_DISPATCH_EXPRS`. That tag accumulates descendant
names during roll-up and lifting, so reading it as "was I dispatched" made the
classification depend on planner visit order.
- Route the two accessors through the existing `sortup` traversal, as
`fallbackReasons` already does, so they no longer render and discard a full
plan string just to read tags off nodes.
- Drop two no-op configs from the new test, fix an inaccurate comment about the
dispatcher's closure payload (that `Literal` is built locally and is never
reachable from `op.expressions`), and trim a comment that restated its code.
The summary read Comet accelerated 13 expressions (13 native, 1 codegen dispatch). The total was the union of the two name sets rather than their sum, because the same function can be lowered natively for one set of arguments and dispatched for another, so a name can be in both. Correct by design, but it presents as an arithmetic error to anyone reading the line. It now reads Accelerated expressions: 13 native, 1 codegen dispatch. which reports the same two counts, has no total to misread, and is grammatical at any count. Also drops the merged-set allocation the total needed. Golden files regenerated for Spark 3.4, 3.5, 4.0, 4.1 and 4.2; the only changed lines are the summary.
…CometSubqueryBroadcast `ExtendedExplainInfo`'s coverage summary miscounted three kinds of node: - `ReusedSubqueryExec` fell through to the Spark bucket, adding a phantom un-accelerated operator at every reuse site. - `CometSubqueryBroadcastExec` did not mix in `CometPlan`, so Comet's own DPP operator was also counted as a Spark operator. - `ReusedExchangeExec` was both ignored by the match and unwrapped by `getActualPlan`, so the reused subtree was traversed and counted in full at every reference. On q14a that inflated the eligible count from 128 to 2302. Both reuse markers are now ignored and rendered as leaves, so the operators they reference are counted once, where the plan they point at is shown. `getActualPlan` keeps its existing unwrapping behaviour for fallback-reason collection; the coverage traversal uses the new `actualPlanForCoverage`.
…t-fixes # Conflicts: # spark/src/main/scala/org/apache/comet/ExtendedExplainInfo.scala # spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q44/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q58/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q67/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q70/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q83/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_0/q14b/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_0/q44/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_0/q54/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_0/q58/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_0/q6/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_0/q83.ansi/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_1/q33/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_1/q49/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_1/q56/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_1/q60/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_1/q66/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q1/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q10/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q11/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q12/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q13/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q14a/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q14b/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q15/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q17/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q18/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q2/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q20/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q21/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q22/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q23a/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q23b/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q24a/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q24b/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q25/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q26/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q27/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q28/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q29/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q30/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q31/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q32/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q33/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q34/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q35/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q36/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q37/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q38/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q39a/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q39b/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q4/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q40/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q44/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q45/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q46/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q47/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q48/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q49/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q5/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q50/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q51/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q53/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q54/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q56/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q57/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q58/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q59/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q6/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q60/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q61/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q63/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q64/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q65/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q66/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q67/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q68/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q69/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q7/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q70/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q71/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q72/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q73/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q74/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q75/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q76/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q77/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q78/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q79/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q8/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q80/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q81/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q82/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q83/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q85/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q86/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q87/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q88/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q89/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q9/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q90/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q91/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q92/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q95/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q97/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q98/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7-spark3_5/q67a/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7-spark3_5/q70a/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7-spark4_0/q14/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7-spark4_0/q6/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7-spark4_1/q14a/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7-spark4_1/q49/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7-spark4_2/q77a/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q10a/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q11/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q12/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q14/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q14a/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q18a/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q20/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q22/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q22a/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q24/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q27a/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q34/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q35/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q35a/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q36a/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q47/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q49/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q51a/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q57/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q5a/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q6/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q64/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q67a/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q70a/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q72/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q74/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q75/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q77a/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q78/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q80a/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q86a/extended.txt # spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q98/extended.txt # spark/src/test/scala/org/apache/comet/CometCodegenSuite.scala
…oadcast Drop the ReusedExchange counting change so this PR only corrects the summary counts and leaves the rendered plan text untouched. The ReusedExchange work moves to a follow-up.
mbutrovich
left a comment
There was a problem hiding this comment.
Just docs changes. I understand if you want to defer those to a fast followup PR.
| - Transition nodes (`CometColumnarToRow`, `CometNativeColumnarToRow`, | ||
| `CometSparkRowToColumnar`, `ColumnarToRow`, `RowToColumnar`), which are | ||
| reported separately as the transition count. | ||
| - Wrappers that do no work of their own: `AdaptiveSparkPlan`, `InputAdapter`, |
There was a problem hiding this comment.
CometSparkToColumnarExec.nodeName renders as either CometSparkColumnarToColumnar or CometSparkRowToColumnar, depending on whether child.supportsColumnar is true. The transition-nodes bullet only lists CometSparkRowToColumnar. Since generateTreeString matches on the Scala type rather than the rendered name, both variants are already excluded from the operator counts today. Could the bullet list both names, so a reader who sees CometSparkColumnarToColumnar in their own explain output can tell it's covered by this list too?
| - Wrappers that do no work of their own: `AdaptiveSparkPlan`, `InputAdapter`, | ||
| `WholeStageCodegen`, query stages, and `AQEShuffleRead`. | ||
| - The reuse marker `ReusedSubquery`. The subquery it points at is counted where | ||
| that subquery is shown, so the marker itself does not add to the totals. |
There was a problem hiding this comment.
ReusedExchangeExec is matched in the same ignore arm as AdaptiveSparkPlanExec, InputAdapter, WholeStageCodegenExec, QueryStageExec, and AQEShuffleReadExec, but it isn't mentioned anywhere in this new section. It also behaves differently from the wrappers that are listed: getActualPlan unwraps it to its child, so a reused exchange subtree is counted once per reference rather than once total (the item 3 double-counting left open in #5203). Since this section's purpose is explaining what's excluded from the counts and why, could it also cover ReusedExchangeExec, including the caveat that it isn't cleanly excluded the way the others are?
Thanks, would prefer separate PR for docs changes to save CI time |
mbutrovich
left a comment
There was a problem hiding this comment.
Approved pending CI, thanks @andygrove. We'll do a docs followup.
…mes in operator-count exclusions (#5240) Follow-up to review feedback on #5206. - The transition-nodes bullet only listed `CometSparkRowToColumnar`, but `CometSparkToColumnarExec` also renders as `CometSparkColumnarToColumnar` when its child supports columnar. Both names are excluded from the operator counts today (the match is on the Scala type, not the rendered name), so list both. - `ReusedExchangeExec` is matched in the same ignore arm as the other wrapper nodes but was not mentioned. It also behaves differently from them: `getActualPlan` unwraps it to its child, so a reused subtree is counted once per reference rather than once per plan. Document it along with that caveat and the link to item 3 of #5203.
Which issue does this PR close?
Part of #5203. Items 1 and 2 from the issue are fixed here; item 3 (counting reused
exchanges once) is deferred to a follow-up, as the issue itself suggested it could be.
Rationale for this change
The operator coverage summary produced by
ExtendedExplainInfo(
Comet accelerated N out of M eligible operators (X%)) puts two node kinds in theun-accelerated Spark bucket even though neither is un-accelerated Spark work. Every
reuse site and every DPP subquery broadcast therefore drags the reported percentage
down, and fully accelerated plans report less than 100%.
The same counts feed the
cometmetrics source viaCometMetricsListener->CometSource, so the skew also biases theacceleration.ratiogauge rather than beingconfined to
EXPLAINoutput.What changes are included in this PR?
ReusedSubqueryExecis no longer counted as an un-accelerated Spark operator.It is pure reuse bookkeeping and a
LeafExecNode, so it joins the ignore listalongside the other wrapper nodes. The subquery it points at is counted where that
subquery is shown.
CometSubqueryBroadcastExecnow mixes inCometPlanand counts as accelerated.On the blast radius the issue asked about: the other
CometPlanmatch sites(
EliminateRedundantTransitions.hasCometNativeChild,foreachUntilCometInput,CometShuffleExchangeExec.isCometPlan) all reach nodes throughchildren, and aBaseSubqueryExechangs off aDynamicPruningExpressionrather thanchildren, sonone of them observe the new type. The one site that does newly match it is
CometExecRule's "never replace" guard, whose only effect is suppressing amisleading info message, and where the new classification is the correct one.
The plan text rendered by extended explain is unchanged. Only the summary line moves:
all 133 golden diffs are confined to that one line, with no tree edits and no golden
files added or removed. 101 of the changed plans now report 100% acceleration, and
across all 158 goldens the count at 100% goes from 21 to 122. No plan's percentage
decreases.
approved-plans-v1_4/q1, the example from the issue:The denominator drops by one because
ReusedSubqueryleaves the eligible set entirely,and the numerator rises by one because
CometSubqueryBroadcastmoves from the Sparkbucket to the Comet bucket.
Also documents in
understanding-comet-plans.mdwhich nodes are excluded from thecounts.
How are these changes tested?
New
CometCoverageStatsSuitecovers both cases directly: aReusedSubquerycontributes no operators of any kind, and wrapping a plan in
CometSubqueryBroadcastExecadds exactly one accelerated operator while leaving theSpark operator and transition counts untouched.
TPC-DS plan stability goldens were regenerated for Spark 3.4, 3.5, 4.0, 4.1 and 4.2.
Both plan stability suites then pass in verification mode against the regenerated
goldens on all five versions (135 tests per version, 10 suite runs, no failures).
CometCodegenSuiteand the new suite also pass on the default profile.