Skip to content

fix: count ReusedSubquery and CometSubqueryBroadcast correctly in extended explain - #5206

Merged
andygrove merged 8 commits into
apache:mainfrom
andygrove:explain-operator-count-fixes
Aug 3, 2026
Merged

fix: count ReusedSubquery and CometSubqueryBroadcast correctly in extended explain#5206
andygrove merged 8 commits into
apache:mainfrom
andygrove:explain-operator-count-fixes

Conversation

@andygrove

@andygrove andygrove commented Aug 1, 2026

Copy link
Copy Markdown
Member

Which issue does this PR close?

Part of #5203. Items 1 and 2 from the issue are fixed here; item 3 (counting reused
exchanges once) is deferred to a follow-up, as the issue itself suggested it could be.

Rationale for this change

The operator coverage summary produced by ExtendedExplainInfo
(Comet accelerated N out of M eligible operators (X%)) puts two node kinds in the
un-accelerated Spark bucket even though neither is un-accelerated Spark work. Every
reuse site and every DPP subquery broadcast therefore drags the reported percentage
down, and fully accelerated plans report less than 100%.

The same counts feed the comet metrics source via CometMetricsListener ->
CometSource, so the skew also biases the acceleration.ratio gauge rather than being
confined to EXPLAIN output.

What changes are included in this PR?

  1. ReusedSubqueryExec is no longer counted as an un-accelerated Spark operator.
    It is pure reuse bookkeeping and a LeafExecNode, so it joins the ignore list
    alongside the other wrapper nodes. The subquery it points at is counted where that
    subquery is shown.

  2. CometSubqueryBroadcastExec now mixes in CometPlan and counts as accelerated.
    On the blast radius the issue asked about: the other CometPlan match sites
    (EliminateRedundantTransitions.hasCometNativeChild, foreachUntilCometInput,
    CometShuffleExchangeExec.isCometPlan) all reach nodes through children, and a
    BaseSubqueryExec hangs off a DynamicPruningExpression rather than children, so
    none of them observe the new type. The one site that does newly match it is
    CometExecRule's "never replace" guard, whose only effect is suppressing a
    misleading info message, and where the new classification is the correct one.

The plan text rendered by extended explain is unchanged. Only the summary line moves:
all 133 golden diffs are confined to that one line, with no tree edits and no golden
files added or removed. 101 of the changed plans now report 100% acceleration, and
across all 158 goldens the count at 100% goes from 21 to 122. No plan's percentage
decreases.

approved-plans-v1_4/q1, the example from the issue:

-Comet accelerated 47 out of 49 eligible operators (95%). ...
+Comet accelerated 48 out of 48 eligible operators (100%). ...

The denominator drops by one because ReusedSubquery leaves the eligible set entirely,
and the numerator rises by one because CometSubqueryBroadcast moves from the Spark
bucket to the Comet bucket.

Also documents in understanding-comet-plans.md which nodes are excluded from the
counts.

How are these changes tested?

New CometCoverageStatsSuite covers both cases directly: a ReusedSubquery
contributes no operators of any kind, and wrapping a plan in
CometSubqueryBroadcastExec adds exactly one accelerated operator while leaving the
Spark operator and transition counts untouched.

TPC-DS plan stability goldens were regenerated for Spark 3.4, 3.5, 4.0, 4.1 and 4.2.
Both plan stability suites then pass in verification mode against the regenerated
goldens on all five versions (135 tests per version, 10 suite runs, no failures).
CometCodegenSuite and the new suite also pass on the default profile.

…ed explain

The extended explain summary reported operator coverage and transition count
but said nothing about expressions, so there was no way to see how much of a
plan's expression evaluation ran in native DataFusion kernels versus Spark's
own generated code inside the codegen dispatcher.

Track both paths on the expression tree with a new `NATIVE_EXPRS` tag, the
counterpart of the existing `CODEGEN_DISPATCH_EXPRS`, and roll them up per
operator onto the converted Comet plan node alongside the info messages. The
summary line now ends with, for example:

  Comet accelerated 14 expressions (14 native, 1 codegen dispatch).

Counts are distinct expression names per plan. A name can appear in both
buckets when the same function is lowered natively for one set of arguments
and dispatched for another, so the total is the union rather than the sum.
Structural nodes (attribute references, literals, aliases, bound references)
are excluded; they appear in nearly every tree and would swamp the names that
matter, and excluding literals also keeps the dispatcher's closure-serialized
payload out of the native count.

Add `ExtendedExplainInfo.getNativeExpressions` and
`getCodegenDispatchExpressions` for programmatic access to the two name lists.

Dispatch tagging is no longer gated on `spark.comet.explain.codegen.enabled`;
that config now controls only whether the `[COMET-INFO: JVM codegen
dispatcher: ...]` segment is rendered, so the counts are available by default.

Enable that config in `CometPlanStabilitySuite` so the TPC-DS goldens record
which expressions took the dispatcher path, not just how many. Golden files
regenerated for Spark 3.4, 3.5, 4.0, 4.1 and 4.2; the only changes are summary
lines and `COMET-INFO` segments.
Quality pass over the previous commit, no behavior change:

- Collapse the three byte-identical "append a name to a Set-valued tag" bodies
  (`withInfo`, `withCodegenDispatchExpr`, `withNativeExpr`) onto one private
  `appendTagValue`, with a bulk `appendTagValues` for lifting a whole name set.
- Add `CometExplainInfo.collectTagValues` and use it from both roll-up sites in
  `CometExecRule` (including the pre-existing `EXTENSION_INFO` line) and from
  `liftCoverageTags`, replacing three spellings of the same union.
- `rollUpInfoMessages` now writes the coverage tags through `appendTagValues`
  rather than raw `setTagValue`, so there is one way to write these tags.
- `liftCoverageTags` walks the promoted tree once instead of twice, and drops
  the identity `collect { case e: Expression => e }`.
- Classify native-vs-dispatched from a dedicated `DISPATCHED_SELF` marker
  instead of sniffing `CODEGEN_DISPATCH_EXPRS`. That tag accumulates descendant
  names during roll-up and lifting, so reading it as "was I dispatched" made the
  classification depend on planner visit order.
- Route the two accessors through the existing `sortup` traversal, as
  `fallbackReasons` already does, so they no longer render and discard a full
  plan string just to read tags off nodes.
- Drop two no-op configs from the new test, fix an inaccurate comment about the
  dispatcher's closure payload (that `Literal` is built locally and is never
  reachable from `op.expressions`), and trim a comment that restated its code.
The summary read

  Comet accelerated 13 expressions (13 native, 1 codegen dispatch).

The total was the union of the two name sets rather than their sum, because the
same function can be lowered natively for one set of arguments and dispatched
for another, so a name can be in both. Correct by design, but it presents as an
arithmetic error to anyone reading the line. It now reads

  Accelerated expressions: 13 native, 1 codegen dispatch.

which reports the same two counts, has no total to misread, and is grammatical
at any count. Also drops the merged-set allocation the total needed.

Golden files regenerated for Spark 3.4, 3.5, 4.0, 4.1 and 4.2; the only changed
lines are the summary.
…CometSubqueryBroadcast

`ExtendedExplainInfo`'s coverage summary miscounted three kinds of node:

- `ReusedSubqueryExec` fell through to the Spark bucket, adding a phantom
  un-accelerated operator at every reuse site.
- `CometSubqueryBroadcastExec` did not mix in `CometPlan`, so Comet's own DPP
  operator was also counted as a Spark operator.
- `ReusedExchangeExec` was both ignored by the match and unwrapped by
  `getActualPlan`, so the reused subtree was traversed and counted in full at
  every reference. On q14a that inflated the eligible count from 128 to 2302.

Both reuse markers are now ignored and rendered as leaves, so the operators
they reference are counted once, where the plan they point at is shown.
`getActualPlan` keeps its existing unwrapping behaviour for fallback-reason
collection; the coverage traversal uses the new `actualPlanForCoverage`.
@andygrove andygrove added this to the 1.0.0 milestone Aug 3, 2026
@andygrove
andygrove marked this pull request as draft August 3, 2026 14:41
…t-fixes

# Conflicts:
#	spark/src/main/scala/org/apache/comet/ExtendedExplainInfo.scala
#	spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q44/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q58/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q67/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q70/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q83/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_0/q14b/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_0/q44/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_0/q54/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_0/q58/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_0/q6/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_0/q83.ansi/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_1/q33/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_1/q49/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_1/q56/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_1/q60/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_1/q66/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q1/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q10/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q11/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q12/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q13/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q14a/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q14b/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q15/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q17/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q18/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q2/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q20/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q21/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q22/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q23a/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q23b/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q24a/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q24b/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q25/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q26/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q27/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q28/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q29/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q30/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q31/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q32/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q33/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q34/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q35/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q36/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q37/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q38/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q39a/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q39b/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q4/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q40/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q44/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q45/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q46/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q47/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q48/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q49/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q5/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q50/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q51/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q53/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q54/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q56/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q57/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q58/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q59/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q6/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q60/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q61/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q63/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q64/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q65/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q66/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q67/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q68/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q69/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q7/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q70/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q71/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q72/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q73/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q74/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q75/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q76/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q77/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q78/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q79/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q8/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q80/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q81/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q82/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q83/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q85/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q86/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q87/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q88/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q89/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q9/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q90/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q91/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q92/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q95/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q97/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q98/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7-spark3_5/q67a/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7-spark3_5/q70a/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7-spark4_0/q14/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7-spark4_0/q6/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7-spark4_1/q14a/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7-spark4_1/q49/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7-spark4_2/q77a/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q10a/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q11/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q12/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q14/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q14a/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q18a/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q20/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q22/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q22a/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q24/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q27a/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q34/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q35/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q35a/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q36a/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q47/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q49/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q51a/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q57/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q5a/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q6/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q64/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q67a/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q70a/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q72/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q74/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q75/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q77a/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q78/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q80a/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q86a/extended.txt
#	spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q98/extended.txt
#	spark/src/test/scala/org/apache/comet/CometCodegenSuite.scala
@andygrove
andygrove marked this pull request as ready for review August 3, 2026 19:46
@andygrove
andygrove marked this pull request as draft August 3, 2026 19:54
…oadcast

Drop the ReusedExchange counting change so this PR only corrects the
summary counts and leaves the rendered plan text untouched. The
ReusedExchange work moves to a follow-up.
@andygrove
andygrove marked this pull request as ready for review August 3, 2026 20:18
@andygrove andygrove changed the title fix: correct extended explain operator counts for reuse wrappers and CometSubqueryBroadcast fix: count ReusedSubquery and CometSubqueryBroadcast correctly in extended explain Aug 3, 2026
@mbutrovich
mbutrovich self-requested a review August 3, 2026 21:04

@mbutrovich mbutrovich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just docs changes. I understand if you want to defer those to a fast followup PR.

Comment on lines +184 to +187
- Transition nodes (`CometColumnarToRow`, `CometNativeColumnarToRow`,
`CometSparkRowToColumnar`, `ColumnarToRow`, `RowToColumnar`), which are
reported separately as the transition count.
- Wrappers that do no work of their own: `AdaptiveSparkPlan`, `InputAdapter`,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CometSparkToColumnarExec.nodeName renders as either CometSparkColumnarToColumnar or CometSparkRowToColumnar, depending on whether child.supportsColumnar is true. The transition-nodes bullet only lists CometSparkRowToColumnar. Since generateTreeString matches on the Scala type rather than the rendered name, both variants are already excluded from the operator counts today. Could the bullet list both names, so a reader who sees CometSparkColumnarToColumnar in their own explain output can tell it's covered by this list too?

Comment on lines +187 to +190
- Wrappers that do no work of their own: `AdaptiveSparkPlan`, `InputAdapter`,
`WholeStageCodegen`, query stages, and `AQEShuffleRead`.
- The reuse marker `ReusedSubquery`. The subquery it points at is counted where
that subquery is shown, so the marker itself does not add to the totals.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ReusedExchangeExec is matched in the same ignore arm as AdaptiveSparkPlanExec, InputAdapter, WholeStageCodegenExec, QueryStageExec, and AQEShuffleReadExec, but it isn't mentioned anywhere in this new section. It also behaves differently from the wrappers that are listed: getActualPlan unwraps it to its child, so a reused exchange subtree is counted once per reference rather than once total (the item 3 double-counting left open in #5203). Since this section's purpose is explaining what's excluded from the counts and why, could it also cover ReusedExchangeExec, including the caveat that it isn't cleanly excluded the way the others are?

@andygrove

Copy link
Copy Markdown
Member Author

Just docs changes. I understand if you want to defer those to a fast followup PR.

Thanks, would prefer separate PR for docs changes to save CI time

@mbutrovich mbutrovich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved pending CI, thanks @andygrove. We'll do a docs followup.

@andygrove
andygrove merged commit ee974a1 into apache:main Aug 3, 2026
70 checks passed
@andygrove
andygrove deleted the explain-operator-count-fixes branch August 3, 2026 22:07
andygrove added a commit that referenced this pull request Aug 3, 2026
…mes in operator-count exclusions (#5240)

Follow-up to review feedback on #5206.

- The transition-nodes bullet only listed `CometSparkRowToColumnar`, but
  `CometSparkToColumnarExec` also renders as `CometSparkColumnarToColumnar`
  when its child supports columnar. Both names are excluded from the operator
  counts today (the match is on the Scala type, not the rendered name), so list
  both.

- `ReusedExchangeExec` is matched in the same ignore arm as the other wrapper
  nodes but was not mentioned. It also behaves differently from them:
  `getActualPlan` unwraps it to its child, so a reused subtree is counted once
  per reference rather than once per plan. Document it along with that caveat
  and the link to item 3 of #5203.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants