Describe the bug
An aggregate over an expression that appears twice fails to plan. Common subexpression
elimination extracts the shared expression as __common_expr_1, and its nullability is
computed one way for the logical plan and another for the physical plan; the aggregate is
the operator that checks the two agree, so the query never runs:
Internal error: Physical input schema should be the same as the one converted from logical
input schema. Differences:
- field nullability at index 0 [__common_expr_1]: (physical) true vs (logical) false.
This issue was likely caused by a bug in DataFusion's code. Please help us to resolve this
by filing a bug report in our issue tracker: https://github.com/apache/datafusion/issues
Reproduces on datafusion-cli 55.1.0 (latest crates.io release) and on the
datafusion 54.0.0 Python wheel.
To Reproduce
Three rows are enough, and no session settings are changed:
CREATE TABLE t(a INT, b INT) AS VALUES (1,1),(2,2),(3,3);
SELECT sum(CASE WHEN (CASE WHEN (a > 1 AND b < 3) THEN a ELSE 0 END)
BETWEEN 1 AND 2 THEN 1 ELSE 0 END)
FROM t;
datafusion-cli -f repro.sql raises the Internal error above.
Three ingredients are each necessary -- removing any one makes the query plan and run
normally. Measured on 55.1.0:
| query |
result |
| aggregate + two columns + duplicated expression (above) |
Internal error |
count() instead of sum() |
Internal error |
BETWEEN written out as >= 1 AND <= 2 |
Internal error |
| the same expression with no aggregate |
planned and ran |
one column used twice (a > 1 AND a < 3) instead of two |
planned and ran |
| a single comparison, so nothing is duplicated |
planned and ran |
one condition in the inner CASE |
planned and ran |
So it needs (1) an aggregate over the expression, (2) an inner CASE condition
referencing two distinct columns -- the same column twice is fine, which points at
nullability inference rather than at CASE itself -- and (3) the expression appearing
twice, which is what invites CSE to extract it. BETWEEN duplicates it implicitly;
writing the two comparisons out by hand fails the same way.
Expected behavior
The query plans and returns a result, as the same expression does without the aggregate.
For the three-row table above the answer is 1.
Additional context
- Reproduced on datafusion-cli 55.1.0 (built from crates.io with
cargo install datafusion-cli --version 55.1.0 --locked) and on the datafusion 54.0.0 Python wheel
(the newest on PyPI). All seven rows of the table above behave identically on both.
- It fires regardless of batch size, target partition count, and optimizer settings.
Describe the bug
An aggregate over an expression that appears twice fails to plan. Common subexpression
elimination extracts the shared expression as
__common_expr_1, and its nullability iscomputed one way for the logical plan and another for the physical plan; the aggregate is
the operator that checks the two agree, so the query never runs:
Reproduces on datafusion-cli 55.1.0 (latest crates.io release) and on the
datafusion 54.0.0 Python wheel.
To Reproduce
Three rows are enough, and no session settings are changed:
datafusion-cli -f repro.sqlraises the Internal error above.Three ingredients are each necessary -- removing any one makes the query plan and run
normally. Measured on 55.1.0:
count()instead ofsum()BETWEENwritten out as>= 1 AND <= 2a > 1 AND a < 3) instead of twoCASESo it needs (1) an aggregate over the expression, (2) an inner
CASEconditionreferencing two distinct columns -- the same column twice is fine, which points at
nullability inference rather than at
CASEitself -- and (3) the expression appearingtwice, which is what invites CSE to extract it.
BETWEENduplicates it implicitly;writing the two comparisons out by hand fails the same way.
Expected behavior
The query plans and returns a result, as the same expression does without the aggregate.
For the three-row table above the answer is 1.
Additional context
cargo install datafusion-cli --version 55.1.0 --locked) and on the datafusion 54.0.0 Python wheel(the newest on PyPI). All seven rows of the table above behave identically on both.