Skip to content

Internal error: CSE-extracted __common_expr_1 has mismatched nullability between logical and physical plans when the expression is under an aggregate #25369

Description

@hanke580

Describe the bug

An aggregate over an expression that appears twice fails to plan. Common subexpression
elimination extracts the shared expression as __common_expr_1, and its nullability is
computed one way for the logical plan and another for the physical plan; the aggregate is
the operator that checks the two agree, so the query never runs:

Internal error: Physical input schema should be the same as the one converted from logical
input schema. Differences:
	- field nullability at index 0 [__common_expr_1]: (physical) true vs (logical) false.
This issue was likely caused by a bug in DataFusion's code. Please help us to resolve this
by filing a bug report in our issue tracker: https://github.com/apache/datafusion/issues

Reproduces on datafusion-cli 55.1.0 (latest crates.io release) and on the
datafusion 54.0.0 Python wheel.

To Reproduce

Three rows are enough, and no session settings are changed:

CREATE TABLE t(a INT, b INT) AS VALUES (1,1),(2,2),(3,3);

SELECT sum(CASE WHEN (CASE WHEN (a > 1 AND b < 3) THEN a ELSE 0 END)
                BETWEEN 1 AND 2 THEN 1 ELSE 0 END)
FROM t;

datafusion-cli -f repro.sql raises the Internal error above.

Three ingredients are each necessary -- removing any one makes the query plan and run
normally. Measured on 55.1.0:

query result
aggregate + two columns + duplicated expression (above) Internal error
count() instead of sum() Internal error
BETWEEN written out as >= 1 AND <= 2 Internal error
the same expression with no aggregate planned and ran
one column used twice (a > 1 AND a < 3) instead of two planned and ran
a single comparison, so nothing is duplicated planned and ran
one condition in the inner CASE planned and ran

So it needs (1) an aggregate over the expression, (2) an inner CASE condition
referencing two distinct columns -- the same column twice is fine, which points at
nullability inference rather than at CASE itself -- and (3) the expression appearing
twice, which is what invites CSE to extract it. BETWEEN duplicates it implicitly;
writing the two comparisons out by hand fails the same way.

Expected behavior

The query plans and returns a result, as the same expression does without the aggregate.
For the three-row table above the answer is 1.

Additional context

  • Reproduced on datafusion-cli 55.1.0 (built from crates.io with cargo install datafusion-cli --version 55.1.0 --locked) and on the datafusion 54.0.0 Python wheel
    (the newest on PyPI). All seven rows of the table above behave identically on both.
  • It fires regardless of batch size, target partition count, and optimizer settings.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions