fix: keep the recovery projection when leaf pushdown would drop a computed same-name column

[adriangb]

Which issue does this PR close?

• Closes Wrong results: leaf expression pushdown removes a computed column that has the same name as its input column #25414.

Rationale for this change

A subquery that computes a column under the same name as its input column gives wrong results. This happens with the default configuration.

create table t(a int, s struct<b varchar>) as values (1, {b: 'x'}), (2, {b: 'y'});

select a, s['b'] from (select -a as a, s from t where a > 0);

Query	DataFusion on main	DuckDB and PostgreSQL
select a, s['b'] from (select -a as a, s from t where a > 0)	1 x / 2 y	-1 x / -2 y
select a, s['b'] from (select -a as a, s from t limit 10)	1 x / 2 y	-1 x / -2 y
select a, s['b'] from (select a * 10 as a, s from t limit 10)	Internal error	10 x / 20 y
select a, s['b'] from (select -a as a, s from t) where a < 0	1 x / 2 y	-1 x / -2 y
select a, count(s['b']) from (select -a as a, s from t where a > 0) group by a	groups 1 and 2	groups -1 and -2

The fourth query returns rows that fail its own filter. The third query stops the plan with this message:

Internal error: Assertion failed: compatible: Failed due to a difference in schemas:
original schema: ... "a", data_type: Int64 ...
new schema: ... "a", data_type: Int32 ...

The results are correct with set datafusion.optimizer.enable_leaf_expression_pushdown = false;.

The cause is in split_and_push_projection in datafusion/optimizer/src/extract_leaf_expressions.rs. The rule pushes the extraction of s['b'] below the projection that computes -a AS a. It then decides if it must keep a recovery projection, and it decides from the set of unqualified field names of the pushed plan.

A name says nothing about the value behind it. For this projection:

Projection: (- t.a) AS a, t.s, get_field(t.s, "b") AS __datafusion_extracted_1
  Filter: t.a > Int32(0)
    TableScan: t

the pushed plan holds the names a, s and __datafusion_extracted_1 again, but its a is the table column t.a, not - t.a. The two sets are equal, the rule drops the recovery projection, and the computed column becomes its own input column:

Projection: a, __datafusion_extracted_1 AS t.s[b]
  Filter: t.a > Int32(0)
    Projection: get_field(t.s, "b") AS __datafusion_extracted_1, t.a, t.s
      TableScan: t

The same comparison ignores data types. That is why a * 10 AS a fails the optimizer schema check instead of giving wrong data.

What changes are included in this PR?

The recovery projection now also stays when a recovery expression computes a value, that is, when it is not a pass-through of a column. passthrough_column gives that answer, and it already accepts a requalification such as t.a AS a.

The name comparison stays as it is. It catches the leaked-column case that it was written for, which the expression check does not see. The comparison still ignores qualifiers, so the SubqueryAlias requalification behaviour does not change.

The change is inside split_and_push_projection only. #25412 touches build_extraction_projection_impl in the same file. The two changes are independent and fix different bugs. The only overlap is that both PRs append a block to the end of datafusion/sqllogictest/test_files/struct.slt, so the second one to merge needs a trivial rebase there.

What is the testing strategy for this PR?

Six SQL shapes go in datafusion/sqllogictest/test_files/struct.slt: through a Filter, through a Limit, a computed column with a different data type, an outer filter on the computed column, a group key, and through a Sort. One EXPLAIN shows that the projection that computes -a stays in the plan.

A counterfactual run confirms each one. With the fix reverted, the new block gives 6 errors: the EXPLAIN, four wrong results and the internal error. The Sort shape was already correct and guards it.

A unit test, test_recovery_kept_for_same_name_computed_column in extract_leaf_expressions.rs, builds the plan shape that loses the computed column and runs the two leaf rules alone, in their production order.

All 883 datafusion-optimizer tests pass. No existing insta snapshot changes. All 520 sqllogictest files pass.

Are there any user-facing changes?

Queries of this shape now give correct results. There are no API changes.

Part of the leaf-pushdown EPIC: #25459

🤖 Generated with Claude Code

[codecov-commenter]

Codecov Report

❌ Patch coverage is 78.57143% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.33%. Comparing base (3a647e4) to head (2bb9d19).

Files with missing lines	Patch %	Lines
...tafusion/optimizer/src/extract_leaf_expressions.rs	78.57%	1 Missing and 5 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #25445      +/-   ##
==========================================
- Coverage   82.33%   82.33%   -0.01%     
==========================================
  Files        1137     1137              
  Lines      432498   432525      +27     
  Branches   432498   432525      +27     
==========================================
+ Hits       356115   356124       +9     
- Misses      54844    54854      +10     
- Partials    21539    21547       +8     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.