test: differential fuzz for leaf expression pushdown and parquet filter pushdown

[adriangb]

Which issue does this PR close?

• Related to Wrong results: leaf expression pushdown removes a computed column that has the same name as its input column #25414, fix: resolve pass-through columns against the input when merging an extraction projection #25412 and Wrong results: filter marked as pushed down at plan time can be silently dropped per-file when schema evolution makes the runtime pushdown checker reject the adapted predicate #25268.

Rationale for this change

ExtractLeafExpressions and PushDownLeafProjections (the option datafusion.optimizer.enable_leaf_expression_pushdown, default true) move expressions such as s['b'] out of a filter, sort, limit, aggregate or join, and push them towards the scan. The rules resolve columns by name. When a plan holds two columns with the same name, or a column that is computed from a column of the same name, the rules can drop a projection or merge the wrong pair. The user sees wrong rows or a planning error, with the default configuration.

The rules had 10 bug fixes since February 2026. Users or an ad-hoc fuzzer found all of them. The 520 sqllogictest files found none, because they do not build these shapes.

A differential test finds this class cheaply: run the same query with the option on and off, and compare the rows. This is a fixed-seed differential regression suite rather than a fuzzer that accumulates coverage; the seeds are the same on every run so a failure is reproducible with one command. Bugs of the class this test covers:

• Wrong results: leaf expression pushdown removes a computed column that has the same name as its input column #25414: a computed column with the same name as its input is dropped. Wrong results.
• fix: resolve pass-through columns against the input when merging an extraction projection #25412: pass-through columns merged in the wrong name space. Planning error.
• Incorrect results: leaf expression pushdown duplicates volatile expressions (random() evaluated twice) #24678: a volatile expression duplicated by leaf extraction. Wrong results.
• Filter on a volatile GROUP BY key returns rows that do not satisfy the filter #25415: a filter on a volatile group key pushed below the aggregate. Wrong results.
• optimize_projections fails with "No field named ..." when join keys contain get_field (ExtractLeafExpressions) #22895: optimize_projections fails with "No field named ..." when the join keys hold a get_field. Planning error.
• push_down_leaf_projections fails for self joins in some scenarios #24241: push_down_leaf_projections fails for a self join. Planning error.

The harness reproduces the two open bugs today. With INCLUDE_SAME_NAME_ALIAS_SHAPES = true, 526 of 3000 seeds fail (476 planning errors, 50 silent wrong results). With INCLUDE_FALSE_BRANCH_UNION_SHAPES = true, 404 of 3000 seeds fail.

What changes are included in this PR?

One new file, datafusion/core/tests/fuzz_cases/leaf_pushdown_fuzz.rs.

The generator builds SQL over three wide tables and one narrow table. Every table has Int32 columns, a Utf8 column and two Struct columns. Two struct fields have the same name as a top level column. The generated shapes are:

• s['f'] in SELECT, WHERE, ORDER BY, GROUP BY, HAVING, JOIN ON and under LIMIT.
• Derived tables, CTEs, IN and EXISTS subqueries, UNION ALL.
• named_struct(...) built in a subquery and read above.
• arrow_field(a) used bare and through ['f'] in the same query.
• random() in a subquery projection, read two times above. The two runs cannot be compared with each other, so the query reports its own consistency in one boolean column.

Each query runs in two SessionContexts built from the same batches. The rows are sorted and compared. An error on one side only is a failure. An error on both sides is a skip, and the test fails if more than 30% of the seeds skip.

Three shapes fail on main today. Each one is behind a named const switch with the link to its issue, so this PR is green:

• INCLUDE_SAME_NAME_ALIAS_SHAPES for Wrong results: leaf expression pushdown removes a computed column that has the same name as its input column #25414.
• INCLUDE_FALSE_BRANCH_UNION_SHAPES for fix: resolve pass-through columns against the input when merging an extraction projection #25412.

Flip the switch to true when the fix lands.

What is the testing strategy for this PR?

This PR is a test. Three test entry points:

test	cases	wall clock
leaf_pushdown_fuzz	250	3.9s
leaf_pushdown_parquet_fuzz	125	included above
leaf_pushdown_parquet_schema_evolution	50	included above
leaf_pushdown_fuzz_extended	5000 + 1000	87s

cargo test --profile ci -p datafusion --features extended_tests --test fuzz -- leaf_pushdown

The whole fuzz_cases tree is already behind the extended_tests feature, so the short tests run in the extended job together with the long one.

Every case comes from a seed. A failure prints the seed, the SQL and both result sets, and names the command that replays it:

LEAF_PUSHDOWN_FUZZ_SEED=<seed> LEAF_PUSHDOWN_FUZZ_CASES=1 \
  cargo test --profile ci -p datafusion --features extended_tests \
  --test fuzz -- leaf_pushdown --nocapture

Are there any user-facing changes?

No.

Update: the swap shape (b AS a, a AS b) fails for a different reason than the same-name computed column. It is #25446, which is open. It now has its own switch, INCLUDE_SWAP_ALIAS_SHAPES, so the fix for #25414 can be verified with INCLUDE_SAME_NAME_ALIAS_SHAPES alone. On a branch with #25445 and #25412 merged and both of those switches on, the only remaining failures are 24 of 250 seeds, all the swap shape with the ambiguity error.

Known gaps from independent review, to be addressed in this PR: the generic generator only emits pass-through spellings (c, q.c, q.c AS c, c AS c), so renames and computed columns come only from the hardcoded same-name arm; all_true passes on an empty result; and the whole fuzz_cases tree runs only in the merge-queue extended_tests job, so the short run should move out from behind that gate.

Part of the leaf-pushdown EPIC: #25459

🤖 Generated with Claude Code

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.33%. Comparing base (3a647e4) to head (cec21a4).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #25453      +/-   ##
==========================================
- Coverage   82.33%   82.33%   -0.01%     
==========================================
  Files        1137     1137              
  Lines      432498   432498              
  Branches   432498   432498              
==========================================
- Hits       356115   356107       -8     
- Misses      54844    54851       +7     
- Partials    21539    21540       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.