Describe the bug
The SUM result has no useful statistics for a downstream HAVING filter. A
simplified Q18 query separates this loss from the already inaccurate group-count
estimate.
To Reproduce
From the repository root, with the CLI fix from
PR #25570 applied:
cargo build --profile ci --locked -p datafusion-benchmarks --bin dfbench
cargo install tpchgen-cli --version 1.1.1 --locked # if not already installed
repro_dir=$(mktemp -d)
tpchgen-cli --scale-factor 1 --format parquet \
--parquet-compression 'ZSTD(1)' --parts 1 --output-dir "$repro_dir/data"
cat > "$repro_dir/repro.sql" <<'SQL'
SET datafusion.execution.target_partitions = 1;
SET datafusion.optimizer.enable_dynamic_filter_pushdown = false;
SELECT l_orderkey, SUM(l_quantity) AS qty FROM lineitem GROUP BY l_orderkey HAVING SUM(l_quantity) > 300;
SQL
target/ci/dfbench statistics \
--path "$repro_dir/data" --query_path "$repro_dir/repro.sql"
Observed with tpchgen-cli 1.1.1 at
6c320561b5.
Inspect the SELECT reports; ignore the empty SET reports.
| Operator |
Node |
Estimated rows |
Actual rows |
| AggregateExec |
0.0.0 |
6,001,215 |
1,500,000 |
FilterExec (SUM > 300) |
0.0 |
1,200,243 |
57 |
Expected behavior
Propagate conservative aggregate-result statistics and use them for downstream
selectivity. Rare SUM tails may require distribution summaries; bounds alone are
insufficient.
Additional context
The q-error grows from 4× at the aggregate to 21,057× at the filter. Improving
group counts alone does not model the SUM distribution.
Part of #25610.
Describe the bug
The SUM result has no useful statistics for a downstream HAVING filter. A
simplified Q18 query separates this loss from the already inaccurate group-count
estimate.
To Reproduce
From the repository root, with the CLI fix from
PR #25570 applied:
Observed with
tpchgen-cli1.1.1 at6c320561b5.
Inspect the SELECT reports; ignore the empty
SETreports.0.0.0SUM > 300)0.0Expected behavior
Propagate conservative aggregate-result statistics and use them for downstream
selectivity. Rare SUM tails may require distribution summaries; bounds alone are
insufficient.
Additional context
The q-error grows from 4× at the aggregate to 21,057× at the filter. Improving
group counts alone does not model the SUM distribution.
Part of #25610.