Describe the bug
Local TopK statistics cap the total estimate at one partition’s limit, while
runtime metrics sum rows across all partitions. A simplified grouped query
exposes the mismatch without the joins in TPC-H Q3.
To Reproduce
From the repository root, with the CLI fix from
PR #25570 applied:
cargo build --profile ci --locked -p datafusion-benchmarks --bin dfbench
cargo install tpchgen-cli --version 1.1.1 --locked # if not already installed
repro_dir=$(mktemp -d)
tpchgen-cli --scale-factor 1 --format parquet \
--parquet-compression 'ZSTD(1)' --parts 1 --output-dir "$repro_dir/data"
cat > "$repro_dir/repro.sql" <<'SQL'
SET datafusion.optimizer.enable_dynamic_filter_pushdown = false;
SET datafusion.execution.target_partitions = 1;
SELECT l_orderkey, SUM(l_quantity) AS qty FROM lineitem GROUP BY l_orderkey ORDER BY qty DESC LIMIT 10;
SET datafusion.execution.target_partitions = 4;
SELECT l_orderkey, SUM(l_quantity) AS qty FROM lineitem GROUP BY l_orderkey ORDER BY qty DESC LIMIT 10;
SQL
target/ci/dfbench statistics \
--path "$repro_dir/data" --query_path "$repro_dir/repro.sql"
Observed with tpchgen-cli 1.1.1 at
6c320561b5.
Inspect the SELECT reports; ignore the empty SET reports.
| SELECT |
Operator / node |
Estimated rows |
Actual rows |
| One partition |
SortExec(TopK), 0.0 |
10 |
10 |
| Four partitions |
SortExec(TopK), 0.0.0 |
10 |
40 |
| Four partitions |
SortPreservingMergeExec, 0 |
10 |
10 |
Expected behavior
Account for the emitter count in local TopK estimates while preserving the
global merge’s limit. Per-partition and overall statistics must use consistent
units.
Additional context
Dynamic filtering is disabled. Early cancellation can reduce intermediate output
in other plans; this reproducer emits all 40 local rows.
Part of #25610.
Describe the bug
Local TopK statistics cap the total estimate at one partition’s limit, while
runtime metrics sum rows across all partitions. A simplified grouped query
exposes the mismatch without the joins in TPC-H Q3.
To Reproduce
From the repository root, with the CLI fix from
PR #25570 applied:
Observed with
tpchgen-cli1.1.1 at6c320561b5.
Inspect the SELECT reports; ignore the empty
SETreports.SortExec(TopK),0.0SortExec(TopK),0.0.0SortPreservingMergeExec,0Expected behavior
Account for the emitter count in local TopK estimates while preserving the
global merge’s limit. Per-partition and overall statistics must use consistent
units.
Additional context
Dynamic filtering is disabled. Early cancellation can reduce intermediate output
in other plans; this reproducer emits all 40 local rows.
Part of #25610.