Skip to content

Account for partitions in local TopK statistics #25619

Description

@gabotechs

Describe the bug

Local TopK statistics cap the total estimate at one partition’s limit, while
runtime metrics sum rows across all partitions. A simplified grouped query
exposes the mismatch without the joins in TPC-H Q3.

To Reproduce

From the repository root, with the CLI fix from
PR #25570 applied:

cargo build --profile ci --locked -p datafusion-benchmarks --bin dfbench
cargo install tpchgen-cli --version 1.1.1 --locked # if not already installed
repro_dir=$(mktemp -d)
tpchgen-cli --scale-factor 1 --format parquet \
  --parquet-compression 'ZSTD(1)' --parts 1 --output-dir "$repro_dir/data"
cat > "$repro_dir/repro.sql" <<'SQL'
SET datafusion.optimizer.enable_dynamic_filter_pushdown = false;
SET datafusion.execution.target_partitions = 1;
SELECT l_orderkey, SUM(l_quantity) AS qty FROM lineitem GROUP BY l_orderkey ORDER BY qty DESC LIMIT 10;
SET datafusion.execution.target_partitions = 4;
SELECT l_orderkey, SUM(l_quantity) AS qty FROM lineitem GROUP BY l_orderkey ORDER BY qty DESC LIMIT 10;
SQL
target/ci/dfbench statistics \
  --path "$repro_dir/data" --query_path "$repro_dir/repro.sql"

Observed with tpchgen-cli 1.1.1 at
6c320561b5.
Inspect the SELECT reports; ignore the empty SET reports.

SELECT Operator / node Estimated rows Actual rows
One partition SortExec(TopK), 0.0 10 10
Four partitions SortExec(TopK), 0.0.0 10 40
Four partitions SortPreservingMergeExec, 0 10 10

Expected behavior

Account for the emitter count in local TopK estimates while preserving the
global merge’s limit. Per-partition and overall statistics must use consistent
units.

Additional context

Dynamic filtering is disabled. Early cancellation can reduce intermediate output
in other plans; this reproducer emits all 40 local rows.

Part of #25610.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions