Skip to content

Propagate window ranking statistics into rank filters #25622

Description

@gabotechs

Describe the bug

Filtering a window ranking to the first ten rows retains the full input
estimate. The reduced example estimates 73,049 rows and emits ten; the window
input itself is estimated correctly.

To Reproduce

From the repository root, with
PR #25570 applied:

cargo build --profile ci --locked -p datafusion-benchmarks --bin dfbench
repro_dir=$(mktemp -d)
mkdir -p "$repro_dir/data"
curl --fail --location \
  https://raw.githubusercontent.com/apache/datafusion-benchmarks/32f67477f692453616d2fa98a05a37c5eb4cae49/tpcds/data/sf1/date_dim.parquet \
  -o "$repro_dir/data/date_dim.parquet"
cat > "$repro_dir/repro.sql" <<'SQL'
SET datafusion.execution.target_partitions = 1;
SET datafusion.optimizer.enable_dynamic_filter_pushdown = false;
SELECT * FROM (
  SELECT d_date_sk, ROW_NUMBER() OVER (ORDER BY d_date_sk) AS rn
  FROM date_dim
) t WHERE rn <= 10;
SELECT * FROM (
  SELECT d_date_sk, RANK() OVER (ORDER BY d_date_sk) AS rn
  FROM date_dim
) t WHERE rn <= 10;
SQL
target/ci/dfbench statistics \
  --path "$repro_dir/data" --query_path "$repro_dir/repro.sql"

Observed at
this revision
with the pinned SF1 data above. Inspect the SELECT reports in order; ignore the
empty SET reports.

Predicate FilterExec node Estimated rows Actual rows
ROW_NUMBER() <= 10 0.0 73,049 10
RANK() <= 10 0.0 73,049 10

Expected behavior

Expose usable ranking-column statistics. A global ROW_NUMBER() <= 10 has an
upper bound of ten rows. RANK needs tie-aware estimation, and partitioned
windows need partition-count estimates; neither can generally use that same
global bound.

Additional context

Reduced from the ranking filters in TPC-DS Q49/Q67/Q70, with ROW_NUMBER added as
a stronger semantic control. BoundedWindowAggExec currently appends unknown
column statistics for its window outputs. This concerns estimates, independently
of whether execution can stop early.

Part of #25610.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions