Skip to content

Estimate equality to a scalar subquery without a blanket 20% fallback #25623

Description

@gabotechs

Describe the bug

Replacing a month-key literal with an equivalent scalar subquery changes the
filter estimate from 31 to 14,610 rows. Both return the same 29 rows.

To Reproduce

From the repository root, with
PR #25570 applied:

cargo build --profile ci --locked -p datafusion-benchmarks --bin dfbench
repro_dir=$(mktemp -d)
mkdir -p "$repro_dir/data"
curl --fail --location \
  https://raw.githubusercontent.com/apache/datafusion-benchmarks/32f67477f692453616d2fa98a05a37c5eb4cae49/tpcds/data/sf1/date_dim.parquet \
  -o "$repro_dir/data/date_dim.parquet"
cat > "$repro_dir/repro.sql" <<'SQL'
SET datafusion.execution.target_partitions = 1;
SET datafusion.optimizer.enable_dynamic_filter_pushdown = false;
SELECT d_date_sk FROM date_dim WHERE d_month_seq = 1201;
SELECT d_date_sk FROM date_dim
WHERE d_month_seq = (
  SELECT MAX(d_month_seq) FROM date_dim WHERE d_year = 2000 AND d_moy = 2
);
SQL
target/ci/dfbench statistics \
  --path "$repro_dir/data" --query_path "$repro_dir/repro.sql"

Observed at
this revision
with the pinned SF1 data above. Inspect the SELECT reports in order; ignore the
empty SET reports.

Predicate FilterExec node Estimated rows Actual rows
Literal equality 0 31 29
Scalar-subquery equality 0.0 14,610 29

Expected behavior

When the scalar value is unavailable during planning, use available
equality-selectivity information, such as the left-hand NDV or a suitable
range-based estimate, instead of an unconditional 20%. This should not require
executing arbitrary subqueries during planning.

Additional context

Reduced from the scalar month lookup in TPC-DS Q6; MAX replaces its DISTINCT
lookup to keep the reproducer scalar by construction. The outer input estimate
is correct at 73,049 rows. This is an estimation-policy improvement, not a
query-result correctness bug.

Part of #25610.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions