Describe the bug
Replacing a month-key literal with an equivalent scalar subquery changes the
filter estimate from 31 to 14,610 rows. Both return the same 29 rows.
To Reproduce
From the repository root, with
PR #25570 applied:
cargo build --profile ci --locked -p datafusion-benchmarks --bin dfbench
repro_dir=$(mktemp -d)
mkdir -p "$repro_dir/data"
curl --fail --location \
https://raw.githubusercontent.com/apache/datafusion-benchmarks/32f67477f692453616d2fa98a05a37c5eb4cae49/tpcds/data/sf1/date_dim.parquet \
-o "$repro_dir/data/date_dim.parquet"
cat > "$repro_dir/repro.sql" <<'SQL'
SET datafusion.execution.target_partitions = 1;
SET datafusion.optimizer.enable_dynamic_filter_pushdown = false;
SELECT d_date_sk FROM date_dim WHERE d_month_seq = 1201;
SELECT d_date_sk FROM date_dim
WHERE d_month_seq = (
SELECT MAX(d_month_seq) FROM date_dim WHERE d_year = 2000 AND d_moy = 2
);
SQL
target/ci/dfbench statistics \
--path "$repro_dir/data" --query_path "$repro_dir/repro.sql"
Observed at
this revision
with the pinned SF1 data above. Inspect the SELECT reports in order; ignore the
empty SET reports.
| Predicate |
FilterExec node |
Estimated rows |
Actual rows |
| Literal equality |
0 |
31 |
29 |
| Scalar-subquery equality |
0.0 |
14,610 |
29 |
Expected behavior
When the scalar value is unavailable during planning, use available
equality-selectivity information, such as the left-hand NDV or a suitable
range-based estimate, instead of an unconditional 20%. This should not require
executing arbitrary subqueries during planning.
Additional context
Reduced from the scalar month lookup in TPC-DS Q6; MAX replaces its DISTINCT
lookup to keep the reproducer scalar by construction. The outer input estimate
is correct at 73,049 rows. This is an estimation-policy improvement, not a
query-result correctness bug.
Part of #25610.
Describe the bug
Replacing a month-key literal with an equivalent scalar subquery changes the
filter estimate from 31 to 14,610 rows. Both return the same 29 rows.
To Reproduce
From the repository root, with
PR #25570 applied:
Observed at
this revision
with the pinned SF1 data above. Inspect the SELECT reports in order; ignore the
empty SET reports.
00.0Expected behavior
When the scalar value is unavailable during planning, use available
equality-selectivity information, such as the left-hand NDV or a suitable
range-based estimate, instead of an unconditional 20%. This should not require
executing arbitrary subqueries during planning.
Additional context
Reduced from the scalar month lookup in TPC-DS Q6; MAX replaces its DISTINCT
lookup to keep the reproducer scalar by construction. The outer input estimate
is correct at 73,049 rows. This is an estimation-policy improvement, not a
query-result correctness bug.
Part of #25610.