Describe the bug
A bare null check falls back to 20% selectivity even when input column
statistics contain a null count. IS NULL and IS NOT NULL receive the same
estimate.
To Reproduce
From the repository root, with
PR #25570 applied:
cargo build --profile ci --locked -p datafusion-benchmarks --bin dfbench
repro_dir=$(mktemp -d)
mkdir -p "$repro_dir/data"
curl --fail --location \
https://raw.githubusercontent.com/apache/datafusion-benchmarks/32f67477f692453616d2fa98a05a37c5eb4cae49/tpcds/data/sf1/store_sales.parquet \
-o "$repro_dir/data/store_sales.parquet"
cat > "$repro_dir/repro.sql" <<'SQL'
SET datafusion.execution.target_partitions = 1;
SET datafusion.optimizer.enable_dynamic_filter_pushdown = false;
SELECT ss_item_sk FROM store_sales WHERE ss_addr_sk IS NULL;
SELECT ss_item_sk FROM store_sales WHERE ss_addr_sk IS NOT NULL;
SQL
target/ci/dfbench statistics \
--path "$repro_dir/data" --query_path "$repro_dir/repro.sql"
Observed at
this revision
with the pinned SF1 data above. Inspect the SELECT reports in order; ignore the
empty SET reports.
| Predicate |
FilterExec node |
Estimated rows |
Actual rows |
| IS NULL |
0 |
576,081 |
129,975 |
| IS NOT NULL |
0 |
576,081 |
2,750,429 |
Expected behavior
Use the input null count for IS NULL and subtract it from the input row count
for IS NOT NULL, retaining the appropriate precision. Here those estimates would
be 129,975 and 2,750,429.
Additional context
Reduced from TPC-DS Q76. The input has 2,880,404 rows; a diagnostic capture
confirms its ss_addr_sk null count is available as Inexact(129975). Related:
#24942 concerns removing
provably redundant null checks; this report concerns estimating checks that
actually remove rows.
Part of #25610.
Describe the bug
A bare null check falls back to 20% selectivity even when input column
statistics contain a null count. IS NULL and IS NOT NULL receive the same
estimate.
To Reproduce
From the repository root, with
PR #25570 applied:
Observed at
this revision
with the pinned SF1 data above. Inspect the SELECT reports in order; ignore the
empty SET reports.
00Expected behavior
Use the input null count for IS NULL and subtract it from the input row count
for IS NOT NULL, retaining the appropriate precision. Here those estimates would
be 129,975 and 2,750,429.
Additional context
Reduced from TPC-DS Q76. The input has 2,880,404 rows; a diagnostic capture
confirms its ss_addr_sk null count is available as Inexact(129975). Related:
#24942 concerns removing
provably redundant null checks; this report concerns estimating checks that
actually remove rows.
Part of #25610.