Skip to content

Use null counts when estimating IS NULL and IS NOT NULL #25620

Description

@gabotechs

Describe the bug

A bare null check falls back to 20% selectivity even when input column
statistics contain a null count. IS NULL and IS NOT NULL receive the same
estimate.

To Reproduce

From the repository root, with
PR #25570 applied:

cargo build --profile ci --locked -p datafusion-benchmarks --bin dfbench
repro_dir=$(mktemp -d)
mkdir -p "$repro_dir/data"
curl --fail --location \
  https://raw.githubusercontent.com/apache/datafusion-benchmarks/32f67477f692453616d2fa98a05a37c5eb4cae49/tpcds/data/sf1/store_sales.parquet \
  -o "$repro_dir/data/store_sales.parquet"
cat > "$repro_dir/repro.sql" <<'SQL'
SET datafusion.execution.target_partitions = 1;
SET datafusion.optimizer.enable_dynamic_filter_pushdown = false;
SELECT ss_item_sk FROM store_sales WHERE ss_addr_sk IS NULL;
SELECT ss_item_sk FROM store_sales WHERE ss_addr_sk IS NOT NULL;
SQL
target/ci/dfbench statistics \
  --path "$repro_dir/data" --query_path "$repro_dir/repro.sql"

Observed at
this revision
with the pinned SF1 data above. Inspect the SELECT reports in order; ignore the
empty SET reports.

Predicate FilterExec node Estimated rows Actual rows
IS NULL 0 576,081 129,975
IS NOT NULL 0 576,081 2,750,429

Expected behavior

Use the input null count for IS NULL and subtract it from the input row count
for IS NOT NULL, retaining the appropriate precision. Here those estimates would
be 129,975 and 2,750,429.

Additional context

Reduced from TPC-DS Q76. The input has 2,880,404 rows; a diagnostic capture
confirms its ss_addr_sk null count is available as Inexact(129975). Related:
#24942 concerns removing
provably redundant null checks; this report concerns estimating checks that
actually remove rows.

Part of #25610.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions