fix: correlated NOT IN with a non-equality correlation returns wrong results by adriangb · Pull Request #25339 · apache/datafusion

adriangb · 2026-09-15T17:49:13Z

Which issue does this PR close?

Closes Wrong results: correlated NOT IN with a non-equality correlation returns no rows (null-aware LeftAnti join ignores the residual filter) #25336.

Rationale for this change

A correlated NOT IN subquery gives wrong results when the correlation is not an equality and the subquery column contains NULL. There is no error and no warning.

CREATE TABLE t1(id INT, z INT) AS VALUES (1,10), (2,20), (NULL,30), (4,40);
CREATE TABLE t2(id INT, z INT) AS VALUES (1,5), (NULL,50);

SELECT id FROM t1 WHERE id NOT IN (SELECT t2.id FROM t2 WHERE t2.z < t1.z) ORDER BY id;
-- main: (no rows)    this PR, DuckDB, PostgreSQL: 2, 4

SELECT id FROM t1 WHERE NOT (id IN (SELECT t2.id FROM t2 WHERE t2.z < t1.z)) OR id = 4 ORDER BY id;
-- main: 2, 4, NULL   this PR, DuckDB, PostgreSQL: 2, 4

The same gap also makes an equality-correlated NOT IN in a WHERE clause fail to plan:

SELECT id FROM t1 WHERE id NOT IN (SELECT t2.id FROM t2 WHERE t2.z = t1.z);
-- main: Error during planning: null_aware LeftAnti joins only support single column join key, got 2 columns

The fix sketch in the issue (turn off null_aware for a LeftAnti join that has a join filter) does not work. I tried it: the plain anti join ignores NULLs completely, so queries that are correct today start to return rows. For example, id NOT IN (SELECT t2.id FROM t2 WHERE t2.z > t1.z) must return no rows, and returns 1, 2, 4, NULL with that change.

What changes are included in this PR?

The hash join already had the right mechanism for correlated NOT IN mark joins with equality correlation keys: a per-build-row bitmap that records "this row's NOT IN is UNKNOWN". This PR uses that mechanism for every correlated null-aware join and makes it apply the join filter. The commits are split for review:

HashJoinExec: a null-aware LeftAnti or LeftMark join is correlated when it has correlation scope keys or a join filter. For a NULL value on either side, the join finds the candidate (build, probe) row pairs through the scope key hash map, or takes all pairs when there are no scope keys. The join filter then decides which pairs make a build row UNKNOWN. The LeftAnti final stage drops those rows. The extra work is only for rows that have a NULL value key, so it is zero when the data has no NULLs. JoinSelection swaps a null-aware LeftAnti only when it has a single key and no filter.
DecorrelatePredicateSubquery: a NOT IN mark join with a non-equality correlation is now planned as null-aware. Four EXPLAIN results in subquery.slt change: the mark join now shows null_aware, and in one of them the join is no longer swapped to RightMark, because null-aware mark joins are never swapped.
Tests.
Benchmarks: correctness asserts for Q05 to Q08 of the null_aware_join suite (bench: SQL benchmark suite for null-aware (NOT IN) joins #25386), and expect_plan null_aware: true for Q08. These checks fail on main and pass with this PR.
Performance, from the review: a build row stays UNKNOWN after it is marked. Thus the join now skips candidate pairs whose build row is already marked, and it does not evaluate the join filter for those pairs. Without scope keys, the pairing also drops the marked build rows after each chunk of pairs, and it stops when no unmarked build row is left. At the default benchmark sizes this takes Q06 from 20.6M candidate pairs to 240K, and Q07 from 20.6M to 169K. A consequence is that the filter is evaluated for fewer pairs, so a filter that gives an error on a skipped pair no longer causes an error. This is the same as the short-circuit behavior of AND.

null_aware_join suite, release build, Apple M4 Pro. Each number is the median of 40 iterations, taken as 4 interleaved rounds. "Before" is commit 4 of this PR and "after" is commit 5. Q01 to Q03 do not use this code path.

Query	before	after
Q01	19.4 ms	17.7 ms
Q02	16.6 ms	15.0 ms
Q03	16.2 ms	14.7 ms
Q04	0.8 ms	0.8 ms	1.00x
Q05	2.7 ms	1.0 ms	2.70x
Q06	94.8 ms	1.0 ms	95x
Q07	93.2 ms	1.0 ms	93x
Q08	14.7 ms	12.0 ms	1.22x

Q06 and Q07 now cost about the same as Q04, which is the same shape with no NULL. Thus the correct result for the non-equality correlation is now nearly free.

Q08 is the remaining case. It has an equality correlation, so the candidate pairs come from the scope-key hash lookup, and the marked build rows are removed only after that lookup. At the default sizes it still enumerates 6.25M pairs, of which 86K survive. #25438 tracks narrowing the lookup itself.

The benchmark bot compares this PR with main (results). On main, Q05 to Q08 give wrong results, so the main column for those rows is the time to calculate an incorrect result (see #25386).

Query	`main`	this PR
Q01	21.9 ms	22.6 ms
Q02	21.6 ms	22.3 ms
Q03	21.1 ms	21.5 ms
Q04	1.43 ms	1.45 ms	1.01x
Q05	1.43 ms (wrong result)	1.93 ms	1.35x
Q06	1.42 ms (wrong result)	1.86 ms	1.32x
Q07	1.39 ms (wrong result)	1.77 ms	1.27x
Q08	1.75 ms (wrong result)	21.2 ms	12.1x

For Q05 to Q07, a correct result costs about 30% more than the incorrect fast result on main.

For an absolute reference, the same eight queries in datafusion-cli (this PR) and in DuckDB 1.5.2, on the same tables and the default sizes, Apple M4 Pro, median of 3 EXPLAIN ANALYZE runs. Both engines give the same result for all eight queries.

Query	DataFusion	DuckDB
Q01	5 ms	14 ms
Q02	3 ms	13 ms
Q03	2 ms	13 ms
Q04	< 1 ms	117 ms
Q05	1 ms	116 ms
Q06	1 ms	114 ms
Q07	1 ms	3540 ms
Q08	12 ms	22 ms

DataFusion is faster than DuckDB for every query of this suite. Thus the cost of the correct result is small in absolute terms, and Q08 is 12 ms against DuckDB's 22 ms.

What is the testing strategy for this PR?

New sqllogictest cases in null_aware_anti_join.slt and null_aware_mark_join.slt: the queries from the issue, NULL outer values with empty and non-empty subquery results, equality plus non-equality correlation, a filter on the subquery value itself, the positive IN form, the mark column through IS NULL / IS TRUE / IS FALSE / NOT ... OR and directly in a SELECT list, and runs with batch_size = 1. I checked all expected results with DuckDB 1.5.2 and PostgreSQL 17.11. 14 of these cases fail on main.
New HashJoinExec unit tests for a null-aware LeftAnti and LeftMark join that has a join filter and no scope keys, at all batch sizes.
The null_aware_join benchmark suite (bench: SQL benchmark suite for null-aware (NOT IN) joins #25386) now checks the result of all eight queries. Each assert compares the NOT IN count with a reference count that does not use NOT IN, so it holds for all values of NAJ_ROWS and NAJ_LARGE_ROWS. The suite passes at the default sizes and at -r 3000 -l 1001. On main, the asserts for Q05 to Q08 fail. See bench: SQL benchmark suite for null-aware (NOT IN) joins #25386 for the performance numbers.

Are there any user-facing changes?

Queries that returned wrong results now return correct results, and correlated NOT IN with an equality correlation in a WHERE clause no longer fails to plan. There are no public API changes.

🤖 Generated with Claude Code

codecov-commenter · 2026-09-15T18:18:41Z

Codecov Report

❌ Patch coverage is 93.72822% with 18 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.34%. Comparing base (edc936f) to head (3b38f6b).

Files with missing lines	Patch %	Lines
...fusion/physical-plan/src/joins/hash_join/stream.rs	93.43%	6 Missing and 7 partials ⚠️
...tafusion/physical-plan/src/joins/hash_join/exec.rs	94.18%	1 Missing and 4 partials ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##             main   #25339    +/-   ##
========================================
  Coverage   82.33%   82.34%            
========================================
  Files        1137     1137            
  Lines      432498   432653   +155     
  Branches   432498   432653   +155     
========================================
+ Hits       356116   356257   +141     
- Misses      54843    54850     +7     
- Partials    21539    21546     +7

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

kosiew

@adriangb,

Thanks for working on this. The approach looks good to me, especially keeping the residual join filter involved when determining the per-build-row UNKNOWN state for correlated NOT IN. I also like the added coverage for both anti and mark joins.

I left one non-blocking performance suggestion below. Nothing that needs to hold up the PR.

kosiew · 2026-09-16T03:57:16Z

+            None => {
+                let probe_rows =
+                    UInt32Array::from_iter_values(0..state.batch.num_rows() as u32);
+                for_each_cross_product(


One potential performance concern here: when there are no scope keys, we evaluate the residual filter for every NULL build row × probe row pair. The symmetric path below does the same for NULL probe rows. For a nullable non-equality-correlated NOT IN, that could add quadratic work, including for build rows that have already been marked UNKNOWN.

Would it be worth adding a small bounded benchmark or targeted performance regression test for this path? As a follow-up optimization, we might also be able to skip build rows that are already marked UNKNOWN, although we'd need to be careful about volatile or erroring filter expressions.

jayzhan211

Thanks @adriangb , 2 non-blocking suggestions

jayzhan211 · 2026-09-16T15:04:31Z

+            None => {
+                let build_rows =
+                    UInt64Array::from_iter_values(0..left_data.batch().num_rows() as u64);
+                for_each_cross_product(


The case with no scope keys re-checks build rows that are already marked UNKNOWN

Without scope keys, case 2 evaluates the filter for every (build row × NULL probe row) pair in every probe batch, including build rows already set in null_indices_bitmap. Those bits never clear, so that work is wasted. 20K outer × 10K NULL inner with i.z < o.z spends 8.85s in join_time (debug build). Skip marked rows and stop once none are left (same for case 1 at :1466):

None => { let num_build_rows = left_data.batch().num_rows(); for probe_rows in null_probe_rows.values().chunks(batch_size.max(1)) { let build_rows = { let bitmap = left_data.null_indices_bitmap().lock(); UInt64Array::from_iter_values( (0..num_build_rows) .filter(|i| !bitmap.get_bit(*i)) .map(|i| i as u64), ) }; if build_rows.is_empty() { break; } let probe_rows = UInt32Array::from(probe_rows.to_vec()); for_each_cross_product(&build_rows, &probe_rows, batch_size, &mut mark)?; } }

Fine to handle in a follow-up

jayzhan211 · 2026-09-16T15:06:40Z

        num_keys: usize,
        has_filter: bool,
    ) -> Result<Self> {
+        let correlated = num_keys > 1 || has_filter;


correlated = num_keys > 1 || has_filter assumes on[0] is the NOT IN value key. When the value has no outer columns, 1 = i.id is pushed into the subquery, so on[0] becomes the correlation key o.g = i.g, and a NULL o.g is marked UNKNOWN:

CREATE TABLE o(id INT, g INT, z INT) AS VALUES (1,1,10),(2,NULL,10),(3,2,10); CREATE TABLE i(id INT, g INT, z INT) AS VALUES (1,1,5),(5,2,5),(NULL,3,5); SELECT id FROM o WHERE 1 NOT IN (SELECT i.id FROM i WHERE i.g = o.g AND i.z < o.z); -- expected 2, 3; returns 3

The form without AND i.z < o.z is also wrong and doesn't go through the new code, so this predates the PR. Fine as a follow-up: in build_join, only set null_aware when the in-predicate's outer side references a left column.

Agreed, this is a bug in main:

main: R1 (with i.z < o.z) → 1 row R2 (equality only) → 1 row PR: R1 → 1 row R2 → 1 row

adriangb · 2026-09-16T20:02:17Z

@jayzhan211 @kosiew I opened #25386 w/ benchmarks for this change. Could we merge that first so we can look at before/afterS?

HashJoinExec's Debug output did not include null_aware, so `expect_plan HashJoinExec` also passed for a plain anti join. Add the field to Debug and require `null_aware: true` on Q02-Q07. Q01 has non-nullable keys, so it is not null-aware. Q08 plans as a plain mark join on main until apache#25339 lands, so it keeps only the HashJoinExec check here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Each assert compares the NOT IN count with a reference count that does not use NOT IN, so it holds at every NAJ_ROWS / NAJ_LARGE_ROWS value. Q05-Q08 give wrong results on main (apache#25336), so their asserts go in apache#25339 together with the fix. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

## Which issue does this PR close? - N/A. This PR adds benchmarks only. It is split out of apache#25339 so that the suite is on `main` first, and that PR can then be measured against it. ## Rationale for this change A `NOT IN` subquery becomes a null-aware join. An outer row that finds no match is TRUE only when neither side has a NULL in scope. If a NULL is in scope, the result is UNKNOWN. This decision is cheap for an uncorrelated `NOT IN`. For a correlated `NOT IN`, the correlation predicate stays behind as a join filter. The join must then evaluate that filter for each candidate (build row x probe row) pair, to find which rows the NULLs reach. A non-equality correlation gives no equality key, so there is no scope key to reduce the number of pairs. The cost then grows with the NULL count multiplied by the size of the opposite table. No benchmark measured this shape, so there was no way to see the cost, or to tell a change from noise. Review on apache#25339 asked for this benchmark. These are the measured results for apache#25339. Each number is the median of 60 iterations, taken as 6 interleaved rounds of 10 iterations on an Apple M4 Pro in release mode. The two sides are the base commit of apache#25339 and its head commit, each with this suite applied, so the comparison isolates the change in that PR. | Query | Shape | base | apache#25339 | | |---|---|---|---|---| | Q01 | uncorrelated, non-nullable keys | 17.9 ms | 17.7 ms | 0.99x | | Q02 | uncorrelated, 1% NULL subquery side | 14.9 ms | 14.9 ms | 1.00x | | Q03 | uncorrelated, 50% NULL outer side | 14.6 ms | 14.6 ms | 1.00x | | Q04 | correlated, nullable keys, no NULL present | 0.9 ms | 0.9 ms | 0.96x | | Q05 | correlated, 1% NULL outer side | 0.9 ms | 2.9 ms | 3.1x | | Q06 | correlated, 50% NULL outer side | 0.9 ms | 96.2 ms | 109x | | Q07 | correlated, 50% NULL subquery side | 0.8 ms | 94.4 ms | 112x | | Q08 | as Q06, with an equality correlation | 1.1 ms | 15.1 ms | 14x | Q01 to Q04 are the comparable rows, and they show no change. The base gives wrong results for Q05 to Q08, which is the bug that apache#25339 corrects. Thus the base numbers for those four rows are the time to calculate an incorrect result. They show the cost of correct results, not a regression. These are the results at the default sizes. DuckDB agrees with the "correct" column. | Query | correct (apache#25339) | base | |---|---|---| | Q05 | 7460 | 7450 | | Q06 | 5010 | 5000 | | Q07 | 10 | 0 | | Q08 | 5530 | 10000 | Q06 and Q07 are the rows that the review of apache#25339 asked about. They also give the baseline to measure any later optimization of that path against. Q08 has the same NULL fraction as Q06 and is 6 times cheaper, which is the value of the equality correlation. ## What changes are included in this PR? A `null_aware_join` SQL benchmark suite. There are no Rust changes. The runner finds suites in `benchmarks/sql_benchmarks/`, and the load SQL makes each table from `range()`, so there is no data generation step. - Q01 to Q03 are uncorrelated `NOT IN` at different NULL fractions. Their cost is linear with the table size. They are the regression guard for the plain null-aware path. - Q04 is the correlated shape with nullable keys that hold no NULL. It separates the baseline cost of the shape from the per-pair filter work. - Q05 to Q07 are the same correlation at 1% and 50% NULL on each side. This is where that work becomes visible. - Q08 has the same NULL fraction as Q06, but adds an equality correlation. The candidate pairs then come from a hash lookup. The difference between Q06 and Q08 shows the value of the scope key. Both table sizes are knobs. `NAJ_ROWS` (default 10000) sets the size for the correlated queries, whose cost grows with its square. `NAJ_LARGE_ROWS` (default 1000000) sets the size for the uncorrelated queries. ```bash ./bench.sh run null_aware_join # One query, with more rows for the correlated shape NAJ_ROWS=20000 ./bench.sh run null_aware_join 6 ``` This PR also adds the suite to `bench.sh` (including `all`) and documents it in `benchmarks/README.md` and `benchmarks/sql_benchmarks/README.md`. There is one Rust change: the `Debug` output of `HashJoinExec` now includes `null_aware`. The suite's `expect_plan` directive matches that output, so Q02 to Q07 can require `null_aware: true`. Before this change, `expect_plan HashJoinExec` also passed for a plain anti join. ## What is the testing strategy for this PR? This PR adds benchmarks, so it adds no new tests. The existing `checked_in_suites_cover_benchmark_directories` test in `benchmarks/src/sql_benchmark_suite.rs` covers suite discovery, and it passes with the new directory. Each query has these checks: - `expect_plan HashJoinExec`. Q02 to Q07 also require `expect_plan null_aware: true`. Q01 has non-nullable keys, so it is not null-aware. Q08 plans as a plain mark join on `main`, so apache#25339 adds its `null_aware: true` check. - Q01 to Q04 have an `assert` correctness canary. The assert compares the `NOT IN` count with a reference count that does not use `NOT IN`, so it is correct for all values of `NAJ_ROWS` and `NAJ_LARGE_ROWS`. I checked each reference against the `NOT IN` result in DuckDB at six pairs of sizes. The same asserts for Q05 to Q08 fail on `main`, so apache#25339 adds them together with the fix. I ran them on this branch merged with apache#25339, and all eight queries pass at the default sizes and at `-r 3000 -l 1001`. I ran all eight queries on this branch at the default sizes and at `-r 1500 -l 101`. As a counterfactual check on `main`, the Q05 to Q08 asserts fail, and `null_aware: true` fails on Q01 and Q08. Each query also runs on `main` as written. Q08 uses the mark join form on purpose. The plain `WHERE ... NOT IN` form with an equality correlation does not plan on `main`, and a query that runs on only one branch cannot compare two branches. ## Are there any user-facing changes? No. This PR changes benchmarks and documentation only. It does not change library code. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>

adriangb · 2026-09-17T16:31:55Z

run benchmark null_aware_join

adriangbot · 2026-09-17T16:36:41Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c5717854886-2404-6n7np 6.12.94+ #1 SMP Tue Aug 4 08:44:15 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing claude/datafusion-issue-25336-ba2e61 (7162cd7) to c4f72ba (merge-base) diff

Run configuration

run benchmark null_aware_join

Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-09-17T17:04:33Z

Benchmark for this request failed before finishing (Kubernetes reason: BackoffLimitExceeded).

Benchmarks requested: null_aware_join

Runner log (last 40 lines)

2026-09-17T16:35:50.774765Z  INFO runner starting benchmark runner bench_type=Datafusion, pr_url=https://github.com/apache/datafusion/pull/25339, benchmarks=null_aware_join
2026-09-17T16:35:50.845987Z  INFO benchmark_controller::runner::bench_datafusion === Cloning PR branch ===
2026-09-17T16:35:50.846113Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["clone", "--depth=200", "https://github.com/apache/datafusion.git", "/workspace/datafusion-branch"], cwd="/"
2026-09-17T16:35:55.861661Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["fetch", "origin", "refs/pull/25339/head:claude/datafusion-issue-25336-ba2e61", "main"], cwd="/workspace/datafusion-branch"
2026-09-17T16:36:00.863032Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["checkout", "claude/datafusion-issue-25336-ba2e61"], cwd="/workspace/datafusion-branch"
2026-09-17T16:36:05.864827Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["merge-base", "HEAD", "origin/main"], cwd="/workspace/datafusion-branch"
2026-09-17T16:36:10.867319Z  INFO benchmark_controller::runner::bench_datafusion === Cloning merge-base ===
2026-09-17T16:36:10.867334Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["clone", "--depth=200", "https://github.com/apache/datafusion.git", "/workspace/datafusion-base"], cwd="/"
2026-09-17T16:36:15.868761Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["-c", "advice.detachedHead=false", "checkout", "c4f72bad34249798182ac7de605eb63ab134f26e"], cwd="/workspace/datafusion-base"
2026-09-17T16:36:20.871452Z  INFO benchmark_controller::runner::shell running command cmd=rustc, args=["--version"], cwd="/"
2026-09-17T16:36:25.873992Z  INFO benchmark_controller::runner::shell running command cmd=cargo, args=["metadata", "--no-deps", "--format-version", "1"], cwd="/workspace/datafusion-branch/benchmarks"
2026-09-17T16:36:30.878134Z  INFO benchmark_controller::runner::bench_datafusion === Compiling dfbench for PR branch and merge-base in parallel ===
2026-09-17T16:36:30.907956Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["rev-parse", "HEAD"], cwd="/workspace/datafusion-branch"
2026-09-17T16:36:35.913246Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["rev-parse", "HEAD"], cwd="/workspace/datafusion-base"
2026-09-17T16:36:40.923635Z  INFO benchmark_controller::runner::controller_client post_comment _repo=apache/datafusion, _pr_number=25339, job_id=2404
2026-09-17T16:36:41.732143Z  INFO benchmark_controller::runner::bench_datafusion === Waiting for builds ===
2026-09-17T16:51:12.728542Z  INFO benchmark_controller::runner::bench_datafusion === Builds complete ===
2026-09-17T16:51:12.728553Z  INFO benchmark_controller::runner::bench_datafusion === Setting up bench runner ===
2026-09-17T16:51:12.728563Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["clone", "--depth=200", "https://github.com/apache/datafusion.git", "/workspace/datafusion-bench"], cwd="/"
2026-09-17T16:51:17.730532Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["-c", "advice.detachedHead=false", "checkout", "origin/main"], cwd="/workspace/datafusion-bench"
2026-09-17T16:51:22.733337Z  INFO benchmark_controller::runner::shell running command cmd=cp, args=["-r", "/data/tpch-answers/.", "/workspace/datafusion-bench/benchmarks/data/tpch_sf1/answers"], cwd="/"
2026-09-17T16:51:27.737470Z  INFO benchmark_controller::runner::shell running command cmd=cp, args=["-r", "/data/tpch-answers/.", "/workspace/datafusion-bench/benchmarks/data/tpch_sf10/answers"], cwd="/"
2026-09-17T16:51:32.739061Z  INFO benchmark_controller::runner::bench_datafusion ** Creating data if needed for null_aware_join **
2026-09-17T16:51:32.739652Z  INFO benchmark_controller::runner::shell running command cmd=/scripts/cache_data.sh, args=["null_aware_join", "/workspace/datafusion-bench/benchmarks"], cwd="/workspace/datafusion-bench/benchmarks"
2026-09-17T16:51:37.742609Z  INFO benchmark_controller::runner::bench_datafusion ** Running null_aware_join baseline **
2026-09-17T16:51:37.742726Z  INFO benchmark_controller::runner::shell running command (monitored) cmd=env, args=["DATAFUSION_DIR=/workspace/datafusion-base", "RESULTS_NAME=HEAD", "DATAFUSION_RUNTIME_TEMP_DIRECTORY=/workspace/spill-base-null_aware_join", "SQL_CARGO_COMMAND=cargo bench --bench sql -- --save-baseline HEAD", "./bench.sh", "run", "null_aware_join"], cwd="/workspace/datafusion-bench/benchmarks"

Kubernetes message

Job has reached the specified backoff limit

File an issue against this benchmark runner

adriangb · 2026-09-17T17:06:46Z

run benchmark null_aware_join

env:
CARGO_PROFILE_BENCH_LTO: "thin"

adriangbot · 2026-09-17T17:09:43Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c5718298180-2405-v2mjl 6.12.94+ #1 SMP Tue Aug 4 08:44:15 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing claude/datafusion-issue-25336-ba2e61 (7162cd7) to c4f72ba (merge-base) diff

Run configuration

run benchmark null_aware_join
env:
  CARGO_PROFILE_BENCH_LTO: "thin"

Results will be posted here when complete

File an issue against this benchmark runner

adriangb · 2026-09-17T17:20:27Z

run benchmark null_aware_join

env:
  CARGO_PROFILE_BENCH_LTO: "thin"
  CARGO_BUILD_JOBS: "3"

adriangbot · 2026-09-17T17:23:32Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c5718476115-2406-hj6rf 6.12.94+ #1 SMP Tue Aug 4 08:44:15 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing claude/datafusion-issue-25336-ba2e61 (7162cd7) to c4f72ba (merge-base) diff

Run configuration

run benchmark null_aware_join
env:
  CARGO_BUILD_JOBS: "3"
  CARGO_PROFILE_BENCH_LTO: "thin"

Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-09-17T17:55:25Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing claude/datafusion-issue-25336-ba2e61 (7162cd7) to c4f72ba (merge-base) diff

Run configuration

run benchmark null_aware_join
env:
  CARGO_BUILD_JOBS: "3"
  CARGO_PROFILE_BENCH_LTO: "thin"

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                  HEAD                                    claude_datafusion-issue-25336-ba2e61
-----                  ----                                    ------------------------------------
null_aware_join/Q01    1.00     22.9±0.81ms        ? ?/sec     1.01     23.1±0.28ms        ? ?/sec
null_aware_join/Q02    1.00     22.4±0.61ms        ? ?/sec     1.03     23.1±0.56ms        ? ?/sec
null_aware_join/Q03    1.00     21.6±0.28ms        ? ?/sec     1.03     22.3±0.76ms        ? ?/sec
null_aware_join/Q04    1.08  1548.9±115.27µs        ? ?/sec    1.00  1437.0±17.62µs        ? ?/sec
null_aware_join/Q05    1.00  1605.8±130.29µs        ? ?/sec    1.60      2.6±0.17ms        ? ?/sec
null_aware_join/Q06    1.00  1450.2±14.87µs        ? ?/sec     12.14    17.6±0.08ms        ? ?/sec
null_aware_join/Q07    1.00  1484.9±100.60µs        ? ?/sec    11.69    17.4±0.09ms        ? ?/sec
null_aware_join/Q08    1.00      2.0±0.18ms        ? ?/sec     10.78    21.7±0.24ms        ? ?/sec

Resource Usage

null_aware_join — base (merge-base)

Metric	Value
Wall time	470.2s
Peak memory	312.2 MiB
Avg memory	50.1 MiB
CPU user	130.0s
CPU sys	6.8s
Peak spill	0 B

null_aware_join — branch

Metric	Value
Wall time	810.2s
Peak memory	365.8 MiB
Avg memory	30.2 MiB
CPU user	141.1s
CPU sys	3.9s
Peak spill	0 B

File an issue against this benchmark runner

adriangbot · 2026-09-17T17:55:39Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing claude/datafusion-issue-25336-ba2e61 (7162cd7) to c4f72ba (merge-base) diff

Run configuration

run benchmark null_aware_join
env:
  CARGO_PROFILE_BENCH_LTO: "thin"

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                  HEAD                                   claude_datafusion-issue-25336-ba2e61
-----                  ----                                   ------------------------------------
null_aware_join/Q01    1.00     23.0±0.47ms        ? ?/sec    1.01     23.1±0.36ms        ? ?/sec
null_aware_join/Q02    1.00     22.8±0.84ms        ? ?/sec    1.02     23.1±0.30ms        ? ?/sec
null_aware_join/Q03    1.00     22.1±0.95ms        ? ?/sec    1.03     22.7±0.43ms        ? ?/sec
null_aware_join/Q04    1.00  1446.1±25.01µs        ? ?/sec    1.01  1458.8±22.26µs        ? ?/sec
null_aware_join/Q05    1.00  1440.2±25.64µs        ? ?/sec    1.72      2.5±0.03ms        ? ?/sec
null_aware_join/Q06    1.00  1448.0±30.19µs        ? ?/sec    12.39    17.9±0.15ms        ? ?/sec
null_aware_join/Q07    1.00  1424.6±36.45µs        ? ?/sec    12.41    17.7±0.06ms        ? ?/sec
null_aware_join/Q08    1.00  1836.8±31.67µs        ? ?/sec    11.75    21.6±0.05ms        ? ?/sec

Resource Usage

null_aware_join — base (merge-base)

Metric	Value
Wall time	850.4s
Peak memory	328.5 MiB
Avg memory	29.2 MiB
CPU user	127.7s
CPU sys	6.4s
Peak spill	0 B

null_aware_join — branch

Metric	Value
Wall time	850.3s
Peak memory	341.5 MiB
Avg memory	29.4 MiB
CPU user	143.5s
CPU sys	4.2s
Peak spill	0 B

File an issue against this benchmark runner

adriangb · 2026-09-17T19:07:14Z

run benchmark null_aware_join

env:
CARGO_PROFILE_BENCH_LTO: "thin"

adriangbot · 2026-09-17T19:15:40Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c5719775194-2408-7h469 6.12.94+ #1 SMP Tue Aug 4 08:44:15 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing claude/datafusion-issue-25336-ba2e61 (439274b) to c4f72ba (merge-base) diff

Run configuration

run benchmark null_aware_join
env:
  CARGO_PROFILE_BENCH_LTO: "thin"

Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-09-17T19:45:35Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing claude/datafusion-issue-25336-ba2e61 (439274b) to c4f72ba (merge-base) diff

Run configuration

run benchmark null_aware_join
env:
  CARGO_PROFILE_BENCH_LTO: "thin"

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                  HEAD                                   claude_datafusion-issue-25336-ba2e61
-----                  ----                                   ------------------------------------
null_aware_join/Q01    1.00     21.9±0.46ms        ? ?/sec    1.03     22.6±0.51ms        ? ?/sec
null_aware_join/Q02    1.00     21.6±0.41ms        ? ?/sec    1.03     22.3±0.40ms        ? ?/sec
null_aware_join/Q03    1.00     21.1±0.28ms        ? ?/sec    1.02     21.5±0.38ms        ? ?/sec
null_aware_join/Q04    1.00  1433.9±26.32µs        ? ?/sec    1.01  1448.6±29.92µs        ? ?/sec
null_aware_join/Q05    1.00  1431.3±26.44µs        ? ?/sec    1.35  1927.8±67.09µs        ? ?/sec
null_aware_join/Q06    1.00  1415.2±12.03µs        ? ?/sec    1.32  1863.6±12.18µs        ? ?/sec
null_aware_join/Q07    1.00  1390.7±19.90µs        ? ?/sec    1.27  1769.3±27.81µs        ? ?/sec
null_aware_join/Q08    1.00  1751.5±19.14µs        ? ?/sec    12.11    21.2±0.16ms        ? ?/sec

Resource Usage

null_aware_join — base (merge-base)

Metric	Value
Wall time	400.2s
Peak memory	346.6 MiB
Avg memory	57.8 MiB
CPU user	122.8s
CPU sys	5.9s
Peak spill	0 B

null_aware_join — branch

Metric	Value
Wall time	595.2s
Peak memory	311.3 MiB
Avg memory	41.1 MiB
CPU user	126.1s
CPU sys	5.7s
Peak spill	0 B

File an issue against this benchmark runner

adriangb · 2026-09-17T21:34:42Z

DuckDB comparison, and what the Q08 number means

To find out whether the remaining cost of this PR is acceptable, I measured the suite's eight queries in datafusion-cli (this branch, release) and in DuckDB 1.5.2 (EXPLAIN ANALYZE, three runs, median, Apple M4 Pro, same tables from the suite's load.sql). The two engines give the same result for all eight queries.

At the suite's default sizes

Query	Shape	DataFusion	DuckDB	DuckDB / DF
Q01	uncorrelated, non-nullable	5 ms	14 ms	2.8x
Q02	uncorrelated, 1% NULL subquery	3 ms	13 ms	4.2x
Q03	uncorrelated, 50% NULL outer	2 ms	13 ms	6.6x
Q04	correlated, no NULL	< 1 ms	117 ms	> 100x
Q05	correlated, 1% NULL outer	1 ms	116 ms	116x
Q06	correlated, 50% NULL outer	1 ms	114 ms	114x
Q07	correlated, 50% NULL subquery	1 ms	3540 ms	~3500x
Q08	as Q06, plus an equality correlation	12 ms	22 ms	1.8x

datafusion-cli reports milliseconds, so the sub-millisecond queries show as 1 ms.

Each engine against itself

Q06 and Q08 have the same NULL fraction. Q08 adds an equality correlation, which gives the join a scope key. Thus the ratio of the two shows what each engine gets from that key.

`NAJ_ROWS`	DF Q06	DF Q08	DF Q08/Q06	DuckDB Q06	DuckDB Q08	DuckDB Q08/Q06
10,000	1 ms	12 ms	12x	114 ms	22 ms	0.19x
30,000	1 ms	38.5 ms	38x	329 ms	49.7 ms	0.15x
100,000	13 ms	418.5 ms	32x	1110 ms	147 ms	0.13x

The equality correlation makes Q08 about 7x cheaper than Q06 in DuckDB. In DataFusion it makes Q08 more than 10x more expensive than Q06. So the Q08 number is not a property of the shape. It is a property of our scope-key path.

The cause is visible in the pair counts. Without a scope key, the join now drops the build rows that are already UNKNOWN after each chunk of candidate pairs, so Q06 evaluates 240K pairs instead of 20.6M. With a scope key, the candidate pairs come from the hash lookup, and the marked rows are removed only after that lookup, so Q08 still enumerates 6.25M pairs and keeps 86K of them. Q08's cost grows with the square of the table size, while DuckDB's grows about linearly. DuckDB is faster than DataFusion for Q08 from about 100,000 rows.

Conclusion

For the non-equality correlation, which is the bug this PR closes, the correct result is now nearly free: Q06 and Q07 cost about the same as Q04 (the same shape with no NULL), and about 30% more than the incorrect fast result on main.
For the uncorrelated shapes (Q01 to Q03) this PR changes nothing.
Q08 is a real gap, not a measurement artifact. It is pre-existing in the scope-key path, and this PR makes it visible because the other correlated queries are now fast. I suggest we track it as a follow-up: narrow the scope-key lookup with the UNKNOWN bitmap, so that the equality correlation reduces work instead of adding it.

Filed as #25438.

adriangb · 2026-09-17T21:40:31Z

Thanks for review @kosiew and @jayzhan211. Since I've made some implementation changes and posted new benches I'll give you an opportunity to re-review before we merge this, but it's looking ready from my end.

…s UNKNOWN A null-aware LeftAnti join with a join filter ignored the filter for NULL keys: one NULL probe key removed every build row, even when the filter excluded that NULL row for every build row. A null-aware LeftAnti join with correlation scope keys failed to plan. Treat a null-aware LeftAnti or LeftMark join as correlated when it has scope keys or a join filter. Correlated joins record the UNKNOWN decision per build row in the null-indices bitmap: the candidate (build, probe) pairs come from the scope map, or from all pairs when there are no scope keys, and the join filter decides which pairs count. The LeftAnti final stage drops the rows marked UNKNOWN. JoinSelection only swaps an uncorrelated null-aware LeftAnti. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…ware The hash join now applies the join filter when it marks UNKNOWN rows, so a NOT IN mark join no longer needs to fall back to a non null-aware join when a non-equality correlation stays behind as a join filter. The fallback gave FALSE instead of NULL, so NOT (x IN (...)) returned extra rows. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Adds sqllogictest regression tests for apache#25336 (expected results checked with DuckDB and PostgreSQL) and HashJoinExec unit tests for null-aware LeftAnti and LeftMark joins that have a join filter and no scope keys. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…ll_aware These asserts fail on main (apache#25336) and pass with the fix. Q08 is a null-aware mark join once the fix lands. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

… joins A build row stays UNKNOWN once it is marked. The candidate pairing now removes pairs whose build row is already marked before it evaluates the join filter. Without correlation scope keys, the pairing also drops the marked build rows after each chunk of pairs, and it stops when no unmarked build row is left. For `NAJ_ROWS=10000` of the `null_aware_join` benchmark, this takes Q06 from 20.6M candidate pairs to 240K, and Q07 from 20.6M to 169K. A consequence is that the join filter is evaluated for fewer pairs, so a filter that gives an error only for a skipped pair no longer gives that error. This is the same as the short-circuit behavior of `AND`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

kosiew

@adriangb,

Thanks for the follow-up. I went through the changes again and don't see any new blocking issues.

The residual-filter-aware UNKNOWN handling for correlated null-aware NOT IN anti/mark hash joins looks good, and the follow-up in 3b38f6be6c addresses the concern about re-checking build rows that are already UNKNOWN by using retain_unmarked / for_each_unmarked_cross_product.

On the performance side, the null_aware_join correctness and benchmark coverage, together with the candidate-pruning follow-up, addresses the earlier benchmark request. The follow-up may evaluate fewer residual-filter pairs once an UNKNOWN result is found, but I wasn't able to establish a deterministic contract violation from that behavior.

Looks good to me. Thanks for working through the review feedback!

github-actions Bot added optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt) physical-plan Changes to the physical-plan crate labels Sep 15, 2026

adriangb mentioned this pull request Sep 15, 2026

Wrong results: constant NOT IN (subquery) in a WHERE clause ignores NULLs in the subquery #25340

Closed

adriangb requested review from kosiew and neilconway and removed request for neilconway September 15, 2026 20:17

adriangb mentioned this pull request Sep 15, 2026

Wrong results: correlated NOT IN (subquery) ignores NULLs in the subquery #25347

Open

kosiew approved these changes Sep 16, 2026

View reviewed changes

adriangb mentioned this pull request Sep 16, 2026

perf: use one null-aware mark join for hashable IN subqueries in projections #25338

Open

jayzhan211 approved these changes Sep 16, 2026

View reviewed changes

adriangb mentioned this pull request Sep 16, 2026

bench: SQL benchmark suite for null-aware (NOT IN) joins #25386

Merged

adriangb force-pushed the claude/datafusion-issue-25336-ba2e61 branch from e9a4ea3 to 31f4dcd Compare September 17, 2026 15:14

adriangb mentioned this pull request Sep 17, 2026

Force thin LTO and a job cap on every build the runner starts adriangb/datafusion-benchmarking#35

Merged

adriangb force-pushed the claude/datafusion-issue-25336-ba2e61 branch from 7162cd7 to 439274b Compare September 17, 2026 19:06

adriangb mentioned this pull request Sep 17, 2026

Null-aware join: an equality correlation makes a correlated NOT IN slower, not faster #25438

Open

adriangb mentioned this pull request Sep 17, 2026

bench: keep the expect_plan check out of the measured region #25441

Open

adriangb and others added 5 commits September 17, 2026 18:08

bench: add correctness asserts to null_aware_join Q05-Q08, pin Q08 nu…

f6d74b4

…ll_aware These asserts fail on main (apache#25336) and pass with the fix. Q08 is a null-aware mark join once the fix lands. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

adriangb force-pushed the claude/datafusion-issue-25336-ba2e61 branch from 439274b to 3b38f6b Compare September 17, 2026 23:11

kosiew approved these changes Sep 18, 2026

View reviewed changes

Conversation

adriangb commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

What is the testing strategy for this PR?

Are there any user-facing changes?

Uh oh!

codecov-commenter commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

kosiew left a comment

Choose a reason for hiding this comment

Uh oh!

kosiew Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

jayzhan211 left a comment

Choose a reason for hiding this comment

Uh oh!

jayzhan211 Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

jayzhan211 Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

adriangb Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

adriangb commented Sep 16, 2026

Uh oh!

adriangb commented Sep 17, 2026

Uh oh!

adriangbot commented Sep 17, 2026

Uh oh!

adriangbot commented Sep 17, 2026

Uh oh!

adriangb commented Sep 17, 2026

Uh oh!

adriangbot commented Sep 17, 2026

Uh oh!

adriangb commented Sep 17, 2026

Uh oh!

adriangbot commented Sep 17, 2026

Uh oh!

adriangbot commented Sep 17, 2026

Uh oh!

adriangbot commented Sep 17, 2026

Uh oh!

adriangb commented Sep 17, 2026

Uh oh!

adriangbot commented Sep 17, 2026

Uh oh!

adriangbot commented Sep 17, 2026

Uh oh!

adriangb commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

DuckDB comparison, and what the Q08 number means

At the suite's default sizes

Each engine against itself

Conclusion

Uh oh!

adriangb commented Sep 17, 2026

Uh oh!

kosiew left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

adriangb commented Sep 15, 2026 •

edited

Loading

codecov-commenter commented Sep 15, 2026 •

edited

Loading

adriangb commented Sep 17, 2026 •

edited

Loading

kosiew left a comment •

edited

Loading