Repository navigation
Arrow scan: a NULL row passes a pushed-down <= filter against a NaN constant #615
Description
What happens?
When a WHERE b <= 'nan'::DOUBLE filter is pushed into an Arrow scan, the scan returns rows where b is NULL.
The same predicate over same rows copied to a native table (correctly) excludes the null rows.
This should exclude the nulls (NULL <= x is false): The bug appears to be in the arrow filter_pushdown: disabling the optimizer/pushdowns returns the correct results.
To Reproduce
import duckdb
import pyarrow as pa
with duckdb.connect() as con:
print(f"duckdb {duckdb.__version__}, pyarrow {pa.__version__}", "source id:", con.execute("SELECT source_id FROM pragma_version()").fetchone()[0])
arrow_table = pa.table({"b": pa.array([1.0, float("nan"), None])})
con.register("arrow_t", arrow_table)
result_native = con.execute("create table native_t as select * from arrow_t;select b from native_t where b <= 'nan'::DOUBLE").fetchall()
result = con.execute("select b from arrow_t where b <= 'nan'::DOUBLE").fetchall()
con.execute("PRAGMA disable_optimizer;")
result_no_optimizer = con.execute("select b from arrow_t where b <= 'nan'::DOUBLE").fetchall()
print(f"{result_native=}")
print(f"{result=}")
print(f"{result_no_optimizer=}")Result: Note that "result" differs from others
duckdb 1.6.0.dev379, pyarrow 25.0.1 source id: a00803f768
result_native=[(1.0,), (nan,)]
result=[(1.0,), (nan,), (None,)]
result_no_optimizer=[(1.0,), (nan,)]
OS:
Windows & Ubuntu WSL
DuckDB Package Version:
1.5.5 and 1.6.0.dev379
Python Version:
3.14
Full Name:
Paul T
Affiliation:
Iqmo
What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.
I have tested with a nightly build
Did you include all relevant data sets for reproducing the issue?
Yes
Did you include all code required to reproduce the issue?
- Yes, I have
Did you include all relevant configuration to reproduce the issue?
- Yes, I have
Activity
mohamd764 commentedon Oct 12, 2026
Reproduced on duckdb 1.5.6 / pyarrow 26.0.0 (Linux). I ran the same table through all six comparison operators against
'nan'::DOUBLE, and only<=lets the NULL row through.<,>=,>,=and<>all match the native table. I see the same thing when the source is a polars DataFrame.I think the cause is the NaN special case in
src/arrow/pyarrow_filter_pushdown.cpp(NaNCompare):A literal
truekeeps NULL rows as well. The other branches go throughis_nan(), which is null for a null input, so those rows get dropped as expected. Returningcol.attr("is_valid")()in that branch should keep every non-null value (NaN included) and drop the nulls, which is what the native table returns.src/arrow/polars_filter_pushdown.cpphas the same branch returningpl.lit(true), so it probably needs the equivalentis_not_null()change. I haven't tested the polars lazy pushdown path on its own, though.NaN handling in Arrow scans (#657)
evertlammerts commentedon Oct 14, 2026
Thanks! Fixed in #657