Describe the bug
Null-aware LeftMark hash joins inspect the physical null bitmap of dictionary-encoded join keys instead of their logical nulls. A DictionaryArray key can be physically valid while pointing to a NULL dictionary value, so the join misses that NULL and produces an incorrect FALSE mark where SQL three-valued logic requires NULL.
The problem affects both sides of an uncorrelated null-aware mark join:
- A logical
NULL build key is treated as non-null when the final mark column is built.
- A logical
NULL probe key is not recorded in the shared probe summary, including when it is seen by a different probe partition from the partition that completes the join.
To Reproduce
At the HashJoinExec level, use dictionary keys whose key bitmap contains no nulls but whose dictionary values contain a null:
- Build values
[1, NULL, 4], probe values [1, 2, 3]: the expected marks are [TRUE, NULL, FALSE], but the build-side logical null is marked FALSE.
- Build values
[1, 4], probe values containing [1, NULL]: the expected marks are [TRUE, NULL], but the unmatched build row is marked FALSE.
Expected behavior
Null-aware LeftMark joins should use Arrow logical-null semantics for dictionary arrays on both build and probe sides, producing the same SQL results as non-dictionary arrays.
Describe the bug
Null-aware
LeftMarkhash joins inspect the physical null bitmap of dictionary-encoded join keys instead of their logical nulls. ADictionaryArraykey can be physically valid while pointing to aNULLdictionary value, so the join misses thatNULLand produces an incorrectFALSEmark where SQL three-valued logic requiresNULL.The problem affects both sides of an uncorrelated null-aware mark join:
NULLbuild key is treated as non-null when the final mark column is built.NULLprobe key is not recorded in the shared probe summary, including when it is seen by a different probe partition from the partition that completes the join.To Reproduce
At the
HashJoinExeclevel, use dictionary keys whose key bitmap contains no nulls but whose dictionary values contain a null:[1, NULL, 4], probe values[1, 2, 3]: the expected marks are[TRUE, NULL, FALSE], but the build-side logical null is markedFALSE.[1, 4], probe values containing[1, NULL]: the expected marks are[TRUE, NULL], but the unmatched build row is markedFALSE.Expected behavior
Null-aware
LeftMarkjoins should use Arrow logical-null semantics for dictionary arrays on both build and probe sides, producing the same SQL results as non-dictionary arrays.