Repository navigation
core: a field reference into a nullable struct, list or map derives a required type #1338
Description
FieldReference.dereferenceStruct, dereferenceList and dereferenceMap type the result as the member's declared type and ignore the container's own nullability, so reading a field of a struct that may be null is typed as though it can never be null.
FieldReference.newRootStructReference(0, N.struct(R.I32, N.FP64)).dereferenceStruct(0).getType();
// I32{nullable=false}
FieldReference.newRootStructReference(0, N.list(R.I32)).dereferenceList(0).getType();
// I32{nullable=false}
FieldReference.newRootStructReference(0, N.map(R.STRING, R.I32))
.dereferenceMap(ExpressionCreator.string(false, "k")).getType();
// I32{nullable=false}Measured on main at 4bb02974. The derivation is StructFieldFinder, ListIndexFinder and MapKeyFinder.getReferencedType in FieldReference.java.
The spec does not settle it. site/docs/expressions/field_references.md (v0.102.0) gives a struct field reference's type as "Type of field referenced" and states no nullability rule for any kind of reference. The type is never serialized (StructField carries only field and child), so every consumer derives it itself. substrait-go (StructFieldRef.GetType) and substrait-python (type_inference.py) also return the declared member type unchanged.
This matters more since #1317: a nullable struct now keeps its required fields through Calcite instead of having them widened to nullable, so this derivation now decides the type of any reference into one. The open question is whether :core should make the derived type nullable when the container is, and if so, whether the spec should say so first.
Activity
nielspardon commentedon Oct 7, 2026
A related case the container rule doesn't cover: a key lookup on a required map with required values still derives a required type, although a missing key yields null. For
mof typeMAP<VARCHAR NOT NULL, INTEGER NOT NULL> NOT NULL, Calcite typesITEM(m, 'missing')as nullableINTEGER, but isthmus converts it to a selection typedi32(required), so a consumer trusting the plan could foldm['k'] IS NULLto false. Measured onmainat c130b20. The spec's Map Key row is silent on missing keys too; substrait-go keeps the declared value type, and substrait-validator has a FIXME on exactly this.nielspardon commentedon Oct 9, 2026
One more case the container rule doesn't cover: a list selection whose offset is out of range returns null even when the list and its elements are required, but it still derives the required element type. For
aof typeINTEGER NOT NULL ARRAY NOT NULL, Calcite typesITEM(a, 4)as nullableINTEGER, while isthmus converts it to alist_elementselection typedi32(required). Measured onmainat eb8d3f7. Once #1261 lands this also coversa[0], negative indexes and a NULL index, which convert tolist_element { offset: 2147483647 }: null on every row, but still typed required. The spec says out-of-range list offsets return null (field_references.md, v0.102.0), so the derived type would have to be nullable to match.