Python SDK read_parquet union_by_name fail after version 1.2.2

What happens?

Title

Regression in 1.3.0+: union_by_name fails with "Can't change source type (NULL) to target type (VARCHAR[])" when reading parquet files with mixed NULL/LIST types

DuckDB Version

Working version: 1.2.2
Broken versions: 1.3.0, 1.3.1 (and later)

Environment

OS: Linux
Python: 3.12.9
pandas: (latest)

Description

Starting with DuckDB 1.3.0, reading multiple parquet files with union_by_name=True fails when:

Some parquet files have a column stored as NULL type (because all values are null in that file)
Other parquet files have the same column properly typed as VARCHAR[] (array/list of strings)

This worked correctly in DuckDB 1.2.2 but now throws:

BinderException: Binder Error: Can't change source type ("NULL") to target type (VARCHAR[]), type conversion not allowed

Expected Behavior

When union_by_name=True is set, DuckDB should merge schemas gracefully, treating NULL-typed columns as compatible with any target type (similar to how pandas handles this).

Actual Behavior

DuckDB 1.3.0+ throws a BinderException and refuses to read the files, even though union_by_name=True is explicitly designed to handle schema variations across multiple files.

Root Cause Analysis

Investigation shows:

When a parquet file has ALL NULL values for a column, it's stored with NULL type (e.g., INT32 with NullType() logical type)
Other files with actual data store the same column as BYTE_ARRAY with StringType() or complex types like ListType()
The error specifically mentions VARCHAR[] (array type) suggesting it happens with nested/complex types
This regression appeared between versions 1.2.2 and 1.3.0

How to Reproduce

attached files to test see duckdbtest.zip

import duckdb
print(f"DuckDB version: {duckdb.__version__}")

# Fails with 1.3.0+
try:
    result = duckdb.read_parquet(
        "duckdb_bug_test_files/*.parquet",
        union_by_name=True
    ).df()
    print(f"SUCCESS: Read {len(result)} rows")
except Exception as e:
    print(f"FAILED: {type(e).__name__}: {e}")

To Reproduce

this is only in python SDK

OS:

Linux x86

DuckDB Version:

v1.2.2, v1.3.0 and later

DuckDB Client:

Python

Hardware:

No response

Full Name:

Zack Dai

Affiliation:

Zack Dai

Did you include all relevant configuration (e.g., CPU architecture, Linux distribution) to reproduce the issue?

Yes, I have

Did you include all code required to reproduce the issue?

Yes, I have

Did you include all relevant data sets for reproducing the issue?

Yes

@szarnyasg

Needs to be moved to https://github.com/duckdb/duckdb-python
Could you move it @szarnyasg ?

@Tishj

@Tishj @szarnyasg Thanks, I created one #257 .

(not sure which issue to post in, one of them is a dupe):

Doesn't this same error/inability to union occur with base duckdb?

❯ duckdb
DuckDB v1.4.3 (Andium) d1dc88f950
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
D select * from read_parquet('*.parquet', union_by_name=True)
┬╖ ;
Binder Error:
Struct remap can only remap nested types, not '"NULL"'
D

Note: reproduced that this would work in 1.2.2 but not in 1.3.0+ through 1.4.3.

@paultiq

@paultiq yes same error raised in cli v1.4.3
I didn't notice my cli was in older version and missed this.

cli v.14.3 raises error

DuckDB v1.4.3 (Andium) d1dc88f950
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
D select * from read_parquet('*.parquet', union_by_name=True)
  ;
Binder Error:
Struct remap can only remap nested types, not '"NULL"'

cli v1.2.2 was working fine

v1.2.2 7c039464e4
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
D select * from read_parquet('*.parquet', union_by_name=True)
  ;
┌────────┬──────────────────────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Index  │         Name         │                                                                                          data_detail                                                                                          │
│ int64  │       varchar        │ struct(address varchar, address_en varchar, alt_business_names varchar[], business_name varchar, business_name_en integer, city varchar, city_en varchar, country_code varchar, easy_number…  │
├────────┼──────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ 177000 │ NULL                 │ NULL                                                                                                                                                                                          │
│ 177001 │ NULL                 │ NULL                                                                                                                                                                                          │
│ 177002 │ NULL                 │ NULL                                                                                                                                                                                          │
│ 177003 │ NULL                 │ NULL                                                                                                                                                                                          │
│ 177004 │ NULL                 │ NULL                                                                                                                                                                                          │
│ 177005 │ NULL                 │ NULL                                                                                                                                                                                          │
│   4061 │ NULL                 │ NULL                                                                                                                                                                                          │
│   4062 │ NULL                 │ NULL                                                                                                                                                                                          │
│   4063 │ NULL                 │ NULL                                                                                                                                                                                          │
│   4064 │ AGUILA CAPITAL GMBH  │ {'address': NULL, 'address_en': am homberg 2 b, 'alt_business_names': [alessu beteiligungs, einhundertdreissigste stv vermogensverwaltungsgesellschaft], 'business_name': aguila capital, '…  │
│   4065 │ NULL                 │ NULL                                                                                                                                                                                          │
│   4066 │ NULL                 │ NULL                                                                                                                                                                                          │
│   5042 │ NULL                 │ NULL

Looks like this is fixed in 1.5.x?

❯ duckdb
DuckDB v1.5.1 (Variegata)
Enter ".help" for usage hints.
memory D SELECT * FROM read_parquet('file*.parquet', union_by_name=true);
┌───────┬────────────┐
│  id   │    tags    │
│ int32 │ varchar[]  │
├───────┼────────────┤
│     1 │ [foo, bar] │
│     2 │ NULL       │
└───────┴────────────┘

Here's an MRE:

import pathlib
import tempfile

import duckdb
import pyarrow as pa
import pyarrow.parquet as pq

print(f"duckdb {duckdb.__version__}, pyarrow {pa.__version__}")

tmp = pathlib.Path(tempfile.mkdtemp())

# f1.parquet: struct field `d.b` has Arrow type `null`
f1 = pa.table({
    "k": [1],
    "d": pa.array(
        [{"a": "x", "b": None}],
        type=pa.struct([("a", pa.string()), ("b", pa.null())]),
    ),
})

# f2.parquet: the same field `d.b` has type `list<string>`
f2 = pa.table({
    "k": [2],
    "d": pa.array(
        [{"a": "y", "b": ["p"]}],
        type=pa.struct([("a", pa.string()), ("b", pa.list_(pa.string()))]),
    ),
})

pq.write_table(f1, tmp / "f1.parquet")
pq.write_table(f2, tmp / "f2.parquet")

# Expected: d.b unifies to VARCHAR[], with NULL for k=1 and ['p'] for k=2
duckdb.sql(f"""
    SELECT *
    FROM read_parquet('{tmp}/*.parquet', union_by_name = true)
    ORDER BY k
""").show()

This is still broken on v1.5.5:

❯ pbpaste | uv run python
duckdb 1.5.5, pyarrow 25.0.1
Traceback (most recent call last):
  File "<stdin>", line 38, in <module>
_duckdb.BinderException: Binder Error: Can't change source type ("NULL") to target type (VARCHAR[]), type conversion not allowed

But fixed in the upcoming v2.0, in duckdb/duckdb#25908:

duckdb 2.0.0.dev2609221243, pyarrow 25.0.1
┌───────┬────────────────────────────────┐
│   k   │               d                │
│ int64 │ struct(a varchar, b varchar[]) │
├───────┼────────────────────────────────┤
│     1 │ {'a': x, 'b': NULL}            │
│     2 │ {'a': y, 'b': [p]}             │
└───────┴────────────────────────────────┘

Navigation Menu

Repository navigation

Python SDK read_parquet union_by_name fail after version 1.2.2 #259

What happens?

Title

DuckDB Version

Environment

Description

Expected Behavior

Actual Behavior

Root Cause Analysis

How to Reproduce

To Reproduce

OS:

DuckDB Version:

DuckDB Client:

Hardware:

Full Name:

Affiliation:

Did you include all relevant configuration (e.g., CPU architecture, Linux distribution) to reproduce the issue?

Did you include all code required to reproduce the issue?

Did you include all relevant data sets for reproducing the issue?

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Python SDK read_parquet union_by_name fail after version 1.2.2 #259

Description

What happens?

Title

DuckDB Version

Environment

Description

Expected Behavior

Actual Behavior

Root Cause Analysis

How to Reproduce

To Reproduce

OS:

DuckDB Version:

DuckDB Client:

Hardware:

Full Name:

Affiliation:

Did you include all relevant configuration (e.g., CPU architecture, Linux distribution) to reproduce the issue?

Did you include all code required to reproduce the issue?

Did you include all relevant data sets for reproducing the issue?

Activity

Tishj commented on Jan 8, 2026

DaiZack commented on Jan 8, 2026

paultiq commented on Jan 9, 2026

DaiZack commented on Jan 12, 2026

evertlammerts commented on Apr 9, 2026

evertlammerts commented on Sep 22, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions