Skip to content

upsert_all() writes NULL to omitted columns in mixed-field batches #873

Description

@glaziermag

upsert_all() overwrites omitted columns with NULL when records in the same batch contain different fields. This reproduces on current main (6bc1d33d583c54bd69fbdd2071117e2d38c354a1): the batch-wide column list fills each record's missing fields with None, and the generated ON CONFLICT ... DO UPDATE assigns those values to the existing rows.

from sqlite_utils import Database

db = Database(memory=True)
table = db.table("dogs")
table.insert_all([
    {"id": 1, "name": "Cleo", "age": 5},
    {"id": 2, "name": "Nixie", "age": 6},
], pk="id")
table.upsert_all([
    {"id": 1, "age": 7},
    {"id": 2, "name": "Nixie II"},
])
print(list(table.rows_where(order_by="id")))

Expected:

[{'id': 1, 'name': 'Cleo', 'age': 7},
 {'id': 2, 'name': 'Nixie II', 'age': 6}]

Actual:

[{'id': 1, 'name': None, 'age': 7},
 {'id': 2, 'name': 'Nixie II', 'age': None}]

The upsert documentation says columns omitted from an upsert dictionary remain unchanged, and describes upsert_all() as performing upserts.

The relevant paths are missing-field expansion and the conflict-update assignments. The update columns need to reflect the fields supplied by each record while preserving record order.

Verified with Python 3.13.12 / SQLite 3.53.0. The existing tests/test_upsert.py suite passes (18 tests).

Found and reproduced using OpenAI Codex.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions