Skip to content

fix: list-mode insert_all/upsert_all with hash_id misaligns values - #880

Closed
harsh-thakkar7 wants to merge 1 commit into
simonw:mainfrom
harsh-thakkar7:fix/list-mode-hash-id-align-values
Closed

harsh-thakkar7 wants to merge 1 commit into
simonw:mainfrom
harsh-thakkar7:fix/list-mode-hash-id-align-values

Conversation

@harsh-thakkar7

@harsh-thakkar7 harsh-thakkar7 commented Sep 24, 2026 •

Copy link
Copy Markdown

insert_all() / upsert_all() in list mode (records as lists, first yielded list is the column names) misassigned every row when hash_id= was used:

>>> from sqlite_utils import Database
>>> db = Database(memory=True)
>>> db["t"].insert_all([["a", "b"], [1, "x"], [2, "y"]], hash_id="id")
>>> list(db["t"].rows)
# Boom: every value shifted one column left, never hashed
[{'id': '1', 'a': 'x', 'b': None}, {'id': '2', 'a': 'y', 'b': None}]

insert_all() correctly made id the first column, but the raw row lists were then paired against that expanded column list — so the value that should have been the hash key was stuffed into id, the data columns all slid left, and the last column went NULL. The stored rows are also silently wrong (no hash was ever computed).

Fix

In list mode, when hash_id is set, compute the hash from the user-declared values (dict(zip(user_columns, row))) and prepend it, so value positions stay aligned with all_columns. Short rows are still padded with None and long rows truncated — but only against the user-declared columns, before the hash is computed.

The stored hash keys now match exactly what dict-mode hash_id produces for the same data, and the values stay in their correct columns:

[{'id': '6773f537...', 'a': 1, 'b': 'x'}, {'id': '150c1d4a...', 'a': 2, 'b': 'y'}]

Tests

Added to tests/test_list_mode.py:

  • test_insert_all_list_mode_with_hash_id_aligns_values — values land in the right columns and id is the correct hash for each row (fails before this change).
  • test_insert_all_list_mode_with_hash_id_pads_and_truncates — short rows are padded before hashing; over-long rows are truncated.
  • test_upsert_all_list_mode_with_hash_id — same guarantees for upserts.
pytest -q tests/test_list_mode.py   # 18 passed
pytest -q                           # 1503 passed, 16 skipped

📚 Documentation preview 📚: https://sqlite-utils--880.org.readthedocs.build/en/880/

When insert_all() or upsert_all() received lists of values plus an
auto-generated hash_id, all_columns had the hash column prepended but
the raw rows were then paired against that expanded list. Every value
shifted one column to the left and the hash_id column ended up holding
the first value instead of a key, e.g. rows (1, 'x') became (1, 'x',
NULL) under columns (id, a, b). Compute the hash from the
user-declared values and prepend it so columns stay aligned.
@harsh-thakkar7

Copy link
Copy Markdown
Author

Closing this — I don't think this change is mature enough to land right now. Thanks for the project!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant