Repository navigation
External store codec columns (<filepath@store>, <blob@store>) fail on MariaDB: TypeError: dict can not be used as parameter #1451
Description
Summary
#1438 / PR #1443 fixed json column handling on MariaDB by recovering attr.json from the :json: comment marker. External store codec columns (<filepath@store>, <blob@store>) have the same root cause but are not covered by that fix — their encode() returns a Python dict that reaches pymysql unserialized, raising TypeError: dict can not be used as parameter.
Reproduction
Using DataJoint 2.2.2 against MariaDB 10.3.28:
import datajoint as dj
schema = dj.Schema("test_filepath_mariadb")
dj.config["stores"] = {
"local": {
"protocol": "file",
"location": "/tmp/dj_test_store",
"stage": "/tmp/dj_test_store",
}
}
@schema
class FileTest(dj.Manual):
definition = """
id : int
---
data_file : <filepath@local>
"""
# Create a dummy file
import os
os.makedirs("/tmp/dj_test_store", exist_ok=True)
with open("/tmp/dj_test_store/test.bin", "wb") as f:
f.write(b"test")
FileTest.insert1({"id": 1, "data_file": "test.bin"})
# TypeError: dict can not be used as parameterSame error for <blob@store> columns.
Root cause
The filepath codec's encode() returns a dict ({path, store, size, timestamp}). The blob codec behaves similarly. On insert, __make_placeholder() (table.py line ~1383) checks attr.json to decide whether to json.dumps() the value:
elif attr.json:
value = json.dumps(value)On MySQL, the column type is reported as json from information_schema, so attr.json is set to True and the dict is serialized. On MariaDB, the column type is reported as longtext. PR #1443 recovers attr.json for plain json columns using the :json: comment marker, but codec columns have a :<filepath@store>: marker instead — their original_type resolves through the codec's get_dtype() → "json", which should hit the same category == "JSON" recovery path, but empirically attr.json remains False for these columns on MariaDB.
The unserialized dict reaches pymysql, which raises TypeError in converters.escape_dict().
Impact
This affects all external store columns on MariaDB:
<filepath@store>— file reference metadata<blob@store>— serialized data references
Both insert and fetch are affected (fetch returns a raw JSON string instead of a dict on MariaDB).
Current workaround
We are patching pymysql's converters at import time to serialize dicts as JSON strings:
import json
import pymysql.converters
def _escape_dict_as_json(val, charset, mapping=None):
return pymysql.converters.escape_str(json.dumps(val, default=str), charset)
pymysql.converters.encoders[dict] = _escape_dict_as_json
pymysql.converters.conversions[dict] = _escape_dict_as_jsonThis must be applied before any DataJoint connection is created. On MySQL, it is inert (DataJoint serializes dicts before they reach pymysql). This workaround is in production at the Sainsbury Wellcome Centre.
Suggested fix
Extend the attr.json recovery from PR #1443 to cover codec columns whose get_dtype() resolves to "json". Alternatively, have __make_placeholder() call json.dumps() on any dict value destined for a longtext column, regardless of attr.json.
Environment
- DataJoint 2.2.2
- MariaDB 10.3.28
- pymysql 1.1.1
- Python 3.11
Activity
dimitri-yatsenko commentedon Jun 10, 2026
Status: already addressed on master (PR #1443) — not officially supported
A note up front: DataJoint does not officially support MariaDB. The Aeon production pipeline runs on MariaDB through Q2 2026, so we avoid hard-breaking it, but MariaDB is not in the supported-backend matrix for 2.3 onward — MySQL and PostgreSQL are. We rely on MariaDB users (you) to confirm fixes work in practice rather than running CI against a MariaDB container ourselves.
What I traced
The fix from #1443 (commit c3df7b3, on master) covers both plain
jsoncolumns and codec columns whoseget_dtype()resolves to"json". The recovery path inheading.py:attr["type"]starts as the DB-reported type (longtexton MariaDB) —attr["json"]isFalse.heading.py:472-491) runsattr["codec"].get_dtype(is_store=True)for<filepath@local>, which returns"json", and assignsattr["type"] = "json".heading.py:494-498) takesoriginal_type = attr["original_type"] or attr["type"]— for codec columnsoriginal_typeisNone, so it falls through toattr["type"], which is now"json".categoryresolves to"JSON", and the recovery atheading.py:519-520setsattr["json"] = True.I verified this trace with a standalone simulation against the codec-resolution code;
attr["json"]ends upTruefor the MariaDB shape (longtext +:<filepath@local>:marker). On 2.2.2 (which predates #1443) the recovery didn't exist, so the bug you observed was real. On master / 2.2.3+ the same code path that fixed plainjsonalso fixes the codec case.What we're asking of you, @esutlie
If you can confirm — on your MariaDB 10.3.28 instance with the current
masterofdatajoint-python(or a 2.2.4 install) — that the reproducer no longer raises and<filepath@local>/<blob@local>inserts work, we'll close this. If it still fails, I'll dig in further (there may be a piece of the trace I'm missing on the actual MariaDB wire). Either outcome is fine; we just want to know.The
pymysqlconverter monkey-patch you have in place at SWC is no longer required if the recovery works.Not slated for additional fix in 2.3
Given the trace, no further fix is needed in 2.3 specifically for this issue. Tracking continues here; close when confirmed.
esutlie commentedon Jun 16, 2026
Hey Dimitri, I tested this on our MariaDB 10.3.28 instance. The fix from PR #1443 (shipping in 2.2.2) resolved plain
jsoncolumns and<filepath@store>columns, but<blob@store>still fails withTypeError: dict can not be used as parameter.I dug into the heading code to understand why. The issue is that the
category == "JSON"recovery at heading.py:519 only fires whenoriginal_typeresolves to something that matches the JSON pattern. For codec columns like<blob@dj_store>, the heading callsattr["codec"].get_dtype(is_store=True)at line 487, which returns"<hash>"(BlobCodec's dtype), not"json". The full chain resolution down to"json"(via HashCodec.get_dtype) only happens later in the insert path throughresolve_dtype(). So at the point where the recovery runs,attr["type"]is"<hash>", category doesn't match"JSON", andattr.jsonstays False.The result is that after the codec chain encodes the value into a metadata dict, the dict falls through to pymysql without
json.dumps()being called.Test results on our MariaDB 10.3.28, all without our pymysql monkey-patch:
Our pymysql converter monkey-patch still works as a workaround for the blob case. We'll keep it in place for now.
CowanCS1 commentedon Jun 24, 2026
I hit the same issue with dj v2.2.4 and mariadb v12.3.2, and confirm the same diagnosis.
We're in testing mode and don't have existing infrastructure, so we'll just swap to the officially-supported Postgre.
dimitri-yatsenko commentedon Jun 24, 2026
@CowanCS1 We do not officially support MariaDB and it's not in our CI/CD. Here are the supported backends: https://docs.datajoint.com/how-to/installation/. Last week, our last remaining clients moved from MariaDB and we are not currently planning to support it.