Skip to content

fix(json): make JSON path annotations and value comparisons portable - #1607

Open
dimitri-yatsenko wants to merge 1 commit into
masterfrom
fix/json-path-portability
Open

dimitri-yatsenko wants to merge 1 commit into
masterfrom
fix/json-path-portability

Conversation

@dimitri-yatsenko

Copy link
Copy Markdown
Member

The same JSON path query could give different results on MySQL and PostgreSQL in two ways.

Type annotations (#1563). Doc.proj(n="data.n:int") worked on PostgreSQL but was a syntax error on MySQL, because the MySQL adapter passed the annotation straight into JSON_VALUE ... RETURNING, which accepts only its own short list of types. The MySQL adapter now maps the common PostgreSQL spellings onto that list: int, integer, smallint, bigint, bool and boolean to signed; numeric(p,s) to decimal(p,s); varchar(n) and text to char(n) and char; double precision to double; timestamp to datetime. In the other direction, the PostgreSQL adapter now maps datetime to timestamp, the one RETURNING type it rejected. Spellings that already worked pass through unchanged, so nothing that ran before changes.

Value comparisons (#1564). In a mapping restriction such as {"s.cal": True} or {"s.ch": 64}, the extracted JSON value is text. PostgreSQL refused to compare text with a bool or a number. MySQL compared 'true' with 1 and silently matched nothing. Bools and numbers (numpy scalars and Decimal included) are now compared as JSON values: json_extract(...) = cast('64' as json) on MySQL, jsonb_extract_path(...) = '64'::jsonb on PostgreSQL.

Why compare JSON values rather than use either approach suggested in the issue:

  • Rendering the value as text breaks 64 against a stored 64.0, and 64.0 against a stored 64, because the texts differ. It would also have broken those matches on MySQL, where numeric coercion makes them work today.
  • Casting the extracted text to the value's type raises on PostgreSQL for any row that holds a non-numeric string at that path ('abc'::numeric), so one odd row would break the whole query.
  • JSON equality compares numbers numerically on both servers (64 = 64.0, and large integers stay exact). It matches a boolean only to a JSON boolean, and a value of another JSON type is unequal rather than an error.

One behavior change: on MySQL, a number no longer matches a JSON string holding the same digits ({"s.ch": 64} against "64"), which it used to do through text coercion. PostgreSQL never matched this case; it raised. Strings still compare as text, so {"s.ch": "64"} and {"s.cal": "true"} work as before, and an annotated key such as {"s.n:int": 7} still compares the cast value. NaN and infinity raise a DataJointError, since JSON cannot represent them.

The new integration tests run on both backends. They cover restrictions by bool, int, float, numpy scalars, Decimal and string, including against stored floats, non-numeric strings and missing keys, and restrictions and projections with each annotation spelling. 42 of the 90 cases fail on the previous code.

Closes #1563. Closes #1564.

The same JSON path query could give different results on MySQL and PostgreSQL in two ways.

Type annotations (#1563). `Doc.proj(n="data.n:int")` worked on PostgreSQL but was a syntax error on MySQL, because the MySQL adapter passed the annotation straight into `JSON_VALUE ... RETURNING`, which accepts only its own short list of types. The MySQL adapter now maps the common PostgreSQL spellings onto that list: `int`, `integer`, `smallint`, `bigint`, `bool` and `boolean` to `signed`; `numeric(p,s)` to `decimal(p,s)`; `varchar(n)` and `text` to `char(n)` and `char`; `double precision` to `double`; `timestamp` to `datetime`. In the other direction, the PostgreSQL adapter now maps `datetime` to `timestamp`, the one RETURNING type it rejected. Spellings that already worked pass through unchanged, so nothing that ran before changes.

Value comparisons (#1564). In a mapping restriction such as `{"s.cal": True}` or `{"s.ch": 64}`, the extracted JSON value is text. PostgreSQL refused to compare text with a bool or a number. MySQL compared `'true'` with `1` and silently matched nothing. Bools and numbers (numpy scalars and Decimal included) are now compared as JSON values: `json_extract(...) = cast('64' as json)` on MySQL, `jsonb_extract_path(...) = '64'::jsonb` on PostgreSQL.

Why compare JSON values rather than use either approach suggested in the issue:

- Rendering the value as text breaks `64` against a stored `64.0`, and `64.0` against a stored `64`, because the texts differ. It would also have broken those matches on MySQL, where numeric coercion makes them work today.
- Casting the extracted text to the value's type raises on PostgreSQL for any row that holds a non-numeric string at that path (`'abc'::numeric`), so one odd row would break the whole query.
- JSON equality compares numbers numerically on both servers (64 = 64.0, and large integers stay exact). It matches a boolean only to a JSON boolean, and a value of another JSON type is unequal rather than an error.

One behavior change: on MySQL, a number no longer matches a JSON string holding the same digits (`{"s.ch": 64}` against `"64"`), which it used to do through text coercion. PostgreSQL never matched this case; it raised. Strings still compare as text, so `{"s.ch": "64"}` and `{"s.cal": "true"}` work as before, and an annotated key such as `{"s.n:int": 7}` still compares the cast value. NaN and infinity raise a DataJointError, since JSON cannot represent them.

The new integration tests run on both backends. They cover restrictions by bool, int, float, numpy scalars, Decimal and string, including against stored floats, non-numeric strings and missing keys, and restrictions and projections with each annotation spelling. 42 of the 90 cases fail on the previous code.

Closes #1563. Closes #1564.
@dimitri-yatsenko dimitri-yatsenko added this to the v2.4 milestone Oct 17, 2026
@dimitri-yatsenko dimitri-yatsenko added the bug Indicates an unexpected problem or unintended behavior label Oct 17, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Indicates an unexpected problem or unintended behavior

Projects

None yet

1 participant