Repository navigation
fix(json): make JSON path annotations and value comparisons portable - #1607
Open
dimitri-yatsenko wants to merge 1 commit into
Open
dimitri-yatsenko wants to merge 1 commit into
dimitri-yatsenko wants to merge 1 commit into
Conversation
The same JSON path query could give different results on MySQL and PostgreSQL in two ways. Type annotations (#1563). `Doc.proj(n="data.n:int")` worked on PostgreSQL but was a syntax error on MySQL, because the MySQL adapter passed the annotation straight into `JSON_VALUE ... RETURNING`, which accepts only its own short list of types. The MySQL adapter now maps the common PostgreSQL spellings onto that list: `int`, `integer`, `smallint`, `bigint`, `bool` and `boolean` to `signed`; `numeric(p,s)` to `decimal(p,s)`; `varchar(n)` and `text` to `char(n)` and `char`; `double precision` to `double`; `timestamp` to `datetime`. In the other direction, the PostgreSQL adapter now maps `datetime` to `timestamp`, the one RETURNING type it rejected. Spellings that already worked pass through unchanged, so nothing that ran before changes. Value comparisons (#1564). In a mapping restriction such as `{"s.cal": True}` or `{"s.ch": 64}`, the extracted JSON value is text. PostgreSQL refused to compare text with a bool or a number. MySQL compared `'true'` with `1` and silently matched nothing. Bools and numbers (numpy scalars and Decimal included) are now compared as JSON values: `json_extract(...) = cast('64' as json)` on MySQL, `jsonb_extract_path(...) = '64'::jsonb` on PostgreSQL. Why compare JSON values rather than use either approach suggested in the issue: - Rendering the value as text breaks `64` against a stored `64.0`, and `64.0` against a stored `64`, because the texts differ. It would also have broken those matches on MySQL, where numeric coercion makes them work today. - Casting the extracted text to the value's type raises on PostgreSQL for any row that holds a non-numeric string at that path (`'abc'::numeric`), so one odd row would break the whole query. - JSON equality compares numbers numerically on both servers (64 = 64.0, and large integers stay exact). It matches a boolean only to a JSON boolean, and a value of another JSON type is unequal rather than an error. One behavior change: on MySQL, a number no longer matches a JSON string holding the same digits (`{"s.ch": 64}` against `"64"`), which it used to do through text coercion. PostgreSQL never matched this case; it raised. Strings still compare as text, so `{"s.ch": "64"}` and `{"s.cal": "true"}` work as before, and an annotated key such as `{"s.n:int": 7}` still compares the cast value. NaN and infinity raise a DataJointError, since JSON cannot represent them. The new integration tests run on both backends. They cover restrictions by bool, int, float, numpy scalars, Decimal and string, including against stored floats, non-numeric strings and missing keys, and restrictions and projections with each annotation spelling. 42 of the 90 cases fail on the previous code. Closes #1563. Closes #1564.
dimitri-yatsenko
requested review from
MilagrosMarin,
lum-agilyti and
ttngu207
October 17, 2026 22:02
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The same JSON path query could give different results on MySQL and PostgreSQL in two ways.
Type annotations (#1563).
Doc.proj(n="data.n:int")worked on PostgreSQL but was a syntax error on MySQL, because the MySQL adapter passed the annotation straight intoJSON_VALUE ... RETURNING, which accepts only its own short list of types. The MySQL adapter now maps the common PostgreSQL spellings onto that list:int,integer,smallint,bigint,boolandbooleantosigned;numeric(p,s)todecimal(p,s);varchar(n)andtexttochar(n)andchar;double precisiontodouble;timestamptodatetime. In the other direction, the PostgreSQL adapter now mapsdatetimetotimestamp, the one RETURNING type it rejected. Spellings that already worked pass through unchanged, so nothing that ran before changes.Value comparisons (#1564). In a mapping restriction such as
{"s.cal": True}or{"s.ch": 64}, the extracted JSON value is text. PostgreSQL refused to compare text with a bool or a number. MySQL compared'true'with1and silently matched nothing. Bools and numbers (numpy scalars and Decimal included) are now compared as JSON values:json_extract(...) = cast('64' as json)on MySQL,jsonb_extract_path(...) = '64'::jsonbon PostgreSQL.Why compare JSON values rather than use either approach suggested in the issue:
64against a stored64.0, and64.0against a stored64, because the texts differ. It would also have broken those matches on MySQL, where numeric coercion makes them work today.'abc'::numeric), so one odd row would break the whole query.One behavior change: on MySQL, a number no longer matches a JSON string holding the same digits (
{"s.ch": 64}against"64"), which it used to do through text coercion. PostgreSQL never matched this case; it raised. Strings still compare as text, so{"s.ch": "64"}and{"s.cal": "true"}work as before, and an annotated key such as{"s.n:int": 7}still compares the cast value. NaN and infinity raise a DataJointError, since JSON cannot represent them.The new integration tests run on both backends. They cover restrictions by bool, int, float, numpy scalars, Decimal and string, including against stored floats, non-numeric strings and missing keys, and restrictions and projections with each annotation spelling. 42 of the 90 cases fail on the previous code.
Closes #1563. Closes #1564.