[jasperblues]

Fixes #314
Fixes #321

Port of liberation-data/drivine4j#500 and liberation-data/drivine4j#512 (drivine4j PRs #503 and #514, review-pass commits included). Drivine for TypeScript has the first in liberation-data/drivine#501.

What changes

A map's null and a missing key are one null (#314). On Postgres and Oracle, a null read out of a map or a list is JSON's null, and a missing key is SQL's null. The two grouped, de-duplicated and sorted apart, where Neo4j has one null. A Json value now carries json_null where it may be JSON's null. That is:

• a property or an item read out of a map or a list that isn't known to be free of nulls;
• an entry of a parameter;
• an unwound item of such a list;
• head() and last() of such a list;
• a null literal;
• a reduce();
• a CASE that may give any of these.

Where it is set:

• Grouping and DISTINCT: a projection that groups or keeps its rows distinct compares a flagged item by grouped_null: NULLIF(x, 'null'::jsonb) on Postgres, and NVL(x, JSON('null')) on Oracle, as in drivine4j's review. The GROUP BY key and the DISTINCT column both read it that way. An order term that is such an item sorts by the same column. _held_first checks the compared form for a subquery.
• UNION without ALL: a part's single flagged item is given as SQL's null (compared_null). This applies to a top-level UNION, a subquery's UNION, and a CALL's UNION. Under a CALL's UNION parts (_distinct_parts, set and restored by _comparing_parts), grouping also compares by SQL's null, so CALL { RETURN null AS v UNION RETURN {}.a AS v } doesn't split.
• Oracle order: json_null_sort_key sorts by [rank, grouped_null(x)], so the next term decides between the two nulls.
• Unchanged: a property read from the store is never flagged, so its SQL is the same as before.

A CALL's UNION ALL marks (#321). _called_union used to take its marks from the first part only. Now:

• the merged column may be JSON's null where any part's value may be;
• it is a list, a list of lists, or flat only where every part's value is, as text/texts already were.

Cypher spec

Synced to v0.20.0 with scripts/sync_cypher_spec.py v0.20.0. It adds two steps to "A null in a map and a missing key are one null" (liberation-data/drivine-cypher-spec#45). All of #314's holds are removed: the Postgres and Oracle holds on that case in queries.yaml, and the SQL hold on "A null in a written row and a missing key are one null" in writes.yaml. #321's two new steps pass with no hold.

Tests

These are in tests/integration/test_cypher_compiled.py, and in its sync copy generated by scripts/unasync.py:

• Compiled Cypher: a null in a map and a missing key group, de-duplicate and sort apart on the SQL stores (port of drivine4j#500) #314, beyond the spec: drivine4j's CypherQueryContract and CypherWriteContract cases of #500 that the spec doesn't hold, plus Drivine for TypeScript's own cases from #501's one-null.yaml beside them: a subquery's UNION, head()/last(), $p['a'], and a CASE giving a null parameter.
• Compiled Cypher: a CALL's UNION ALL keeps its null and list marks from the first part only (port of drivine4j#512) #321, from #514:
  • "a CALL's UNION ALL whose later part gives a list's null keeps it one null with a missing key";
  • "a CALL's UNION ALL whose first part gives a list and a later one a number or a map sorts each by its kind". It skips Postgres, as drivine4j does (drivine4j#307), and compares rows without an ORDER BY as a multiset.

SQL comparison

I compiled every statement of spec v0.20.0 (1,496) and of the openCypher TCK corpus (4,119) on origin/main (79f5d87, with #320) and on this branch, for Postgres and Oracle.

• Refusals: none changed in either direction. No statement went from compiled to refused or back.
• SQL that changed: 36 statements on Postgres and 59 on Oracle.
  • 30 on Postgres and 34 on Oracle are the spec's Compiled Cypher: a null in a map and a missing key group, de-duplicate and sort apart on the SQL stores (port of drivine4j#500) #314 and Compiled Cypher: a CALL's UNION ALL keeps its null and list marks from the first part only (port of drivine4j#512) #321 steps, in its two "one null" cases in queries.yaml and writes.yaml.
  • The rest group, keep distinct or (on Oracle) sort a value these fixes flag: an unwound item of a list that may hold a null (UNWIND ['b', null, 'B'], UNWIND [n.name, 'B'], UNWIND [i, i, 3], a pattern comprehension's list); an item taken by index ([...][0], values[i]); head(collect(...)) (TCK Return4 [11], With4 [6]); and an entry of a map that may hold a null (m.a, who.name, p {.name} then p.name).
• Why the rest is safe: every one of these reads a value that may be JSON's null where the rows are compared or sorted, which is what drivine4j's fix flags too. All of them pass the spec on both stores. One unit test pinned the old Oracle order of UNWIND [2, 'b', null] … ORDER BY t DESC. It now checks that a list without a null sorts as before, and that one holding a null sorts by NVL(…, JSON('null')).

Differences from drivine4j and Drivine for TypeScript

• Where the flag lives. drivine-python has no separate writer, so _distinct_parts is on the compiler. _begin resets it, and _comparing_parts sets and restores it, including for a top-level UNION, as Drivine for TypeScript does. drivine4j sets the writer's flag once instead. The effect is the same.
• Method name. drivine4j's compared(value) is _row_compared here, as it is rowCompared in Drivine for TypeScript, because _compared already names the sides of a binary comparison.
• Compiled Cypher: a CALL's UNION ALL keeps its null and list marks from the first part only (port of drivine4j#512) #321's list marks. Drivine for TypeScript's #501 merges only the JSON-null mark. Its list, list-of-lists and flat marks are Compiled Cypher: a CALL's UNION ALL keeps its list marks from the first part only, so mixed kinds sort and compare wrong (port of drivine4j#512) drivine#505, still open. This PR merges all four, as drivine4j does.
• Where the extra cases live. Drivine for TypeScript kept its extra cases in a held one-null.yaml, because it ran an older spec. Here the spec runs at v0.20.0, so only the cases beyond the spec are added, as Python tests.