Skip to content

Ensure the store key's constraint again when another manager made it meanwhile, and write again a stamp it refuses (#298) - #308

Merged
jasperblues merged 2 commits into
mainfrom
fix/298-store-identity-race
Oct 31, 2026
Merged

jasperblues merged 2 commits into
mainfrom
fix/298-store-identity-race

Conversation

@jasperblues

Copy link
Copy Markdown
Member

Fixes #298. Port of liberation-data/drivine4j#473 (drivine4j PR liberation-data/drivine4j#477), following liberation-data/drivine#489.

What happened

When several managers resolve the store identity at once on a store with no DrivineStore(key) constraint, they all try to make it. A loser whose statement was refused went on without the constraint (_ensure_unique_key returned False). Then _assign_if_absent raised any failure of the MERGE when unguarded. So a stamp refused by the winner's constraint, now in force, failed the boot.

On main, the new concurrency test (16 managers per round) fails on two stores, with tasks and with threads:

  • FalkorDB: 75 (async) and 59 (sync) Could not ensure the DrivineStore key constraint: Constraint already exists. Identity is best-effort.
  • Postgres: 11 or more duplicate key value violates unique constraint "pg_class_relname_nsp_index" best-effort warnings in one run. In the other, a loser's stamp was refused by the winner's constraint, and DrivineError: duplicate key value violates unique constraint "drivine_uq_drivinestorekey_…" was raised out of resolve().

Neo4j, Memgraph and Oracle pass on main. The schema executor already runs a schema statement again on a transient Bolt error, through when_not_contended (drivine/schema/schema_executor.py), and that includes Neo4j's DeadlockDetected. On Oracle the race gives only ORA-04020, which ddl_contended already retries.

What this changes

  1. _ensure_unique_key: the constraint is ensured again, at most 3 times, when its making was refused for something another manager made meanwhile. That covers FalkorDB's Constraint already exists and already indexed, Postgres's 23505 on a pg_ catalog constraint, and Oracle's ORA-00955. The second ensure finds the winner's constraint, so the stamp is written under it. Deadlocks are not retried here, because the schema executor has already retried them.
    • A Postgres failure is judged by its SQLSTATE and diag.constraint_name alone. psycopg's message carries the DETAIL, Key (...)=(...) already exists., of any violation, so matching the text would count a violation in a user's table as made meanwhile.
  2. _assign_if_absent: a uniqueness violation (unique constraint|already exists, not any "constraint") is merged again even when the constraint wasn't confirmed, because the violation proves a constraint is in force. This applies only to a manager with schema management. A write conflict is still merged again only under a confirmed constraint. Merges stay bounded at 5.

Neo4jConnection.release already logs a closing failure at debug, so it needs no change here.

Tests

  • Unit (tests/unit/test_store_identity_race.py), async and sync: a fake Neo4j store, run through the real ConstraintManager and SchemaExecutor. Cases:

    • a deadlock every time, then a violation: written again, two merges;
    • a deadlock once: the constraint is made again, and a conflict on the first merge is written again;
    • refused as made meanwhile (Constraint already exists., already indexed, a Postgres 23505 on pg_class_relname_nsp_index): ensured again, no best-effort warning;
    • a 23505 on a user's table (orders_pkey, with psycopg's DETAIL): not ensured again;
    • refused as made meanwhile each time, though nothing made it: 3 attempts, then the best-effort warning;
    • a constraint refusal of another kind: not ensured again;
    • a SyntaxError, or a not-null violation, on the MERGE: raised at once;
    • a store without schema management: raised at once.

    On main, 10 of the 22 fail: the violation, made-meanwhile and 3-attempt cases.

  • Integration: 16 managers resolve at once on a store with the constraint dropped and the stamps wiped. That is 5 rounds on Neo4j, Memgraph and FalkorDB, 3 on Postgres and 2 on Oracle. Each round must agree on one id, leave one stamp, and log no best-effort warning. The test runs with tasks (tests/integration/test_schema.py, async_only) and with threads (tests/sync/test_store_identity_race.py). It fails on main on FalkorDB and Postgres, and passes on every store with this change.

The full gate passed with ORACLE_TESTS=true: ruff check, ruff format --check, pyright, unasync.py --check, and pytest (5674 passed, 298 skipped).

…meanwhile, and write again a stamp it refuses (#298)

Port of liberation-data/drivine4j#473 (drivine4j PR #477), following drivine PR #489.

Managers resolving the store identity at once on a store without the
DrivineStore(key) constraint race to make it. On FalkorDB and Postgres the
losers' statements were refused for what the winner made meanwhile
("Constraint already exists.", a 23505 on pg_class_relname_nsp_index), so
they went on without the constraint, warning that identity was best-effort,
and a stamp the winner's constraint then refused was raised out of resolve().

- _ensure_unique_key ensures the constraint again, at most 3 times, when its
  making was refused as made meanwhile. A Postgres failure counts only as a
  23505 on a pg_ catalog constraint, judged by SQLSTATE and constraint name,
  since psycopg's message carries "already exists" for any violation.
  Deadlocks are left to the schema executor's contention retry, which already
  runs Neo4j's DeadlockDetected again.
- _assign_if_absent merges again on a uniqueness violation even when the
  constraint wasn't confirmed, on a manager with schema management only. A
  write conflict is still merged again only under a confirmed constraint.

Tests: unit tests with a fake Neo4j store through the real ConstraintManager
and SchemaExecutor, async and sync; and 16 managers racing per round on every
store (5 rounds; 3 on Postgres, 2 on Oracle), with tasks and with threads.

Fixes #298.
@jasperblues
jasperblues merged commit 6192af8 into main Oct 31, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Neo4j: managers resolving the store identity at once can fail when the key constraint's creation deadlocks

1 participant