Skip to content

Wait out a busy lock when making the store's unique index on Oracle, so racing first boots don't run without it - #319

Merged
jasperblues merged 4 commits into
mainfrom
fix/oracle-identity-flake
Nov 1, 2026
Merged

jasperblues merged 4 commits into
mainfrom
fix/oracle-identity-flake

Conversation

@jasperblues

Copy link
Copy Markdown
Member

Fixes the Oracle store-identity CI flake on main: tests/sync/test_store_identity_race.py:29 failing in run 41190521832, where the "no best-effort warning" assertion saw Could not ensure the DrivineStore key constraint: ORA-00054 … and ORA-04020 …. The same change is made in Drivine for TypeScript in liberation-data/drivine#504.

Cause

Managers that make the DrivineStore(key) constraint at the same time each run CREATE UNIQUE INDEX IF NOT EXISTS over the node table. Oracle's DDL doesn't wait for a lock, so a manager whose statement meets another session's lock on the node table (another manager building the index, or writing its stamp) is turned away with ORA-00054. The schema executor runs the statement again up to six times, about a second in all. Under CI's load the index build takes longer than that, so the manager runs out of attempts and carries on without the constraint. The manager has truly been running without it, so this is a real bug, if a narrow one.

How drivine4j differs

drivine4j's OracleGraph.ensuring waits out a busy lock inside its PL/SQL block. It runs each statement again a tenth of a second after ORA-00054, for up to BUSY_ATTEMPTS = 300 tries. When it can't ensure the key's constraint, it also looks for it and carries on without a warning if another manager's is in force. That fallback misses a loser that looks while the winner is still building, so drivine4j shares the weakness: liberation-data/drivine4j#517.

Fix

_made() in oracle_schema_grammar.py is the block that makes an index over the node table, both the shared range index and the unique index. After ORA-00054 it now runs its statement again a tenth of a second later, for up to BUSY_ATTEMPTS = 300 tries, which is drivine4j's loop and bound. It then finds what the other session made. A deadlock (ORA-04020) is still run again by the schema executor. The change touches only Oracle's DDL path. The goldens are copied from liberation-data/drivine@c34cf55 with scripts/sync_golden.py, byte for byte.

The test is unchanged, including its "no best-effort warning" assertion.

Proof

  • Main, no fix: both Oracle race tests (the sync one and tests/integration/test_schema.py's) passed 10 times out of 10 locally. They also passed with 25 rounds × 4 runs and the Oracle container held to 0.5 CPU, and with the node table filled with 300,000 nodes and emptied again (5 out of 5 each). I couldn't reproduce the CI failure locally.
  • Fixed code: both Oracle race tests passed 10 times out of 10 each (20 out of 20).
  • Gate: ruff check, ruff format --check, pyright (0 errors), scripts/unasync.py --check, and ORACLE_TESTS=true uv run pytest -q (5909 passed, 298 skipped) all pass.

… lock turns it away, so racing managers keep the store key's constraint

Managers making the DrivineStore(key) constraint together on Oracle were turned away by each other's lock on the
node table (ORA-00054), as Oracle's DDL doesn't wait for a lock, and once by a library cache deadlock (ORA-04020).
The schema executor's six attempts, about a second in all, ran out under CI's load, and a manager went on without
the constraint ("Identity is best-effort"), failing tests/sync/test_store_identity_race.py on main (run 41190521832).

The block that makes an index over the node table now runs its statement again a tenth of a second after a busy
lock, for up to 300 tries, as drivine4j's OracleGraph.ensuring waits out a busy lock, and then finds what the other
session made. The goldens are copied from Drivine for TypeScript, which makes the same change.
@jasperblues
jasperblues merged commit 62170cd into main Nov 1, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant