Repository navigation
Wait out a busy lock when making the store's unique index on Oracle, so racing first boots don't run without it - #319
Merged
Conversation
… lock turns it away, so racing managers keep the store key's constraint
Managers making the DrivineStore(key) constraint together on Oracle were turned away by each other's lock on the
node table (ORA-00054), as Oracle's DDL doesn't wait for a lock, and once by a library cache deadlock (ORA-04020).
The schema executor's six attempts, about a second in all, ran out under CI's load, and a manager went on without
the constraint ("Identity is best-effort"), failing tests/sync/test_store_identity_race.py on main (run 41190521832).
The block that makes an index over the node table now runs its statement again a tenth of a second after a busy
lock, for up to 300 tries, as drivine4j's OracleGraph.ensuring waits out a busy lock, and then finds what the other
session made. The goldens are copied from Drivine for TypeScript, which makes the same change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the Oracle store-identity CI flake on main:
tests/sync/test_store_identity_race.py:29failing in run 41190521832, where the "no best-effort warning" assertion sawCould not ensure the DrivineStore key constraint: ORA-00054 …andORA-04020 …. The same change is made in Drivine for TypeScript in liberation-data/drivine#504.Cause
Managers that make the
DrivineStore(key)constraint at the same time each runCREATE UNIQUE INDEX IF NOT EXISTSover the node table. Oracle's DDL doesn't wait for a lock, so a manager whose statement meets another session's lock on the node table (another manager building the index, or writing its stamp) is turned away with ORA-00054. The schema executor runs the statement again up to six times, about a second in all. Under CI's load the index build takes longer than that, so the manager runs out of attempts and carries on without the constraint. The manager has truly been running without it, so this is a real bug, if a narrow one.How drivine4j differs
drivine4j's
OracleGraph.ensuringwaits out a busy lock inside its PL/SQL block. It runs each statement again a tenth of a second after ORA-00054, for up toBUSY_ATTEMPTS = 300tries. When it can't ensure the key's constraint, it also looks for it and carries on without a warning if another manager's is in force. That fallback misses a loser that looks while the winner is still building, so drivine4j shares the weakness: liberation-data/drivine4j#517.Fix
_made()inoracle_schema_grammar.pyis the block that makes an index over the node table, both the shared range index and the unique index. After ORA-00054 it now runs its statement again a tenth of a second later, for up toBUSY_ATTEMPTS = 300tries, which is drivine4j's loop and bound. It then finds what the other session made. A deadlock (ORA-04020) is still run again by the schema executor. The change touches only Oracle's DDL path. The goldens are copied from liberation-data/drivine@c34cf55 withscripts/sync_golden.py, byte for byte.The test is unchanged, including its "no best-effort warning" assertion.
Proof
tests/integration/test_schema.py's) passed 10 times out of 10 locally. They also passed with 25 rounds × 4 runs and the Oracle container held to 0.5 CPU, and with the node table filled with 300,000 nodes and emptied again (5 out of 5 each). I couldn't reproduce the CI failure locally.scripts/unasync.py --check, andORACLE_TESTS=true uv run pytest -q(5909 passed, 298 skipped) all pass.