Conversation
Both sleeps happened while the original table was write-locked, so they were pure table downtime: - executeWriteFuncs slept 1s whenever both queues were empty, which is the steady state once row copy is done. The cut-over sentinel arrives on applyEventsQueue and had to wait that sleep out. It now blocks on both queues with a 1s timeout instead. - waitForRename watched for the blocking RENAME through retryOperation, which backs off a flat 1s. The first check usually runs before the RENAME shows up. It now checks every 10ms, via a new retryOperationWithInterval (retryOperation with the attempt count and wait made explicit). "Lock & rename duration" over the 73 localtests that reach cut-over. Before, it was ~1s on essentially every cut-over: mariadb:11.8 p50 9ms p90 21ms p99 90ms max 90ms mysql:8.4.3 p50 23ms p90 31ms p99 96ms max 96ms The tail is not this code path. It is waitForEventsUpToLock: on both flavours the same four cases, which happen to see no DML during the migration, spend 54-77ms waiting for the sentinel to come back through an otherwise idle binlog stream. No case with concurrent DML exceeds 50ms. That delay reproduces with a plain binlog reader, so it is in binlog delivery, not in gh-ost. TestCutOverLossDataCaseLockGhostBeforeRename now locks the ghost table before un-postponing, instead of relying on cut-over being slow. Fixes github#1630
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A Pull Request should be associated with an Issue.
Related issue: #1630
Description
This PR removes two fixed 1-second sleeps that run while the original table is write-locked, i.e. while every query on it is blocked.
1. The idle sleep in
executeWriteFuncs(the one reported in #1630)When both the events queue and the row-copy queue are empty, the loop slept a flat second. Once row copy is done that is the steady state, so the
AllEventsUpToLockProcessedsentinel — which arrives onapplyEventsQueueafter the tables are locked — had to wait out the remainder of that sleep:It now blocks on both queues with a 1s timeout instead, which is the shape
executeDMLWriteFuncsalready uses a few lines below. Same log line after:2. The
retryOperationbackoff inwaitForRename, one step laterWith #1 fixed the cut-over still took ~1s, and the remaining second is in the next phase. After issuing the atomic RENAME in a goroutine, gh-ost polls
SHOW PROCESSLISTfor it throughretryOperation, whose backoff is a flat 1 second. The first poll usually loses the race against the RENAME registering its metadata-lock wait, so it paid that full second — again, with the table locked:Across the full
localtestssuite this hit 6 of the 73 cases that reach cut-over (~8%), which is why it does not show on every run.It now polls every 10ms, for a budget of twice the RENAME's own
lock_wait_timeout— past that the RENAME has errored out and settableRenameKnownToHaveFailed, sowaitForRenamereturns immediately and the attempt count never runs out.Both call sites go through a new
retryOperationWithInterval, which is the existingretryOperationbody with the attempt count and wait made explicit;retryOperationnow delegates to it with its previousMaxRetries()/1s values. No behaviour change for its other callers.Results
Lock & rename durationover the 73localtestscases that reach cut-over. Before, it was ~1s on essentially every cut-over:The remaining tail is not this code path. It is
waitForEventsUpToLock: on both flavours the same four cases, which happen to see no DML during the migration, spend 54–77ms waiting for the sentinel to come back through an otherwise idle binlog stream. No case with concurrent DML exceeds 50ms on either flavour (0/60 and 0/59). That delay reproduces with a plainmariadb-binlog --read-from-remote-serverclient, so it is in binlog delivery rather than in gh-ost, and it predates this change.Test change
TestCutOverLossDataCaseLockGhostBeforeRenametook its lock on the ghost table one second after removing the postpone flag, relying on cut-over still being in flight. With cut-over now ~10ms the ghost table was already renamed, givingTable 'test._testing_gho' doesn't exist. It now takes the lock before un-postponing, which is what the test name describes anyway and no longer depends on cut-over being slow.Verification
script/cibuild— clean (gofmt, build,go test -race ./go/...).script/docker-gh-ost-replica-tests run— full suite passes onmariadb:11.4,mariadb:11.8,mysql:8.0.41andmysql:8.4.3.Lock & rename durationmeasured onmariadb:11.8andmysql:8.4.3for the table above.script/cibuildreturns with no formatting errors, build errors or unit test errors.