Repository navigation
test(master): leader failover rebuild and sustained watchdog isolation - #371
Merged
beinan merged 1 commit intoOct 9, 2026
Merged
Conversation
The two test gaps left open across review rounds 5 and 7.
Leader failover (spec test 4): master A leads, builds the demand cache
and shadow placements over three tables, and is aborted without
releasing so its lease expires naturally. Its token is then refused
("leadership lost"). Master B campaigns with only etcd to go on and
must rebuild an identical demand cache, identical shadow placements
(target, executor, bytes), identical executor totals, and must keep a
durable Bound reservation written by the old leader. Mutations
checked: a second leader that places differently fails on the
placements assertion; a rebuild that drops Bound reservations fails on
the durable-reservation assertion.
Sustained concurrent isolation: four real index preparations at once
for several idle windows - two throttled-but-alive at different rates,
one frozen from the start, one frozen partway after real progress.
Alive builds must finish past the idle window with real IO and the
slower one must take longer; frozen-start is cancelled in [IDLE, IDLE+4)
from the start; frozen-mid is cancelled in that window measured from
its own freeze, not from the start and not from a neighbour's progress;
alive builds outlive both cancellations. The whole scenario is bounded
so a never-cancelled build fails rather than hangs. Mutation checked: a
shared progress counter across watchdogs fails the test.
Both pass against real etcd (9 s and 16 s); planner, watchdog and
isolation suites green. Clippy -D warnings, fmt, typos clean.
Refs lance-format#333.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
17 of 18 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #333. Closes the two test gaps left open across review rounds 5 and 7. Test-only; no behaviour change.
Leader failover (spec test 4)
Master A leads, builds cache + shadow placements over three tables, is aborted without releasing (lease expires naturally). Its token is then refused. Master B campaigns with only etcd and must rebuild identical demand cache, shadow placements (target/executor/bytes), executor totals, and keep a durable
Boundreservation written by the old leader.Mutations caught: second leader placing differently → placements assertion; rebuild dropping
Bound→ durable-reservation assertion.Sustained concurrent isolation
Four real index preparations at once over several idle windows: throttled-alive ×2 (different rates), frozen-from-start, frozen-midway after real progress. Asserts: alive pair finishes past the idle window with real IO and the slower one takes longer; frozen-start cancelled in
[IDLE, IDLE+4); frozen-mid cancelled in that window measured from its own freeze; alive builds outlive both cancellations. Whole scenario bounded at 60 s so a never-cancelled build fails fast.Mutation caught: a shared progress counter across watchdogs fails the test.
Validation
Both green against real etcd (9 s, 16 s); planner/watchdog/isolation suites green; clippy
-D warnings, fmt, typos clean.🤖 Generated with Claude Code