Don't lose worker startup exception when its process exit is observed first (flaky test_local_cluster_redundant_kwarg)#9327
Merged
crusaderky merged 2 commits intoJul 9, 2026
Conversation
… first
When the worker process fails to start, it sends the exception to the
Nanny through init_result_q, flushes the queue, and terminates. If
mark_stopped(), fired by the process exit callback, ran before
WorkerProcess._wait_until_connected polled the message out of the queue
(the poll interval is 50ms), the exception was lost: instantiate()
returned Status.stopped and Nanny.start_unsafe raised a bare
RuntimeError("Nanny failed to start worker process") instead of
chaining from the original exception.
Fix by polling init_result_q one last time after observing that the
process stopped; the child flushes the queue before exiting, so the
message (if any) is guaranteed to be readable by then.
Follow-up to dask#9313; fixes the remaining flakiness in
test_local_cluster_redundant_kwarg.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
test_local_cluster_redundant_kwarg)test_local_cluster_redundant_kwarg)
test_local_cluster_redundant_kwarg)test_local_cluster_redundant_kwarg)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix race condition in
Nannywhen a worker fails to start.Follow-up to #9313; fixes the remaining flakiness in
test_local_cluster_redundant_kwarg(example failure).In-depth commentary (AI-generated)
Root cause
When the worker process fails to start, the child sends the startup exception to the Nanny through
init_result_q, flushes the queue, and terminates.WorkerProcess._wait_until_connectedpolls that queue every 50ms and, in each iteration, first bails out withNoneif the status is no longerStatus.starting. Ifmark_stopped()(fired by the process exit callback) flipped the status during one of the 50ms sleeps, before the poll picked up the message, the exception was lost:instantiate()returnedStatus.stoppedandNanny.start_unsaferaised a bareRuntimeError("Nanny failed to start worker process")instead of chaining from the worker's original exception, breaking the test'sraises_with_cause(RuntimeError, None, TypeError, "unexpected keyword argument")expectation.Fix
init_result_qone last time after observing that the process stopped. The child always flushes the queue before exiting (close()+join_thread()), so by the time the exit callback has run, the message — if any — is guaranteed to be readable.init_result_qinWorkerProcess.start(), sincemark_stopped()releasesself.init_result_q.Status.runningif the process already died:mark_stopped()set it toStatus.stoppedand released the queues, and overwriting it would make the nextkill()crash on the released queues, leaving the Nanny stuck inStatus.closing(same failure mode described in Fix Nanny race conditions when worker fails to start (flakytest_failure_during_worker_initialization) #9313).