Skip to content

Fix loadscope scheduler deadlock after a worker crash - #1363

Open
larsoner wants to merge 1 commit into
pytest-dev:masterfrom
larsoner:lock
Open

Fix loadscope scheduler deadlock after a worker crash#1363
larsoner wants to merge 1 commit into
pytest-dev:masterfrom
larsoner:lock

Conversation

@larsoner

Copy link
Copy Markdown
  • Make sure to include reasonable tests for your change if necessary
  • We use towncrier for changelog management, so please add a news file into the changelog folder ...

I used Claude Fable 5 to look into #1313 (after hitting a very reliable hang in MNE-Python in mne-tools/mne-python#14063 during a ~20m run). With some guidance, Claude was able to boil it down into a minimal repro (now adapted into a unit test) and suggest this fix based on its conclusion:

The bug is that mark_test_complete() never prunes fully-completed work units from assigned_work, so when a worker crashes, remove_node() requeues everything it ever ran. Replacement (or endgame) workers then get assigned an already-completed unit as an empty runtests command, which can never be marked complete — so _reschedule() never fires again and the session deadlocks with idle workers.

I then tested in mne-tools/mne-python#14101 that 1) it still hung in MNE-Python with latest pytest-xdist master (it did), then 2) used this PR and saw it succeed in ending properly after an error rather than hanging. It coming back red allowed me to narrow down the bug to a Qt6<->Cocoa issue (with multiple runs, each ending properly on this branch) and work around it to get to green 🎉 !

Related to #1313 and addressed what I commented on there, but not sure if it actually closes it or not. @clemlesne you could try my branch and check if you have the spare cycles (I pushed upstream tags to git so the dynamic dev version gets resolved correctly, and used pip install "pytest-xdist @ git+https://github.com/larsoner/pytest-xdist.git@lock" in my own testing.)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant