Skip to content

Don't leave Nanny stuck in closing status when worker kill times out (fix flaky test_chaos_rechunk)#9317

Merged
crusaderky merged 2 commits into
dask:mainfrom
crusaderky:test_chaos_rechunk
Jul 9, 2026
Merged

Don't leave Nanny stuck in closing status when worker kill times out (fix flaky test_chaos_rechunk)#9317
crusaderky merged 2 commits into
dask:mainfrom
crusaderky:test_chaos_rechunk

Conversation

@crusaderky

@crusaderky crusaderky commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator

test_chaos_rechunk frequently hangs, e.g. https://github.com/dask/distributed/actions/runs/28594679995/job/84786875121?pr=9315

This is due to an underlying race condition in Nanny which is triggered on very slow hosts (like Windows CI frequently is).

@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

   19 files   -     21     19 suites   - 21   7h 8m 50s ⏱️ - 7h 6m 35s
  100 tests  -  4 055    100 ✅  -  3 870    0 💤  -   180  0 ❌  - 5 
1 900 runs   - 77 520  1 800 ✅  - 73 412  100 💤  - 4 103  0 ❌  - 5 

Results for commit b1bef38. ± Comparison against base commit 2d43f7f.

This pull request removes 4155 and adds 100 tests. Note that renamed tests count towards both.
distributed.cli.tests.test_dask_scheduler ‑ test_dashboard
distributed.cli.tests.test_dask_scheduler ‑ test_dashboard_allowlist
distributed.cli.tests.test_dask_scheduler ‑ test_dashboard_non_standard_ports
distributed.cli.tests.test_dask_scheduler ‑ test_dashboard_port_zero
distributed.cli.tests.test_dask_scheduler ‑ test_defaults
distributed.cli.tests.test_dask_scheduler ‑ test_hostport
distributed.cli.tests.test_dask_scheduler ‑ test_idle_timeout
distributed.cli.tests.test_dask_scheduler ‑ test_interface
distributed.cli.tests.test_dask_scheduler ‑ test_multiple_protocols
distributed.cli.tests.test_dask_scheduler ‑ test_multiple_workers
…
distributed.tests.test_stress ‑ test_chaos_rechunk[1-100]
distributed.tests.test_stress ‑ test_chaos_rechunk[10-100]
distributed.tests.test_stress ‑ test_chaos_rechunk[100-100]
distributed.tests.test_stress ‑ test_chaos_rechunk[11-100]
distributed.tests.test_stress ‑ test_chaos_rechunk[12-100]
distributed.tests.test_stress ‑ test_chaos_rechunk[13-100]
distributed.tests.test_stress ‑ test_chaos_rechunk[14-100]
distributed.tests.test_stress ‑ test_chaos_rechunk[15-100]
distributed.tests.test_stress ‑ test_chaos_rechunk[16-100]
distributed.tests.test_stress ‑ test_chaos_rechunk[17-100]
…

♻️ This comment has been updated with latest results.

@crusaderky crusaderky mentioned this pull request Jul 3, 2026
4 tasks
@crusaderky crusaderky added flaky test Intermittent failures on CI. tests Unit tests and/or continuous integration labels Jul 9, 2026
@crusaderky crusaderky changed the title Fix flaky test_chaos_rechunk [Unsupervised AI] Fix flaky test_chaos_rechunk Jul 9, 2026
@crusaderky crusaderky changed the title [Unsupervised AI] Fix flaky test_chaos_rechunk [unsupervised AI] Fix flaky test_chaos_rechunk Jul 9, 2026
@crusaderky crusaderky force-pushed the test_chaos_rechunk branch from b1bef38 to 5b013ee Compare July 9, 2026 13:52
@crusaderky crusaderky changed the title [unsupervised AI] Fix flaky test_chaos_rechunk Fix flaky test_chaos_rechunk Jul 9, 2026
Fix test_chaos_rechunk flakiness on slow CI hosts with a Nanny stuck in Status.closing at teardown.
@crusaderky crusaderky force-pushed the test_chaos_rechunk branch from 5b013ee to 5c735a7 Compare July 9, 2026 13:58
@crusaderky crusaderky changed the title Fix flaky test_chaos_rechunk Don't leave Nanny stuck in closing status when worker kill times out (fix flaky test_chaos_rechunk) Jul 9, 2026
@crusaderky crusaderky marked this pull request as ready for review July 9, 2026 13:58
@crusaderky crusaderky requested a review from fjetter as a code owner July 9, 2026 13:58
@crusaderky crusaderky merged commit 90e1a84 into dask:main Jul 9, 2026
44 of 48 checks passed
@crusaderky crusaderky deleted the test_chaos_rechunk branch July 9, 2026 15:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

flaky test Intermittent failures on CI. tests Unit tests and/or continuous integration

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant