fix(pd): recover NIXL transfer task failures - #1474
Open
sufubao wants to merge 1 commit into
Open
Conversation
Route status-query and completion-notification errors through the existing failed-task cleanup path so pages and transfer handles are reclaimed. Install a fatal thread exception hook in transfer subprocesses so any unhandled daemon-thread failure is visible to process supervision.
sufubao
force-pushed
the
fix/pd-nixl-task-failure-recovery
branch
from
August 21, 2026 08:38
d5603da to
e30ea00
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Make NIXL transfer failures observable and recoverable without silently losing the transfer status thread.
Root cause
An exception in the prefill transfer status loop escaped the daemon thread. The subprocess remained alive, so process liveness checks could not detect that no later NIXL transfer would complete. The failed task could also retain its page and transfer handle.
Runtime invariant
Handled per-transfer failures reach normal page and handle cleanup. Any unexpected exception that still escapes a transfer daemon thread terminates the transfer subprocess, making the failure visible to existing process supervision.
Scope
This PR is rebased onto the latest
upstream/main, contains one production-code commit, and adds no unit-test files or test changes. The fault-injection tests from the production branch depend on the separate NIXL peer-reconnect test fixture and are intentionally not included here.Validation
31 passedblack,flake8): passedgit diff --check: passed