Skip to content

fix(pd): recover NIXL transfer task failures - #1474

Open
sufubao wants to merge 1 commit into
ModelTC:mainfrom
sufubao:fix/pd-nixl-task-failure-recovery
Open

fix(pd): recover NIXL transfer task failures#1474
sufubao wants to merge 1 commit into
ModelTC:mainfrom
sufubao:fix/pd-nixl-task-failure-recovery

Conversation

@sufubao

@sufubao sufubao commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

Summary

Make NIXL transfer failures observable and recoverable without silently losing the transfer status thread.

  • Route status-query exceptions to the existing failed-task queue.
  • Route completion-notification failures to the same cleanup path.
  • Treat telemetry collection as diagnostic so telemetry errors do not turn a completed transfer into a stuck task.
  • Install a fatal thread exception hook in prefill and decode transfer subprocesses.

Root cause

An exception in the prefill transfer status loop escaped the daemon thread. The subprocess remained alive, so process liveness checks could not detect that no later NIXL transfer would complete. The failed task could also retain its page and transfer handle.

Runtime invariant

Handled per-transfer failures reach normal page and handle cleanup. Any unexpected exception that still escapes a transfer daemon thread terminates the transfer subprocess, making the failure visible to existing process supervision.

Scope

This PR is rebased onto the latest upstream/main, contains one production-code commit, and adds no unit-test files or test changes. The fault-injection tests from the production branch depend on the separate NIXL peer-reconnect test fixture and are intentionally not included here.

Validation

  • Existing PD regression tests: 31 passed
  • Changed-file pre-commit hooks (black, flake8): passed
  • Python syntax compilation: passed
  • git diff --check: passed

Route status-query and completion-notification errors through the existing failed-task cleanup path so pages and transfer handles are reclaimed. Install a fatal thread exception hook in transfer subprocesses so any unhandled daemon-thread failure is visible to process supervision.
@sufubao
sufubao force-pushed the fix/pd-nixl-task-failure-recovery branch from d5603da to e30ea00 Compare August 21, 2026 08:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant