Skip to content

fix(pd): fail requests when assigned nodes disconnect - #1473

Open
sufubao wants to merge 1 commit into
ModelTC:mainfrom
sufubao:fix/pd-node-disconnect-lifecycle
Open

fix(pd): fail requests when assigned nodes disconnect#1473
sufubao wants to merge 1 commit into
ModelTC:mainfrom
sufubao:fix/pd-node-disconnect-lifecycle

Conversation

@sufubao

@sufubao sufubao commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

Summary

Bind each PD registration to the WebSocket that created it.

  • Ignore stale disconnect cleanup from an older connection after the same endpoint reconnects.
  • Fail every in-flight request assigned to a replaced or disconnected prefill/decode node.
  • Wake prefill, decode, and token waiters and surface the recorded connection error immediately.

Root cause

PD nodes were keyed only by endpoint. When a node reconnected, the old WebSocket cleanup could remove the new registration. Requests also retained the old node object and had no failure notification, so they could wait until a long timeout after the transport was already gone.

Runtime invariant

A disconnect may remove only the registration owned by that exact WebSocket. Any request that references the removed node is terminally failed before the node disappears from request processing.

Scope

This PR is based directly on upstream/main, contains one production-code commit, and adds no unit-test files or test changes.

Validation

  • Existing PD regression suite: 39 passed
  • Repository pre-commit hooks: passed
  • Python syntax compilation: passed

Track registrations by their owning WebSocket so a stale disconnect cannot evict a replacement connection. Wake requests assigned to a replaced or disconnected node and surface the connection failure at every PD wait stage.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant