fix(pd): fail requests when assigned nodes disconnect - #1473
Open
sufubao wants to merge 1 commit into
Open
Conversation
Track registrations by their owning WebSocket so a stale disconnect cannot evict a replacement connection. Wake requests assigned to a replaced or disconnected node and surface the connection failure at every PD wait stage.
This was referenced Aug 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Bind each PD registration to the WebSocket that created it.
Root cause
PD nodes were keyed only by endpoint. When a node reconnected, the old WebSocket cleanup could remove the new registration. Requests also retained the old node object and had no failure notification, so they could wait until a long timeout after the transport was already gone.
Runtime invariant
A disconnect may remove only the registration owned by that exact WebSocket. Any request that references the removed node is terminally failed before the node disappears from request processing.
Scope
This PR is based directly on
upstream/main, contains one production-code commit, and adds no unit-test files or test changes.Validation
39 passed