fix(pd): propagate worker background failures - #1480
Closed
sufubao wants to merge 2 commits into
Closed
Conversation
Supervise token forwarding, heartbeat, and per-request generation tasks while the worker waits for PD master messages. Tear down the connection when a background task fails so the master observes a disconnect instead of waiting on a silently dead request.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景
PD worker 的 token forwarding、heartbeat 和单请求 generation 都运行在后台 task 中,但连接主循环只等待
websocket.recv()。当 generation 抛出异常时,
_pd_process_generate()原先只记录日志并吞掉异常;当 forwarding 或 heartbeat task 异常退出时,主循环也不会观察其结果。因此 worker 连接可能仍然保持存活,而请求不再产生 token 或终止信号。修改
依赖
依赖 #1473。该 PR 负责在 PD 节点断连后唤醒并失败掉分配给该节点的在途请求;本 PR 负责确保 worker 后台 task 失败能够转化为可被 PD master 观察到的连接断开。
建议先合并 #1473。
验证
python -m pytest -q unit_tests/server/httpserver/test_pd_connection_tasks.py