Skip to content

fix(pd): propagate worker background failures - #1480

Closed
sufubao wants to merge 2 commits into
ModelTC:mainfrom
sufubao:fix/pd-worker-background-failures
Closed

fix(pd): propagate worker background failures#1480
sufubao wants to merge 2 commits into
ModelTC:mainfrom
sufubao:fix/pd-worker-background-failures

Conversation

@sufubao

@sufubao sufubao commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator

背景

PD worker 的 token forwarding、heartbeat 和单请求 generation 都运行在后台 task 中,但连接主循环只等待 websocket.recv()

当 generation 抛出异常时,_pd_process_generate() 原先只记录日志并吞掉异常;当 forwarding 或 heartbeat task 异常退出时,主循环也不会观察其结果。因此 worker 连接可能仍然保持存活,而请求不再产生 token 或终止信号。

修改

  • 将 WebSocket receive 与 forwarding、heartbeat、generation failure 纳入同一个 connection lifecycle;
  • 任一连接级后台 task 异常退出时,取消当前 receive 并关闭连接;
  • generation task 不再吞掉非预期异常,并在完成回调中上报失败;
  • 连接退出时统一取消并等待所有子 task,避免遗留 task;
  • 增加测试,覆盖后台失败打断阻塞 receive,以及 generation 异常向上传播。

依赖

依赖 #1473。该 PR 负责在 PD 节点断连后唤醒并失败掉分配给该节点的在途请求;本 PR 负责确保 worker 后台 task 失败能够转化为可被 PD master 观察到的连接断开。

建议先合并 #1473

验证

  • python -m pytest -q unit_tests/server/httpserver/test_pd_connection_tasks.py
  • 结果:2 passed

Supervise token forwarding, heartbeat, and per-request generation tasks while the worker waits for PD master messages. Tear down the connection when a background task fails so the master observes a disconnect instead of waiting on a silently dead request.
@sufubao sufubao closed this Aug 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant