fix(pd): harden production cutover lifecycles

[sufubao]

Summary

This PR hardens PD separation for production cutover across request admission, first-token arbitration, WebSocket ownership, and NIXL peer/transfer lifecycles.

• Preserve prefill cache metadata while preventing duplicate first-token output, with a bounded fallback when the prefill token is missing.
• Preserve a terminal decode marker so KV-transfer failures cannot be hidden by first-token deduplication.
• Cancel worker generation tasks even while they are waiting for shared-memory request slots, and release partially allocated slots on cancellation.
• Serialize NIXL peer admission/removal and protect reconnects with peer generations and active-transfer tracking.
• Bind PD registrations to their owning WebSocket, fail requests assigned to replaced or disconnected nodes, and ignore stale disconnects.
• Supervise worker generation, token-forwarding, and heartbeat tasks so background failures tear down the connection instead of becoming silent hangs.
• Route NIXL status and completion-notification failures through normal task cleanup; make any remaining uncaught daemon-thread failure visible to process supervision.

Production failure modes addressed

Before these changes, a request could remain stuck after its assigned PD node disconnected, a stale WebSocket cleanup could remove a newer registration, and worker generation exceptions could disappear while the receive loop stayed alive. NIXL status-thread exceptions could also strand pages and transfer handles while the transfer subprocess still appeared healthy.

The resulting lifecycle guarantees are:

1. A WebSocket owns exactly the registration it created.
2. Requests are awakened immediately when either assigned node is replaced or disconnected.
3. A worker connection cannot remain healthy after one of its critical background tasks fails.
4. A handled NIXL task failure reaches the existing failed-task cleanup path.
5. An unhandled transfer-thread failure terminates the subprocess and becomes observable by existing liveness supervision.

Inline comments document the non-obvious request-slot cancellation and NIXL failure-recovery invariants.

Audited fixes, one commit each

• f6a44175 fix(pd): fail requests when assigned nodes disconnect
• 623d640d fix(pd): propagate worker background failures
• feedc43e fix(pd): recover NIXL transfer task failures

Each commit includes focused regression coverage and a commit body describing its root cause and runtime behavior.

Validation

PYTHONPATH=. python -m pytest -q \
  unit_tests/server/httpserver/test_pd_pending_abort.py \
  unit_tests/server/httpserver/test_pd_connection_tasks.py \
  unit_tests/test_nixl_kv_transporter_thread_safety.py \
  unit_tests/server/httpserver/test_pd_master_cached_tokens.py \
  test/test_pd_selector/test_pd_master_multi_choice.py \
  unit_tests/server/test_pd_master_mode.py \
  unit_tests/server/test_pd_cache_aware.py

51 passed

The touched files also pass the repository flake8 configuration and git diff --check.

No benchmark was run because this change is correctness and lifecycle focused.

[sufubao]

Superseded by three independent PRs, each based directly on upstream main and containing one fix only:

• fix(pd): fail requests when assigned nodes disconnect #1473 — fail requests when assigned PD nodes disconnect
• fix(pd): propagate worker background failures #1475 — propagate PD worker background failures
• fix(pd): recover NIXL transfer task failures #1474 — recover NIXL transfer task failures

The replacement PRs intentionally omit the added unit-test changes.