[Fix] generate: propagate the disconnect 400 instead of returning null - #4782
[Fix] generate: propagate the disconnect 400 instead of returning null#4782AmirF194 wants to merge 1 commit into
Conversation
Fixes InternLM#4781 _inner_call()'s disconnect branch returns create_error_response(400) from inside its async with block, but the call site never captures it and the nonlocal response it otherwise sets is only reachable on the normal completion path. Capture _inner_call()'s return value and return it directly when not None.
|
Looked at the two failing jobs (pr_functions_test, unit_test). Both fail identically before any test runs: the turbomind CUDA extension build via cmake/ninja fails during |
|
This has been open a week with no review yet, no rush. The one failing job here (unit_test) is an unrelated CI-runner disk-space issue during the turbomind CUDA build, not from this diff (it only touches api_server.py and a test file, no C++/CUDA). Happy to rebase or split it if that would help review. |
Motivation
Fixes #4781. The non-streaming
/generateendpoint aborts the session on client disconnect but discards the400response it computed, so the caller getsHTTP 200with anullbody instead. Same discard-the-inner-return shape as #4776/#4777, in the sibling/generateendpoint.Modification
generate's_inner_call()returns the disconnect error response from inside itsasync withblock, but the call site (await _inner_call()) never captured it, and thenonlocal responseit otherwise sets is only reachable on the non-disconnect path. Capture_inner_call()'s return value and return it directly when notNone, falling back toresponseonly on the normal completion path:Verification
Docker (
python:3.11-slim, CPU-only deps, no GPU), currentmain(9140d372). Addedtest_generate_disconnect_returns_client_disconnected_responsetotests/test_lmdeploy/serve/test_session_cleanup.py, mirroring the existingtest_completions_v1_disconnect_returns_client_disconnected_responsein the same file: a fakeAsyncEngine.generate()yields one item then blocks, and a fakeraw_request.is_disconnected()returnsTrue. Confirmed the test fails on unmodifiedmain(resultisNone) and passes with this change. The fulltests/test_lmdeploy/serve/test_session_cleanup.pyfile (10 tests) stays green, andpre-commit run --fileson both changed files passes.Not verified: the CUDA/GPU CI legs (no GPU on this machine) and the pre-existing failures in
tests/test_lmdeploy/serve/openai/responses/(unrelatedpydanticvalidation errors in the/responsesAPI, reproduced as already red on unmodifiedmain, untouched by this change).BC-breaking
None.
Checklist