Repository navigation
Say why a run retries and keep its connection when the run ends - #716
Merged
Merged
Conversation
A proxy answering for a Chopin that went away sends 502 with Content-Length 0.
Bun reads that empty body as JSON null, so re-registering the machine threw a
TypeError reading `error` of null instead of refusing with the status. The
TypeError happened to be retried, but carries no status or code, so every
"Still trying to reach Chopin" line of a run said "no answer". Registration now
treats a body that is not a JSON object as empty and refuses with the HTTP
status, and a failure's kind names a refusal's status ("HTTP 502") and looks
for a status, server code or network code in anything the error wraps.
A heartbeat still in flight when a run finished was cancelled by the run, and
the MCP client rejects that as a request timeout, which counted as a failure:
the terminal said the connection was lost and Chopin out of reach right after
"Finished", and the healthy shared connection was dropped. A call its own
caller gave up is no longer a failure: it keeps the connection and says
nothing. A call that ran out of time still counts.
Assistant-model: Claude Opus 5.5 (fast)
Assistant-workflow: inline (fast subagents)
Assistant-verification: bun test (connector suite and full suite) passed
Assistant-verification: bun run types and bun run ci passed
Co-authored-by: Alex Lavaee <alexlavaee@microsoft.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two connector log bugs seen while a build ran through a two-and-a-half-minute Chopin outage. Both are in the
chopinCLI, so they ship with the next CLI release. No protocol change."Still trying to reach Chopin" said "no answer" behind a 502 gateway
While Chopin was down, the proxy in front of it answered every request with 502 within about 50 ms. Every retry line said
no answerinstead of the status.Once the run's connection drops, each heartbeat registers the machine again before it calls Chopin. The proxy's 502 has an empty body sent with
Content-Length: 0, and Bun'sresponse.json()reads that asnull, not a parse error. Sopost()inmachine.tsthrewTypeError: null is not an object (evaluating 'value.error'). It did not throw theConnectorError("http-502", …, 502)it was meant to throw. That TypeError happened to be classified as retryable, but it has no status or code, sofailureKindfell back tono answer. A 502 with an HTML body, or a refused connection, never showed the problem.post()now treats any body that is not a JSON object as empty, so registration refuses with the HTTP status.failureKindnow reads the status of a refused registration (HTTP 502, the same wording as the MCP client's errors, orHTTP 503 unavailablewhen there is a server code). It also finds a status, server code or network code in anything the error wraps as itscause. It still never reports message text.False "Lost the connection" / "Lost contact" right after "✓ Finished"
When a run ends,
runWorkstops its heartbeat, which cancels arenew_*call that is still in flight. The MCP client rejects that withMCP error -32001: AbortError.link.callonly checked the connector's own stop signal, andclassifyConnectorErrortreats -32001 as retryable. So the connector printed "Lost the connection to Chopin during a run" and dropped the healthy shared connection, andrunReports.attemptprinted "Lost contact with Chopin".Now a call that its own caller gave up is not a connection failure: it keeps the connection and prints nothing. A call that timed out (its signal's reason is a
TimeoutError) still counts as a failure, as before.Tests
Each test failed on the unfixed code for the reason above:
run.test.ts: "a run behind a gateway answering 502 with no body says HTTP 502 while it retries". It runs the realserveWork→runWorkheartbeat with a connect that registers and then opens MCP, the waychopin connectdoes, against a gateway that sends an empty 502. Before the fix the lines saidno answer.run.test.ts: "a heartbeat in flight when a run finishes is cancelled without losing the connection". Before the fix it printed the sameMCP error -32001: AbortErrorlines and registered the machine a second time.connection.test.ts:failureKindfor wrapped refusals and network errors.