Skip to content

Say why a run retries and keep its connection when the run ends - #716

Merged
lavaman131 merged 1 commit into
mainfrom
connector-log-fixes
Oct 21, 2026
Merged

lavaman131 merged 1 commit into
mainfrom
connector-log-fixes

Conversation

@lavaman131

Copy link
Copy Markdown
Collaborator

Two connector log bugs seen while a build ran through a two-and-a-half-minute Chopin outage. Both are in the chopin CLI, so they ship with the next CLI release. No protocol change.

"Still trying to reach Chopin" said "no answer" behind a 502 gateway

While Chopin was down, the proxy in front of it answered every request with 502 within about 50 ms. Every retry line said no answer instead of the status.

Once the run's connection drops, each heartbeat registers the machine again before it calls Chopin. The proxy's 502 has an empty body sent with Content-Length: 0, and Bun's response.json() reads that as null, not a parse error. So post() in machine.ts threw TypeError: null is not an object (evaluating 'value.error'). It did not throw the ConnectorError("http-502", …, 502) it was meant to throw. That TypeError happened to be classified as retryable, but it has no status or code, so failureKind fell back to no answer. A 502 with an HTML body, or a refused connection, never showed the problem.

  • post() now treats any body that is not a JSON object as empty, so registration refuses with the HTTP status.
  • failureKind now reads the status of a refused registration (HTTP 502, the same wording as the MCP client's errors, or HTTP 503 unavailable when there is a server code). It also finds a status, server code or network code in anything the error wraps as its cause. It still never reports message text.

False "Lost the connection" / "Lost contact" right after "✓ Finished"

When a run ends, runWork stops its heartbeat, which cancels a renew_* call that is still in flight. The MCP client rejects that with MCP error -32001: AbortError. link.call only checked the connector's own stop signal, and classifyConnectorError treats -32001 as retryable. So the connector printed "Lost the connection to Chopin during a run" and dropped the healthy shared connection, and runReports.attempt printed "Lost contact with Chopin".

Now a call that its own caller gave up is not a connection failure: it keeps the connection and prints nothing. A call that timed out (its signal's reason is a TimeoutError) still counts as a failure, as before.

Tests

Each test failed on the unfixed code for the reason above:

  • run.test.ts: "a run behind a gateway answering 502 with no body says HTTP 502 while it retries". It runs the real serveWork → runWork heartbeat with a connect that registers and then opens MCP, the way chopin connect does, against a gateway that sends an empty 502. Before the fix the lines said no answer.
  • run.test.ts: "a heartbeat in flight when a run finishes is cancelled without losing the connection". Before the fix it printed the same MCP error -32001: AbortError lines and registered the machine a second time.
  • connection.test.ts: failureKind for wrapped refusals and network errors.

A proxy answering for a Chopin that went away sends 502 with Content-Length 0.
Bun reads that empty body as JSON null, so re-registering the machine threw a
TypeError reading `error` of null instead of refusing with the status. The
TypeError happened to be retried, but carries no status or code, so every
"Still trying to reach Chopin" line of a run said "no answer". Registration now
treats a body that is not a JSON object as empty and refuses with the HTTP
status, and a failure's kind names a refusal's status ("HTTP 502") and looks
for a status, server code or network code in anything the error wraps.

A heartbeat still in flight when a run finished was cancelled by the run, and
the MCP client rejects that as a request timeout, which counted as a failure:
the terminal said the connection was lost and Chopin out of reach right after
"Finished", and the healthy shared connection was dropped. A call its own
caller gave up is no longer a failure: it keeps the connection and says
nothing. A call that ran out of time still counts.

Assistant-model: Claude Opus 5.5 (fast)
Assistant-workflow: inline (fast subagents)
Assistant-verification: bun test (connector suite and full suite) passed
Assistant-verification: bun run types and bun run ci passed
Co-authored-by: Alex Lavaee <alexlavaee@microsoft.com>
@lavaman131 lavaman131 self-assigned this Oct 21, 2026
@lavaman131
lavaman131 merged commit 5195fb6 into main Oct 21, 2026
24 checks passed
@lavaman131
lavaman131 deleted the connector-log-fixes branch October 21, 2026 20:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant