Skip to content

Baseline fetch (fetchPreviousRun) can hang forever — 10s timeout doesn't survive a half-open socket during body read #163

Description

@veksen

This was generated by AI during triage.

Summary

fetchPreviousRun (the CI baseline fetch, src/reporters/site-api.ts / driven from src/main.ts) can hang indefinitely. Its 10s AbortSignal.timeout(10000) does not survive a half-open socket during body read, so a transient Site API stall becomes a full CI-job hang that only ends at the job's 20-minute timeout.

Evidence

Observed in GitHub Actions run 28626422344 (repo veksen-enterprises/d2armory, PR #907, test job 84893821211, attempt 1). Tests passed in ~70s; the job then hung on the final query-doctor/analyzer@main step and was killed by the job's timeout-minutes: 20 ("The operation was canceled" at 23:08:21, exactly 20 min after job start 22:48:09).

Log trace:

22:50:24.09  Site API ingestion successful: {"id":"019f2506-..."}   # POST /ci/runs = HTTP 200, ~1s
22:50:24.09  Fetching previous run from https://api.querydoctor.com/ci/runs/latest?repo=veksen-enterprises%2Fd2armory&branch=main&excludeId=...
22:50:25.85  [main] API connection broken during CI run: Error: RPC session was shut down by disposing the main stub
             ...total silence until the 23:08:21 kill (~18 min)

Key point: none of fetchPreviousRun's own failure/retry log lines appeared — no Failed to fetch previous run, no Retrying baseline fetch (attempt 2/3). So the 10s timeout never took effect. Execution was sitting in await response.json() on the multi-MB /ci/runs/latest response after headers had already arrived, and the abort did not interrupt the stalled body read once the server (capnweb / Cloudflare-Workers "main stub") was disposed mid-stream and left a half-open TCP socket (no FIN/RST).

The ingest POST succeeded first — this is not the dup-hash ingest 500.

Fix

Enforce a hard wall-clock deadline covering the entire fetch + parse, not just the initial request:

  • Promise.race([fetchAndParse(), rejectAfter(deadlineMs)]), or a total-deadline AbortSignal that also aborts body consumption, so await response.json() on a half-open socket is guaranteed to reject.
  • On deadline, fall through to the existing { kind: "error" }comparisonUnavailable path ("comparison temporarily unavailable — re-run") instead of hanging.

Result: a stalled baseline fetch degrades in ~10s instead of hanging 18+ minutes.

Severity

Medium. Not currently firing (the run self-resolved on re-run with no client change, consistent with a Site API restart), but it recurs on any api.querydoctor.com blip during the baseline fetch, and each recurrence is a 20-minute hung job + red PR check.

Related

  • Root-cause counterpart (server side): Query-Doctor/Site — RPC "main stub" disposal leaving a half-open response socket.
  • Optional CI guardrail: veksen-enterprises/d2armory — step-level timeout on the analyzer step.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions