Skip to content

Start the gateway outage test's outage after the agent's session report lands - #722

Merged
lavaman131 merged 1 commit into
mainfrom
deflake-gateway-retry-test
Oct 21, 2026
Merged

lavaman131 merged 1 commit into
mainfrom
deflake-gateway-retry-test

Conversation

@lavaman131

@lavaman131 lavaman131 commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

The connector test "a run behind a gateway answering 502 with no body says HTTP 502 while it retries" failed on unrelated PRs. In one run it printed attempt 6, 28 s where it expected attempt 6, 24 s. In another, the attempt 6 line never appeared. Locally it passed.

Cause

The fake gateway moved the test's clock four seconds for every refused request and ended the outage after eight. The test expected all eight to come from the heartbeat. But the agent sends its session report (report_implementation_build, state running) when it starts. On a fast runner, that report arrived during the outage. Each of its retries registered the machine again and was refused, which moved the clock and used up one of the eight refusals. The heartbeat then printed other attempt numbers and elapsed times, or the outage ended before its sixth attempt. Both CI failures fit one or two refusals of this kind.

The connector is not at fault here: a report retried during an outage should keep trying to reach Chopin.

Fix

This changes only the test. Chopin now goes away only once it has heard the agent's session report. After that, every refused request comes from the heartbeat, so the printed lines are the same on every run, and the exact assertions stay. The test now waits for the agent to start before the outage, so it takes longer and gets a 10 second budget, as other slow tests in the file do.

Evidence

  • Before: with only the test process paused again and again (SIGSTOP 30 ms, then run 10 ms, which lets the agent start while the outage is still going), the old test failed 19 of 20 runs. One failure showed the exact CI mismatch: attempt 6, 28 s instead of attempt 6, 24 s.
  • After, on the same test logic:
    • 50 plain runs: 50 passed.
    • 50 runs with the same pausing: 50 passed.
    • 20 runs with heavier pausing (stop 50 ms, run 5 ms): 20 passed.
    • 50 runs pinned to one CPU beside three busy loops: 50 passed. The slowest took 5.6 s, which is why the 10 second budget is needed.
  • Before the 10 second budget was added, 1 of 50 pinned runs timed out at bun's default 5 s.
  • bun test apps/connector, bun run types and bun run ci pass.

…rt lands

The test that has a gateway answer 502 while Chopin is away moved its fake clock
four seconds for every refused request and ended the outage after eight. It
expected all eight to come from the heartbeat, but the agent's session report is
sent when the agent starts, and on a fast runner that happened during the
outage. Each of the report's retries registered the machine again, was refused,
and moved the clock and the count, so the heartbeat printed different attempt
numbers and elapsed times, or stopped before its sixth attempt.

Chopin now goes away only after it has heard the agent's session, so the
heartbeat makes every refused request and the printed lines are the same on
every run. Waiting for the agent before the outage makes the test longer, so it
gets a 10 second budget, as other slow tests in the file do.

Assistant-model: Claude Opus 5.5 (fast)
Assistant-workflow: inline (fast subagents)
Assistant-verification: the old test failed 19 of 20 runs with the test process repeatedly paused, including the exact attempt and elapsed-time mismatch seen in CI
Assistant-verification: the new test passed 50 plain runs, 50 runs with the test process repeatedly paused, and 50 runs pinned to one CPU beside three busy loops
Assistant-verification: bun test apps/connector, bun run types and bun run ci pass
Co-authored-by: Alex Lavaee <alexlavaee@microsoft.com>
@lavaman131 lavaman131 self-assigned this Oct 21, 2026
@lavaman131
lavaman131 merged commit 2f180df into main Oct 21, 2026
24 checks passed
@lavaman131
lavaman131 deleted the deflake-gateway-retry-test branch October 21, 2026 22:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant