Skip to content

test: fix flaky run() compaction test and hang-on-failure - #4

Merged
mcollina merged 1 commit into
mainfrom
fix/flaky-run-tests-release-timeout
Oct 13, 2026
Merged

mcollina merged 1 commit into
mainfrom
fix/flaky-run-tests-release-timeout

Conversation

@mcollina

Copy link
Copy Markdown
Member

Problem

Release run 38384169229 hung in the Test step for ~18 minutes until it was cancelled. The log shows:

✖ run() compacts per level using time-bucketed scheduling (870.05405ms)
##[error]The operation was canceled.

Two bugs combined:

  1. Flaky assertion. The test starts replica.run(), writes for ~150 ms, then polls restore(). On a slow runner the first sync hadn't uploaded anything yet, so restore() correctly threw NoMatchingBackupError and waitForRestoredRows rethrew it instead of retrying.
  2. Hang on failure. ac.abort() only ran on the success path, so after the failure the run() loop kept going and the test process never exited. The release job had no timeout.

Reproduced locally by delaying the first writeLtxFile by 300 ms: same NoMatchingBackupError, then the process hangs.

Changes

  • waitForRestoredRows (compaction-scheduling and compaction-retention tests) retries transient restore failures until its deadline and includes the last error in the timeout message. The retention test also needs this because retention can delete files while a restore is reading them.
  • The compaction and leasing run() tests always abort and await the run loop in finally, so a failed assertion fails fast instead of hanging.
  • release.yml: drop the Test step (CI already runs the tests on every push to main) and add timeout-minutes: 15.

Verification

  • pnpm lint, pnpm typecheck:all and pnpm test:all pass (119 + 18 tests).
  • 25 back-to-back runs of the three changed test files: 0 failures.
  • With the 300 ms first-upload delay the scheduling test now passes. With a forced assertion failure it exits in under a second instead of hanging.

🤖 Generated with Claude Code

https://claude.ai/code/session_0162xGMZU6wtdADEwqhqNqnY

The time-bucketed scheduling test polled restore() while run() was still
performing its first sync. On a slow runner the replica was still empty,
so restore() threw NoMatchingBackupError and the helper rethrew it
instead of retrying. Because ac.abort() only ran on the success path,
the failed test left run() looping forever and the process never exited,
which hung the release workflow.

- waitForRestoredRows retries transient restore failures until its
  deadline and reports the last error on timeout
- always abort and await run() in finally (compaction and leasing tests)
- release workflow: skip tests (CI covers them) and add a job timeout

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0162xGMZU6wtdADEwqhqNqnY
@mcollina
mcollina merged commit dc6d463 into main Oct 13, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant