fix(ci): stabilize flaky up-docker-wsl on windows-latest - #1051
Conversation
The up-docker-wsl job was intermittently failing with the container exiting (exit code 0) right after start, followed by "[inject] EOF" and 3 failed restart attempts. The rerun of the same job with identical code passed, confirming an environmental flake on windows-latest + Podman/WSL. Root cause: podman `machine start` returns before the WSL VM and its API socket are fully ready. Containers created in that window get SIGTERM'd mid-startup; the daemon (PID 1) catches SIGTERM via signal.NotifyContext and exits 0, which cascades into the inject EOF and the restart loop. This is the Windows analogue of the Linux Podman-machine flakiness PR #1039 addresses (which is gated runtime.GOOS == linux). Changes: * .github/workflows/pr-ci.yml: mirror the Linux rootful setup for the Windows Podman step - retry machine init/start (init is not idempotent, so only on the first attempt), gate the e2e suite behind a `podman info` readiness probe, and run a throwaway preflight container before the suite starts. * pkg/devcontainer/single.go: DefaultEntrypoint now execs the daemon from $DEVSY_AGENT_PATH (falling back to /usr/local/bin/devsy). This wires in LocalDockerDelivery's pre-start volume binary, so the daemon arms immediately from the volume instead of waiting for post-start shell injection to place /usr/local/bin/devsy - shrinking the startup race window. DEVSY_AGENT_PATH was previously set but had zero readers. * pkg/config/env.go: add the EnvAgentPath constant for the above. * pkg/agent/delivery/local_docker.go: use the new constant. * pkg/driver/docker/lifecycle.go: surface restart-attempt failures at warn instead of debug so the exit code + tail logs are visible without --debug (an exit code 0 here typically means PID 1 caught a SIGTERM). Co-authored-by: openhands <openhands@all-hands.dev>
✅ Deploy Preview for images-devsy-sh canceled.
|
✅ Deploy Preview for devsydev canceled.
|
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Thank you for your contribution. Before we can merge it, please read our Contributor License Agreement. If you agree, post a comment on this pull request with exactly the following text: I have read the CLA Document and I hereby sign the CLA You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot. |
|
If you're new to commit signing, there are different ways to set it up: Sign commits with
|
Up to standards ✅🟢 Issues
|
| Metric | Results |
|---|---|
| Complexity | 0 |
| Duplication | 0 |
AI Reviewer: run a review on demand. To trigger the first review automatically, go to your organization or repository integration settings. AI can make mistakes. Always validate suggestions.
TIP This summary will be updated as you push new changes.
Summary
The
up-docker-wsljob (windows-latest) was intermittently failing with the container exiting (exit code 0) right after start, followed by[inject] EOFand 3 failed restart attempts. The rerun of the same job with identical code passed, confirming an environmental flake.Repro log: run 31775426829 / job 94693528470 (failed) → rerun job
94699202314(succeeded, same code).Root cause
podman machine startreturns before the WSL VM and its API socket are fully ready. Containers created in that window get SIGTERM'd mid-startup. The daemon (PID 1, after the entrypoint'sexec) catches SIGTERM viasignal.NotifyContext(..., os.Interrupt, syscall.SIGTERM)and exits0, which cascades into the injectEOFand the 3-restart loop (each restart re-execs the daemon, which gets SIGTERM'd again).This is the Windows analogue of the Linux Podman-machine flakiness PR #1039 addresses — but #1039 is gated
runtime.GOOS == linuxand doesn't touch the Windows WSL path.Changes
Primary fix — gate the e2e suite behind Podman readiness (
.github/workflows/pr-ci.yml):Mirrors the Linux rootful setup for the Windows Podman step:
machine init/startup to 3× (init is not idempotent, so only on the first attempt; later attempts stop + restart the existing machine).podman infofor up to 60s.busyboxcontainer before the suite starts.Secondary fix — wire the pre-start volume binary into the entrypoint (
pkg/devcontainer/single.go,pkg/config/env.go,pkg/agent/delivery/local_docker.go):Latent bug found during the investigation:
LocalDockerDelivery.DeliverPreStartmounts the agent binary at/opt/devsy/devsyand setsDEVSY_AGENT_PATH=/opt/devsy/devsy, butDEVSY_AGENT_PATHhad zero readers anywhere in the codebase — the entrypoint waited for andexec'd/usr/local/bin/devsy(placed later by the post-start legacyinject.sh). So the pre-start volume binary was effectively dead weight, and daemon startup was coupled to the post-start inject (where theEOFhappens).DefaultEntrypointnowexecs the daemon from/usr/local/bin/devsy, so the daemon arms immediately from the pre-start volume binary, decoupling it from the post-start inject.sh. The legacy inject.sh still runs to provision/usr/local/bin/devsyfor the setup execs (docker exec /usr/local/bin/devsy ...), so no call sites needed changing. This shrinks the startup race window that contributes to the flake. Added theEnvAgentPathconstant.Diagnostics (
pkg/driver/docker/lifecycle.go):Surface per-restart-attempt failures at
warninstead ofdebugso the container's exit code + tail logs are visible without--debug(an exit code 0 here typically means PID 1 caught a SIGTERM).Verification
go build ./...✅go veton all changed packages ✅go test ./pkg/devcontainer/... ./pkg/agent/... ./pkg/config/... ./pkg/driver/docker/...✅ (all pass)go test ./pkg/... ./cmd/...: the only failure ispkg/git/TestRepoCloneFromInfoHelper, which is pre-existing and environmental (nogit-lfsin the test sandbox — fails identically on the unmodifiedmaintree). Unrelated to this change.This PR was created by an AI agent (OpenHands) on behalf of the user.
@skevetter can click here to continue refining the PR