Skip to content

fix(ci): stabilize flaky up-docker-wsl on windows-latest - #1051

Draft
skevetter wants to merge 1 commit into
mainfrom
fix/up-docker-wsl-flaky
Draft

fix(ci): stabilize flaky up-docker-wsl on windows-latest#1051
skevetter wants to merge 1 commit into
mainfrom
fix/up-docker-wsl-flaky

Conversation

@skevetter

Copy link
Copy Markdown
Contributor

Summary

The up-docker-wsl job (windows-latest) was intermittently failing with the container exiting (exit code 0) right after start, followed by [inject] EOF and 3 failed restart attempts. The rerun of the same job with identical code passed, confirming an environmental flake.

Repro log: run 31775426829 / job 94693528470 (failed) → rerun job 94699202314 (succeeded, same code).

Root cause

podman machine start returns before the WSL VM and its API socket are fully ready. Containers created in that window get SIGTERM'd mid-startup. The daemon (PID 1, after the entrypoint's exec) catches SIGTERM via signal.NotifyContext(..., os.Interrupt, syscall.SIGTERM) and exits 0, which cascades into the inject EOF and the 3-restart loop (each restart re-execs the daemon, which gets SIGTERM'd again).

This is the Windows analogue of the Linux Podman-machine flakiness PR #1039 addresses — but #1039 is gated runtime.GOOS == linux and doesn't touch the Windows WSL path.

Note: the quadrupled backslashes in the SSH-tunnel DEBUG logs ("\"D:\\\\a\\\\...") are just nested-%q log noise (agent.go builds the command with %q, sshtunnel.go logs it again with %q) — cosmetic, not a cause.

Changes

Primary fix — gate the e2e suite behind Podman readiness (.github/workflows/pr-ci.yml):
Mirrors the Linux rootful setup for the Windows Podman step:

  • Retry machine init/start up to 3× (init is not idempotent, so only on the first attempt; later attempts stop + restart the existing machine).
  • Readiness probe: poll podman info for up to 60s.
  • Preflight: run a throwaway busybox container before the suite starts.

Secondary fix — wire the pre-start volume binary into the entrypoint (pkg/devcontainer/single.go, pkg/config/env.go, pkg/agent/delivery/local_docker.go):

Latent bug found during the investigation: LocalDockerDelivery.DeliverPreStart mounts the agent binary at /opt/devsy/devsy and sets DEVSY_AGENT_PATH=/opt/devsy/devsy, but DEVSY_AGENT_PATH had zero readers anywhere in the codebase — the entrypoint waited for and exec'd /usr/local/bin/devsy (placed later by the post-start legacy inject.sh). So the pre-start volume binary was effectively dead weight, and daemon startup was coupled to the post-start inject (where the EOF happens).

DefaultEntrypoint now execs the daemon from /usr/local/bin/devsy, so the daemon arms immediately from the pre-start volume binary, decoupling it from the post-start inject.sh. The legacy inject.sh still runs to provision /usr/local/bin/devsy for the setup execs (docker exec /usr/local/bin/devsy ...), so no call sites needed changing. This shrinks the startup race window that contributes to the flake. Added the EnvAgentPath constant.

Diagnostics (pkg/driver/docker/lifecycle.go):
Surface per-restart-attempt failures at warn instead of debug so the container's exit code + tail logs are visible without --debug (an exit code 0 here typically means PID 1 caught a SIGTERM).

Verification

  • go build ./...
  • go vet on all changed packages ✅
  • go test ./pkg/devcontainer/... ./pkg/agent/... ./pkg/config/... ./pkg/driver/docker/... ✅ (all pass)
  • Full go test ./pkg/... ./cmd/...: the only failure is pkg/git/TestRepoCloneFromInfoHelper, which is pre-existing and environmental (no git-lfs in the test sandbox — fails identically on the unmodified main tree). Unrelated to this change.
  • Workflow YAML validated.

This PR was created by an AI agent (OpenHands) on behalf of the user.

@skevetter can click here to continue refining the PR

The up-docker-wsl job was intermittently failing with the container
exiting (exit code 0) right after start, followed by "[inject] EOF" and
3 failed restart attempts. The rerun of the same job with identical code
passed, confirming an environmental flake on windows-latest + Podman/WSL.

Root cause: podman `machine start` returns before the WSL VM and its API
socket are fully ready. Containers created in that window get SIGTERM'd
mid-startup; the daemon (PID 1) catches SIGTERM via signal.NotifyContext
and exits 0, which cascades into the inject EOF and the restart loop.
This is the Windows analogue of the Linux Podman-machine flakiness PR
#1039 addresses (which is gated runtime.GOOS == linux).

Changes:
* .github/workflows/pr-ci.yml: mirror the Linux rootful setup for the
  Windows Podman step - retry machine init/start (init is not idempotent,
  so only on the first attempt), gate the e2e suite behind a `podman info`
  readiness probe, and run a throwaway preflight container before the
  suite starts.
* pkg/devcontainer/single.go: DefaultEntrypoint now execs the daemon from
  $DEVSY_AGENT_PATH (falling back to /usr/local/bin/devsy). This wires in
  LocalDockerDelivery's pre-start volume binary, so the daemon arms
  immediately from the volume instead of waiting for post-start shell
  injection to place /usr/local/bin/devsy - shrinking the startup race
  window. DEVSY_AGENT_PATH was previously set but had zero readers.
* pkg/config/env.go: add the EnvAgentPath constant for the above.
* pkg/agent/delivery/local_docker.go: use the new constant.
* pkg/driver/docker/lifecycle.go: surface restart-attempt failures at
  warn instead of debug so the exit code + tail logs are visible without
  --debug (an exit code 0 here typically means PID 1 caught a SIGTERM).

Co-authored-by: openhands <openhands@all-hands.dev>
@skevetter skevetter added the ci label Aug 14, 2026 — with OpenHands AI
@netlify

netlify Bot commented Aug 14, 2026

Copy link
Copy Markdown

Deploy Preview for images-devsy-sh canceled.

Name Link
🔨 Latest commit d6c435a
🔍 Latest deploy log https://app.netlify.com/projects/images-devsy-sh/deploys/6a7f1b3b94006e0008a471db

@netlify

netlify Bot commented Aug 14, 2026

Copy link
Copy Markdown

Deploy Preview for devsydev canceled.

Name Link
🔨 Latest commit d6c435a
🔍 Latest deploy log https://app.netlify.com/projects/devsydev/deploys/6a7f1b3b95216600089ec92e

@coderabbitai

coderabbitai Bot commented Aug 14, 2026

Copy link
Copy Markdown

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 0f3b744e-fa96-4f48-b7ed-f3a8d10f1376

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@devsy-app

devsy-app Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution. Before we can merge it, please read our Contributor License Agreement. If you agree, post a comment on this pull request with exactly the following text:


I have read the CLA Document and I hereby sign the CLA


You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot.

@github-actions

Copy link
Copy Markdown

⚠️ This PR contains unsigned commits. To get your PR merged, please sign those commits (git rebase --exec 'git commit -S --amend --no-edit -n' @{upstream}) and force push them to this branch (git push --force-with-lease).

If you're new to commit signing, there are different ways to set it up:

Sign commits with gpg

Follow the steps below to set up commit signing with gpg:

  1. Generate a GPG key
  2. Add the GPG key to your GitHub account
  3. Configure git to use your GPG key for commit signing
Sign commits with ssh-agent

Follow the steps below to set up commit signing with ssh-agent:

  1. Generate an SSH key and add it to ssh-agent
  2. Add the SSH key to your GitHub account
  3. Configure git to use your SSH key for commit signing
Sign commits with 1Password

You can also sign commits using 1Password, which lets you sign commits with biometrics without the signing key leaving the local 1Password process.

Learn how to use 1Password to sign your commits.

Watch the demo

@codacy-production

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 0 complexity · 0 duplication

Metric Results
Complexity 0
Duplication 0

View in Codacy

AI Reviewer: run a review on demand. To trigger the first review automatically, go to your organization or repository integration settings. AI can make mistakes. Always validate suggestions.

Run reviewer

TIP This summary will be updated as you push new changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants