Skip to content

ci: run staging integration tests on self-hosted runners #256

Open

Description

@rfay

Short summary (TL;DR)

Run the integration tests on self-hosted GitHub Actions runners on the staging box. GitHub then queues jobs, so the polling ci-slot-* lock, its time limits, and most janitor tuning can go away.

Is there an existing issue for this?

  • I have searched the existing issues in this repository

Is your feature request related to a problem?

The integration workflows run on GitHub-hosted runners and create real workspaces on staging-coder.ddev.com through the Coder API. GitHub has no view of the staging box's capacity, so every job starts at once, and the repository limits load with its own queue on top:

  1. Each job pushes an inactive template version.
  2. The job claims one of CI_LOCK_SLOTS lock workspaces (ci-slot-1..ci-slot-N) with scripts/ci-acquire-staging-lock.sh. While all slots are busy, it polls every 30 seconds.
  3. The job creates its workspace, tests it, deletes it, and releases its slot.
  4. A scheduled janitor (ci-reap.yml) deletes orphaned workspaces and archives old template versions. A waiting job also takes over a slot that looks stale.

Every part of this has its own time limit, and the limits must agree with each other and with how long jobs take. When one limit is wrong, jobs fail for reasons that have nothing to do with the change under test.

Failures on 2026-11-16 and 2026-11-17

Of 163 GitHub-hosted integration jobs that finished between 2026-11-16 18:00 UTC and 2026-11-17 12:50 UTC, 85 passed. Most failures came from the queueing machinery, not from the tests:

Failure Jobs Cause
Gave up waiting for a lock slot, or reached the job timeout while waiting 24 Lock waits were limited to 45 minutes, and the 75-minute job timeout included the wait
Reached the job timeout during setup, after a long lock wait 15 Same
Push template rate-limited 13 Coder allows 12 template uploads per minute per user, and every job uploads at once
Workspace deleted mid-test (agent is shutting down) 8 The janitor deleted running workspaces older than 20 minutes, and contrib setup takes about 20 minutes

Quiet days pass almost completely: 9 to 39 jobs a day, close to 100%. Busy days fail: 90 jobs on 2026-11-05 (63% passed), 152 on 2026-11-16 (49%). One push to main starts about 9 heavy jobs and one PR push starts 6 to 9, so two pushes close together saturate 3 slots.

The box itself was not the limit. During one of those runs, with 3 CI workspaces in their heaviest phase, the 8-core host had load 3.2 / 5.2 / 5.1, cores 30–47% busy, and 5.5 GB of 62.6 GB memory in use. Much of the load came from containerd unpacking images, with processes in uninterruptible disk wait (D state).

History

  • Before June 2026: docs/content/self-hosting/server-setup.md (Step 13) describes five self-hosted runners labeled sysbox on staging, "one runner instance per concurrent matrix job". The workflows still contain those jobs (runs-on: [self-hosted, sysbox]), but each has if: false and the comment "Disabled: self-hosted sysbox runner; use …-gh instead". The repository history starts with one large commit (b6c1e74), so it doesn't record why they were disabled.
  • ddev/coder-ddev#172 (2026-06-29): added the janitor, because cancel-in-progress frees a GitHub runner but leaves the workspace and template version on staging.
  • ddev/coder-ddev#188 (2026-07-27): serialized workspace creation with job-level concurrency groups, then with scripts/ci-wait-for-staging-box.sh, which waited until no ci-bot workspace existed.
  • ddev/coder-ddev#197 (2026-08-08): that wait was a check-then-act race, and three workspaces once ran on a box sized for two. It was replaced with the ci-lock template and ci-slot-N workspaces, an atomic semaphore because Coder workspace names are unique per owner.
  • ddev/coder-ddev#232 (2026-11-05): the per-run reaper deleted another ref's live workspaces, so workspace names now carry a ref tag.
  • ddev/coder-ddev#252 (2026-11-16): retries template pushes after the Coder upload rate limit.
  • ddev/coder-ddev#253 (2026-11-16): raised the janitor and stale-slot limits from 20 and 30 minutes to 80. Before that, a waiting job took over any slot older than 30 minutes, so more jobs than CI_LOCK_SLOTS often ran at once. That overcommit had kept the queue moving.
  • ddev/coder-ddev#254 (2026-11-17): the Drupal templates rejected CI's GITHUB_TOKEN, so Composer always downloaded anonymously.
  • ddev/coder-ddev#255 (open): removes the 45-minute lock wait, raises the job timeout to 360 minutes, and skips the heavy workflows for docs-only changes.

Each of these fixed a real problem, and each added a limit that the next change had to keep in step.

Describe your solution

Register N self-hosted runners for the staging box, and move the integration jobs back to them.

  • GitHub does the queueing. A job waits in GitHub's queue until a runner is free, for up to 24 hours. Nothing polls, and nothing fails while it waits.
  • The runner count limits parallel jobs. Start with 4, which today's load numbers support.
  • The lock layer can go. Once no GitHub-hosted job touches staging, remove ci-lock, ci-acquire-staging-lock.sh, ci-release-staging-lock.sh and the stale-slot takeover. Remove the matching janitor limits as well.
  • Jobs can read the host's load directly. If needed later, a runner can wait while /proc/loadavg is high before it creates a workspace.
  • The tests stay the same. They must still create workspaces through Coder, because the templates are what they test.

Points to decide:

  1. Where the runners run. Directly on the staging host as the github-runner user (Step 13 of server-setup.md), or each in its own container. A container limits what a job can reach on the host.
  2. Fork PRs. The -gh jobs already skip PRs from forks. The self-hosted jobs must keep that rule, so untrusted code never runs on the box.
  3. Template pushes. Each job still uploads a template version. With 4 runners, uploads stay under Coder's 12-per-minute limit, and ci: retry template pushes when Coder rate-limits uploads #252's retry handles bursts.
  4. Runner upkeep. Runner versions need updates, and a reboot must restart the runner services. fix: restart running workspaces after a host reboot so sudo works #247 handled reboots for workspaces, not for runners.
  5. Migration. Turn the sysbox jobs back on beside the -gh jobs, compare them for a week, then remove the -gh jobs and the lock layer.

Describe alternatives

Additional context

  • The staging host has 8 cores and 62.6 GB of memory. A CI workspace uses about 1 CPU and a few GB once it runs. Starts are the expensive part: image builds, image unpacking, composer install and ddev poser, which takes about 19 minutes for drupal-contrib.
  • Current setting: CI_LOCK_SLOTS is 3.

🤖 Generated with Claude Code

https://claude.ai/code/session_01T3Cc2F1jt7n3f7sxKHo6p1


Generated by Claude Code

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions