Repository navigation
Description
Short summary (TL;DR)
Run the integration tests on self-hosted GitHub Actions runners on the staging box. GitHub then queues jobs, so the polling ci-slot-* lock, its time limits, and most janitor tuning can go away.
Is there an existing issue for this?
- I have searched the existing issues in this repository
Is your feature request related to a problem?
The integration workflows run on GitHub-hosted runners and create real workspaces on staging-coder.ddev.com through the Coder API. GitHub has no view of the staging box's capacity, so every job starts at once, and the repository limits load with its own queue on top:
- Each job pushes an inactive template version.
- The job claims one of
CI_LOCK_SLOTSlock workspaces (ci-slot-1..ci-slot-N) withscripts/ci-acquire-staging-lock.sh. While all slots are busy, it polls every 30 seconds. - The job creates its workspace, tests it, deletes it, and releases its slot.
- A scheduled janitor (
ci-reap.yml) deletes orphaned workspaces and archives old template versions. A waiting job also takes over a slot that looks stale.
Every part of this has its own time limit, and the limits must agree with each other and with how long jobs take. When one limit is wrong, jobs fail for reasons that have nothing to do with the change under test.
Failures on 2026-11-16 and 2026-11-17
Of 163 GitHub-hosted integration jobs that finished between 2026-11-16 18:00 UTC and 2026-11-17 12:50 UTC, 85 passed. Most failures came from the queueing machinery, not from the tests:
| Failure | Jobs | Cause |
|---|---|---|
| Gave up waiting for a lock slot, or reached the job timeout while waiting | 24 | Lock waits were limited to 45 minutes, and the 75-minute job timeout included the wait |
| Reached the job timeout during setup, after a long lock wait | 15 | Same |
Push template rate-limited |
13 | Coder allows 12 template uploads per minute per user, and every job uploads at once |
Workspace deleted mid-test (agent is shutting down) |
8 | The janitor deleted running workspaces older than 20 minutes, and contrib setup takes about 20 minutes |
Quiet days pass almost completely: 9 to 39 jobs a day, close to 100%. Busy days fail: 90 jobs on 2026-11-05 (63% passed), 152 on 2026-11-16 (49%). One push to main starts about 9 heavy jobs and one PR push starts 6 to 9, so two pushes close together saturate 3 slots.
The box itself was not the limit. During one of those runs, with 3 CI workspaces in their heaviest phase, the 8-core host had load 3.2 / 5.2 / 5.1, cores 30–47% busy, and 5.5 GB of 62.6 GB memory in use. Much of the load came from containerd unpacking images, with processes in uninterruptible disk wait (D state).
History
- Before June 2026:
docs/content/self-hosting/server-setup.md(Step 13) describes five self-hosted runners labeledsysboxon staging, "one runner instance per concurrent matrix job". The workflows still contain those jobs (runs-on: [self-hosted, sysbox]), but each hasif: falseand the comment "Disabled: self-hosted sysbox runner; use …-gh instead". The repository history starts with one large commit (b6c1e74), so it doesn't record why they were disabled. - ddev/coder-ddev#172 (2026-06-29): added the janitor, because
cancel-in-progressfrees a GitHub runner but leaves the workspace and template version on staging. - ddev/coder-ddev#188 (2026-07-27): serialized workspace creation with job-level
concurrencygroups, then withscripts/ci-wait-for-staging-box.sh, which waited until noci-botworkspace existed. - ddev/coder-ddev#197 (2026-08-08): that wait was a check-then-act race, and three workspaces once ran on a box sized for two. It was replaced with the
ci-locktemplate andci-slot-Nworkspaces, an atomic semaphore because Coder workspace names are unique per owner. - ddev/coder-ddev#232 (2026-11-05): the per-run reaper deleted another ref's live workspaces, so workspace names now carry a ref tag.
- ddev/coder-ddev#252 (2026-11-16): retries template pushes after the Coder upload rate limit.
- ddev/coder-ddev#253 (2026-11-16): raised the janitor and stale-slot limits from 20 and 30 minutes to 80. Before that, a waiting job took over any slot older than 30 minutes, so more jobs than
CI_LOCK_SLOTSoften ran at once. That overcommit had kept the queue moving. - ddev/coder-ddev#254 (2026-11-17): the Drupal templates rejected CI's
GITHUB_TOKEN, so Composer always downloaded anonymously. - ddev/coder-ddev#255 (open): removes the 45-minute lock wait, raises the job timeout to 360 minutes, and skips the heavy workflows for docs-only changes.
Each of these fixed a real problem, and each added a limit that the next change had to keep in step.
Describe your solution
Register N self-hosted runners for the staging box, and move the integration jobs back to them.
- GitHub does the queueing. A job waits in GitHub's queue until a runner is free, for up to 24 hours. Nothing polls, and nothing fails while it waits.
- The runner count limits parallel jobs. Start with 4, which today's load numbers support.
- The lock layer can go. Once no GitHub-hosted job touches staging, remove
ci-lock,ci-acquire-staging-lock.sh,ci-release-staging-lock.shand the stale-slot takeover. Remove the matching janitor limits as well. - Jobs can read the host's load directly. If needed later, a runner can wait while
/proc/loadavgis high before it creates a workspace. - The tests stay the same. They must still create workspaces through Coder, because the templates are what they test.
Points to decide:
- Where the runners run. Directly on the staging host as the
github-runneruser (Step 13 ofserver-setup.md), or each in its own container. A container limits what a job can reach on the host. - Fork PRs. The
-ghjobs already skip PRs from forks. The self-hosted jobs must keep that rule, so untrusted code never runs on the box. - Template pushes. Each job still uploads a template version. With 4 runners, uploads stay under Coder's 12-per-minute limit, and ci: retry template pushes when Coder rate-limits uploads #252's retry handles bursts.
- Runner upkeep. Runner versions need updates, and a reboot must restart the runner services. fix: restart running workspaces after a host reboot so sudo works #247 handled reboots for workspaces, not for runners.
- Migration. Turn the
sysboxjobs back on beside the-ghjobs, compare them for a week, then remove the-ghjobs and the lock layer.
Describe alternatives
- Keep the lock and tune its limits (ci: stop the staging janitor from deleting workspaces of running jobs #253, ci: let staging jobs wait for a slot, skip them for docs changes #255). This works, but it keeps several limits that must agree, and a GitHub-hosted job still occupies a runner while it polls.
- Admit jobs by host load instead of a fixed slot count. Better use of the box, but a GitHub-hosted job can't read the host's load. It would need a new endpoint on staging. Load also lags admission by several minutes, so it still needs a hard cap and a gap between starts.
- A single queue workflow that starts jobs one after another. GitHub's
concurrencykeeps only one pending run per group and cancels older pending runs, so this needs custom dispatch logic. - Run fewer jobs. ci: let staging jobs wait for a slot, skip them for docs changes #255 skips docs-only changes, and feat(ci): reap stale CI workspaces and template versions on staging #172 already runs a smaller matrix on PRs. This reduces load but doesn't remove the bursts.
Additional context
- The staging host has 8 cores and 62.6 GB of memory. A CI workspace uses about 1 CPU and a few GB once it runs. Starts are the expensive part: image builds, image unpacking,
composer installandddev poser, which takes about 19 minutes for drupal-contrib. - Current setting:
CI_LOCK_SLOTSis 3.
🤖 Generated with Claude Code
https://claude.ai/code/session_01T3Cc2F1jt7n3f7sxKHo6p1
Generated by Claude Code
Activity