Repository navigation
CI performance dashboard #1182
Description
Maintained by the CI profiler routine, which runs every 4h. Each run replaces this body and carries the history rows over.
Headline (run 2026-10-22 12:20 UTC)
Window: 2026-10-21 12:20 → 2026-10-22 12:20. 155 workflow runs (all listed, 2 pages). Since the 08:16 run: 0 merge_group runs, 0 merges, 0 push-to-main runs, 4 full PR CI runs, 3 CI stubs and 7 audit runs. The CI workflow_dispatch on ci-janitor/uv-settings-test-windows that was running at 08:16 ended cancelled at 09:48.
Still no merges (16.4 h quiet); PR runs steady on fast hosts; nothing new to file
- No queue activity since 10-21 19:57. At least 22 non-draft PRs are open (among the 30 most recently updated), 20 of them labelled Ready for review (7 of those
ci-janitor/ci-perf). CI isn't what's holding them: every PR run since 00:16 reachedci-okin under 8.2 min. The 24h merge figures are still the 7 merges of 18:59–19:57 on 10-21. - PR run created →
ci-ok: 7.35 / 7.47 / 7.67 / 8.13 min (n=4: 39756025591, 39759518927, 39768264362, 39774668592). 29 jobs each, 32.0–34.2 Linux job-min per run;coverage5.75–6.16 min. - Depot slow starts: 2/116 jobs since 08:16, both in 39756025591, with an 18 s gap at most. That's 0.7% across the last 16 h (2/290), after 65% at 20:16–00:16. CI perf (settings): slow Depot 16-core hosts since 10-13 15:20 add ~1.5–2.5 min to the merge-queue critical path (~0.6k Linux job-min/day) #1512 stays open (it has come back after a quiet day before). It already has today's comment (00:18), so no new one.
- CI dispatch 39745825228 (full tier, 492 jobs, on the Fix uv settings test failing on Windows nightly #1663 branch):
test-release (2)failed at 08:15, then the run was cancelled at 09:48 after ~99 jobs had run. Cost: 85 L / 47 W / 53 M job-min. That's a one-off manual run, not a recurring cost.
Sample: 2 pages of runs (all 155), jobs (filter=all) for the 4 new full CI runs and the dispatch run, plus the recent PR list. About 12 REST requests. No logs read: no new slow job or failure pattern to look into.
| metric | 24h |
|---|---|
| PRs merged | 7 (18:59–19:57 on 10-21; none since) |
| Merge queue enqueue → merge | p50 20.9 / p90 70.2 min (n=7, carried) |
merge_group created → ci-ok (success) |
p50 9.0 / p90 10.8 (n=8, carried; 13 mg runs in 24h, 8 success / 4 cancelled / 1 failure) |
PR run created → ci-ok |
since 08:16: 7.35 / 7.47 / 7.67 / 8.13 (n=4) |
| Nightly | 10-22 05:47 failure (Windows test shard 1; #1663), 22.7 min wall (carried) |
| Linux queue wait p50 / p90 | Depot 0.01 / 0.01 (n=207 PR + dispatch); start-up gap over 15 s in 2% (2/116) of PR jobs |
| Windows queue wait p50 / p90 | 0.03 / 0.05 (dispatch + nightly) |
| macOS queue wait p50 / p90 | 0.05 / 0.06 (dispatch, n=4) |
| Job-min/day (Linux / Windows / macOS) | ~2.2k / 0.16k / 0.08k (estimated: the 08:16 figure, minus one 10-21 08:16–12:20 full PR run (~33 L) leaving the window, plus 4 PR runs (132 L) and the dispatch run (85 / 47 / 53)) |
Merge-queue settings (ruleset 24668460): not re-read this run. At 10-21 20:16 they were build 5 / merge 5, min 1, min wait 5 min, ALLGREEN, timeout 90, SQUASH.
History
| run (UTC) | merges/24h | MQ enq→merge p50/p90 (min) | mg CI success p50 | PR push→ci-ok p50/p90 | mac queue p50/p90 | linux queue p50/p90 | job-min/day L/W/M |
|---|---|---|---|---|---|---|---|
| 2026-10-22 12:20 | 7 PRs merged in the window (18:59–19:57 on 10-21); 0 merge_group runs since (16.4 h quiet; 22+ non-draft PRs open, 20 Ready for review) | 20.9 / 70.2 (n=7, carried) | 9.0 (n=8, carried) | since 08:16: 7.35 / 7.67 / 8.13 (n=4, fast hosts; coverage 5.75–6.16) | 0.05 / 0.06 (CI dispatch, n=4) | 0.01 / 0.01 (Depot, n=207, max 0.03) | ~2.2k / 0.16k / 0.08k (est.; PR 132 L in 4 full runs since 08:16; CI dispatch on ci-janitor/uv-settings-test-windows 85 L / 47 W / 53 M, cancelled; Depot slow-start 2% (2/116), max gap 18 s) |
| 2026-10-22 08:16 | 7 PRs merged in the window (18:59–19:57 on 10-21); 0 merge_group runs since (12.3 h quiet) | 20.9 / 70.2 (n=7, carried) | 9.0 (n=8, carried) | since 04:16: 7.28 / 7.83 / 8.17 (n=3, fast hosts) | 0.03 / 0.05 (10-22 nightly) | 0.01 / 0.02 (Depot, n=178, max 0.02) | ~2.0k / 0.11k / 0.03k (est.; PR 95 L in 3 full runs since 04:16; nightly 10-22 failed (Windows shard 1, #1663) 87 / 61 / 32; Depot slow-start 0% (0/87)) |
| 2026-10-22 04:16 | 7 PRs merged in the window (18:59–19:57 on 10-21); 0 merge_group runs since (8.3 h quiet) | 20.9 / 70.2 (n=7, carried) | 9.0 (n=8, carried) | since 00:16: 7.73 (n=1, fast host; coverage 5.98) | 0.04 / 0.05 (carried) | 0.02 / 0.02 (Depot, n=29, max 0.02) | ~2.0k / 0.10k / 0.04k (est.; PR 1 full run ~33 L + 10 stubs/audits since 00:16; Depot slow-start 0% (0/29)) |
| 2026-10-22 00:16 | 7 PRs merged in the window (18:59–19:57 on 10-21); 0 merge_group runs since (4.3 h quiet) | 20.9 / 70.2 (n=7, carried) | 9.0 (n=8, carried) | since 20:16: 9.01–11.23 (n=4; 1 Setup PHP failure, re-run passed at 14.13) |
0.04 / 0.05 (carried) | 0.02 / 0.02 (Depot, n=145, max 0.33) | ~2.1k / 0.10k / 0.04k (est.; PR 252 L in 4 runs since 20:16; Depot slow-start 65% (94/145)) |
| 2026-10-21 20:16 | 7 PRs merged 18:59–19:57 (queue reopened at 18:47 after 23.7 h idle; all 7 had been evicted in the 10-20 stall, so ~26 h of extra latency each; 1 lost-Depot-runner eviction (#1593), 1 real test failure (#1632)) | 20.9 / 70.2 today's first enqueue (n=7) | 9.0 (n=8, p90 10.8) | 11.0 / 11.8 (n=10 since 16:16, range 7.92–13.85; slow hosts) | 0.04 / 0.05 (10-21 nightly, carried) | 0.02 / 0.02 (Depot, n=755, max 0.9) | ~2.0k / 0.10k / 0.04k (est.; since 16:16 mg 595 L / 43 W in 13 runs, PR 520 L, push 66 L; Depot slow-start 47% (353/748), 19h 52%) |
| 2026-10-21 16:16 | 10 PRs merged in the window, all 10-20 18:38–19:02; 0 merge_group runs since 19:02 (21.2 h quiet; 23 non-draft PRs open) | 172 / 191 first enqueue (n=10, carried) | 11.4 (n=16, carried) | 10.9 / 15.8 24h (carried); since 12:22: 8.98–10.52 (n=4, exact; coverage 7.6–8.9 on slow hosts) | 0.04 / 0.05 (10-21 nightly, carried) | 0.02 / 0.02 (Depot; since 12:22 max 2 s, n=145) | ~7.3k / 0.49k / 0.12k (mg carried; PR +195 L in 5 runs since 12:22; Depot slow-start 25% (36/145), one run 62% (18/29)) |
| 2026-10-21 12:22 | 10 PRs merged in the window, all 10-20 18:38–19:02; 0 merge_group runs since 19:02 (17.3 h quiet; 22 non-draft PRs open) | 172 / 191 first enqueue (n=10, carried) | 11.4 (n=16, carried) | 10.9 / 15.8 24h (carried); since 08:16: 7.88 (n=1, exact) | 0.04 / 0.05 (10-21 nightly, carried) | 0.02 / 0.02 (Depot; since 08:16 max 1 s, n=29) | ~7.3k / 0.49k / 0.12k (mg 4,876 L / 337 W carried; PR +33 L in 1 full run since 08:16 plus 8 stubs/audits; Depot slow-start 0/29 in the new run) |
| 2026-10-21 08:16 | 10 PRs merged in the window, all 10-20 18:38–19:02; 0 merge_group runs since 19:02 (13.2 h quiet; 21 non-draft PRs open) | 172 / 191 first enqueue (n=10, carried) | 11.4 (n=16, carried) | 10.9 / 15.8 24h (carried, n=14); since 04:16: 7.28, 8.43 (n=2, exact) | 0.04 / 0.05 (10-21 nightly, green, 23.3 min) | 0.02 / 0.02 (Depot; since 04:16 max 1 s) | ~7.3k / 0.49k / 0.12k (mg 4,876 L / 337 W carried; PR +66 L in 2 full runs since 04:16 (32, 34); 10-21 nightly 85 L / 54 W / 39 M; Depot slow-start 0 in the new sample) |
| 2026-10-21 04:16 | 10 PRs merged in the window, all 10-20 18:38–19:02; 0 merge_group runs since 19:02 (9.2 h quiet) | 172 / 191 first enqueue (n=10, carried: no new merges) | 11.4 (n=16, carried) | 10.9 / 15.8 24h (carried, n=14); since 00:16: 7.20–8.38 (n=4 full runs, exact) | 0.08 / 0.13 (nightly + 1 CI dispatch only, carried) | 0.02 / 0.02 (Depot; since 00:16 max 1 s, n=116) | ~7.3k / 0.49k / 0.12k (mg 4,876 L / 337 W carried, 84% wasted in the 10-20 stall; PR +130 L in 4 full runs since 00:16 (31.6–33.6 each) plus 7 stubs; Depot slow-start 0% (0/116) since 00:16) |
| 2026-10-21 00:16 | 10 PRs merged in the window, all 18:38–19:02; 0 merge_group runs since 19:02 (5.2 h quiet; 15 non-draft PRs open) | 172 / 191 first enqueue (n=10, carried: no new merges) | 11.4 (n=16, range 7.4–12.6; slow hosts) | 10.9 / 15.8 (n=14 full runs; since 20:16: 7.4–9.98, one 15.75 with a 148 s Depot start gap) | 0.08 / 0.13 (nightly + 1 CI dispatch only) | 0.02 / 0.02 (Depot, n=4,387, max 0.8) | ~7.2k / 0.49k / 0.12k (mg 4,876 L / 337 W in 120 runs, 4,087 L (84%) in 104 cancelled/failed runs; PR ~1.77k L (since 20:16 exact, earlier ×3); push 42 L; nightly 264 / 80 / 58; CI dispatch on #1647 280 / 71 / 64; Depot slow-start 57% (2,231/3,926) in 24h, 23% (79/348) since 20:16) |
| 2026-10-20 20:16 | 10 PRs merged (18:38–19:02; queue stalled 15:54→18:38, 0 merges: 17 heads on red base 14eee88 evicted by the uv 0.13.2 pipenv leg; fixed by the uv pin riding in #1605) |
172 / 191 first enqueue (n=10); 12–110 last re-enqueue | 11.1 (n=18, range 7.4–12.6; slow hosts) | 11.3 / 14.8 (n=9 full runs, sampled 1 in 3) | 0.15 / 0.18 (nightly only) | 0.02 / 0.02 (Depot, n=3,759, max 0.8) | ~6.7k / 0.42k / 0.06k (mg 4,820 L / 334 W in 129 runs, 4,320 (84%) in 111 cancelled/failed runs; PR ~1.55k L scaled ×3; push 42 L; nightly carried; Depot slow-start 62% (2,119/3,445)) |
| 2026-10-20 16:16 | 0 PRs merged (last merges #1429/#1592 on 10-19 13:3x; 13 merge_group CI runs failed or were cancelled from 15:54, 2 heads evicted (#1591, #1593): main is red from uv 0.13.2, fix #1639 not yet queued) | — (n=0) | — (0 mg successes; failing runs reached ci-ok at 5.7–13.4) |
9.6 (n=1 full run, slow hosts) | 0.22 max (nightly only) | 0.02 / 0.03 (Depot) | ~0.70k / 0.15k / 0.06k (mg 549 L / 37 W in 22 min, all wasted; PR 65 L; nightly 10-20 failed (pipenv leg), 306 / 91 / 57 carried; Depot slow-start 57% (235/409), the 16:01 mg batch 92%, median 50 s) |
| 2026-10-18 00:15 | 1 PR merged (#1590 at 14:47, none since; 0 mg failures, 0 evictions) | 7.93 (n=1, #1590) | 7.48 (n=1) | 7.43 / 7.83 (n=14 full runs, range 7.27–7.88) | 0.13 / 0.13 (nightly only, carried) | 0.02 / 0.02 (Depot, max 0.07) | ~0.84k / 0.09k / 0.06k (PR ~0.47k L in 39 runs (14 full at ~31.4 L, 1 cancelled, 24 stubs); mg 32.2 L / 1.8 W + watcher 8.0 L; push 7.6 L; nightly 306 / 91 / 57 carried; Depot slow-start 10-17 day ~2% (8/~420), 21h 0/29) |
| 2026-10-17 20:16 | 1 PR merged (#1590 at 14:47, none since; 0 mg failures, 0 evictions; 17 PRs open and not draft) | 7.93 (n=1, #1590) | 7.47 (n=1) | 7.47 / 7.88 (n=15 full runs, range 7.27–8.32) | 0.13 / 0.13 (nightly only, carried) | 0.02 / 0.02 (Depot, max 0.07) | ~0.85k / 0.09k / 0.06k (PR ~0.48k L in 41 runs (15 full at ~31.5 L, 25 stubs); mg 32.2 L / 1.8 W + watcher 8.0 L; push 7.6 L; nightly 306 / 91 / 57 carried; Depot slow-start 17h 28% (8/29, 4-core, +0.4 min), 19h 0% (0/58)) |
| 2026-10-17 16:16 | 1 PR merged (#1590 at 14:47; 1 mg CI success, 0 failures, 0 evictions; first merge after 42.8 h quiet; 15 PRs open and not draft) | 7.93 (n=1, #1590) | 7.47 (n=1) | 7.48 / 7.88 (n=12 full runs, range 7.27–8.32) | 0.13 / 0.13 (nightly only, carried) | 0.02 / 0.02 (Depot, max 0.07) | ~0.75k / 0.09k / 0.06k (PR ~0.38k L in 37 runs (12 full at ~31 L, 25 stubs); mg 32.2 L / 1.8 W + fail-fast watcher 8.0 L; push 7.6 L; nightly 306 / 91 / 57 carried; Depot slow-start 0/69 in the 5 new runs (13h–14h)) |
| 2026-10-17 12:16 | 0 PRs merged (0 mg CI runs, 0 push runs; none since 10-15 19:59, 40.3 h quiet; 14 PRs Ready for review) | 9.22 / 11.52 (carried, n=5; no merges in window) | — (no mg runs) | 7.53 / 8.31 (n=13 full runs, range 7.25–8.55) | 0.13 / 0.13 (nightly only) | 0.02 / 0.02 (Depot, n≈450) | ~0.75k / 0.09k / 0.06k (PR ~0.45k L in 40 runs; nightly 306 / 91 / 57 carried; Depot slow-start 15h 71% (20/28), 10h 0% (0/29); 1 new full run since 08:16, 7.25 min) |
| 2026-10-17 08:16 | 0 PRs merged (0 mg CI runs, 0 push runs; none since 10-15 19:59, 36.3 h quiet; 13 PRs Ready for review) | 9.22 / 11.52 (carried, n=5; no merges in window) | — (no mg runs) | 7.53 / 8.30 (n=12 full runs, range 7.32–8.48) | 0.13 / 0.13 (nightly only) | 0.02 / 0.02 (Depot, n=495) | ~0.76k / 0.09k / 0.06k (PR 457 L in 37 runs; nightly 306 / 91 / 57 carried; Depot slow-start 15h 71% (20/28), 21h 7% (4/58), 05h nightly 0% (0/92); 10-17 nightly green, 24.15 min) |
| 2026-10-17 04:16 | 0 PRs merged (0 mg CI runs, 0 push runs; none since 10-15 19:59, 32.3 h quiet; 12 PRs Ready for review) | 9.22 / 11.52 (carried, n=5; no merges in window) | — (no mg runs) | 7.80 / 8.48 (n=11 full runs, range 7.32–9.18) | 0.14 / 0.16 (nightly only) | 0.02 / 0.02 (Depot, n=433) | ~0.69k / 0.09k / 0.06k (PR 383 L in 34 runs; nightly 306 / 91 / 57 carried; Depot slow-start 05h 67% (nightly), 15h 69% (20/29), 01h–02h 0% (0/87)) |
Open ci-perf issues by ROI
ROI on one scale: (thousands of job-min/day weighted Linux ×1, Windows ×2, macOS ×3, plus 1 per critical-path or PR-latency minute) × confidence ÷ effort (S=1, M=2, L=3).
| ROI | issue | saving | status |
|---|---|---|---|
| ~9.3 | #1645 floating toolchain versions red-main the queue | ~0.45k job-min/day and ~15 MQ-min/day amortized. The 10-20 incident alone cost 4.3k job-min, 2.7 h of 0 merges, and ~26 h of extra latency for the 7 PRs it evicted | Fix #1647 is open and waiting on review. It was skipped in the 10-21 7-PR batch and is still unmerged at 12:20 on 10-22 (37.5 h idle). Fresh numbers posted. |
| ~2.5 | #1512 (settings) slow Depot hosts | ~2–3 min on each merge that lands on a slow host | Intermittent: slow-start 2% (2/116) since 08:16 (4 runs, max gap 18 s), 0/87 at 04:16–08:16 and 0/29 at 00:16–04:16, after 65% (94/145) at 20:16–00:16 and 47% at 16:16–20:16. PR push→ci-ok 9.0–11.2 against 7.3–8.4 on fast hosts, and mg ci-ok p50 9.0 against 7.5. A lost runner evicted #1593 (+22 min) on 10-21. Fresh numbers posted at 00:16. Needs a human to raise it with Depot. |
| ~1.0 | #1248 (settings) merge-queue build concurrency | ~6 min per entry past the 5th in a burst | Today's burst of 7 (enqueued 18:47:43–18:48:03): the 6th entry (#1593) waited 8.8 min for a build slot. Needs a human decision. |
Open ci-perf / ci-janitor PRs: #1647 pins the toolchains (#1645) and is waiting on review (no activity since 10-20 22:51, 37.5 h). #1666 (new) skips cargo shapes on a failed --precise pin; its PR run reached ci-ok in 7.67 min. #1663 fixes the uv settings test that turned the 10-22 nightly red on Windows; its full-tier CI dispatch (39745825228) failed in test-release (2) and was cancelled, so the fix has no full-tier green yet. #1659 uses the runner image's PHP in the composer e2e legs (would remove Setup PHP download failures like the one that cost a re-run on 10-21). #1654 retries crates.io fetches in coverage's cargo e2e. #1652 keeps the transport cause in registry errors. #1643 fixes the update-lock flake. #1639 is redundant with main's uv pin and can be closed. No ci-perf or ci-janitor PR has merged since 10-20 19:02.
Merged ci-perf / ci-janitor PRs (last 7 days) and measured effect
| PR | merged | before → after |
|---|---|---|
#1620 ci-watch.sh works without GraphQL (ci-janitor) |
10-20 19:02 | Tooling; no CI-time effect expected. |
| #1634 keep reuse self-test notices out of CI logs (ci-janitor) | 10-20 19:02 | Log hygiene; no CI-time effect expected. |
| #1590 move action pins off deprecated Node 20 (#1588, ci-janitor) | 10-17 14:47 | Maintenance. No measurable change, as expected. |
| #1585 core covgap tests join their suites (#1089) | 10-15 18:58 | No measurable change; it claimed no saving. |
| #1568 fold last top-level covgap test binaries (#1089) | 10-15 14:21 | coverage 5.82–6.03 → 5.87–6.25 on fast hosts. No measurable change; it claimed no saving. |
| #1563, #1565 vex / cli suite tests assert codes (#1089) | 10-15 | Test refactors. No CI-time effect expected. |
| #1560, #1556 scan mod / hosted tests assert codes (#1089) | 10-15 | Test refactors. No CI-time effect expected. |
| #1546, #1544 apply / vendor suites assert codes (#1089) | 10-14 | Test refactors. No CI-time effect expected. |
| #1525 fix nightly old-uv orphan-unwind failure (ci-janitor) | 10-14 09:01 | 10-15 nightly green. Delivered. |
Method notes
- GraphQL is blocked in this environment, so merge-queue state comes from REST timelines of the merged PRs. "First enqueue" means the PR's first
added_to_merge_queueevent in the window. - Jobs are fetched with
filter=all. Queue wait = jobstarted_at−created_at; skipped jobs are excluded. "Start-up gap" = jobstarted_at→ first stepstarted_at; over 15 s counts as a slow host. - Critical path = merge_group run
created_at→ci-okcompleted_at. - This run (10-22 12:20) the 24h window held 155 runs, all listed (2 pages). The 08:16 run: 149. The 04:16 run: 149. The 00:16 run: 158 runs. Earlier runs: 10-21 16:16 held 574 runs and listed the newest 300. The 10-21 12:22 run: the 24h window held 592 runs.
CIruns were listed through the ci.yml workflow endpoint (2 pages, 172 runs). The other workflows' runs were seen only from 10-20 17:11 onwards (3 pages of the all-runs list). PR job-minutes since 20:16 are exact; earlier ones are scaled ×3 from a 1-in-3 sample.
Generated by Claude Code
Activity
mikolalysenko commentedon Oct 8, 2026
[agent] There are now two open dashboard issues: this one and #1175, both titled "CI performance dashboard" and labeled
ci-perf-dashboard. The body says it is maintained in place, so the profiler routine should update one issue rather than open a new one each run. I'm not closing either one, because I can't tell which issue the routine will update next. Whoever owns the profiler should close the copy it no longer updates.Generated by Claude Code