Repository navigation
perf(test): enforce a per-suite wall-clock budget in CI, warn then fail #10230
Description
Parent: #3716 (slice 3 of 3). Runs in parallel with slice 1.
Runs in: lane.
Problem
#3716 criterion 6: a documented per-suite wall-clock budget that warns above a threshold and fails above a ceiling, so a suite cannot grow past what a contributor can run. Today nothing enforces a per-suite time: scripts/plan-test-lanes.sh uses scripts/suite-seconds.txt only to size and balance CI legs (ceil(seconds / budget)), and fleet-state.test.sh once passed the 600 s tool maximum on its own (#3716 comment). #8757 covers the separate local affected-run budget; this slice is the per-suite CI check.
Fix
- Add a CI check that reads each suite's measured wall seconds from the test legs of the run and compares it with a documented threshold and ceiling: above the threshold prints a warning naming the suite; above the ceiling fails. Seed the threshold and ceiling from the current
scripts/suite-seconds.txtdistribution so no suite on main fails on day one; state the numbers and their basis where the check is documented. - A suite that legitimately needs more gets a named per-suite exception in one listed file, so an exception is visible in review.
- Document the budget in the scripts' docs where contributors read how to run tests.
Done when
- A test for the check shows a suite above the threshold produces a warning and exit 0, and a suite above the ceiling exits non-zero
- The check runs on pull requests and is green on main's current suite times
- The threshold, ceiling and their basis are written in one documented place, and the exception list is one file
-
ci-statusis green on the PR head
Split by the backlog HITL session (2026-10-25, owner rule Q93: split large work before a run picks it up). This was generated by AI.
Activity
github-actions commentedon Oct 26, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
I understood this as a bounded change: a CI check that compares each suite's measured wall seconds with a documented warn threshold and fail ceiling, with a single per-suite exceptions file and docs for contributors. The numbers would be seeded from the current suite-seconds data.
Some of it may already exist. A suite-budget exceptions file is already in scripts/, and a previous pull request (#10904) merged with
Refs:. Before more work starts, someone should check which of the done-when items that change already covers: the test for warn vs fail, the pull-request run, the documented numbers and basis, and a green ci-status.No likely duplicates found among the open issues.
Automated triage, applied by the lanes App.
melodic-automation-lanes commentedon Oct 29, 2026
This records a re-fire of an earlier refusal, not a new refusal. The lanes kill switch stopped run 40797519639, the work-resolve-issue run that
agent-readystarted, before it did any work; that run ended at 2026-10-29T01:32:15Z. work-unblock has now removed and addedagent-readyback to start a fresh run. A kill-switch refusal is no fault in this issue or the lanes, so it never counts against later re-fires; after limits.limit-stops-per-issue of them since a person last addedagent-readyor a run last reported, work-unblock waits longer before each one, up to 24 hours.melodic-automation-lanes commentedon Oct 29, 2026
The lanes App's pull request #10904 merged with
Refs:, so this issue stays open and nothing starts the lane on it again. work-unblock removedstatus: readyandagent-ready, so the hourly triage sweep sends it back to intake triage. Admission then parks it atstatus: confirmedfor a check of what that pull request left, until a**Decision recorded.**comment says what is left or someone closes it.github-actions commentedon Oct 29, 2026
The triage sweep dispatched intake triage for this issue (attempt 1 of 2).
melodic-automation-lanes commentedon Oct 29, 2026
Not admitted to the issue lane: merged pull request #10904 names it with
Refs:after its newest decision or refinement comment, so its work may already be done: close the issue if it is, or post a comment whose first line is**Decision recorded.**saying what is left, which sends it back through intake. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).melodic-automation-lanes commentedon Oct 29, 2026
The latest triage verdict holds nothing: admission moved the issue to
status: confirmed; its comment says why.kyle-sexton commentedon Oct 30, 2026
All four Done-when criteria hold on main (eb1ac00); closing as completed by #10904 (merge 0795dc1).
scripts/run-plugin-tests.test.sh:641-662.scripts/run-plugin-tests.sh:259-262turns the check on wheneverGITHUB_ACTIONS=true;pr-require-checks.ymlis the only workflow that runs the runner, on PRs and in the merge queue (green merge-queue run 41067381006). Every shell suite median inscripts/suite-seconds.txtis under its ceiling: the two over 300 s (align.test.sh340,demo-video.test.sh334) have 480 s rows inscripts/suite-budget-exceptions.txt.scripts/suite-budget-exceptions.txt.ci-statusgreen on the PR head: perf(test): enforce a per-suite wall-clock budget on CI shell suites, warn then fail #10904 merged through the merge queue, which requires it.Outside the Done-when: the check times shell suites only. The slowest Python suite median is 158 s (
test_share.py), under the 300 s ceiling, so no Python suite is over budget today.