Skip to content

feat(cron): a rotation with a time budget, one process per problem (#435) - #466

Merged
ocots merged 3 commits into
mainfrom
feat/cron-rotation
Nov 20, 2026
Merged

ocots merged 3 commits into
mainfrom
feat/cron-rotation

Conversation

@ocots

@ocots ocots commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Refs #435 (to close once a scheduled run is clean). Refs #449 (T5). Applied conventions: ct-conventions v0.14.0, ct-agents v0.5.0.

Why

The ModelingToolkit cron never finished:

  • its Ubuntu runner died after ~1h20, with no failed step and no log ("The hosted runner lost communication with the server … starves it for CPU/Memory");
  • its Windows job hit the 6 h limit.

The whole ModelingToolkit/Corleone suite runs in one Julia process. Locally, the process of beam alone reaches 4.4 GB.

What

A cron with a budget input (seconds) becomes a rotation:

  1. Selection (tools/cron/select.py, a new select job): the problems of the modeling (a file in ext/<M>Models/), ordered from the one tested longest ago to the last tested. The date is that of its last result on results, and a problem never tested comes first.
  2. Tests (test/runtests.jl, test/common/selection.jl): with budget=<s>, each problem runs in its own Julia process, in that order, and no new one starts once the budget is spent. A process that crashes, or runs past its timeout (timeout=, 45 min by default), is reported as an error of its problem only (TestStatus.write_process_report), so the problem at fault is named. Each problem writes its own report (<env>-cron-<problem>.md).
  3. Reports and issues (tools/cron/report.py --rotation): the state keeps the last rows, results and convergence findings of every problem (latest), and the reports and summary issues are built from them.
    • A problem that did not run keeps its findings, and is never "resolved" by a run that did not test it.
    • The regression issues of a problem change only when it runs.
    • The scope report ends with the date each problem was last tested, or "not tested yet".

CronModelingToolkit.yml becomes daily, with a budget of 1 h. The other crons are unchanged (budget: 0, every problem, as before). They can move to a rotation one at a time.

The problems input of a rotation puts those problems first; the run then goes on as usual. Without a budget, it keeps its meaning (a restricted run, reports and issues left as they are).

Test plan

  • core (Julia 1.12): 2827 passed. test_runner.jl: the parsing of budget=/timeout=, the budget loop with a fake clock (the first problem always runs, the rest is left for later), and run_process (:success, :failed, :timeout).
  • tools/cron/test: 42 tests. select.py (order, scope, no results branch). The rotation of report.py:
    • a problem that did not run keeps its failure, then is resolved when it passes;
    • a problem out of the scope is dropped;
    • a crashed process is a failure of its problem;
    • the results of the problems that ran replace their own only.
  • Locally, a real run: Pkg.test(; test_args=["ModelingToolkit", "cron", "problem=beam,chain", "budget=1"]).
    • Output: Problem beam: success, Budget spent, left for a later run: chain, and its own report ModelingToolkit-cron-beam.md.
    • These reports through report.py --rotation beam,chain: the table with beam (this run) and chain (not tested yet).
  • select.py on the real results branch: the ModelingToolkit problems never tested come first.
  • CI.
  • CronModelingToolkit.yml dispatched on this branch (run 44747569284): the select job ordered the problems and passed the budget; on Windows beam succeeded, bioreactor timed out after 2700 s, the 8 others were left for a later run; the report and [Cron] cron-failure #435 show the rotation line and the timeout row of bioreactor. The Ubuntu runner died during bioreactor (MTK solver cost, a separate PR) and left no report, which the report did not show: fixed in 0e3a5ae, a runner of runs_on without a report is a workflow error of its platform (--runners, test added).

🤖 Generated with Claude Code

ocots and others added 2 commits November 20, 2026 14:32
)

A cron with a `budget` input tests the problems of its modeling tested
longest ago first (tools/cron/select.py, from the dates on `results`),
each in its own Julia process, while the budget lasts. A process that
crashes or runs past its timeout is an `error` of its problem only.
The reports and issues of a rotation are built from the last result of
every problem (`--rotation`, the `latest` entry of the state): a problem
that did not run keeps its findings, and its regression issues change
only when it runs.

The ModelingToolkit cron becomes a daily rotation with a budget of 1 h:
in one process, its Ubuntu runner died after ~1h20 (most likely out of
memory) and its Windows job hit the 6 h limit.

Refs #435, #449

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs #435

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ocots ocots added the run ci github-runner Trigger the CI workflow on GitHub-hosted runners label Nov 20, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ocots
ocots merged commit a02d4d9 into main Nov 20, 2026
14 checks passed
@ocots
ocots deleted the feat/cron-rotation branch November 20, 2026 17:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run ci github-runner Trigger the CI workflow on GitHub-hosted runners

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant