Repository navigation
feat(cron): a rotation with a time budget, one process per problem (#435) - #466
Merged
Merged
Conversation
) A cron with a `budget` input tests the problems of its modeling tested longest ago first (tools/cron/select.py, from the dates on `results`), each in its own Julia process, while the budget lasts. A process that crashes or runs past its timeout is an `error` of its problem only. The reports and issues of a rotation are built from the last result of every problem (`--rotation`, the `latest` entry of the state): a problem that did not run keeps its findings, and its regression issues change only when it runs. The ModelingToolkit cron becomes a daily rotation with a budget of 1 h: in one process, its Ubuntu runner died after ~1h20 (most likely out of memory) and its Windows job hit the 6 h limit. Refs #435, #449 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs #435 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
16 of 53 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #435 (to close once a scheduled run is clean). Refs #449 (T5). Applied conventions:
ct-conventions v0.14.0,ct-agents v0.5.0.Why
The ModelingToolkit cron never finished:
The whole ModelingToolkit/Corleone suite runs in one Julia process. Locally, the process of
beamalone reaches 4.4 GB.What
A cron with a
budgetinput (seconds) becomes a rotation:tools/cron/select.py, a newselectjob): the problems of the modeling (a file inext/<M>Models/), ordered from the one tested longest ago to the last tested. The date is that of its last result onresults, and a problem never tested comes first.test/runtests.jl,test/common/selection.jl): withbudget=<s>, each problem runs in its own Julia process, in that order, and no new one starts once the budget is spent. A process that crashes, or runs past its timeout (timeout=, 45 min by default), is reported as anerrorof its problem only (TestStatus.write_process_report), so the problem at fault is named. Each problem writes its own report (<env>-cron-<problem>.md).tools/cron/report.py --rotation): the state keeps the last rows, results and convergence findings of every problem (latest), and the reports and summary issues are built from them.CronModelingToolkit.ymlbecomes daily, with a budget of 1 h. The other crons are unchanged (budget: 0, every problem, as before). They can move to a rotation one at a time.The
problemsinput of a rotation puts those problems first; the run then goes on as usual. Without a budget, it keeps its meaning (a restricted run, reports and issues left as they are).Test plan
core(Julia 1.12): 2827 passed.test_runner.jl: the parsing ofbudget=/timeout=, the budget loop with a fake clock (the first problem always runs, the rest is left for later), andrun_process(:success,:failed,:timeout).tools/cron/test: 42 tests.select.py(order, scope, noresultsbranch). The rotation ofreport.py:Pkg.test(; test_args=["ModelingToolkit", "cron", "problem=beam,chain", "budget=1"]).Problem beam: success,Budget spent, left for a later run: chain, and its own reportModelingToolkit-cron-beam.md.report.py --rotation beam,chain: the table withbeam(this run) andchain(not tested yet).select.pyon the realresultsbranch: the ModelingToolkit problems never tested come first.CronModelingToolkit.ymldispatched on this branch (run 44747569284): the select job ordered the problems and passed the budget; on Windowsbeamsucceeded,bioreactortimed out after 2700 s, the 8 others were left for a later run; the report and [Cron] cron-failure #435 show the rotation line and the timeout row ofbioreactor. The Ubuntu runner died duringbioreactor(MTK solver cost, a separate PR) and left no report, which the report did not show: fixed in 0e3a5ae, a runner ofruns_onwithout a report is aworkflowerror of its platform (--runners, test added).🤖 Generated with Claude Code