Repository navigation
Description
Agents: read the guide for agents in #310 first (order, difficulty, how to claim an issue, working method). The specification is in this description and in the comments below.
Goal
Run the slow tests (cron mode of #304) regularly, one cron per scope, staggered over the week; report automatically through issues and report files, so that an upstream change (OptimalControl, CTDirect, CTParser, CTSolvers, ExaModels…) breaking a problem is seen within a week, and drafts ready to become :valid are noticed.
Part of #310, step 3. Depends on #304 (environments, cron mode, draft report) and #312 (CI infrastructure).
Reference
CTFlows.jl .github/workflows/CronIntegration.yml: weekly, CTActions ci.yml with test_args: 'cron' on ubuntu, macos, windows, plus ci-gpu.yml (kkt, else occidata); a notify-on-failure job opens one issue labelled cron-failure unless one is open. Runs take 30–90 minutes; it opened #391 and #426, since closed.
Several crons, one per scope
Each scope runs core plus one environment, in cron mode (test_args: '<Env> cron'), so that each run stays well under the 6-hour job limit and a failure points at one modeling.
| Workflow | Environment | Runners | Proposed schedule (UTC) |
|---|---|---|---|
CronJuMP.yml |
JuMP |
ubuntu, macos, windows | Tuesday 04:00 |
CronOptimalControl.yml |
OptimalControl |
ubuntu, macos, windows + GPU | Wednesday 04:00 |
CronOptimalControlScalarized.yml |
OptimalControlScalarized |
ubuntu, macos, windows + GPU | Thursday 04:00 |
CronComparison.yml |
comparison |
ubuntu (+ GPU if comparisons on GPU exist) | Friday 04:00 |
CronModelingToolkit.yml |
ModelingToolkit (solved with Corleone) |
ubuntu | Saturday 04:00 (once #308 exists) |
- Stagger to avoid loading everything, and the GPU runners in particular, at the same time. Local survey of the organization's schedules (2026-11-04): busy slots are daily 00:00, 01:00 and 03:15, and Monday 00:00, 02:30, 03:00 (CTFlows cron) and 06:00. Re-check before choosing.
- Write the common logic once: a local reusable workflow (e.g.
.github/workflows/cron-reusable.yml, input: environment name, GPU or not) called by the thin scheduled files above; each caller also hasworkflow_dispatch. - The crons ignore the scope labels of [Tests] Test layout per environment and consistency tests #304: everything of the scope runs.
- Add the new workflows to CTActions
docs/workflows/per-repo.md(package-specific workflows).
What makes a cron fail
- A
:validproblem failing its tests in that environment. - A configuration (modeler × scheme × solver × CPU/GPU, from
test/common/configs.jl) failing for a:validproblem, unless declared in the metadata as a known failure, e.g.:known_failures => [(modeler=:exa, parameter=:gpu, reason="…")](field added by [Metadata] Computed features, reference objective, shared initial guess #306). A declared failure that no longer happens is reported, not failed. Same principle as the statuses: a claim, checked by the tests; the capability matrix is never written by hand. :draftproblems never fail the cron.
Reports
- The tests write report files ([Tests] Test layout per environment and consistency tests #304): per problem × modeling × configuration → pass / fail / error, time, objective; drafts that pass.
- CTActions change:
ci.yml(andci-gpu.yml) keep no file produced by the tests. Add an optional input (e.g.upload_artifact_path) that uploads a file or directory as a workflow artifact. Open the issue/PR in CTActions; useful to other packages. - A final job of each cron collects the artifacts and:
-
updates three issues, each unique (update the open one instead of opening another), with the run link:
Label When Content cron-failuresomething above failed environments, problems, configurations failing cron-draftsa draft passed all its tests problems and modelings ready to become :validcron-capabilitiesa declared known failure no longer happens the entries of :known_failuresthat can be removedOne issue per label for the whole repository; the body has one section per scope, replaced by each run of that scope.
-
stores the reports in the repository for the documentation: commit them to the
cron/directory of the sharedresultsbranch ([Doc] Results pipeline for the documentation, and a gallery of problems #314: orphan branch rewritten with one commit, single writer; one file per scope, plus the date and the run link), withcontents: write. The documentation reads them to build a status page ([Doc] Rewrite the documentation: getting started, user guide, problems, developers #307).
-
- Labels
cron-failure,cron-drafts,cron-capabilities: create them here and add them to the label plan of [Chore] Harmonize repository metadata and ct-conventions integration with CTBase #313.
Bug issues per problem (idempotent)
Besides the summary issues above, the final job opens bug issues for failing problems, idempotently:
- One issue per (problem, modeling), not per configuration; failing configurations are listed in its body. Labels
bugandcron-bug. - Identified by a hidden key in the body, e.g.
<!-- cron-key: beam|JuMP -->; search by key (open and closed issues), never by title, which may be edited by hand.
| Situation | Action |
|---|---|
| Failure, no issue with this key | open it |
| Failure, issue open | edit the body (failing configurations, last run link, date); comment only when the set of failing configurations changes, to avoid weekly notifications |
| Failure, issue closed as completed | reopen it once, with a "regression" comment |
| Failure, issue closed as not planned | do nothing: the human decision stands |
| Success, issue open | after two consecutive successful runs, close it automatically with a comment (run links) |
Safeguards:
- Flaky problems: after 3 automatic reopenings, add the label
flakyand stop reopening; runs only update the body. - General breakage: if more than 5 (problem, modeling) pairs fail in one run, open no per-problem issue; report everything in
cron-failureas a likely upstream breakage. - Drafts and declared
:known_failuresnever get a bug issue. - State between runs (consecutive successes, reopen count) is read from and written to the
resultsbranch (cron/directory, [Doc] Results pipeline for the documentation, and a gallery of problems #314). - Labels
cron-bugandflaky: create them here and add them to the label plan of [Chore] Harmonize repository metadata and ct-conventions integration with CTBase #313.
Out of scope
- Running the documentation examples: done by the documentation build ([Doc] Rewrite the documentation: getting started, user guide, problems, developers #307, [Doc] review of examples published on the website #229).
- The status page itself: [Doc] Rewrite the documentation: getting started, user guide, problems, developers #307.
Done when
- The five scheduled workflows (ModelingToolkit once [Problems] Proof of concept: a ModelingToolkit modeling, solved with Corleone, with our own bryson_denham #308 exists) and the reusable one; staggered schedules;
workflow_dispatchon each. - CTActions
ci.yml/ci-gpu.ymlcan upload a report artifact (issue/PR in CTActions). - Each issue of the table is created once and updated in place; shown by two manual runs, one failing on purpose.
- Reports committed to the
resultsbranch (cron/) after each run. - Known-failure logic shown on a deliberately declared failure, and on one that no longer happens.
- Bug issues: shown on manual runs to be opened once, updated without comment, commented when the failing set changes, reopened on regression, left alone when closed as not planned, closed with a comment after two successes; the general-breakage threshold shown too.
- Workflows listed in CTActions
docs/workflows/per-repo.md.
Activity
ocots commentedon Nov 4, 2026
Decision (while detailing #312): the weekly cron runs on GPU too (
ci-gpu.yml:kkt, elseoccidata), as in CTFlows.jlCronIntegration.yml, with every problem available under Exa on GPU.[-][Tests] Weekly cron for the slow tests (as in CTFlows)[/-][+][Tests] Crons per scope for the slow tests, with reports[/+]ocots commentedon Nov 4, 2026
Documentation (decision in #307): this issue also writes
.github/workflows/README.md(crons per scope, reports on theresultsbranch, automatic issues, how to add a scope). Procedures only. Add it to the 'Done when' checklist.ocots commentedon Nov 5, 2026
From #304 (PRs #319, #324) — what the crons can rely on:
Pkg.test(; test_args=["cron"])runstest/cron/<group>/test_*.jlin the environment oftest/suite/<group>/; environments and file globs combine (["comparison", "cron"]). The cron ignores the scope labels.test/reports/<environment>[-cron].md(git-ignored):| test | problem | modelings | status | outcome |, one line per test and problem (:validlines say "see the test summary"; drafts say pass/fail/error; checks that report instead of failing, e.g. the comparison rule, say why). The console lists the lines worth reading. A draft withpasson every line is ready to become:valid— the report to publish and the trigger of the bug issues.bryson_denham(JuMP) andmountain_car(scalarized) intest_solution.ocots commentedon Nov 5, 2026
Taken by an agent; draft PR to follow.
ocots commentedon Nov 5, 2026
Implementation is underway in the dedicated worktree/branch
test/305-crons-reports-bug-issues. The repository already has the requested cron labels (cron-failure,cron-drafts,cron-capabilities,cron-bug,flaky). The required CTActions artifact-upload interface is tracked by control-toolbox/CTActions#135.ocots commentedon Nov 5, 2026
Draft PR opened: #335. The CTActions dependency is implemented in draft PR control-toolbox/CTActions#136.
fix: only touch issues labelled cron-bug when matching a hidden key
ocots commentedon Nov 5, 2026
To do (found while working on #330, PR #354): point the cron GPU job to GPU configurations
CronOptimalControl.ymlandCronOptimalControlScalarized.ymlsetgpu: true, socron-reusable.ymlruns<scope> cronon the automatic GPU runner (ci-gpu.yml). Buttest/cron/<scope>/test_convergence.jlloops overCONFIGS, which holds the CPU configurations only (ADNLP + Ipopt, Exa + MadNLP on CPU): the GPU runner solves on the CPU, and no cron test exercises Exa + MadNLP on the GPU at the default grid.#330 adds the opt-in environment
GPU(test/suite/GPU/,GPU_CONFIGSintest/common/configs.jl,TestCapabilities) with the small grid. Remaining for the cron:test/cron/GPU/test_convergence.jl(default grid of each problem,GPU_CONFIGS, two solves as in the other crons);gpujob ofcron-reusable.ymlrunsGPU croninstead of<scope> cron, and its report is named so that the publish step finds it (cron-<scope>-gputoday);gpu: true(probably not: it would then run twice on the GPU runner).Depends on #354.
🤖 Generated with Claude Code
ocots commentedon Nov 5, 2026
Delivered by #335 (cron workflows and reports), #346 (fixes found by the demonstration runs, Ubuntu + Windows, texts that explain what to do) and the CTActions interfaces #136 (
upload_artifact_*) and #140 (fail_fast). Tracked in #310, step 3.Done when
CronJuMP,CronOptimalControl,CronOptimalControlScalarized,CronComparison(Tuesday to Friday, 04:00 UTC,workflow_dispatcheach) andCronModelingToolkit(manual and disabled until [Problems] Proof of concept: a ModelingToolkit modeling, solved with Corleone, with our own bryson_denham #308), all thin callers ofcron-reusable.yml.ci.yml/ci-gpu.ymluploadtest/reportsas an artifact, including on failure ([Dev] Exa and problems #136), andfail_fast: falsekeeps every platform running ([AUTO] JuliaFormatter.jl run #140); shown on run 42012648736 (Windows report lost) and then 42022818017 (both platforms reported).results: firstCron JuMPrun 42010703158 created the orphan branch with one commit:cron/JuMP.md(platform, test, problem, status, outcome, run link, commit, Julia and package versions) andcron/state.json.cron-drafts([Cron] cron-drafts #348,bryson_denhamready, passing on Ubuntu and Windows) andcron-failure([Cron] cron-failure #350, then [Cron] cron-failure #352 after [Cron] cron-failure #350 was closed); one section per scope, replaced by each run; texts say who acts (a maintainer; nothing is automatic) and what to do, with a link to.github/workflows/README.md.chain, [Cron] Regression: bioreactor (JuMP) #356bioreactor, run 42027539655); closed with a comment after two consecutive successful runs (runs 42028717070, 42029607436); reopened with a "regression" comment (run 42030556878); left alone when closed as not planned (runs 42021780211, 42022818017, [Cron] Regression: chain (JuMP) #351); more than 5 failing pairs open no per-problem issue and are listed incron-failure(run 42023615281). The "failing configurations changed" comment is covered by the unit tests only.:known_failures): not active; the metadata field comes with [Metadata] Computed features, reference objective, shared initial guess #306. Thecron-capabilitiesissue is therefore never created yet.docs/workflows/per-repo.md: done in [Dev] Exa and problems #136..github/workflows/README.mddocuments the procedures.Found by the demonstrations and fixed (#346): the first run failed to create the results tree (
git rmon an empty orphan worktree); an issue that quotes a key as an example (#305 itself) was taken for a bug issue and its description rewritten; the matrix of the crons now runs without macOS.Deviations and limits
CronOptimalControlandCronOptimalControlScalarizedhave not been exercised yet: their first scheduled run (Wednesday and Thursday) is the check.tools/cron/test/and run in thescopejob ofCI.yml.cron-failure) and [Cron] Regression: bioreactor (JuMP) #356 (bioreactor, reopened by the last planted failure); they are demo artefacts, not real failures.