Skip to content

[Tests] Crons per scope for the slow tests, with reports #305

Closed

Description

@ocots

Agents: read the guide for agents in #310 first (order, difficulty, how to claim an issue, working method). The specification is in this description and in the comments below.

Goal

Run the slow tests (cron mode of #304) regularly, one cron per scope, staggered over the week; report automatically through issues and report files, so that an upstream change (OptimalControl, CTDirect, CTParser, CTSolvers, ExaModels…) breaking a problem is seen within a week, and drafts ready to become :valid are noticed.

Part of #310, step 3. Depends on #304 (environments, cron mode, draft report) and #312 (CI infrastructure).

Reference

CTFlows.jl .github/workflows/CronIntegration.yml: weekly, CTActions ci.yml with test_args: 'cron' on ubuntu, macos, windows, plus ci-gpu.yml (kkt, else occidata); a notify-on-failure job opens one issue labelled cron-failure unless one is open. Runs take 30–90 minutes; it opened #391 and #426, since closed.

Several crons, one per scope

Each scope runs core plus one environment, in cron mode (test_args: '<Env> cron'), so that each run stays well under the 6-hour job limit and a failure points at one modeling.

Workflow Environment Runners Proposed schedule (UTC)
CronJuMP.yml JuMP ubuntu, macos, windows Tuesday 04:00
CronOptimalControl.yml OptimalControl ubuntu, macos, windows + GPU Wednesday 04:00
CronOptimalControlScalarized.yml OptimalControlScalarized ubuntu, macos, windows + GPU Thursday 04:00
CronComparison.yml comparison ubuntu (+ GPU if comparisons on GPU exist) Friday 04:00
CronModelingToolkit.yml ModelingToolkit (solved with Corleone) ubuntu Saturday 04:00 (once #308 exists)
  • Stagger to avoid loading everything, and the GPU runners in particular, at the same time. Local survey of the organization's schedules (2026-11-04): busy slots are daily 00:00, 01:00 and 03:15, and Monday 00:00, 02:30, 03:00 (CTFlows cron) and 06:00. Re-check before choosing.
  • Write the common logic once: a local reusable workflow (e.g. .github/workflows/cron-reusable.yml, input: environment name, GPU or not) called by the thin scheduled files above; each caller also has workflow_dispatch.
  • The crons ignore the scope labels of [Tests] Test layout per environment and consistency tests #304: everything of the scope runs.
  • Add the new workflows to CTActions docs/workflows/per-repo.md (package-specific workflows).

What makes a cron fail

  • A :valid problem failing its tests in that environment.
  • A configuration (modeler × scheme × solver × CPU/GPU, from test/common/configs.jl) failing for a :valid problem, unless declared in the metadata as a known failure, e.g. :known_failures => [(modeler=:exa, parameter=:gpu, reason="…")] (field added by [Metadata] Computed features, reference objective, shared initial guess #306). A declared failure that no longer happens is reported, not failed. Same principle as the statuses: a claim, checked by the tests; the capability matrix is never written by hand.
  • :draft problems never fail the cron.

Reports

  • The tests write report files ([Tests] Test layout per environment and consistency tests #304): per problem × modeling × configuration → pass / fail / error, time, objective; drafts that pass.
  • CTActions change: ci.yml (and ci-gpu.yml) keep no file produced by the tests. Add an optional input (e.g. upload_artifact_path) that uploads a file or directory as a workflow artifact. Open the issue/PR in CTActions; useful to other packages.
  • A final job of each cron collects the artifacts and:
    • updates three issues, each unique (update the open one instead of opening another), with the run link:

      Label When Content
      cron-failure something above failed environments, problems, configurations failing
      cron-drafts a draft passed all its tests problems and modelings ready to become :valid
      cron-capabilities a declared known failure no longer happens the entries of :known_failures that can be removed

      One issue per label for the whole repository; the body has one section per scope, replaced by each run of that scope.

    • stores the reports in the repository for the documentation: commit them to the cron/ directory of the shared results branch ([Doc] Results pipeline for the documentation, and a gallery of problems #314: orphan branch rewritten with one commit, single writer; one file per scope, plus the date and the run link), with contents: write. The documentation reads them to build a status page ([Doc] Rewrite the documentation: getting started, user guide, problems, developers #307).

  • Labels cron-failure, cron-drafts, cron-capabilities: create them here and add them to the label plan of [Chore] Harmonize repository metadata and ct-conventions integration with CTBase #313.

Bug issues per problem (idempotent)

Besides the summary issues above, the final job opens bug issues for failing problems, idempotently:

  • One issue per (problem, modeling), not per configuration; failing configurations are listed in its body. Labels bug and cron-bug.
  • Identified by a hidden key in the body, e.g. <!-- cron-key: beam|JuMP -->; search by key (open and closed issues), never by title, which may be edited by hand.
Situation Action
Failure, no issue with this key open it
Failure, issue open edit the body (failing configurations, last run link, date); comment only when the set of failing configurations changes, to avoid weekly notifications
Failure, issue closed as completed reopen it once, with a "regression" comment
Failure, issue closed as not planned do nothing: the human decision stands
Success, issue open after two consecutive successful runs, close it automatically with a comment (run links)

Safeguards:

Out of scope

Done when

  • The five scheduled workflows (ModelingToolkit once [Problems] Proof of concept: a ModelingToolkit modeling, solved with Corleone, with our own bryson_denham #308 exists) and the reusable one; staggered schedules; workflow_dispatch on each.
  • CTActions ci.yml / ci-gpu.yml can upload a report artifact (issue/PR in CTActions).
  • Each issue of the table is created once and updated in place; shown by two manual runs, one failing on purpose.
  • Reports committed to the results branch (cron/) after each run.
  • Known-failure logic shown on a deliberately declared failure, and on one that no longer happens.
  • Bug issues: shown on manual runs to be opened once, updated without comment, commented when the failing set changes, reopened on regression, left alone when closed as not planned, closed with a comment after two successes; the general-breakage threshold shown too.
  • Workflows listed in CTActions docs/workflows/per-repo.md.

Activity

  1. added
    featureNew feature or request
    devInternal change, no user-facing impact
    on Nov 4, 2026
  2. ocots commented on Nov 4, 2026

    @ocots
    MemberAuthor

    Decision (while detailing #312): the weekly cron runs on GPU too (ci-gpu.yml: kkt, else occidata), as in CTFlows.jl CronIntegration.yml, with every problem available under Exa on GPU.

  3. changed the title [-][Tests] Weekly cron for the slow tests (as in CTFlows)[/-] [+][Tests] Crons per scope for the slow tests, with reports[/+] on Nov 4, 2026
  4. ocots commented on Nov 4, 2026

    @ocots
    MemberAuthor

    Documentation (decision in #307): this issue also writes .github/workflows/README.md (crons per scope, reports on the results branch, automatic issues, how to add a scope). Procedures only. Add it to the 'Done when' checklist.

  5. ocots commented on Nov 5, 2026

    @ocots
    MemberAuthor

    From #304 (PRs #319, #324) — what the crons can rely on:

    • Mode: Pkg.test(; test_args=["cron"]) runs test/cron/<group>/test_*.jl in the environment of test/suite/<group>/; environments and file globs combine (["comparison", "cron"]). The cron ignores the scope labels.
    • Reports: each run writes test/reports/<environment>[-cron].md (git-ignored): | test | problem | modelings | status | outcome |, one line per test and problem (:valid lines say "see the test summary"; drafts say pass/fail/error; checks that report instead of failing, e.g. the comparison rule, say why). The console lists the lines worth reading. A draft with pass on every line is ready to become :valid — the report to publish and the trigger of the bug issues.
    • Local cron time (Julia 1.12, Apple silicon): 27 min 50 s — JuMP 2 min, OptimalControl 5 min 30 s, OptimalControlScalarized 8 min 16 s, comparison 11 min 54 s.
    • Current drafts failing in the cron: bryson_denham (JuMP) and mountain_car (scalarized) in test_solution.
    • Cancelling superseded runs of a PR: Cancel superseded runs of a pull request in the reusable workflows CTActions#134.
  6. ocots commented on Nov 5, 2026

    @ocots
    MemberAuthor

    Taken by an agent; draft PR to follow.

  7. ocots commented on Nov 5, 2026

    @ocots
    MemberAuthor

    Implementation is underway in the dedicated worktree/branch test/305-crons-reports-bug-issues. The repository already has the requested cron labels (cron-failure, cron-drafts, cron-capabilities, cron-bug, flaky). The required CTActions artifact-upload interface is tracked by control-toolbox/CTActions#135.

  8. ocots commented on Nov 5, 2026

    @ocots
    MemberAuthor

    Draft PR opened: #335. The CTActions dependency is implemented in draft PR control-toolbox/CTActions#136.

  9. ocots commented on Nov 5, 2026

    @ocots
    MemberAuthor

    To do (found while working on #330, PR #354): point the cron GPU job to GPU configurations

    CronOptimalControl.yml and CronOptimalControlScalarized.yml set gpu: true, so cron-reusable.yml runs <scope> cron on the automatic GPU runner (ci-gpu.yml). But test/cron/<scope>/test_convergence.jl loops over CONFIGS, which holds the CPU configurations only (ADNLP + Ipopt, Exa + MadNLP on CPU): the GPU runner solves on the CPU, and no cron test exercises Exa + MadNLP on the GPU at the default grid.

    #330 adds the opt-in environment GPU (test/suite/GPU/, GPU_CONFIGS in test/common/configs.jl, TestCapabilities) with the small grid. Remaining for the cron:

    • a test/cron/GPU/test_convergence.jl (default grid of each problem, GPU_CONFIGS, two solves as in the other crons);
    • the gpu job of cron-reusable.yml runs GPU cron instead of <scope> cron, and its report is named so that the publish step finds it (cron-<scope>-gpu today);
    • decide whether the CPU cron of the two OptimalControl scopes keeps its gpu: true (probably not: it would then run twice on the GPU runner).

    Depends on #354.

    🤖 Generated with Claude Code

  10. ocots commented on Nov 5, 2026

    @ocots
    MemberAuthor

    Delivered by #335 (cron workflows and reports), #346 (fixes found by the demonstration runs, Ubuntu + Windows, texts that explain what to do) and the CTActions interfaces #136 (upload_artifact_*) and #140 (fail_fast). Tracked in #310, step 3.

    Done when

    Found by the demonstrations and fixed (#346): the first run failed to create the results tree (git rm on an empty orphan worktree); an issue that quotes a key as an example (#305 itself) was taken for a bug issue and its description rewritten; the matrix of the crons now runs without macOS.

    Deviations and limits

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

devInternal change, no user-facing impactfeatureNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions