Repository navigation
Description
Agents: read the guide for agents in #310 first (order, difficulty, how to claim an issue, working method). The specification is in this description and in the comments below.
Goal
Split the tests into one environment per modeling, so that each extension is shown to load alone and heavy dependencies stay isolated; make the tests follow the statuses of #303 (:draft / :valid); add the consistency tests that fail as soon as the collection drifts; select what runs on a PR with labels. Keep the PR CI fast: full solves move to the weekly cron (#305).
Part of #310, step 1. Depends on #303 (registry, three backends, statuses) and #312 (label-gated CI). Lands on main, no release.
Current state
- One environment,
test/Project.toml;test/runtests.jlincludestest_<name>.jlforaqua(disabled),kwargs,JuMP,OptimalControl,OptimalControl_s,init,solution,quick,parameters; helpers intest/utils.jl(697 lines:comparison, tolerances, solver options, plots intest/figures/). - Every test loops over
LIST_OF_PROBLEMSand callseval(f)/eval(Symbol(f, :_s)). - The CI solves the 19 problems with Ipopt and MadNLP, in three modelings, on every run: 40 to 70 minutes on
main. - No package of the organization has several test environments yet: this is new. CTActions
ci.ymlrunsPkg.testwithtest_argsonly.
Layout
test/
├── Project.toml, runtests.jl # light environment: Test, Aqua, CTBase; dispatcher
├── common/ # shared through include: tolerances, solver options, configs.jl, draft report
├── core/ # in the main process, no extension loaded
├── JuMP/ # each: Project.toml, runtests.jl, test_*.jl (fast), cron/test_*.jl (slow)
├── OptimalControl/
├── OptimalControlScalarized/
├── comparison/ # every modeling loaded
└── ModelingToolkit/ # created by #308 (solved with Corleone)
Dispatcher (test/runtests.jl)
- Runs
core/in the main process, then each selected environment in a subprocess:julia --project=test/<Env>,Pkg.develop(path=<package root>)(Julia 1.10 does not read[sources]),Pkg.instantiate(), thentest/<Env>/runtests.jlwith the remaining arguments. Same Julia binary (Base.julia_cmd()), coverage flag forwarded to the child (Base.JLOptions().code_coverage), non-zero exit of a child fails the run. test_argsgrammar:- environment names (
JuMP,OptimalControl,OptimalControlScalarized,comparison,ModelingToolkit): run those only; none given → all exceptModelingToolkit; cron: runcron/test_*.jlinstead oftest_*.jlin each environment (as in CTFlows.jltest/runtests.jl), stripped before forwarding;- anything else: file globs, forwarded to
CTBase.run_testsin each environment.
- environment names (
corealways runs.
test/common/
configs.jl: the single matrix of configurations (modelers × schemes × solvers × options), read by every test; for now Ipopt and MadNLP, trapezoid (extended by [Paper] Port to OptimalControl v2 and new problem interface #283, [Paper] JuMP models for Euler, trapeze and midpoint, and NLP equivalence tests #284).- Tolerances and solver options (today
TOL = 1e-8,MU_STRATEGY,MAX_ITER,MAX_WALL_TIME). - The status-aware loop and the draft report (below).
Tests follow the statuses
Every test function loops over problems(backend; status=:all):
:valid: checked with@test; a failure fails CI.:draft: run insidetry, outcome recorded, no@test(not@test_broken, which errors as soon as a draft passes); a failure never fails CI.- Report at the end of each run: table problem × modeling × test → pass / fail / error, printed and written to a file (
test/report-<env>.mdor similar, git-ignored), so that the cron ([Tests] Crons per scope for the slow tests, with reports #305) can report drafts ready to become:valid. - A comparison between two modelings is
:valid(CI-failing) only when the problem is:validin both.
What runs where
Fast (test_*.jl, PR CI) |
Slow (cron/test_*.jl, weekly) |
|
|---|---|---|
core |
consistency tests (below), Aqua, parameters (test_parameters), precompilation cache test of #303, environment contract |
— |
JuMP |
build every problem, JuMP model contract (below), one solve on a small grid (N≈20, Ipopt) | convergence on the default grid, Ipopt and MadNLP (today test_JuMP) |
OptimalControl, OptimalControlScalarized |
build every problem, one solve on a small grid (N≈20, Ipopt), kwargs |
convergence, every configuration of configs.jl, GPU |
comparison |
NLP equivalence on a small grid (filled by #284), objectives on a small grid (today test_quick), initial guesses (today test_init) |
solutions OptimalControl / JuMP / scalarized on the default grid (today test_solution), objectives against reference_objective |
ModelingToolkit |
see #308 | see #308 |
The plots of test/utils.jl (test/figures/) are not kept in the tests; plotting moves to #175.
Consistency tests (core)
- Files: every model file has a metadata file and a description, and conversely; directory names match the modeling table of [Dev] Per-modeling lists of problems, discovered automatically, with a status per modeling #303.
- Metadata schema: required fields and their types (extend
METADATA_TYPES);:statuslists exactly the modelings where the problem has a file, with values in(:draft, :valid). - Methods: after loading the extensions (in each modeling environment), every problem with a file has a real method defined in the extension module, not only the generic fallback.
- Comparison rule: every problem
:validsomewhere has at least one reference: another modeling where it is:valid, or a sourcedreference_objective([Metadata] Computed features, reference objective, shared initial guess #306; until then, report instead of fail). - No hand-written list: no list of problem names in
src/,ext/,test/(search for the problem names outsideext/and the metadata). - Environment contract (rule
TST6): on a GPU runner (RUNNER_NAMEcontainskktoroccidata), the GPU must be functional, otherwise fail; seeTestCapabilitiesin CTFlows.jltest/runtests.jlandtest/suite/environment/.
JuMP model contract (JuMP, no solve)
For every problem and every scheme available in JuMP:
- keys present in the model:
:time_grid,:state_components,:costate_components,:control_components,:variable_components,:scheme; - types (vectors of
String,time_gridcallable); - every component name resolves to a variable or a constraint of the model;
- lengths consistent with
grid_sizeand the scheme (stateN+1; controlN+1for trapezoid,Nfor midpoint and Euler, see [Paper] JuMP models for Euler, trapeze and midpoint, and NLP equivalence tests #284; costate constraintsN); time_grid()hasN+1points, fromt0totf;- dimensions equal to those of the OptimalControl model when the problem is available there (checked in
comparison).
Today presence and name resolution are checked only inside the slow convergence test (test/test_JuMP.jl:27-47).
Labels
Two independent families:
| Family | Labels | None set |
|---|---|---|
| Where (runner), defined in #312 | run ci github-runner, run ci kkt-runner, run ci occidata-runner |
nothing runs |
| What (scope), modifiers | test JuMP backend, test OptimalControl backend, test OptimalControlScalarized backend, test ModelingToolkit backend, test comparison |
everything except ModelingToolkit |
- Scope labels add up;
corealways runs; ModelingToolkit only with its label (and in the cron). - On a GPU runner, environments without GPU tests are skipped (e.g. JuMP).
- Adding or removing a scope label reruns the CI when a runner label is set.
- The cron ignores labels.
- Implementation: in
CI.yml, a small job reads the PR labels and outputstest_args(e.g.JuMP comparison), passed to CTActionsci.yml; no CTActions change. - Create the scope labels here, and add them to the label plan of [Chore] Harmonize repository metadata and ct-conventions integration with CTBase #313 (theme 2) as package-specific labels.
- GPU jobs in
CI.yml:run ci kkt-runner,run ci occidata-runneron PRs, and one GPU run onmainthrough CTActionsci-gpu.yml, as in CTFlows.jl. They become useful once Exa GPU tests exist (after [Paper] Port to OptimalControl v2 and new problem interface #283); add the jobs now if cheap, otherwise with [Paper] Port to OptimalControl v2 and new problem interface #283.
To check (not in this issue): a release PR must test everything, so the release gate should refuse a release PR carrying a test … label. That is a rule for CTActions release-gate.yml; open an issue there.
Out of scope
- NLP equivalence itself ([Paper] JuMP models for Euler, trapeze and midpoint, and NLP equivalence tests #284): this issue provides the
comparisonenvironment and the loop; [Paper] JuMP models for Euler, trapeze and midpoint, and NLP equivalence tests #284 fills it. - Cron workflow ([Tests] Crons per scope for the slow tests, with reports #305): this issue provides the
cronmode andcron/directories. - Plots ([General] Plots #175), ModelingToolkit environment content ([Problems] Proof of concept: a ModelingToolkit modeling, solved with Corleone, with our own bryson_denham #308).
Done when
- Layout above;
Pkg.test()runs core and every environment except ModelingToolkit;Pkg.test(test_args=["JuMP"]),["cron"],["JuMP", "cron"], and a file glob work. - Each modeling environment loads only its own extension (e.g.
JuMP/does not have OptimalControl in itsProject.toml). - Coverage of extension code is collected from the subprocesses.
- A
:draftproblem that fails does not fail CI and appears in the report; a:validone that fails does. - Consistency tests: each one shown to fail on a deliberately broken case (missing description, wrong
:status, …) before the fix is reverted. - JuMP model contract runs without solving, for every JuMP problem.
- PR CI (fast tests, three OSes) clearly shorter than today's 40–70 minutes; time reported in the PR.
- Scope labels created;
CI.ymlturns them intotest_args; behaviour shown on a test PR (no scope label, one, two). - Issue opened in CTActions about scope labels on release PRs.
Activity
[-][Tests] Consistency tests: files, metadata, comparison rule, JuMP model contract[/-][+][Tests] Test layout per environment and consistency tests[/+]ocots commentedon Nov 4, 2026
Scope update (decided while detailing #303): this issue also takes the test layout per environment, so that each extension is shown to load alone and heavy dependencies stay isolated:
cron/subdirectory, selected by the"cron"test argument as in CTFlows.:draftproblems are reported without failing CI; failures of:validproblems, or of a comparison between two modelings where the problem is:valid, fail CI.Pkg.test()only knowstest/Project.toml(hence the top-level dispatcher, unless CTActions already supports several environments); Julia 1.10 does not read[sources](usePkg.develop(path=...)).This issue will be detailed in turn.
ocots commentedon Nov 4, 2026
Additions (decided while detailing #312):
run ci jump,run ci optimalcontrol,run ci optimalcontrol-scalarized,run ci corleone, names to settle). They are package-specific labels: create them here, and keep them in the label plan of [Chore] Harmonize repository metadata and ct-conventions integration with CTBase #313 (theme 2).run ci kkt-runnerandrun ci occidata-runner, and one GPU run onmainthrough CTActionsci-gpu.yml, as in CTFlows.jlCI.yml. Requires the Exa GPU tests, after the port to v2 ([Paper] Port to OptimalControl v2 and new problem interface #283). GPU tests must fail visibly when the GPU is not functional on a GPU runner (ruleTST6, seeTestCapabilitiesin CTFlows.jltest/runtests.jl).ocots commentedon Nov 4, 2026
Taken by an agent; draft PR to follow. Built on top of the branch of #303 (in progress, another agent), to be rebased onto
mainonce #303 is merged.test: add the dispatcher and the core environment
test: add the JuMP environment
ocots commentedon Nov 4, 2026
Documentation (decision in #307): this issue also writes
test/README.md(environments, labelsrun ci …andtest …, how to run each environment, how to add a test). Procedures only, no hand-written list of tests. Add it to the 'Done when' checklist.test: add the OptimalControl and OptimalControlScalarized environments
test: follow the suite/cron layout of the conventions
test: add a provisional test runner and shared test options
test: add the dispatcher and the core environment
test: add the JuMP environment
test: add the OptimalControl and OptimalControlScalarized environments
test: follow the suite/cron layout of the conventions
test: add a provisional test runner and shared test options
test: add the dispatcher and the core environment
test: add the JuMP environment
test: add the OptimalControl and OptimalControlScalarized environments
test: follow the suite/cron layout of the conventions
Merge pull request #319 from control-toolbox/test/304-test-layout
ci: rerun the CI when a scope label is added or removed
ocots commentedon Nov 5, 2026
Status of #304 once #324 is merged (part 1: #319 via #320; part 2: #324).
Done
test/suite/<group>/(fast) andtest/cron/<group>/(slow); one environment per modeling pluscomparison; the dispatcher runs each in a subprocess; coverage of the extension code is collected from them (22 files ofext/JuMPModels/checked).:validis checked with@test,:draftis recorded in a report (test/reports/<env>[-cron].md) and never fails the run. Each check was shown failing on a deliberately broken case (see the PR comments, cases A–H).core(files, metadata schema, comparison rule, no hand-written list); JuMP model contract without solving (682 checks); the single matrixtest/common/configs.jl;test/README.md.test <environment> backend/test comparison, turned intotest_argsbyCI.yml; adding or removing one reruns the CI. The test argumentproblem=a,bruns only the given problems.SMALL_GRID = 50;space_shuttleJuMP bounds fixed (tf_l,tf_u);mountain_caris:draftfor the scalarized modeling, with a note.Not done, or done differently
gliderandcart_pendulumhave several local solutions, and the CI ended on a different one on Windows. Thecomparisonenvironment checks structure, grid, trajectories and objective at the initial guess (zero iterations). The equivalence of the models is the NLP equivalence of [Paper] JuMP models for Euler, trapeze and midpoint, and NLP equivalence tests #284; known local solutions go with [Metadata] Computed features, reference objective, shared initial guess #306.:schemekey of the JuMP contract (no model has it before [Paper] JuMP models for Euler, trapeze and midpoint, and NLP equivalence tests #284), the GPU environment contractTST6and GPU jobs (no GPU test before [Paper] Port to OptimalControl v2 and new problem interface #283), the precompilation cache test ([Dev] Test the automatic discovery of problems and the precompilation cache #323).test/utils.jlare not kept (see [General] Plots #175).Discovered on the way
Base.Docs.docneedsREPL; CTParser 0.8.2 and ExaModels ≥ 0.9.5 break precompilation, so ExaModels is bounded to ≤ 0.9.4 in the OptimalControl test environment until [Paper] Port to OptimalControl v2 and new problem interface #283.@testsetdoes not accept a qualified type name.bryson_denhamJuMP (draft):variable_components = []is aVector{Any}, the contract wantsString[].test …labels before merging (Release gate: refuse scope labels that restrict the tests of a release PR CTActions#133); superseded runs are not cancelled (Cancel superseded runs of a pull request in the reusable workflows CTActions#134).Follow-ups: #327 (test only the problems a PR changes), #326, #323, #283, #284, #305, #306, #311 (notes posted there).