Skip to content

[Tests] Test layout per environment and consistency tests #304

Closed

Description

@ocots

Agents: read the guide for agents in #310 first (order, difficulty, how to claim an issue, working method). The specification is in this description and in the comments below.

Goal

Split the tests into one environment per modeling, so that each extension is shown to load alone and heavy dependencies stay isolated; make the tests follow the statuses of #303 (:draft / :valid); add the consistency tests that fail as soon as the collection drifts; select what runs on a PR with labels. Keep the PR CI fast: full solves move to the weekly cron (#305).

Part of #310, step 1. Depends on #303 (registry, three backends, statuses) and #312 (label-gated CI). Lands on main, no release.

Current state

  • One environment, test/Project.toml; test/runtests.jl includes test_<name>.jl for aqua (disabled), kwargs, JuMP, OptimalControl, OptimalControl_s, init, solution, quick, parameters; helpers in test/utils.jl (697 lines: comparison, tolerances, solver options, plots in test/figures/).
  • Every test loops over LIST_OF_PROBLEMS and calls eval(f) / eval(Symbol(f, :_s)).
  • The CI solves the 19 problems with Ipopt and MadNLP, in three modelings, on every run: 40 to 70 minutes on main.
  • No package of the organization has several test environments yet: this is new. CTActions ci.yml runs Pkg.test with test_args only.

Layout

test/
├── Project.toml, runtests.jl   # light environment: Test, Aqua, CTBase; dispatcher
├── common/                      # shared through include: tolerances, solver options, configs.jl, draft report
├── core/                        # in the main process, no extension loaded
├── JuMP/                        # each: Project.toml, runtests.jl, test_*.jl (fast), cron/test_*.jl (slow)
├── OptimalControl/
├── OptimalControlScalarized/
├── comparison/                  # every modeling loaded
└── ModelingToolkit/             # created by #308 (solved with Corleone)

Dispatcher (test/runtests.jl)

  • Runs core/ in the main process, then each selected environment in a subprocess: julia --project=test/<Env>, Pkg.develop(path=<package root>) (Julia 1.10 does not read [sources]), Pkg.instantiate(), then test/<Env>/runtests.jl with the remaining arguments. Same Julia binary (Base.julia_cmd()), coverage flag forwarded to the child (Base.JLOptions().code_coverage), non-zero exit of a child fails the run.
  • test_args grammar:
    • environment names (JuMP, OptimalControl, OptimalControlScalarized, comparison, ModelingToolkit): run those only; none given → all except ModelingToolkit;
    • cron: run cron/test_*.jl instead of test_*.jl in each environment (as in CTFlows.jl test/runtests.jl), stripped before forwarding;
    • anything else: file globs, forwarded to CTBase.run_tests in each environment.
  • core always runs.

test/common/

Tests follow the statuses

Every test function loops over problems(backend; status=:all):

  • :valid: checked with @test; a failure fails CI.
  • :draft: run inside try, outcome recorded, no @test (not @test_broken, which errors as soon as a draft passes); a failure never fails CI.
  • Report at the end of each run: table problem × modeling × test → pass / fail / error, printed and written to a file (test/report-<env>.md or similar, git-ignored), so that the cron ([Tests] Crons per scope for the slow tests, with reports #305) can report drafts ready to become :valid.
  • A comparison between two modelings is :valid (CI-failing) only when the problem is :valid in both.

What runs where

Fast (test_*.jl, PR CI) Slow (cron/test_*.jl, weekly)
core consistency tests (below), Aqua, parameters (test_parameters), precompilation cache test of #303, environment contract —
JuMP build every problem, JuMP model contract (below), one solve on a small grid (N≈20, Ipopt) convergence on the default grid, Ipopt and MadNLP (today test_JuMP)
OptimalControl, OptimalControlScalarized build every problem, one solve on a small grid (N≈20, Ipopt), kwargs convergence, every configuration of configs.jl, GPU
comparison NLP equivalence on a small grid (filled by #284), objectives on a small grid (today test_quick), initial guesses (today test_init) solutions OptimalControl / JuMP / scalarized on the default grid (today test_solution), objectives against reference_objective
ModelingToolkit see #308 see #308

The plots of test/utils.jl (test/figures/) are not kept in the tests; plotting moves to #175.

Consistency tests (core)

  1. Files: every model file has a metadata file and a description, and conversely; directory names match the modeling table of [Dev] Per-modeling lists of problems, discovered automatically, with a status per modeling #303.
  2. Metadata schema: required fields and their types (extend METADATA_TYPES); :status lists exactly the modelings where the problem has a file, with values in (:draft, :valid).
  3. Methods: after loading the extensions (in each modeling environment), every problem with a file has a real method defined in the extension module, not only the generic fallback.
  4. Comparison rule: every problem :valid somewhere has at least one reference: another modeling where it is :valid, or a sourced reference_objective ([Metadata] Computed features, reference objective, shared initial guess #306; until then, report instead of fail).
  5. No hand-written list: no list of problem names in src/, ext/, test/ (search for the problem names outside ext/ and the metadata).
  6. Environment contract (rule TST6): on a GPU runner (RUNNER_NAME contains kkt or occidata), the GPU must be functional, otherwise fail; see TestCapabilities in CTFlows.jl test/runtests.jl and test/suite/environment/.

JuMP model contract (JuMP, no solve)

For every problem and every scheme available in JuMP:

  • keys present in the model: :time_grid, :state_components, :costate_components, :control_components, :variable_components, :scheme;
  • types (vectors of String, time_grid callable);
  • every component name resolves to a variable or a constraint of the model;
  • lengths consistent with grid_size and the scheme (state N+1; control N+1 for trapezoid, N for midpoint and Euler, see [Paper] JuMP models for Euler, trapeze and midpoint, and NLP equivalence tests #284; costate constraints N);
  • time_grid() has N+1 points, from t0 to tf;
  • dimensions equal to those of the OptimalControl model when the problem is available there (checked in comparison).

Today presence and name resolution are checked only inside the slow convergence test (test/test_JuMP.jl:27-47).

Labels

Two independent families:

Family Labels None set
Where (runner), defined in #312 run ci github-runner, run ci kkt-runner, run ci occidata-runner nothing runs
What (scope), modifiers test JuMP backend, test OptimalControl backend, test OptimalControlScalarized backend, test ModelingToolkit backend, test comparison everything except ModelingToolkit

To check (not in this issue): a release PR must test everything, so the release gate should refuse a release PR carrying a test … label. That is a rule for CTActions release-gate.yml; open an issue there.

Out of scope

Done when

  • Layout above; Pkg.test() runs core and every environment except ModelingToolkit; Pkg.test(test_args=["JuMP"]), ["cron"], ["JuMP", "cron"], and a file glob work.
  • Each modeling environment loads only its own extension (e.g. JuMP/ does not have OptimalControl in its Project.toml).
  • Coverage of extension code is collected from the subprocesses.
  • A :draft problem that fails does not fail CI and appears in the report; a :valid one that fails does.
  • Consistency tests: each one shown to fail on a deliberately broken case (missing description, wrong :status, …) before the fix is reverted.
  • JuMP model contract runs without solving, for every JuMP problem.
  • PR CI (fast tests, three OSes) clearly shorter than today's 40–70 minutes; time reported in the PR.
  • Scope labels created; CI.yml turns them into test_args; behaviour shown on a test PR (no scope label, one, two).
  • Issue opened in CTActions about scope labels on release PRs.

Activity

  1. added
    featureNew feature or request
    devInternal change, no user-facing impact
    on Nov 4, 2026
  2. changed the title [-][Tests] Consistency tests: files, metadata, comparison rule, JuMP model contract[/-] [+][Tests] Test layout per environment and consistency tests[/+] on Nov 4, 2026
  3. ocots commented on Nov 4, 2026

    @ocots
    MemberAuthor

    Scope update (decided while detailing #303): this issue also takes the test layout per environment, so that each extension is shown to load alone and heavy dependencies stay isolated:

    test/
    ├── runtests.jl          # Pkg.test(): runs core, then each environment in a subprocess
    ├── core/                # no extension: registry, metadata, file consistency
    ├── JuMP/
    ├── OptimalControl/
    ├── OptimalControlScalarized/
    ├── comparison/          # all modelings loaded: equivalence and comparisons
    └── Corleone/            # own environment (#308)
    

    This issue will be detailed in turn.

  4. ocots commented on Nov 4, 2026

    @ocots
    MemberAuthor

    Additions (decided while detailing #312):

    • Labels per modeling, to run the tests of one backend only on a PR (for instance run ci jump, run ci optimalcontrol, run ci optimalcontrol-scalarized, run ci corleone, names to settle). They are package-specific labels: create them here, and keep them in the label plan of [Chore] Harmonize repository metadata and ct-conventions integration with CTBase #313 (theme 2).
    • GPU tests on PRs: jobs run ci kkt-runner and run ci occidata-runner, and one GPU run on main through CTActions ci-gpu.yml, as in CTFlows.jl CI.yml. Requires the Exa GPU tests, after the port to v2 ([Paper] Port to OptimalControl v2 and new problem interface #283). GPU tests must fail visibly when the GPU is not functional on a GPU runner (rule TST6, see TestCapabilities in CTFlows.jl test/runtests.jl).
  5. ocots commented on Nov 4, 2026

    @ocots
    MemberAuthor

    Taken by an agent; draft PR to follow. Built on top of the branch of #303 (in progress, another agent), to be rebased onto main once #303 is merged.

  6. ocots commented on Nov 4, 2026

    @ocots
    MemberAuthor

    Documentation (decision in #307): this issue also writes test/README.md (environments, labels run ci … and test …, how to run each environment, how to add a test). Procedures only, no hand-written list of tests. Add it to the 'Done when' checklist.

  7. ocots commented on Nov 5, 2026

    @ocots
    MemberAuthor

    Status of #304 once #324 is merged (part 1: #319 via #320; part 2: #324).

    Done

    • Layout test/suite/<group>/ (fast) and test/cron/<group>/ (slow); one environment per modeling plus comparison; the dispatcher runs each in a subprocess; coverage of the extension code is collected from them (22 files of ext/JuMPModels/ checked).
    • Tests follow the statuses of [Dev] Per-modeling lists of problems, discovered automatically, with a status per modeling #303: :valid is checked with @test, :draft is recorded in a report (test/reports/<env>[-cron].md) and never fails the run. Each check was shown failing on a deliberately broken case (see the PR comments, cases A–H).
    • Consistency tests in core (files, metadata schema, comparison rule, no hand-written list); JuMP model contract without solving (682 checks); the single matrix test/common/configs.jl; test/README.md.
    • Scope labels test <environment> backend / test comparison, turned into test_args by CI.yml; adding or removing one reruns the CI. The test argument problem=a,b runs only the given problems.
    • PR CI on the last commit (Julia 1.10 and 1.13, three OSes): six jobs green, 27 to 44 minutes (40 to 70 before).
    • What the strict tests revealed and the maintainer decided: SMALL_GRID = 50; space_shuttle JuMP bounds fixed (tf_l, tf_u); mountain_car is :draft for the scalarized modeling, with a note.

    Not done, or done differently

    Discovered on the way

    Follow-ups: #327 (test only the problems a PR changes), #326, #323, #283, #284, #305, #306, #311 (notes posted there).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

devInternal change, no user-facing impactfeatureNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions