Skip to content

[Tests] Report convergence instead of testing it; keep hard checks for breakage and model equivalence #360

Closed

Description

@ocots

Part of #310. Priority: it simplifies the work that follows (#354 GPU, #284 schemes, #305 crons, #306 metadata).

Context

Most solve tests are binary: @test succeeded(sol) in every environment (test_solve), in the crons (test_convergence) and on the GPU (#354). Convergence depends on the grid, the modeler, the solver, the linear solver, the AD backend, the parameters and the options: it cannot be guaranteed for every combination, and it should not be, since the problems are meant for benchmarking. Binary checks lead to tolerance tuning and special cases: #354 went through GPU_TOL, a list of known failures (rejected by no hand-written list), then a fallback from 1e-8 to 1e-6, because bioreactor only converges at 1e-6 on the condensed KKT system (#357) while mountain_car is declared infeasible at 1e-6. Several problems are draft only because of a convergence gap (mountain_car scalarized, 1.6e-4 on the objective).

Proposal

Three parts, one role each. Only the first two can fail a run.

1. "Does not break" — always (local, pull request, push, cron); hard failure

  • Every problem of every modeling (not only those shared by several modelings), with every configuration of its environment (GPU included, environment GPU).
  • Small grid, a few iterations: no exception, finite objective (and finite derivatives at the initial guess).
  • Replaces the current */test_solve (full convergence on the grid 50). It also covers what the 0-iteration solves of comparison/test_init miss today: the configurations other than the first one of each modeling (e.g. OptimalControl with Exa + MadNLP), problems available in one modeling only, and actual iterations (Hessian). The number of iterations is to be measured so that the Hessian and one real iteration are exercised in each solver.

2. Model equivalence — always; hard failure, the most important part

3. Convergence — off by default; a report, no verdict

  • A solve to convergence on the default grid of each problem: used to tune the initial guesses (trial and error: look at the results and adapt; helper scripts may come later) and to see that the solvers go to the end.
  • Off locally and on pull requests; on in the cron, always; on demand with Pkg.test(; test_args=["convergence"]) locally and a label test convergence on a pull request.
  • The report gives, per problem and configuration: a result status (error: exception, the only one that fails; converged: the solver's flag says so; not_converged: with the solver's reason), the values (objective, iterations, time, configuration), and the gap to the reference only when a reference exists (sourced reference_objective of [Metadata] Computed features, reference objective, shared initial guess #306, or a reference configuration); no reference, no gap.
  • In the cron, the change from the previous run (results branch, cron/state.json): a clear degradation (converged → not converged, gap growing) is flagged as the cron regressions are today, without failing the run.
  • The current cron/*/test_convergence become this part.

Statuses

  • draft means "the model is not finished or not equivalent", no longer "does not converge"; the problem statuses and the reports of TestStatus are revised accordingly.

Initial guesses

Several initial guesses (provided, close to the solution, arbitrary) can later become a dimension of the configurations (#306).

What else stays hard

core (consistency, metadata, API, Problem), structure checks such as solution(model, pb) (#329).

Scope

  • Replace */test_solve (and the GPU solves) by the "does not break" part; turn cron/*/test_convergence into the convergence part; add the convergence switch (test argument, label test convergence).
  • Result statuses and report format (values, gap if a reference exists, change in the cron).
  • Revise draft statuses that are only convergence gaps.
  • Document in test/README.md.
  • Then adapt test: GPU tests of the OptimalControl modelings (#330) #354 (no tolerance tuning on the GPU).

Done when

  • No test fails on non-convergence; an exception or a non-equivalence still fails.
  • Convergence off by default locally and on pull requests, on in the cron, on demand with the argument or the label.
  • Reports give status, values and gap to the reference when it exists; the cron shows the change from the previous run.
  • draft only for unfinished or non-equivalent models.
  • Documented in test/README.md.

🤖 Generated with Claude Code

Activity

  1. self-assigned this
    on Nov 5, 2026
  2. ocots commented on Nov 5, 2026

    @ocots
    MemberAuthor

    Done in #363 and #365 (merged).

    What was done

    • Does not break (every run, hard): suite/<modeling>/test_smoke.jl and the GPU test_smoke_*: every problem of every modeling, every configuration, grid SMOKE_GRID = 10, SMOKE_MAX_ITER = 3, finite objective. Measured first: all 85 problem × modeling × configuration combinations run at grids 5, 10, 20 and 1, 3, 5 iterations (765 runs).
    • Model equivalence (every run, hard): comparison unchanged (at the initial guess); the NLP equivalence is [Paper] JuMP models for Euler, trapeze and midpoint, and NLP equivalence tests #284.
    • Convergence (off by default; a report): cron/<modeling>/test_convergence.jl record one result per solve (TestStatus.result!: solver status and reason, objective, iterations, time, gap to :reference_objective only when the problem has one); results table in the report and <report>-results.tsv. On demand with Pkg.test(; test_args=["convergence"]) and the label test convergence; always in the cron.
    • Cron (ci: cron report of the convergence results (#360, part 2) #365): the scope report lists the results with their change since the previous run (new, lost, gained, objective change above 1e-4), the history is kept in state.json on results, and the summary issue cron-convergence flags what got worse; a convergence test never makes a draft ready.
    • Statuses: :draft means not finished or not equivalent; mountain_car scalarized became :valid. Documented in test/README.md and .github/workflows/README.md.

    Evidence

    Left

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions