Repository navigation
Description
Part of #310. Priority: it simplifies the work that follows (#354 GPU, #284 schemes, #305 crons, #306 metadata).
Context
Most solve tests are binary: @test succeeded(sol) in every environment (test_solve), in the crons (test_convergence) and on the GPU (#354). Convergence depends on the grid, the modeler, the solver, the linear solver, the AD backend, the parameters and the options: it cannot be guaranteed for every combination, and it should not be, since the problems are meant for benchmarking. Binary checks lead to tolerance tuning and special cases: #354 went through GPU_TOL, a list of known failures (rejected by no hand-written list), then a fallback from 1e-8 to 1e-6, because bioreactor only converges at 1e-6 on the condensed KKT system (#357) while mountain_car is declared infeasible at 1e-6. Several problems are draft only because of a convergence gap (mountain_car scalarized, 1.6e-4 on the objective).
Proposal
Three parts, one role each. Only the first two can fail a run.
1. "Does not break" — always (local, pull request, push, cron); hard failure
- Every problem of every modeling (not only those shared by several modelings), with every configuration of its environment (GPU included, environment
GPU). - Small grid, a few iterations: no exception, finite objective (and finite derivatives at the initial guess).
- Replaces the current
*/test_solve(full convergence on the grid 50). It also covers what the 0-iteration solves ofcomparison/test_initmiss today: the configurations other than the first one of each modeling (e.g. OptimalControl with Exa + MadNLP), problems available in one modeling only, and actual iterations (Hessian). The number of iterations is to be measured so that the Hessian and one real iteration are exercised in each solver.
2. Model equivalence — always; hard failure, the most important part
- Comparing models is fair only if they are the same problem: same continuous problem, same discretized NLP where it makes sense, compared at the definition, not at convergence.
comparison/test_init(at the initial guess) today; the NLP equivalence of [Paper] JuMP models for Euler, trapeze and midpoint, and NLP equivalence tests #284 (JuMP / ADNLP / Exa) next.
3. Convergence — off by default; a report, no verdict
- A solve to convergence on the default grid of each problem: used to tune the initial guesses (trial and error: look at the results and adapt; helper scripts may come later) and to see that the solvers go to the end.
- Off locally and on pull requests; on in the cron, always; on demand with
Pkg.test(; test_args=["convergence"])locally and a labeltest convergenceon a pull request. - The report gives, per problem and configuration: a result status (
error: exception, the only one that fails;converged: the solver's flag says so;not_converged: with the solver's reason), the values (objective, iterations, time, configuration), and the gap to the reference only when a reference exists (sourcedreference_objectiveof [Metadata] Computed features, reference objective, shared initial guess #306, or a reference configuration); no reference, no gap. - In the cron, the change from the previous run (
resultsbranch,cron/state.json): a clear degradation (converged → not converged, gap growing) is flagged as the cron regressions are today, without failing the run. - The current
cron/*/test_convergencebecome this part.
Statuses
draftmeans "the model is not finished or not equivalent", no longer "does not converge"; the problem statuses and the reports ofTestStatusare revised accordingly.
Initial guesses
Several initial guesses (provided, close to the solution, arbitrary) can later become a dimension of the configurations (#306).
What else stays hard
core (consistency, metadata, API, Problem), structure checks such as solution(model, pb) (#329).
Scope
- Replace
*/test_solve(and the GPU solves) by the "does not break" part; turncron/*/test_convergenceinto the convergence part; add theconvergenceswitch (test argument, labeltest convergence). - Result statuses and report format (values, gap if a reference exists, change in the cron).
- Revise
draftstatuses that are only convergence gaps. - Document in
test/README.md. - Then adapt test: GPU tests of the OptimalControl modelings (#330) #354 (no tolerance tuning on the GPU).
Done when
- No test fails on non-convergence; an exception or a non-equivalence still fails.
- Convergence off by default locally and on pull requests, on in the cron, on demand with the argument or the label.
- Reports give status, values and gap to the reference when it exists; the cron shows the change from the previous run.
-
draftonly for unfinished or non-equivalent models. - Documented in
test/README.md.
🤖 Generated with Claude Code
Activity
test: GPU solves run a few iterations and report convergence instead …
Merge pull request #363 from control-toolbox/test/360-report-convergence
ci: the cron report lists the convergence results and flags what got …
Merge pull request #365 from control-toolbox/test/360-cron-convergence
ocots commentedon Nov 5, 2026
Done in #363 and #365 (merged).
What was done
suite/<modeling>/test_smoke.jland the GPUtest_smoke_*: every problem of every modeling, every configuration, gridSMOKE_GRID = 10,SMOKE_MAX_ITER = 3, finite objective. Measured first: all 85 problem × modeling × configuration combinations run at grids 5, 10, 20 and 1, 3, 5 iterations (765 runs).comparisonunchanged (at the initial guess); the NLP equivalence is [Paper] JuMP models for Euler, trapeze and midpoint, and NLP equivalence tests #284.cron/<modeling>/test_convergence.jlrecord one result per solve (TestStatus.result!: solver status and reason, objective, iterations, time, gap to:reference_objectiveonly when the problem has one); results table in the report and<report>-results.tsv. On demand withPkg.test(; test_args=["convergence"])and the labeltest convergence; always in the cron.new,lost,gained, objective change above 1e-4), the history is kept instate.jsononresults, and the summary issuecron-convergenceflags what got worse; a convergence test never makes a draft ready.:draftmeans not finished or not equivalent;mountain_carscalarized became:valid. Documented intest/README.mdand.github/workflows/README.md.Evidence
kkt16 min 24 s (20 min 11 s before, with 100 iterations per GPU solve), and a run withtest convergence(JuMP 22, OptimalControl 44, scalarized 19 solves reported, green). Locally the smoke tests did not save time on the CPU: the cost is the compilation per problem and configuration, not the iterations.Cron JuMP) on ci: cron report of the convergence results (#360, part 2) #365: the convergence table onresults(44 rows), then onelost(moonlanderon Windows,ITERATION_LIMIT) reported in [Cron] cron-convergence #367 without failing.Left
known_failure/cron-capabilitiesintools/cron/report.pyis probably obsolete with this issue (nothing fails on convergence any more); kept, for the maintainer to decide.test(mtk): compare with OptimalControl without discretizing, shared t…