You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Dev] Precompile the problems so that users and tests do not recompile them every time #373
Part of #310. Not for now: to do before the collection grows (the roadmap aims at many more problems, #179, #287, #308).
Context
Solving a problem of the OptimalControl modelings costs mostly compilation, and it is specific to each problem: each @def creates its own functions, and the modeler (ADNLP, Exa) and the solver specialize on them. Measured in #360 (phase A, macOS, Julia 1.12): the first pass over the 85 problem × modeling × configuration combinations took 211 s; the eight next passes took 0.1 to 0.2 s each. About 2.5 s per combination, almost all compilation. JuMP models are interpreted and do not pay it (whole JuMP suite: 24 s). This is also why the smoke tests of #363 did not save CPU time: 3 iterations or convergence cost about the same.
At 200 problems × 3 configurations this would be about 25 min of compilation per run (extrapolated, not measured), for the tests and for a user who solves many problems (benchmarks).
Options
Precompilation in the package (PrecompileTools), recommended: a @compile_workload in the extensions OptimalControlModels and OptimalControlScalarizedModels builds each problem and runs a few iterations of the usual configurations. Since Julia 1.9 the native code is stored in the package images in the depot, including the specializations of ADNLP, Exa and the solvers on our functions. A user pays once per install or update; the CI keeps the images in its cache (julia-actions/cache, made reliable by ci: delete the Actions caches of closed pull requests #336 and ci: keep the test environments in GitHub Actions so that the depot cache holds their packages #340).
Drawback: a long install for everyone. A switch is needed, e.g. a preference (Preferences.jl) to precompile all problems (on in the CI and for benchmarks), off or limited to a list by default.
Limit: an extension precompiles only for the packages loaded together; the solvers and modelers must be triggers or loaded in the workload.
A system image (PackageCompiler) as a CI artifact: the largest gain at startup, but one image per Manifest, OS and Julia version, rebuilt at every dependency change; CI only.
Less specialization: not realistic, automatic differentiation relies on it.
GPU: CUDA kernels are compiled at run time and are not cached the same way; option 1 does not help much there.
Scope
Measure the compilation per problem and configuration, and what a warm cache saves (locally and in the CI).
Prototype @compile_workload on a few problems; check the images are reused by a second session and by the next CI run.
Decide the default for a user (switch, list, or nothing) and document it.
Done when
Measurements: compilation per problem and configuration, cold and warm.
A prototype on a few problems, reused across sessions and CI runs, with its install-time cost.
builds JuMP and OptimalControl models for every problem
Total
25 min 18 s
Per solve: about 1 s with JuMP, about 10 s with OptimalControl per problem and configuration.
During the smoke measurements for #360, the first pass over the 85 combinations (problem × modeling × configuration) took 211 s; later passes took 0.1–0.2 s per combination. Almost all the time is compilation, not solving. This is also why the smoke tests (grid 10, 3 iterations) did not shorten the CPU environments: the solve was already the cheap part.
Why JuMP does not pay this cost
JuMP builds a data structure: a MathOptInterface expression graph. Derivatives come from MOI's reverse-mode AD, which walks the graph. That code is the same for every problem and is compiled once; Ipopt is a C library. A new problem is new data, not new code.
OptimalControl: @def generates Julia functions (dynamics, constraints, cost), each with its own type. Everything downstream specializes on these types and is compiled per problem:
the discretization builds functions specialized on the dynamics;
ADNLP generates derivative code for these functions;
ExaModels encodes the expressions in types and compiles kernels for each;
MadNLP, written in Julia, is parameterized by the model type, so part of the interior-point algorithm is probably recompiled for each Exa model (not measured separately).
Trade-off and consequences
Specialized code is fast to evaluate and lets Exa run on the GPU; JuMP pays for walking the graph at every evaluation. The compilation cost is paid again for every new problem and every new session: this is what precompiling the problems would amortize.
For benchmarks, time the solve after compilation (the convergence tests already solve twice for this reason); otherwise the comparison mostly measures the Julia compiler against JuMP's interpreter.
Before choosing an approach here, measure the split between @def/discretization, AD, and solver compilation (for example with SnoopCompile), to know which layer a workload would need to cover.
Written by Claude (Claude Code) at the request of ocots.
Phase A: where the compilation goes, stage by stage
macOS, Julia 1.12.7, main at 5e14a52. Grid 10, trapezoidal rule, 3 iterations. Two fresh sessions over the same three problems, in opposite orders: the first problem of a session also pays what is shared, the next ones only what is specific to them. @timed, time and compilation time in seconds. Script: stages.jl (to be committed with the prototype).
Session 1, then session 2 (reversed order), time (compile is within 0.05 s of it everywhere):
Stage
vanderpol (1st)
ducted_fan
insurance
insurance (1st)
ducted_fan
vanderpol
@def (build)
1.00
1.01
0.66
2.35
0.60
0.12
discretize
0.92
0.21
0.43
0.91
0.24
0.20
ADNLP nlp_model
5.01
3.86
4.39
6.72
3.64
3.22
ADNLP first evaluation
2.06
1.81
1.86
2.62
1.77
1.29
ADNLP + Ipopt, solve
6.90
0.99
0.77
6.87
0.77
0.71
Exa nlp_model
2.41
2.39
1.86
2.91
1.81
0.93
Exa first evaluation
0.59
1.94
0.94
0.99
1.88
0.58
Exa + MadNLP, solve
5.41
5.98
4.90
5.80
5.58
4.57
second solve (any)
0.00
0.00
0.00
0.00
0.00
0.00
Loading the packages: 4 to 6 s, almost no compilation (already in the package images).
Reading
Everything is compilation: a second solve takes 0.00 s.
Shared, paid once per session: the Ipopt solve path (about 6 s, then under 1 s per problem), and part of @def and discretize.
Paid again for every problem, about 13 s on average:
ADNLP: nlp_model and first evaluation, 4.5 to 6 s (AD of the problem's functions);
Exa: nlp_model and first evaluation, 2.5 to 4 s;
MadNLP: 4.6 to 6 s per problem, never shared. MadNLP is parameterized by the type of the Exa model, which is new for every problem, so the interior-point method is compiled again for each one. This is the largest single per-problem item, ahead of the AD.
Consequence for option 1 (PrecompileTools): a workload pays off only for the problems it runs (the costly part is specific to each one), and it must cover the solve with MadNLP. MadNLP is not a trigger of the extension, so this needs an extension with two triggers (OptimalControl + MadNLP, and the same with NLPModelsIpopt for the Ipopt part). Whether the specializations of MadNLP on our types are kept in our package image, and what that costs at load time, is what the prototype (phase B) has to show.
Two extensions run a PrecompileTools.@compile_workload that solves vanderpol, ducted_fan, insurance (grid 10, 1 iteration): PrecompileIpopt (ADNLP + Ipopt), PrecompileMadNLP (Exa + MadNLP).
Triggers. With OptimalControl + the solver only, the workload fails: the strategies of CTSolvers for MadNLP and Ipopt are extensions of CTSolvers (CTSolversMadNLP = ["MadNLP", "NLPModels"], CTSolversIpopt = ["NLPModels", "NLPModelsIpopt"]), and Julia loads them during the precompilation of our extension only if their packages are among its triggers. Adding NLPModels was not enough; adding CTSolvers as well works. Final triggers: ["OptimalControl", "CTSolvers", "NLPModels", "NLPModelsIpopt"] and ["OptimalControl", "CTSolvers", "MadNLP", "NLPModels"]. They are all in the Manifest of any environment with OptimalControl and the solver, so a user loads nothing more.
Fresh session after precompilation (same machine and script as phase A)
Stage
vanderpol
ducted_fan
insurance
rocket (not in the workload)
@def, discretize
0.00
0.00
0.00
0.56
ADNLP nlp_model + first evaluation
0.04
0.01
0.01
4.46
ADNLP + Ipopt, solve
0.29
0.09
0.09
0.95
Exa nlp_model + first evaluation
0.28
0.77
0.48
2.28
Exa + MadNLP, solve
0.17
0.16
0.14
5.55
Total
0.78
1.03
0.72
13.8
Phase A, the same problems not first in their session: about 13 to 15 s each. Second session, rocket first, then insurance: rocket 14.3 s (the shared Ipopt path is now in the image too: 1.0 s instead of 6.9 s), insurance 0.73 s.
Gain: about 13 s per precompiled problem, down to under 1 s. A problem not in the workload pays as before.
Loading the packages: 4.2 to 5.9 s, as before.
Cost at install: 38 s (PrecompileIpopt) and 42 s (PrecompileMadNLP), in parallel: 43 s for 3 problems, about 13 s per problem and configuration.
Size of the images: 73 MB (PrecompileIpopt) and 65 MB (PrecompileMadNLP) for 3 problems, about 23 MB per problem and configuration.
Extrapolation to every problem (not measured)
22 problems × 2 configurations + 19 scalarized × 1: about 63 problem-configurations, so about 13 min of precompilation (less on several cores: the extensions build in parallel, but each one is sequential) and about 1.4 GB of images per Julia version. The size is the limiting factor: it goes into every user's depot, and into the CI cache (10 GB per repository, one copy per pull request, #336, #340).
To decide (phase C)
The default for users: precompile nothing (the current behaviour), a short list, or everything.
The CI: everything (fast tests, about 1.4 GB of cache), or nothing (the current time).
Written by Claude (Claude Code) at the request of ocots.
Closed as not planned, by decision of ocots, after the measurements above (phase A) and the prototype (phase B, #404, closed). Precompiling with PrecompileTools works: under 1 s per precompiled problem in a fresh session, against 13 to 15 s. But it costs about 13 s of precompilation and 23 MB of image per problem and configuration, about 13 min and 1.4 GB for the whole collection, in every user's depot and in the CI cache. The gain is mainly for the CI, whose time will be reduced instead by testing only what a pull request changes (issue to follow).
Kept for later, if the question comes back: the measurement script test/extras/compile_time.jl and the extensions on the branch feat/373-precompile; the triggers needed (CTSolvers and NLPModels with the solver, so that the solver strategies of CTSolvers load during the precompilation).
Activity
ocots commentedon Nov 5, 2026
Measurements: where the time goes
Full local test suite (macOS, after #363), package precompilation excluded:
Per solve: about 1 s with JuMP, about 10 s with OptimalControl per problem and configuration.
During the smoke measurements for #360, the first pass over the 85 combinations (problem × modeling × configuration) took 211 s; later passes took 0.1–0.2 s per combination. Almost all the time is compilation, not solving. This is also why the smoke tests (grid 10, 3 iterations) did not shorten the CPU environments: the solve was already the cheap part.
Why JuMP does not pay this cost
@defgenerates Julia functions (dynamics, constraints, cost), each with its own type. Everything downstream specializes on these types and is compiled per problem:Trade-off and consequences
@def/discretization, AD, and solver compilation (for example withSnoopCompile), to know which layer a workload would need to cover.ocots commentedon Nov 7, 2026
Taken by an agent; plan to follow, then a draft PR (ct-conventions v0.10.0, ct-agents v0.4.0).
ocots commentedon Nov 7, 2026
Written by Claude (Claude Code) at the request of ocots.
Phase A: where the compilation goes, stage by stage
macOS, Julia 1.12.7,
mainat 5e14a52. Grid 10, trapezoidal rule, 3 iterations. Two fresh sessions over the same three problems, in opposite orders: the first problem of a session also pays what is shared, the next ones only what is specific to them.@timed, time and compilation time in seconds. Script:stages.jl(to be committed with the prototype).Session 1, then session 2 (reversed order),
time(compileis within 0.05 s of it everywhere):@def(build)discretizenlp_modelnlp_modelLoading the packages: 4 to 6 s, almost no compilation (already in the package images).
Reading
@defanddiscretize.nlp_modeland first evaluation, 4.5 to 6 s (AD of the problem's functions);nlp_modeland first evaluation, 2.5 to 4 s;OptimalControl+MadNLP, and the same withNLPModelsIpoptfor the Ipopt part). Whether the specializations of MadNLP on our types are kept in our package image, and what that costs at load time, is what the prototype (phase B) has to show.ocots commentedon Nov 7, 2026
Written by Claude (Claude Code) at the request of ocots.
Phase B: prototype (draft PR #404)
Two extensions run a
PrecompileTools.@compile_workloadthat solvesvanderpol,ducted_fan,insurance(grid 10, 1 iteration):PrecompileIpopt(ADNLP + Ipopt),PrecompileMadNLP(Exa + MadNLP).Triggers. With
OptimalControl+ the solver only, the workload fails: the strategies of CTSolvers for MadNLP and Ipopt are extensions of CTSolvers (CTSolversMadNLP = ["MadNLP", "NLPModels"],CTSolversIpopt = ["NLPModels", "NLPModelsIpopt"]), and Julia loads them during the precompilation of our extension only if their packages are among its triggers. AddingNLPModelswas not enough; addingCTSolversas well works. Final triggers:["OptimalControl", "CTSolvers", "NLPModels", "NLPModelsIpopt"]and["OptimalControl", "CTSolvers", "MadNLP", "NLPModels"]. They are all in the Manifest of any environment with OptimalControl and the solver, so a user loads nothing more.Fresh session after precompilation (same machine and script as phase A)
@def,discretizenlp_model+ first evaluationnlp_model+ first evaluationPhase A, the same problems not first in their session: about 13 to 15 s each. Second session,
rocketfirst, theninsurance:rocket14.3 s (the shared Ipopt path is now in the image too: 1.0 s instead of 6.9 s),insurance0.73 s.PrecompileIpopt) and 42 s (PrecompileMadNLP), in parallel: 43 s for 3 problems, about 13 s per problem and configuration.PrecompileIpopt) and 65 MB (PrecompileMadNLP) for 3 problems, about 23 MB per problem and configuration.Extrapolation to every problem (not measured)
22 problems × 2 configurations + 19 scalarized × 1: about 63 problem-configurations, so about 13 min of precompilation (less on several cores: the extensions build in parallel, but each one is sequential) and about 1.4 GB of images per Julia version. The size is the limiting factor: it goes into every user's depot, and into the CI cache (10 GB per repository, one copy per pull request, #336, #340).
To decide (phase C)
ocots commentedon Nov 7, 2026
Written by Claude (Claude Code) at the request of ocots.
Closed as not planned, by decision of ocots, after the measurements above (phase A) and the prototype (phase B, #404, closed). Precompiling with PrecompileTools works: under 1 s per precompiled problem in a fresh session, against 13 to 15 s. But it costs about 13 s of precompilation and 23 MB of image per problem and configuration, about 13 min and 1.4 GB for the whole collection, in every user's depot and in the CI cache. The gain is mainly for the CI, whose time will be reduced instead by testing only what a pull request changes (issue to follow).
Kept for later, if the question comes back: the measurement script
test/extras/compile_time.jland the extensions on the branchfeat/373-precompile; the triggers needed (CTSolversandNLPModelswith the solver, so that the solver strategies of CTSolvers load during the precompilation).