Skip to content

[Dev] Precompile the problems so that users and tests do not recompile them every time #373

Closed as not planned

Description

@ocots

Part of #310. Not for now: to do before the collection grows (the roadmap aims at many more problems, #179, #287, #308).

Context

Solving a problem of the OptimalControl modelings costs mostly compilation, and it is specific to each problem: each @def creates its own functions, and the modeler (ADNLP, Exa) and the solver specialize on them. Measured in #360 (phase A, macOS, Julia 1.12): the first pass over the 85 problem × modeling × configuration combinations took 211 s; the eight next passes took 0.1 to 0.2 s each. About 2.5 s per combination, almost all compilation. JuMP models are interpreted and do not pay it (whole JuMP suite: 24 s). This is also why the smoke tests of #363 did not save CPU time: 3 iterations or convergence cost about the same.

At 200 problems × 3 configurations this would be about 25 min of compilation per run (extrapolated, not measured), for the tests and for a user who solves many problems (benchmarks).

Options

  1. Precompilation in the package (PrecompileTools), recommended: a @compile_workload in the extensions OptimalControlModels and OptimalControlScalarizedModels builds each problem and runs a few iterations of the usual configurations. Since Julia 1.9 the native code is stored in the package images in the depot, including the specializations of ADNLP, Exa and the solvers on our functions. A user pays once per install or update; the CI keeps the images in its cache (julia-actions/cache, made reliable by ci: delete the Actions caches of closed pull requests #336 and ci: keep the test environments in GitHub Actions so that the depot cache holds their packages #340).
    • Drawback: a long install for everyone. A switch is needed, e.g. a preference (Preferences.jl) to precompile all problems (on in the CI and for benchmarks), off or limited to a list by default.
    • Limit: an extension precompiles only for the packages loaded together; the solvers and modelers must be triggers or loaded in the workload.
  2. A system image (PackageCompiler) as a CI artifact: the largest gain at startup, but one image per Manifest, OS and Julia version, rebuilt at every dependency change; CI only.
  3. Less specialization: not realistic, automatic differentiation relies on it.
  4. GPU: CUDA kernels are compiled at run time and are not cached the same way; option 1 does not help much there.

Scope

  • Measure the compilation per problem and configuration, and what a warm cache saves (locally and in the CI).
  • Prototype @compile_workload on a few problems; check the images are reused by a second session and by the next CI run.
  • Decide the default for a user (switch, list, or nothing) and document it.

Done when

  • Measurements: compilation per problem and configuration, cold and warm.
  • A prototype on a few problems, reused across sessions and CI runs, with its install-time cost.
  • A decision on the default for users, documented.

🤖 Generated with Claude Code

Activity

  1. self-assigned this
    on Nov 5, 2026
  2. ocots commented on Nov 5, 2026

    @ocots
    MemberAuthor

    Measurements: where the time goes

    Full local test suite (macOS, after #363), package precompilation excluded:

    Environment Test time Smoke solves
    JuMP 24 s (748 checks) 22 problems × 1 configuration (Ipopt)
    OptimalControl 9 min 24 s 22 problems × 2 configurations (ADNLP + Ipopt, Exa + MadNLP)
    OptimalControlScalarized 5 min 35 s 19 problems × 1 configuration (Exa + MadNLP)
    comparison 8 min 42 s builds JuMP and OptimalControl models for every problem
    Total 25 min 18 s

    Per solve: about 1 s with JuMP, about 10 s with OptimalControl per problem and configuration.

    During the smoke measurements for #360, the first pass over the 85 combinations (problem × modeling × configuration) took 211 s; later passes took 0.1–0.2 s per combination. Almost all the time is compilation, not solving. This is also why the smoke tests (grid 10, 3 iterations) did not shorten the CPU environments: the solve was already the cheap part.

    Why JuMP does not pay this cost

    • JuMP builds a data structure: a MathOptInterface expression graph. Derivatives come from MOI's reverse-mode AD, which walks the graph. That code is the same for every problem and is compiled once; Ipopt is a C library. A new problem is new data, not new code.
    • OptimalControl: @def generates Julia functions (dynamics, constraints, cost), each with its own type. Everything downstream specializes on these types and is compiled per problem:
      • the discretization builds functions specialized on the dynamics;
      • ADNLP generates derivative code for these functions;
      • ExaModels encodes the expressions in types and compiles kernels for each;
      • MadNLP, written in Julia, is parameterized by the model type, so part of the interior-point algorithm is probably recompiled for each Exa model (not measured separately).

    Trade-off and consequences

    • Specialized code is fast to evaluate and lets Exa run on the GPU; JuMP pays for walking the graph at every evaluation. The compilation cost is paid again for every new problem and every new session: this is what precompiling the problems would amortize.
    • For benchmarks, time the solve after compilation (the convergence tests already solve twice for this reason); otherwise the comparison mostly measures the Julia compiler against JuMP's interpreter.
    • Before choosing an approach here, measure the split between @def/discretization, AD, and solver compilation (for example with SnoopCompile), to know which layer a workload would need to cover.
  3. ocots commented on Nov 7, 2026

    @ocots
    MemberAuthor

    Taken by an agent; plan to follow, then a draft PR (ct-conventions v0.10.0, ct-agents v0.4.0).

  4. ocots commented on Nov 7, 2026

    @ocots
    MemberAuthor

    Written by Claude (Claude Code) at the request of ocots.

    Phase A: where the compilation goes, stage by stage

    macOS, Julia 1.12.7, main at 5e14a52. Grid 10, trapezoidal rule, 3 iterations. Two fresh sessions over the same three problems, in opposite orders: the first problem of a session also pays what is shared, the next ones only what is specific to them. @timed, time and compilation time in seconds. Script: stages.jl (to be committed with the prototype).

    Session 1, then session 2 (reversed order), time (compile is within 0.05 s of it everywhere):

    Stage vanderpol (1st) ducted_fan insurance insurance (1st) ducted_fan vanderpol
    @def (build) 1.00 1.01 0.66 2.35 0.60 0.12
    discretize 0.92 0.21 0.43 0.91 0.24 0.20
    ADNLP nlp_model 5.01 3.86 4.39 6.72 3.64 3.22
    ADNLP first evaluation 2.06 1.81 1.86 2.62 1.77 1.29
    ADNLP + Ipopt, solve 6.90 0.99 0.77 6.87 0.77 0.71
    Exa nlp_model 2.41 2.39 1.86 2.91 1.81 0.93
    Exa first evaluation 0.59 1.94 0.94 0.99 1.88 0.58
    Exa + MadNLP, solve 5.41 5.98 4.90 5.80 5.58 4.57
    second solve (any) 0.00 0.00 0.00 0.00 0.00 0.00

    Loading the packages: 4 to 6 s, almost no compilation (already in the package images).

    Reading

    • Everything is compilation: a second solve takes 0.00 s.
    • Shared, paid once per session: the Ipopt solve path (about 6 s, then under 1 s per problem), and part of @def and discretize.
    • Paid again for every problem, about 13 s on average:
      • ADNLP: nlp_model and first evaluation, 4.5 to 6 s (AD of the problem's functions);
      • Exa: nlp_model and first evaluation, 2.5 to 4 s;
      • MadNLP: 4.6 to 6 s per problem, never shared. MadNLP is parameterized by the type of the Exa model, which is new for every problem, so the interior-point method is compiled again for each one. This is the largest single per-problem item, ahead of the AD.
    • Consequence for option 1 (PrecompileTools): a workload pays off only for the problems it runs (the costly part is specific to each one), and it must cover the solve with MadNLP. MadNLP is not a trigger of the extension, so this needs an extension with two triggers (OptimalControl + MadNLP, and the same with NLPModelsIpopt for the Ipopt part). Whether the specializations of MadNLP on our types are kept in our package image, and what that costs at load time, is what the prototype (phase B) has to show.
  5. ocots commented on Nov 7, 2026

    @ocots
    MemberAuthor

    Written by Claude (Claude Code) at the request of ocots.

    Phase B: prototype (draft PR #404)

    Two extensions run a PrecompileTools.@compile_workload that solves vanderpol, ducted_fan, insurance (grid 10, 1 iteration): PrecompileIpopt (ADNLP + Ipopt), PrecompileMadNLP (Exa + MadNLP).

    Triggers. With OptimalControl + the solver only, the workload fails: the strategies of CTSolvers for MadNLP and Ipopt are extensions of CTSolvers (CTSolversMadNLP = ["MadNLP", "NLPModels"], CTSolversIpopt = ["NLPModels", "NLPModelsIpopt"]), and Julia loads them during the precompilation of our extension only if their packages are among its triggers. Adding NLPModels was not enough; adding CTSolvers as well works. Final triggers: ["OptimalControl", "CTSolvers", "NLPModels", "NLPModelsIpopt"] and ["OptimalControl", "CTSolvers", "MadNLP", "NLPModels"]. They are all in the Manifest of any environment with OptimalControl and the solver, so a user loads nothing more.

    Fresh session after precompilation (same machine and script as phase A)

    Stage vanderpol ducted_fan insurance rocket (not in the workload)
    @def, discretize 0.00 0.00 0.00 0.56
    ADNLP nlp_model + first evaluation 0.04 0.01 0.01 4.46
    ADNLP + Ipopt, solve 0.29 0.09 0.09 0.95
    Exa nlp_model + first evaluation 0.28 0.77 0.48 2.28
    Exa + MadNLP, solve 0.17 0.16 0.14 5.55
    Total 0.78 1.03 0.72 13.8

    Phase A, the same problems not first in their session: about 13 to 15 s each. Second session, rocket first, then insurance: rocket 14.3 s (the shared Ipopt path is now in the image too: 1.0 s instead of 6.9 s), insurance 0.73 s.

    • Gain: about 13 s per precompiled problem, down to under 1 s. A problem not in the workload pays as before.
    • Loading the packages: 4.2 to 5.9 s, as before.
    • Cost at install: 38 s (PrecompileIpopt) and 42 s (PrecompileMadNLP), in parallel: 43 s for 3 problems, about 13 s per problem and configuration.
    • Size of the images: 73 MB (PrecompileIpopt) and 65 MB (PrecompileMadNLP) for 3 problems, about 23 MB per problem and configuration.

    Extrapolation to every problem (not measured)

    22 problems × 2 configurations + 19 scalarized × 1: about 63 problem-configurations, so about 13 min of precompilation (less on several cores: the extensions build in parallel, but each one is sequential) and about 1.4 GB of images per Julia version. The size is the limiting factor: it goes into every user's depot, and into the CI cache (10 GB per repository, one copy per pull request, #336, #340).

    To decide (phase C)

    • The default for users: precompile nothing (the current behaviour), a short list, or everything.
    • The CI: everything (fast tests, about 1.4 GB of cache), or nothing (the current time).
  6. ocots commented on Nov 7, 2026

    @ocots
    MemberAuthor

    Written by Claude (Claude Code) at the request of ocots.

    Closed as not planned, by decision of ocots, after the measurements above (phase A) and the prototype (phase B, #404, closed). Precompiling with PrecompileTools works: under 1 s per precompiled problem in a fresh session, against 13 to 15 s. But it costs about 13 s of precompilation and 23 MB of image per problem and configuration, about 13 min and 1.4 GB for the whole collection, in every user's depot and in the CI cache. The gain is mainly for the CI, whose time will be reduced instead by testing only what a pull request changes (issue to follow).

    Kept for later, if the question comes back: the measurement script test/extras/compile_time.jl and the extensions on the branch feat/373-precompile; the triggers needed (CTSolvers and NLPModels with the solver, so that the solver strategies of CTSolvers load during the precompilation).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions