Repository navigation
feat(models): ModelingToolkit convergence cron, trajectories, and documentation (#439) - #444
Conversation
Add dynamic_trajectory and trajectory functions in SolveModelingToolkit to reconstruct state and control trajectories from Corleone solutions for the documentation results pipeline (#314). Add unit tests in test_methods.jl.
…439) Add test/cron/ModelingToolkit/test_convergence.jl to record convergence and trajectories for ModelingToolkit problems against analytic and discretized references. Enable the Saturday cron schedule in CronModelingToolkit.yml and update cron report scope description.
Add ModelingToolkit to docs Project.toml, make.jl, api_reference.jl, and update the ModelingToolkit adapter in modelings.jl with Corleone direct shooting explanation and snippet. Fix Documenter cross-references to OptimalControlProblems.Problems.Problem.
Critical review (agent self-review)Applied: ct-conventions v0.11.0, ct-agents v0.4.0. CI ( Blocking
Should fix
Minor
SummaryThe core of the issue (trajectory extraction, cron, schedule) works and is tested, but items 1–2 must be fixed before review, and 3 is a maintainability risk. Proposed next step: address them in separate commits on this branch. |
Brings in #446 (JuMP and comparison tests rationalized), #447 (label dispatch) and #448 (statuses permission for the crons). No conflict. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BSmPZipCYwgepwKpMP9sep
ocots
left a comment
There was a problem hiding this comment.
Critical review after merging main (#446, #447, #448)
Applied: ct-conventions v0.11.0, ct-agents v0.4.0. main was merged into this branch (df5f3b3, no conflict). Not run locally (no Julia in this environment): the CI must run again on the merge, and the "Verification" numbers of the description predate #446.
The self-review of 10 Nov is still entirely open: no commit since. Its blocking items stand and I confirm them by reading the code: the docs Manifest commits path = "/Users/ocots/…" and OptimalControlProblems was added to docs/Project.toml (against SITE18: the package comes from LOAD_PATH, see docs/make.jl); the snippet hardcodes (0.0, 1.0) and does not discretize the path constraints it says it discretizes; dynamic_trajectory duplicates prepare; timing includes compilation; violation = 0.0; .github/workflows/README.md still says "manual only". I do not repeat them below.
New: what this PR does not do that #446 now expects
#446 set the philosophy of the tests (test/README.md, "Three kinds of tests"). The ModelingToolkit environment does not follow it:
- No model equivalence for ModelingToolkit — "the most important part, since comparing models is fair only if they are the same problem".
suite/comparison/test_modelingschecks JuMP against both OptimalControl modelings at the continuous level; nothing checks that the ModelingToolkitSystemis the same problem as the OptimalControl model. The cron will publish ModelingToolkit results next to the others (and the docs saychecked_against=:OptimalControl) with no evidence that the problem is the same. → Add atest_modelingsin the ModelingToolkit environment (opt-in, so the comparison environment stays free of ModelingToolkit), reusingcommon/continuous.jl: evaluate the equations, the integrand, the boundary and path constraints of theSystemat the points ofContinuous.trajectoriesand compare withContinuous's OptimalControl functions. This needsOptimalControlintest/suite/ModelingToolkit/Project.toml. - Duplication instead of shared tests. #446 wrote the Problem API, smoke and convergence tests once (
common/optimalcontrol_tests.jl), parameterized by backend.suite/ModelingToolkit/test_problem.jlis a near copy (includingchanged),test_smoke.jland the new cron copy the loops. → Generalize the shared module to a modeling-agnostic one (model type check, solve module, outcome and trajectory as arguments), or at least sharechangedand the loops; the cron should become the same thin wrapper ascron/OptimalControl/test_convergence.jl. - Convergence is tested on every PR.
suite/ModelingToolkit/test_reference_solution.jl(from #442) assertssucceeded(res)and a 5 % gap: against #360 ("convergence is reported, not tested"), and 5 % is too loose to detect a wrong model. Its natural place is the cron (reported), while an equivalence check (point 1) catches model errors on every PR. - Initial guess ignored (inline): the solves start from
0.0, never frominitial_guess(pb), so the cron does not measure what the report is meant to measure (#377, #378). - Generic code tested on a named problem (inline): fine with one problem; once there are more, select with
Representative.selectoverproblems(ModelingToolkitBackend())andfor_each_problem(...; only=...)(#446).
Other
- The cron's grid (
grid_size_data= 500 forbryson_denham) is a collocation grid reused as the shooting control grid and the path-constraint grid; the PR gives no duration for a cron run. Please state it, or choose a grid for shooting explicitly. - Gaps are only
println-ed (inline); the trapezoidal reference is not a meaningful comparison for a shooting solve; only the analytic gap is. - The trajectory tests are structural only (inline): nothing checks the reconstructed values.
tools/cron/report.py:ModelingToolkitis absent fromDEFAULT_SCOPES, so the summary issues get no placeholder section for it before its first run; minor, but the cadence edit alone does not integrate the scope.docs/modelings.jl:checked_against=:OptimalControlis false today;packagesmissesOptimizationMOIandOrdinaryDiffEqTsit5(inline).
Verdict
Not ready to leave draft. Order I would suggest: (a) the self-review's blocking items (Manifest/Project, snippet); (b) one prepare used by solve, trajectory and initial guess; (c) the continuous equivalence test MTK vs OptimalControl; (d) align the suite and cron with the shared tests of #446; (e) re-run CI with test ModelingToolkit backend and report the cron duration.
Generated by Claude Code
|
|
||
| # Controls and initial values | ||
| ctrl_syms = filter(MTK.isinput, MTK.unknowns(sys)) | ||
| inits = [u => 0.0 for u in ctrl_syms] |
There was a problem hiding this comment.
The initial guess of the problem is never used: here and in prepare (l. 138) every control starts at 0.0, whatever initial_guess(pb) says. test_problem checks that the init of the metadata (#377) is recorded and that init= is honoured, but no solve reads it, so the convergence cron does not measure "does the initial guess lead to the best known solution", which is what the report is for (#378). JuMP and OptimalControl start from the init. → Use OCP.initial_guess(pb).control (a number, a vector or a function of t: evaluate it at the switching times), in one place.
Also, this function rebuilds the whole problem a second time (already in the self-review): if grid_size differs from the one of the solve, reconstruct(res.u) silently gives a wrong trajectory or throws, and the cron pays a second symbolic build at N = 500. Return the CorleoneDynamicOptProblem from prepare and reuse it.
Generated by Claude Code
| Main.TestStatus.for_each_problem(test, backend) do problem | ||
| pb = getfield(OptimalControlProblems, problem)(backend) | ||
| N = OptimalControlProblems.grid_size_data(problem) | ||
| time = @elapsed sol = S.solve( |
There was a problem hiding this comment.
Since #446 the convergence tests of a modeling are written once (common/optimalcontrol_tests.jl) and the cron files are thin wrappers. This file re-implements the loop with three divergences from it:
- the first (compiling) solve is timed, OptimalControl and JuMP time the second one;
N = grid_size_data(problem)is the collocation grid of the problem (500 forbryson_denham), reused here as 499 control switches and 500 discretized path constraints in single shooting with ForwardDiff: is that what we want to report, and how long does it take? Nothing in the PR says;- the gaps are
println-ed: they never reach the report (scheme = :shooting_singlehas no reference, soresult!records no gap), and comparing a shooting solve with the trapezoidal reference at N = 500 mixes two discretizations — only the gap to the analytic solution is meaningful. Put it in the report (e.g. a reference for the scheme, or a column), not in the log.
Generated by Claude Code
| res = solve(prob, Ipopt.Optimizer()) | ||
| res.objective""", | ||
| model_start=r"^\s*ModelingToolkit\.@named sys = ModelingToolkit\.System\(", | ||
| checked_against=:OptimalControl, |
There was a problem hiding this comment.
checked_against=:OptimalControl is false (from #440, but this PR publishes the modeling in the docs): by the header of this file it means "the modeling whose NLP this one is checked equal to on every pull request", and nothing compares the ModelingToolkit model with the OptimalControl one — suite/comparison/test_modelings (#446) is JuMP vs OptimalControl/scalarized only. Either add that check (see the review summary) or set nothing until it exists.
Also: packages omits OptimizationMOI and OrdinaryDiffEqTsit5, which the snippet loads.
Generated by Claude Code
| Test.@test isfinite(out.objective) | ||
|
|
||
| # 8. trajectory reconstruction | ||
| dyn_traj = Main.SolveModelingToolkit.dynamic_trajectory(res, pb; grid_size=10) |
There was a problem hiding this comment.
These checks are all structural (keys, lengths). Nothing checks that the reconstructed trajectory is the solution: e.g. that x₁(0), x₂(0) are the initial conditions, the final ones match the boundary constraints, and the control is the one of res.u. With a 2-iteration solve the values are meaningless anyway; a check on the converged solve of test_reference_solution (or against analytic_reference(problem).state at a loose tolerance) would actually test the reconstruction.
The control extension vals[min(i, length(vals))] assumes Corleone returns n - 1 control values; if it returns n, the code is still right, if it returns them on another grid it is silently wrong. Worth one assertion.
Also, the problem and its component names are hardcoded (bryson_denham, "x₁"): #446 never names a problem, it computes the selection from the catalogue (common/representative.jl). Fine with one problem today, but take the names from state_components(sys).
Generated by Claude Code
…stops at the model (#439) The asset Manifest committed a local path and docs/Project.toml listed the package, which the docs load from LOAD_PATH (SITE18). The snippet solved with a hardcoded time span and without the discretization of the path constraints the tests use: it now stops at the model, and checked_against is nothing since no check runs on every pull request. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BSmPZipCYwgepwKpMP9sep
…modelings (#439) common/modeling_tests.jl holds the tests of common/optimalcontrol_tests.jl (#441), parameterized by the backend and an adapter, the solves of the modeling, so that ModelingToolkit uses them too. result! takes the reference of the gap, by default the reference objective of the discretization. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BSmPZipCYwgepwKpMP9sep
…nd measure the violation (#439) prepare returns the Optimization problem and the Corleone one; solve returns both with the result, and the trajectory is read from the problem solved instead of a copy of prepare. The controls start from the initial guess of the problem (#377), not 0. The violation is the largest violation of the constraints and bounds of the NLP, no longer 0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BSmPZipCYwgepwKpMP9sep
…ests, thin cron (#439) test_modelings reads the System as the data of a JuMP problem and compares it with the OptimalControl model by Continuous.compare (#441). test_problem, test_smoke and the cron are wrappers of the shared tests; the cron times the compiled solve and records the gap to the analytic objective. test_reference_solution, which tested convergence on every run (#360), is removed. OptimalControl joins the ModelingToolkit environment for the comparison. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BSmPZipCYwgepwKpMP9sep
…he tests (#439) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BSmPZipCYwgepwKpMP9sep
…thing runs When a shared file runs every environment, the test arguments were empty, which runs every environment except the opt-in ones: the label `test ModelingToolkit backend` was lost, and the ModelingToolkit environment never ran on this pull request. The environments are now named, with the opt-in ones of the labels. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BSmPZipCYwgepwKpMP9sep
Update after the review (head
|
…ed (#439) A System does not say its time span when the cost has no integral (a Mayer cost): tspan reads it from the parameters of the problem, as the continuous comparison does. A test checks that the controls of the problem solved start from the initial guess of the problem (#377). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…odeling (#439) A better sample than bryson_denham alone: a Lagrange cost, state bounds with boundary conditions, a Mayer cost with state and control bounds. They pass the continuous comparison with OptimalControl and the smoke solve; the comparison fails, on the right check, when a parameter of the ModelingToolkit model is changed. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Warning
This description is obsolete. It was written before the critical review and the fixes that followed (merge of
main, tests aligned with #446, model equivalence test, shared tests, CI scope fix). The current state, the list of changes and the verification are in this comment. The original description is kept below as a trace; several of its points no longer hold (for instance the documentation snippet,docs/Project.toml, thereference_objective(..., scheme=:trapeze)comparison of the cron, and the verification numbers).Original description (obsolete)
Closes #439.
Part 3 of 3 of #308. Supports #314.
Applied conventions:
ct-conventions v0.11.0,ct-agents v0.4.0.Summary of Changes
Trajectory Reconstruction (
#314):SolveModelingToolkit.dynamic_trajectoryandSolveModelingToolkit.trajectoryintest/common/solve_modelingtoolkit.jlto reconstruct state and control trajectories from Corleone solutions.Dict{String, Vector{Float64}}()) because Corleone is a direct shooting solver without explicit costate dynamics.test/suite/ModelingToolkit/test_methods.jl.Convergence Cron & CI Schedule:
test/cron/ModelingToolkit/test_convergence.jllooping overMain.TestStatus.for_each_problemwithModelingToolkitBackend().analytic_reference(problem).objectiveandreference_objective(problem; grid_size=N, scheme=:trapeze).0 4 * * 6) in.github/workflows/CronModelingToolkit.ymland updated cadence intools/cron/report.py.Documentation Adapter & References:
docs/modelings.jlwith note explaining that Corleone enforces state path constraints on constraint grid points with piecewise-constant controls, along with a complete solve snippet.ModelingToolkittodocs/Project.toml,docs/make.jl, anddocs/api_reference.jl.OptimalControlProblems.Problems.Problem.Developer Guides:
ModelingToolkitModelsinext/README.md.ModelingToolkittest suite and cron intest/README.md.Verification
Pkg.test(test_args=["ModelingToolkit", "*methods*"]): 34 passed.Pkg.test(test_args=["ModelingToolkit", "cron"]): report and trajectories generated successfully.Pkg.test(test_args=["core"]): 1306 passed, 0 failed.python3 -m unittest discover -s tools/ci/test -p 'test_*.py': 47 passed.julia --project=docs docs/make.jl: documentation built cleanly with Vitepress.🤖 Generated with Claude Code