Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions test/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,40 @@ Only the first two can fail a run (#360).
exception still fails. This is what helps tune an initial guess: look at the results and
adapt.

### What each modeling tests

The tests every modeling shares are written once, in `common/modeling_tests.jl`, and given the
solves of the modeling by an adapter (`common/jump_tests.jl`, `optimalcontrol_tests.jl`,
`modelingtoolkit_tests.jl`); the file of a modeling is a wrapper (#449 T4). The other tests are
specific, for the reason given.

| Test | JuMP | OptimalControl | OptimalControl, scalarized | ModelingToolkit |
| :--- | :---: | :---: | :---: | :---: |
| `test_draft` (shared) | ✓ | ✓ | ✓ | ✓ |
| `test_smoke` (shared) | ✓ | ✓ | ✓ | ✓ |
| `cron/…/test_convergence` (shared) | ✓ | ✓ | ✓ | ✓ |
| `robustness/…/test_robustness` (shared) | ✓ | ✓ | — ¹ | — ¹ |
| `test_problem` (shared) | — ² | ✓ | ✓ | ✓ |
| `test_methods` | ✓ | ✓ | ✓ | ✓ |
| `test_modelings` (the continuous problem) | in `comparison` | reference | in `comparison` | ✓ ³ |
| `test_features` (features, components) | components, in `comparison` | ✓ ⁴ | in `comparison` | components, in `test_modelings` |
| `test_reference_solution` | — ⁵ | ✓ ⁵ | — ⁵ | — ⁵ |
| `test_contract`, `test_build`, `test_tableaux`, `test_signs` | ✓ ⁶ | — | — | — |

1. The robustness of the initial guess is reported for one modeling of each kind of
transcription, monthly: JuMP and OptimalControl. The function is shared, so another
modeling needs a wrapper and a cron.
2. A JuMP problem function returns the discretized model, not a `Problem`.
3. Against the OptimalControl model, in its own environment, so that `comparison` stays free
of ModelingToolkit.
4. The features are computed from the OptimalControl model (`OptimalControlProblems.features`):
checked against the functions of OptimalControl; the scalarized model has the same
features, and the other modelings the same components and dimensions.
5. The analytic solutions are properties of the continuous problem: checked once, on the
OptimalControl model.
6. The JuMP discretization is written by this package (see "What is compared, and what is
not").

The reference objectives are computed from these results, never typed: see
[`tools/references/README.md`](../tools/references/README.md). `core/test_consistency.jl`
fails when a reference no longer matches the default grid or parameters of its problem, or
Expand Down
37 changes: 37 additions & 0 deletions test/common/jump_tests.jl
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# The tests of the JuMP modeling, written once in `common/modeling_tests.jl` (#449 T4) and
# given the JuMP solves (`Main.SolveJuMP`): the draft warning, the smoke solves, the
# convergence and robustness reports. A JuMP problem function returns the discretized model,
# built on its grid: the adapter builds it (`build`) before it is solved. A test file of the
# modeling is a wrapper (TST1).
isdefined(Main, :SolveJuMP) || include(joinpath(@__DIR__, "solve_jump.jl"))
isdefined(Main, :ModelingTests) || include(joinpath(@__DIR__, "modeling_tests.jl"))

module JuMPTests

using JuMP: JuMP
using OptimalControlProblems: OptimalControlProblems

const OCP = OptimalControlProblems

"""What the shared tests need of the JuMP solves (see `common/modeling_tests.jl`)."""
function adapter()
S = Main.SolveJuMP
return (
build=(problem, grid_size) -> getfield(OCP, problem)(OCP.JuMPBackend(); grid_size),
solve=(model, config; grid_size, kwargs...) ->
S.solve!(model; solver=config.solver, kwargs...),
objective=model -> JuMP.has_values(model) ? JuMP.objective_value(model) : NaN,
settings=S.settings,
scheme=config -> Main.TestConfigs.SCHEME,
outcome=S.outcome,
trajectory=(model, problem) -> Main.TestStatus.trajectory(model),
reference=(problem, grid_size, scheme) -> nothing,
)
end

test_draft() = Main.ModelingTests.test_draft(OCP.JuMPBackend(), adapter())
test_smoke() = Main.ModelingTests.test_smoke(OCP.JuMPBackend(), adapter())
test_convergence() = Main.ModelingTests.test_convergence(OCP.JuMPBackend(), adapter())
test_robustness() = Main.ModelingTests.test_robustness(OCP.JuMPBackend(), adapter())

end # module JuMPTests
91 changes: 83 additions & 8 deletions test/common/modeling_tests.jl
Original file line number Diff line number Diff line change
@@ -1,15 +1,21 @@
# The tests every modeling that returns a `Problem` shares (#441, #439): the API of a problem,
# the smoke solves, the convergence report. Written once and parameterized by the backend and
# by an adapter, a named tuple of what differs between the modelings:
# The tests every modeling shares (#441, #439, #449 T4): the draft warning, the smoke solves,
# the convergence and robustness reports, and for the modelings that return a `Problem` its
# API. Written once and parameterized by the backend and by an adapter, a named tuple of what
# differs between the modelings:
#
# - `solve(pb, config; grid_size[, max_iter])`: a solution of `pb` with `config`;
# - `build(problem, grid_size)` (optional): what a problem function gives, solved by `solve`;
# by default `problem(backend)`, the `Problem`, which takes no grid (JuMP: the model, built
# on the grid);
# - `solve(pb, config; grid_size[, max_iter, options])`: a solution of `pb` with `config`;
# - `objective(sol)`: its objective;
# - `settings(config)`: `(modeler, solver, options)`, as `TestStatus.result!` records them;
# - `settings(config[; options])`: `(modeler, solver, options)`, as `TestStatus.result!`
# records them;
# - `scheme(config)`: the discretization, for the report;
# - `outcome(sol)`: `(converged, reason, objective, iterations, violation)`;
# - `trajectory(sol, pb)`: the trajectory for the documentation, or `nothing`;
# - `reference(problem, grid_size, scheme)`: the objective the gap of the report is computed
# against; `nothing` for the reference objective of the problem for this discretization.
# against; `nothing` for the reference objective of the problem for this discretization;
# - `warmup(pb, config)` (optional): a cheaper solve that compiles before the timed one.
#
# The configurations come from `Main.TestConfigs.CONFIGS.<modeling>`; the names of the test
# sets and of the lines of the report are those of the modeling.
Expand All @@ -28,6 +34,34 @@ modeling(backend) = Main.TestStatus.modeling(backend)
"""Configurations to solve with for `backend`, see `Main.TestConfigs.CONFIGS`."""
configs(backend) = getfield(Main.TestConfigs.CONFIGS, modeling(backend))

"""What the problem function of `problem` gives for `backend`, to solve on `grid_size` steps."""
function build(adapter, backend, problem::Symbol, grid_size::Int)
haskey(adapter, :build) && return adapter.build(problem, grid_size)
return getfield(OCP, problem)(backend)
end

"""
A problem that is a draft in the modeling of `backend` warns when it is built, and one that
is valid does not: the call `_warn_if_draft` of every model file (checked by the linter).
"""
function test_draft(backend, adapter)
name = modeling(backend)
Test.@testset "$name draft warning" verbose = VERBOSE showtiming = SHOWTIMING begin
problem = first(OCP.problems(backend))
data = OCP.problem_metadata(problem)
original = data[:status]
grid_size = Main.TestConfigs.SMOKE_GRID
Test.@test_logs build(adapter, backend, problem, grid_size)
data[:status] = merge(original, NamedTuple{(name,)}((:draft,)))
try
Test.@test_logs (:warn, Regex("$problem is a draft for $name")) match_mode =
:any build(adapter, backend, problem, grid_size)
finally
data[:status] = original
end
end
end

"""
`params` with its last nonzero numerical value changed by a relative 1e-6, small enough to
keep every bound of the problem ordered; `nothing` if there is no such value.
Expand Down Expand Up @@ -102,8 +136,8 @@ function test_smoke(backend, adapter)
Test.@testset "$(Main.TestConfigs.label(config))" begin
test = "test_smoke $(Main.TestConfigs.label(config))"
Main.TestStatus.for_each_problem(test, backend) do problem
pb = getfield(OCP, problem)(backend)
o = Main.TestConfigs
pb = build(adapter, backend, problem, o.SMOKE_GRID)
sol = adapter.solve(
pb, config; grid_size=o.SMOKE_GRID, max_iter=o.SMOKE_MAX_ITER
)
Expand All @@ -127,8 +161,8 @@ function test_convergence(backend, adapter)
Test.@testset "$(Main.TestConfigs.label(config))" begin
test = "test_convergence $(Main.TestConfigs.label(config))"
Main.TestStatus.for_each_problem(test, backend) do problem
pb = getfield(OCP, problem)(backend)
N = OCP.grid_size_data(problem, backend)
pb = build(adapter, backend, problem, N)
scheme = adapter.scheme(config)
warmup = get(adapter, :warmup, nothing)
if isnothing(warmup)
Expand Down Expand Up @@ -158,4 +192,45 @@ function test_convergence(backend, adapter)
end
end

"""
Robustness of the initial guess (#378): every problem solved on the default grid of the
modeling under the solver options of `Main.TestConfigs.ROBUSTNESS`, in each configuration.
Reported, not tested (#360): which known solution each solve reached is in the report
(`solution`); an exception still fails.
"""
function test_robustness(backend, adapter)
C = Main.TestConfigs
name = modeling(backend)
Test.@testset "$name robustness report (default grid)" verbose = VERBOSE showtiming =
SHOWTIMING begin
for config in configs(backend), options in getfield(C.ROBUSTNESS, config.solver)
label = C.label(merge(config, options))
Test.@testset "$label" begin
test = "test_robustness $label"
Main.TestStatus.for_each_problem(test, backend) do problem
N = OCP.grid_size_data(problem, backend)
pb = build(adapter, backend, problem, N)
sol = nothing
time = @elapsed sol = adapter.solve(
pb, config; grid_size=N, options=options
)
Main.TestStatus.result!(
test,
problem,
backend,
label;
adapter.settings(config; options=options)...,
grid_size=N,
scheme=adapter.scheme(config),
adapter.outcome(sol)...,
time=time,
reference=adapter.reference(problem, N, adapter.scheme(config)),
)
Test.@test true
end
end
end
end
end

end # module ModelingTests
3 changes: 3 additions & 0 deletions test/common/modelingtoolkit_tests.jl
Original file line number Diff line number Diff line change
Expand Up @@ -73,4 +73,7 @@ test_smoke(backend) = Main.ModelingTests.test_smoke(backend, adapter())
"""Every problem solved on its default grid, and reported, not tested (#360)."""
test_convergence(backend) = Main.ModelingTests.test_convergence(backend, adapter())

"""A draft warns when it is built, a valid problem does not."""
test_draft(backend) = Main.ModelingTests.test_draft(backend, adapter())

end # module ModelingToolkitTests
11 changes: 9 additions & 2 deletions test/common/optimalcontrol_tests.jl
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# The tests of the OptimalControl modelings (OptimalControl and OptimalControlScalarized), written
# once and parameterized by the backend (#441): the API of a problem, the smoke solves, the
# convergence report. A test file of a modeling is a wrapper that gives its backend (TST1).
# once and parameterized by the backend (#441): the API of a problem, the draft warning, the
# smoke solves, the convergence and robustness reports. A test file of a modeling is a wrapper
# that gives its backend (TST1).
# The tests themselves are shared with every modeling that returns a `Problem`
# (`common/modeling_tests.jl`, #439); this module gives them the OptimalControl solves
# (`Main.SolveOptimalControl`).
Expand Down Expand Up @@ -43,4 +44,10 @@ Every problem solved on its default grid in each configuration, and reported, no
"""
test_convergence(backend) = Main.ModelingTests.test_convergence(backend, adapter())

"""A draft warns when it is built, a valid problem does not."""
test_draft(backend) = Main.ModelingTests.test_draft(backend, adapter())

"""The robustness report of the initial guess (#378), on the default grid of the modeling."""
test_robustness(backend) = Main.ModelingTests.test_robustness(backend, adapter())

end # module OptimalControlTests
47 changes: 4 additions & 43 deletions test/cron/JuMP/test_convergence.jl
Original file line number Diff line number Diff line change
@@ -1,49 +1,10 @@
isdefined(Main, :SolveJuMP) ||
include(joinpath(@__DIR__, "..", "..", "common", "solve_jump.jl"))
isdefined(Main, :JuMPTests) ||
include(joinpath(@__DIR__, "..", "..", "common", "jump_tests.jl"))

module TestConvergenceJuMP

using Test: Test
using JuMP: JuMP
using OptimalControlProblems: OptimalControlProblems

const VERBOSE = isdefined(Main, :TestOptions) ? Main.TestOptions.VERBOSE : true
const SHOWTIMING = isdefined(Main, :TestOptions) ? Main.TestOptions.SHOWTIMING : true

function test_convergence()
Test.@testset "JuMP convergence report (default grid)" verbose = VERBOSE showtiming =
SHOWTIMING begin
for config in Main.TestConfigs.CONFIGS.JuMP
Test.@testset "$(Main.TestConfigs.label(config))" begin
test = "test_convergence $(Main.TestConfigs.label(config))"
Main.TestStatus.for_each_problem(
test, OptimalControlProblems.JuMPBackend()
) do problem
build = getfield(OptimalControlProblems, problem)
model = build(OptimalControlProblems.JuMPBackend())
Main.SolveJuMP.solve!(model; solver=config.solver)
time = @elapsed Main.SolveJuMP.solve!(model; solver=config.solver) # warm
# convergence is reported, not tested (#360); an exception still fails
Main.TestStatus.result!(
test,
problem,
OptimalControlProblems.JuMPBackend(),
Main.TestConfigs.label(config);
Main.SolveJuMP.settings(config)...,
grid_size=OptimalControlProblems.grid_size_data(
problem, OptimalControlProblems.JuMPBackend()
),
scheme=Main.TestConfigs.SCHEME,
Main.SolveJuMP.outcome(model)...,
time=time,
trajectory=Main.TestStatus.trajectory(model),
)
Test.@test true
end
end
end
end
end
# The convergence report on the default grid of the modeling, written once in `common/modeling_tests.jl` (#449 T4).
test_convergence() = Main.JuMPTests.test_convergence()

end # module TestConvergenceJuMP

Expand Down
47 changes: 4 additions & 43 deletions test/robustness/JuMP/test_robustness.jl
Original file line number Diff line number Diff line change
@@ -1,49 +1,10 @@
isdefined(Main, :SolveJuMP) ||
include(joinpath(@__DIR__, "..", "..", "common", "solve_jump.jl"))
isdefined(Main, :JuMPTests) ||
include(joinpath(@__DIR__, "..", "..", "common", "jump_tests.jl"))

module TestRobustnessJuMP

using Test: Test
using JuMP: JuMP
using OptimalControlProblems: OptimalControlProblems

const VERBOSE = isdefined(Main, :TestOptions) ? Main.TestOptions.VERBOSE : true
const SHOWTIMING = isdefined(Main, :TestOptions) ? Main.TestOptions.SHOWTIMING : true

# Robustness of the initial guess (#378): every problem solved on its default grid under the
# solver options of `TestConfigs.ROBUSTNESS`. Reported, not tested (#360): which known
# solution each solve reached is in the report (`solution`); an exception still fails.
function test_robustness()
C = Main.TestConfigs
backend = OptimalControlProblems.JuMPBackend()
Test.@testset "JuMP robustness report (default grid)" verbose = VERBOSE showtiming =
SHOWTIMING begin
for config in C.CONFIGS.JuMP, options in getfield(C.ROBUSTNESS, config.solver)
label = C.label(merge(config, options))
Test.@testset "$label" begin
test = "test_robustness $label"
Main.TestStatus.for_each_problem(test, backend) do problem
model = getfield(OptimalControlProblems, problem)(backend)
time = @elapsed Main.SolveJuMP.solve!(
model; solver=config.solver, options=options
)
Main.TestStatus.result!(
test,
problem,
backend,
label;
Main.SolveJuMP.settings(config; options=options)...,
grid_size=OptimalControlProblems.grid_size_data(problem, backend),
scheme=C.SCHEME,
Main.SolveJuMP.outcome(model)...,
time=time,
)
Test.@test true
end
end
end
end
end
# The robustness report of the initial guess (#378), written once in `common/modeling_tests.jl` (#449 T4).
test_robustness() = Main.JuMPTests.test_robustness()

end # module TestRobustnessJuMP

Expand Down
Loading
Loading