Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions .github/workflows/CronModelingToolkit.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,12 @@ name: Cron ModelingToolkit

on:
schedule:
- cron: "0 4 * * 6"
# daily: a rotation (#435), each run tests the problems tested longest ago for an hour
- cron: "0 4 * * *"
workflow_dispatch:
inputs:
problems:
description: "Only these problems, comma-separated (e.g. beam): publishes their results, leaves the reports, the state and the issues"
description: "These problems first, comma-separated (e.g. beam), then the others while the budget lasts"
required: false
default: ''
type: string
Expand All @@ -26,5 +27,6 @@ jobs:
with:
environment: ModelingToolkit
problems: ${{ inputs.problems }}
note: "`CronModelingToolkit.yml` · weekly (Saturday 04:00 UTC) · Julia 1.13, Ubuntu + Windows, CPU"
budget: 3600
note: "`CronModelingToolkit.yml` · daily (04:00 UTC), a rotation with a budget of 1 h (#435) · Julia 1.13, Ubuntu + Windows, CPU"
secrets: inherit
30 changes: 28 additions & 2 deletions .github/workflows/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ scope labels. They are staggered at 04:00 UTC:
| `CronOptimalControl.yml` | `OptimalControl` | Wednesday | Yes |
| `CronOptimalControlScalarized.yml` | `OptimalControlScalarized` | Thursday | Yes |
| `CronComparison.yml` | `comparison` | Friday | No |
| `CronModelingToolkit.yml` | `ModelingToolkit` | Saturday | No |
| `CronModelingToolkit.yml` | `ModelingToolkit` | Daily, a rotation (see below) | No |
| `CronRobustness.yml` | `JuMP-robustness`, `OptimalControl-robustness` | 1st of the month | No |

Each thin workflow calls `cron-reusable.yml` (inputs `mode`: `cron`, or `robustness` for
Expand Down Expand Up @@ -69,8 +69,34 @@ a run restricted to some problems (`problem=`) leaves the other problems as they
environment is removed once no entry refers to it. Everything is written by
`tools/cron/report.py` (`--results-dir`), tested in `tools/cron/test/`.

## Rotation with a time budget

A cron with a `budget` input (seconds; 0, the default, runs every problem) is a rotation
(#435): `ModelingToolkit` for now, with 1 h, daily. Its `select` job orders the problems of
the modeling, those with a file in `ext/<M>Models/`, from the one tested longest ago to the
last tested. The date is that of its last result on `results`, and a problem never tested
comes first (`tools/cron/select.py`). The tests then run each problem in its own Julia
process, in that order, and start no new one once the budget is spent
(`test_args`: `problem=<order> budget=<s>`, see `test/README.md`). A process that crashes
(e.g. out of memory) or runs past its timeout (45 min) is reported as an `error` of its
problem only, so the problem at fault is named and the others are tested. A runner killed
before it uploads its report (run 44747569284: the Ubuntu runner died during a problem) is a
`workflow` error of its platform: the report expects one report from each runner of the
`runs_on` input (`--runners` of `tools/cron/report.py`).

A run of a rotation tests some problems only, so its reports and issues are built from the
**last result of every problem** (`--rotation` of `tools/cron/report.py`, the `latest` entry
of `cron/state.json`):
- a problem that did not run keeps its findings, and is never "resolved" by a run that
did not test it;
- the regression issues of a problem change only when it runs;
- the scope report ends with the date each problem was last tested.

A run that crashed wrote no result, so its problems come first again in the next run. The
`problems` input of a rotation puts those problems first; the run then goes on as usual.

A convergence cron dispatched by hand takes an optional `problems` input (`beam` or
`beam,chain`): it solves those problems only, and publishes their results and environments.
`beam,chain`). Without a budget, it solves those problems only, and publishes their results and environments.
Its report and its state are written aside, unpublished, and the issues are left as they
are: a restricted run cannot tell about the problems it did not solve, so `cron/<scope>.md`,
`cron/state.json` and the issues stay those of the last complete run. Use it to check a
Expand Down
92 changes: 85 additions & 7 deletions .github/workflows/cron-reusable.yml
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,23 @@ on:
required: false
default: ''
type: string
runs_on:
description: >-
The runners of the CPU job, a JSON list. The report expects one report from each:
a runner that leaves none (killed, #435) is a finding. macOS is left out for now
(scarce runners, long queues); it comes back with #344.
required: false
type: string
default: '["ubuntu-latest", "windows-latest"]'
budget:
description: >-
Time budget of a run, in seconds (#435). 0, the default: every problem, in one
process per environment. Otherwise a rotation: the problems of the modeling, tested
longest ago first (tools/cron/select.py), one process each, while the budget lasts;
the reports and issues keep the last result of every problem.
required: false
default: 0
type: number
secrets:
SSH_KEY:
required: false
Expand All @@ -52,11 +69,64 @@ permissions:
contents: read

jobs:
# The arguments of the tests: in a rotation (budget > 0), the problems of the modeling, the
# ones tested longest ago first, read from the `results` branch (#435).
select:
if: inputs.enabled
runs-on: ubuntu-latest
permissions:
contents: read
outputs:
test_args: ${{ steps.args.outputs.test_args }}
problems: ${{ steps.args.outputs.problems }}
steps:
- name: Checkout package
uses: actions/checkout@v4

- name: Test arguments
id: args
shell: bash
env:
ENVIRONMENT: ${{ inputs.environment }}
MODE: ${{ inputs.mode }}
PROBLEMS: ${{ inputs.problems }}
BUDGET: ${{ inputs.budget }}
SCOPE: ${{ inputs.scope || inputs.environment }}
run: |
set -euo pipefail
args="$ENVIRONMENT $MODE"
problems=""
if [ "$BUDGET" != "0" ]; then
results=""
if git ls-remote --exit-code origin results >/dev/null 2>&1; then
git fetch --depth 1 origin results
git worktree add --detach "$RUNNER_TEMP/results" FETCH_HEAD
results="$RUNNER_TEMP/results"
fi
problems=$(python3 tools/cron/select.py --scope "$SCOPE" \
--models-dir "ext/${ENVIRONMENT}Models" --metadata-dir ext/MetaData \
${results:+--results-dir "$results"})
if [ -z "$problems" ]; then
echo "::error::a budget needs the problems of ext/${ENVIRONMENT}Models/"
exit 1
fi
# problems given by hand come first, in their order
order="${PROBLEMS:+$PROBLEMS,}$problems"
order=$(echo "$order" | tr ',' '\n' | awk 'NF && !seen[$0]++' | paste -sd, -)
args="$args problem=$order budget=$BUDGET"
elif [ -n "$PROBLEMS" ]; then
args="$args problem=$PROBLEMS"
fi
echo "test_args=$args" >> "$GITHUB_OUTPUT"
echo "problems=$problems" >> "$GITHUB_OUTPUT"
echo "Test arguments: $args"

# The called CTActions workflows request `actions`, `contents` and `pull-requests` read
# (their `changes` job) and `statuses` write (their `status-start` and `status-end` jobs); a
# nested job cannot be granted less than it requests.
cpu:
if: inputs.enabled
needs: select
permissions:
actions: read
contents: read
Expand All @@ -65,19 +135,19 @@ jobs:
uses: control-toolbox/CTActions/.github/workflows/ci.yml@main
with:
versions: '["1.13"]'
# macOS is left out for now (scarce runners, long queues); it comes back with #344.
runs_on: '["ubuntu-latest", "windows-latest"]'
runs_on: ${{ inputs.runs_on }}
# keep every platform running after a failure: one report per platform is needed
fail_fast: false
use_ct_registry: true
test_args: ${{ inputs.environment }} ${{ inputs.mode }}${{ inputs.problems && format(' problem={0}', inputs.problems) || '' }}
test_args: ${{ needs.select.outputs.test_args }}
upload_artifact_path: test/reports
upload_artifact_name: cron-${{ inputs.scope || inputs.environment }}
secrets:
SSH_KEY: ${{ secrets.SSH_KEY }}

gpu:
if: inputs.enabled && inputs.gpu
needs: select
permissions:
actions: read
contents: read
Expand All @@ -87,7 +157,7 @@ jobs:
with:
versions: '["1.13"]'
use_ct_registry: true
test_args: ${{ inputs.environment }} ${{ inputs.mode }}${{ inputs.problems && format(' problem={0}', inputs.problems) || '' }}
test_args: ${{ needs.select.outputs.test_args }}
upload_artifact_path: test/reports
upload_artifact_name: cron-${{ inputs.scope || inputs.environment }}-gpu
fail_fast: false
Expand All @@ -98,7 +168,7 @@ jobs:

publish:
if: always() && inputs.enabled
needs: [cpu, gpu]
needs: [select, cpu, gpu]
runs-on: ubuntu-latest
permissions:
contents: write
Expand All @@ -113,8 +183,12 @@ jobs:
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
TESTS_FAILED: ${{ needs.cpu.result != 'success' || needs.gpu.result == 'failure' }}
RESULTS_DIR: /tmp/cron-results
# a restricted run writes its report and state aside, unpublished (`problems` input)
PARTIAL: ${{ inputs.problems != '' }}
RUNNERS: ${{ join(fromJSON(inputs.runs_on), ',') }}
# a restricted run writes its report and state aside, unpublished (`problems` input);
# not in a rotation, where every run tests some problems and updates their results
PARTIAL: ${{ inputs.problems != '' && inputs.budget == 0 }}
# a rotation (#435): every problem of the scope, for the reports and issues
ROTATION: ${{ needs.select.outputs.problems }}
steps:
- name: Checkout package
uses: actions/checkout@v4
Expand Down Expand Up @@ -178,6 +252,8 @@ jobs:
--julia-version '1.13' \
--package-version "$package_version" \
--scope-note "$SCOPE_NOTE" \
--rotation "$ROTATION" \
--runners "$RUNNERS" \
"${test_failed_arg[@]}"

- name: Publish report, results and observed state
Expand Down Expand Up @@ -216,6 +292,8 @@ jobs:
--julia-version '1.13' \
--package-version "$(sed -n 's/^version = \"\(.*\)\"/\1/p' Project.toml | head -1)" \
--scope-note "$SCOPE_NOTE" \
--rotation "$ROTATION" \
--runners "$RUNNERS" \
--issues \
"${test_failed_arg[@]}"

Expand Down
12 changes: 10 additions & 2 deletions test/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ test/
├── Project.toml # light environment of the dispatcher and of core (Test, Aqua, CTBase)
├── common/ # shared code, included by the tests (configurations, statuses, solvers)
├── suite/<group>/ # fast tests, run on every pull request
├── cron/<group>/ # slow tests of the same group, run by the weekly cron
├── cron/<group>/ # slow tests of the same group, run by the crons
└── robustness/<group>/ # robustness report of the group, run by the monthly cron (#378)
```

Expand Down Expand Up @@ -42,6 +42,7 @@ Pkg.test(; test_args=["OptimalControl", "convergence"]) # suite and convergence
Pkg.test(; test_args=["JuMP", "cron", "*convergence*"]) # environments, mode and file globs
Pkg.test(; test_args=["problem=beam"]) # one problem, in every environment
Pkg.test(; test_args=["JuMP", "problem=beam,robot"]) # some problems, in one environment
Pkg.test(; test_args=["ModelingToolkit", "cron", "problem=beam,chain", "budget=3600"]) # a rotation
```

`test_args` is a list of:
Expand All @@ -57,6 +58,13 @@ Pkg.test(; test_args=["JuMP", "problem=beam,robot"]) # some problems, in one
- `problem=a,b`: test only these problems, for instance the one you add. A name that is
not in the catalogue is an error (a selection matching nothing would test nothing and
pass). `core` still runs entirely.
- `budget=<seconds>` (with `problem=`), and optionally `timeout=<seconds>` (45 min by
default): the rotation of the crons (#435). Each problem runs in its own Julia process,
in the order of `problem=`, and no new one starts once the budget is spent; a process past
its timeout is stopped. A problem whose process crashed or was stopped gets the report row
`process (crashed)` or `process (timeout after … s)`, an `error`
(`TestStatus.write_process_report`). Each problem writes its own report,
`test/reports/<environment>-cron-<problem>.md`.

In cron mode, an environment without a `cron/<name>/` directory is skipped.

Expand Down Expand Up @@ -275,7 +283,7 @@ the `System` of every problem is read as the data of a JuMP problem
(`common/continuous_modelingtoolkit.jl`) and compared with the OptimalControl model by
`Continuous.compare`, as `suite/comparison/test_modelings` does for JuMP. It runs in this
environment, which therefore loads OptimalControl too, so that the `comparison` environment
stays free of ModelingToolkit. Its convergence report (`cron/ModelingToolkit`, weekly) has no
stays free of ModelingToolkit. Its convergence report (`cron/ModelingToolkit`, daily, a rotation with a budget, #435) has no
reference objective for a shooting discretization: the gap is to the objective of the analytic
solution, when the problem has one, and `violation` is the largest violation of the
constraints of the NLP solved. The costates are not computed by Corleone and are left empty.
Expand Down
75 changes: 75 additions & 0 deletions test/common/selection.jl
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,81 @@ end

const MODES = ("cron", "convergence", "robustness")

# The time a problem may take in a rotation (#435) before its process is stopped, by default.
const DEFAULT_TIMEOUT = 2700.0

"""
split_budget(args) -> (budget, rest)

Take the `budget=<seconds>` and `timeout=<seconds>` arguments out of `args` (#435). With a
budget, the problems of an environment run one per process, in the order of `problem=`,
while the budget lasts (see [`run_within_budget`](@ref)); `timeout` is the time a problem
may take before its process is stopped ([`DEFAULT_TIMEOUT`](@ref) by default). `budget` is
`nothing` without `budget=`, a `NamedTuple` `(seconds, timeout)` otherwise.
"""
function split_budget(args)
values = Dict{String,Float64}()
rest = String[]
for arg in args
key = findfirst(k -> startswith(arg, k * "="), ("budget", "timeout"))
if isnothing(key)
push!(rest, arg)
continue
end
name = ("budget", "timeout")[key]
value = tryparse(Float64, chopprefix(arg, name * "="))
(isnothing(value) || value <= 0) &&
throw(ArgumentError("`$arg`: $name must be a positive number of seconds"))
values[name] = value
end
haskey(values, "budget") || return nothing, rest
return (seconds=values["budget"], timeout=get(values, "timeout", DEFAULT_TIMEOUT)), rest
end

"""
run_within_budget(f, problems, seconds; clock=time) -> (outcomes, skipped)

Call `f(problem)` for the `problems` in their order while less than `seconds` have passed
since the start (#435): the first one always runs, a problem starts only if the budget is
not spent. Returns the `problem => f(problem)` pairs, in order, and the problems left for a
later run.
"""
function run_within_budget(f, problems, seconds::Real; clock=time)
start = clock()
outcomes = Pair{String,Symbol}[]
skipped = String[]
for problem in problems
if clock() - start >= seconds
push!(skipped, problem)
else
push!(outcomes, problem => f(problem))
end
end
return outcomes, skipped
end

"""
run_process(cmd; timeout) -> Symbol

Run `cmd`, its output shown, and stop it after `timeout` seconds. Returns `:success`,
`:failed` (a non-zero exit code: a test failed), `:crashed` (stopped by a signal, e.g. out of
memory) or `:timeout`.
"""
function run_process(cmd::Cmd; timeout::Real)
process = run(ignorestatus(cmd); wait=false)
start = time()
while process_running(process)
if time() - start > timeout
kill(process)
wait(process)
return :timeout
end
sleep(1)
end
success(process) && return :success
return process.termsignal == 0 ? :failed : :crashed
end

"""
directories(group, suite, cron, robustness; mode) -> Vector{String}

Expand Down
Loading
Loading