Skip to content

[Doc] Results pipeline for the documentation, and a gallery of problems #314

Closed

Description

@ocots

Agents: read the guide for agents in #310 first (order, difficulty, how to claim an issue, working method). The specification is in this description and in the comments below.

Updated on 2026-11-06 after #360 (convergence is reported, not tested) and #375 (reference objectives); results now come from the convergence cron. Where an older comment contradicts this description, the description wins; what changed is listed in the update comment.

Goal

Separate the computations shown in the documentation (solving every problem, plots), which are long and may fail, from the documentation build, which must be fast and never fail because of a problem. Jobs compute results and store them; the documentation only reads what exists; each problem page shows the results if they exist and are up to date, says so otherwise. Problem pages are reached through a gallery instead of a list in the sidebar.

One source of results: the convergence cron (decision of 2026-11-06). Since #360 and #375, the weekly cron of each scope already solves every problem, in every modeling and configuration, on its default grid, and publishes the results on the results branch (*-results.tsv: solver status, objective, iterations, time, gap to the reference). The documentation results come from these runs, not from a second computation: the cron jobs also write result.json, the plots and the environment of the run (below). No separate computation workflow to keep in sync with the tests.

Part of #307. Related: #305 (crons write their reports to the same branch), #306 (features for the gallery filters), #134 (visualize), #311 (VitePress).

Storage: a results branch

  • The orphan branch results (exists since [Tests] Crons per scope for the slow tests, with reports #305: cron/ with the reports and state.json; protected against deletion by the ruleset protect-results), rewritten with a single commit on each update so that it does not grow. Not a tracked directory in main (regenerated SVGs would bloat the history).
  • Written only by the publish job of the crons (.github/workflows/cron-reusable.yml, concurrency group cron-results).
  • Locally: a git-ignored directory (e.g. docs/results/) filled by a script, so the documentation also builds offline and in previews.
results/                         # branch `results`
├── manifest.json                # date, commit, list of runs
├── env/<run id>/                # environment of each computation run
│   ├── Project.toml, Manifest.toml
│   ├── versioninfo.txt          # output of versioninfo()
│   └── status.txt, status-manifest.txt   # Pkg.status(), Pkg.status(mode=PKGMODE_MANIFEST)
├── problems/<problem>/
│   ├── result.json              # per modeling and configuration: solver status, objective, iterations, time, gap to the reference; fingerprint; run id
│   ├── state.svg, control.svg, costate.svg
│   ├── visualize.svg            # if the problem has a visualisation (#134)
│   └── thumbnail.svg            # for the gallery
└── cron/                        # reports of the crons (#305)

Robustness

  • One writer. The test jobs of the crons run in parallel (per scope and platform) and only upload workflow artifacts; the publish job collects them and makes the single commit (as today for the reports). No concurrent writes; a failing problem does not prevent the others from being published. Plots need Plots in the cron environments (or a step after the solves): to decide when taking the issue.

  • Freshness. Each result.json records a fingerprint: a hash of the problem files (metadata, model files of every modeling, description) and the versions of the packages used. At build time the documentation recomputes the fingerprint and shows, per problem:

    • up to date: the results;
    • stale: the results, marked "computed for commit X";
    • missing: "results not computed yet".

    The build never fails on missing or stale results.

  • Triggers of the computations: those of the crons (weekly per scope, workflow_dispatch); to add if needed: a push to main that changes a problem (only problems whose fingerprint changed), through the same cron workflows with problem=…. The documentation workflow never triggers or waits for them.

  • VitePress: copy the results into docs/src/public/results/ before makedocs (VitePress silently drops files it does not recognise outside public/, see the notes on the VitePress backend in ct-conventions rules/site.md).

Problem pages

Generated, never hand-written (docs/problems.jl today):

No heavy computation at documentation build time: the only code executed by the build is short (tutorials, trivial calls).

Gallery

Replaces the list of problems in the sidebar, since the collection will grow quickly:

  • one card per problem: thumbnail (thumbnail.svg, or the visualisation), name, modelings and statuses, tags;
  • filters by modeling, status, tag and features (dimensions, free final time, constraints);
  • a Vue component (see docs/src/components/ in CTBase.jl), with the existing interactive table (docs/browser.jl) as a second view of the same page, or replaced by it.

Done when

  • The cron workflows write result.json, the plots and env/<run id>/ for each run, through the existing publish job; branch results rewritten with one commit.
  • Fingerprint computed identically by the jobs and by the documentation build; up to date / stale / missing shown on the pages, shown on purpose for each case.
  • Local script filling docs/results/; documentation builds with no results at all.
  • Problem pages generated with results, plots, visualisation and Reproducibility block.
  • Gallery page with filters; problems removed from the sidebar list.
  • Documentation build time reported before and after.

Activity

  1. ocots commented on Nov 4, 2026

    @ocots
    MemberAuthor

    Reference for the Reproducibility block: the dev documentation of OptimalControl.jl, https://control-toolbox.org/OptimalControl.jl/dev/ (home page, section "Reproducibility"; source docs/src/index.md and docs/make.jl on main). The stable site is not up to date.

  2. ocots commented on Nov 4, 2026

    @ocots
    MemberAuthor

    Every modeling handled generically (OptimalControl, OptimalControlScalarized, JuMP, and Corleone later, #308):

  3. ocots commented on Nov 4, 2026

    @ocots
    MemberAuthor

    Correction: where this issue says "Corleone" as a modeling, read ModelingToolkit (ModelingToolkitBackend, #308): the problem is a ModelingToolkit system, and the documentation adapter shows how to get it and solve it with Corleone. Installation page: ModelingToolkit for the modeling, Corleone to solve it.

  4. ocots commented on Nov 4, 2026

    @ocots
    MemberAuthor

    Documentation (decision in #307): this issue also writes the README of the results pipeline (how results are computed, the result.json format, the results branch, the documentation adapter of a modeling), placed next to the code of the pipeline. Procedures only. Add it to the 'Done when' checklist.

  5. ocots commented on Nov 6, 2026

    @ocots
    MemberAuthor

    Update: results come from the convergence cron (decision of the maintainer, 2026-11-06)

    Principle since #360: three kinds of tests. Does not break (every CI run, short) and model equivalence (every CI run, no solve) can fail; the convergence report (cron, or label test convergence) gives the solver status and the relative gap of the objective to the reference objective (#375), never fails, no tolerance.

    Since #360 and #375 the weekly cron of each scope solves every problem in every modeling and configuration on its default grid, and publishes the results on the results branch. A separate computation workflow for the documentation would duplicate it. The description is updated:

    • One source of results: the cron jobs also write result.json, the plots and env/<run id>/; the existing publish job makes the single commit.
    • result.json holds the solver status and the gap to the reference; the pages show it without ✓/✗ on convergence ([Doc] Rewrite the documentation: getting started, user guide, problems, developers #307).
    • The results branch exists (cron/, state.json) and is protected against deletion (ruleset protect-results).
    • Triggers: those of the crons; a push that changes a problem can be added through the same workflows with problem=….
    • To decide when taking the issue: Plots in the cron environments, or a step after the solves.

    🤖 Generated with Claude Code

  6. ocots commented on Nov 7, 2026

    @ocots
    MemberAuthor

    Written by Claude (Claude Code) at the request of ocots.

    Note from #307 (draft PR #408), for whoever takes this issue: the problem pages are now generated by docs/problems.jl with one tab per modeling and a Compare tab, and already show the results per configuration from ext/MetaData/references/<problem>.toml. What remains here: plots (SITE12 is set in docs/make.jl), the freshness badge, the Reproducibility block per problem, and the gallery, which replaces the sidebar list of the problems. A modeling tab is a <ProblemPanel id="<modeling>"> in the generated page: the results and plots of #314 go there.

    🤖 Generated with Claude Code

  7. ocots commented on Nov 7, 2026

    @ocots
    MemberAuthor

    Taken by an agent. Written by Claude (Claude Code) at the request of ocots.

    Plan: results pipeline and gallery (#314)

    Conventions: ct-conventions v0.11.0; ct-agents v0.4.0. From main 0f6b1ec.

    What and why

    The problem pages (#408) already show the objective of every configuration, read from
    ext/MetaData/references/<problem>.toml. What #314 still adds: plots of the solutions, the
    freshness of the results, the Reproducibility block of the run, and a gallery instead of the
    sidebar list. The source stays the convergence crons, which already publish to the results
    branch and, since #412 (#409 part 1), write <report>-results.tsv (how each solve was
    configured) and <report>-environment.toml (versions, platform, commit) for every run.

    Decisions (agreed with the maintainer, 2026-11-07)

    1. Store trajectories, not images. The crons write the solution of each solve as data
      (problems/<problem>/<modeling>/<configuration>.json: time grid, state, control, costate,
      on the solve's grid, downsampled to at most ~500 points). The pages draw them in the
      browser
      (Vue component), interactively: the reader chooses the curves, the
      configurations and the modelings to overlay. Julia computes, it does not draw.
      • No Plots in the cron environments, no Julia in the publish job.
      • The plots change without rerunning the crons, and the build stays fast.
      • The gallery thumbnails are drawn from the same data.
    2. result.json per problem, merged per scope. Each cron scope publishes on its own, so
      the publish job reads the existing problems/<problem>/result.json, replaces the entries of
      its scope, and keeps the others; same single-commit rewrite as today.
    3. One source of truth, one tool. The fingerprint is one internal function of the
      package (src/), used by the tests (written with each result) and by the documentation:
      SHA-256 of the files of the problem (metadata, model files, description). Versions are
      recorded and shown, not compared: the docs environment is not the crons'. The publication
      extends tools/cron/report.py, which already publishes the reports; no second publisher.
    4. Split into four pull requests, each useful alone.

    Phases

    A — The crons record trajectories and the environment of the run

    • test/common/status.jl: result! takes the solution; write_report writes the trajectories
      (test/reports/trajectories/…json) next to the .tsv.
    • test/common/run_environment.jl: also copies Project.toml, Manifest.toml, versioninfo(),
      Pkg.status() and Pkg.status(mode=PKGMODE_MANIFEST) of the test environment into
      test/reports/env/ (Reproducibility block).
    • src/: _fingerprint(problem), written into each result entry.
    • Test: a smoke run (problem=beam, convergence) produces the files; a test of the
      fingerprint shows it changes when a model file changes.

    B — The publish job writes problems/ and env/ on the results branch

    • tools/cron/report.py, extended: from the downloaded artifacts, also writes problems/<problem>/result.json (entries of the scope replaced),
      the trajectories, env/<run id>/, and manifest.json (date, commit, runs).
    • .github/workflows/cron-reusable.yml: one more step in publish, before the commit.
    • Unit tests in tools/cron/test/.
    • Check: one cron dispatched on the branch (problem=beam), the results tree inspected.
      A dispatched cron updates the cron issues (test: record how each solve was configured and the environment of the run (#409, part 1) #412 notes it): run it on one problem only.

    C — The documentation reads the results

    • docs/results.jl: fills the git-ignored docs/results/ from the results branch, copied
      to docs/src/public/results/; the build works without it ("results not computed yet").
    • docs/problems.jl, in each modeling tab and in Compare: an interactive plot component
      (ProblemPlot.vue) reading the trajectories, the freshness badge (up to date / stale for commit X / missing), and the Reproducibility
      block of the run (::: details, downloads served from docs/src/public/).
    • Build time before and after.

    D — Gallery

    • docs/src/components/ProblemsGallery.vue: one card per problem (thumbnail drawn from the
      trajectories, name, modelings and statuses, features); filters by modeling, status and
      features; the browser table as a second view of the same page.
    • The problem pages leave the sidebar.
    • README of the pipeline next to tools/cron/report.py (procedures only).

    Out of scope

    Typed references and reference solutions (#409 part 2, in progress elsewhere: tools/references
    and ext/MetaData/references are not touched); visualize (#134); JuMP costate signs (#326).

    Human checkpoints

    • ⛔ Before dispatching a cron on the branch (phase B).

    🤖 Generated with Claude Code

  8. added 5 commits that reference this issue on Nov 7, 2026
  9. ocots commented on Nov 8, 2026

    @ocots
    MemberAuthor

    Done — merged in four PRs: #417 (record trajectories), #419 (publish results on results), #420 (problem pages read the results), #422 (gallery).

    Done when

    • The cron workflows write result.json, the trajectories and env/<run id>/ for each run, through the existing publish job; branch results rewritten with one commit — verified on the manual runs of the three scopes (JuMP, OptimalControl, OptimalControlScalarized).
    • Fingerprint computed identically by the jobs and by the documentation build; up to date / stale / missing shown on purpose — missing everywhere until the first runs, current once published, stale on every page now since refactor: split src into submodules, keeping the public API (#413) #421 changed the model files (the next weekly runs refresh to current).
    • Local script filling docs/results/ (docs/results.jl); documentation builds with no results at all.
    • Problem pages generated with results, plots, visualisation and Reproducibility block — e.g. beam.
    • Gallery page with filters (overview); problems removed from the sidebar list.
    • Documentation build time reported before and after: 105 s before (0f6b1ec, local build) vs 111 s after (docs: a gallery of the problems (#314, phase D) #422).

    Deviations from the description, as decided along the way:

    • visualize.svg is left to [General] Visualisation #134 (per the plan).
    • No README next to tools/cron/report.py: the pipeline is documented in .github/workflows/README.md, next to the crons, to avoid a second place to keep up to date.
    • Plots are drawn in the browser from the published trajectories instead of SVGs produced by Julia (decision 1): state.svg / control.svg / costate.svg / thumbnail.svg are replaced by the trajectory JSONs and results/thumbnails.json.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

documentationImprovements or additions to documentation

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions