Skip to content

[CI] Test only what a pull request changes: environments and problems; labels only add #405

Closed

Description

@ocots

Written by Claude (Claude Code) at the request of ocots.

Part of #310. Follows #327 (changed problems), #341 (what a run tested), #360 (convergence off by default). Replaces the precompilation of #373 (closed, not planned) as the way to shorten the CI.

Why

On a pull request, the CI should test what the pull request changes, and labels should only force more. Today (tools/ci/scope.py):

  • the problems are narrowed automatically only when every changed file is a per-problem file; any other file runs every problem;
  • the environments are never chosen automatically: without a scope label every environment runs. Changing ext/JuMPModels/schemes/build.jl or test/suite/JuMP/ runs OptimalControl, OptimalControlScalarized and comparison too; changing docs/ runs everything;
  • the scope labels restrict (test JuMP backend runs JuMP only): without a label, everything; with a label, only what it names. That is the opposite of "automatic unless overridden".

Scope

1. Automatic scope: environments × problems, from the changed files

Each changed file gives a set of environments and a set of problems; the run is the union over the files. A file that is not understood, a removed or renamed file, a truncated or unreadable file list, the release label or an event other than a pull request run everything, as today: narrowing happens only on positive evidence.

The rules are computed from the repository, not written as a list of paths: the environments are the directories of test/suite/ with a Project.toml; the modeling of ext/<M>Models/ is the environment M; the files that a test file includes (include(...)) are read from the sources.

Changed file Environments Problems
ext/<M>Models/<p>.jl (a problem of a modeling) M, comparison p
ext/<M>Models/** other (e.g. schemes/) M, comparison all
ext/<M>Models.jl (entry point of an extension) every environment of the modelings it loads, comparison all
ext/MetaData/<p>.jl, ext/Descriptions/<p>.md all p
test/suite/<env>/** <env> all
test/common/<f>.jl the environments whose tests include <f>, directly or not all
test/cron/**, test/robustness/** core only (not run on a pull request) —
docs/**, *.md outside ext/, tools/cron/** core only —
tools/ci/** core only (its unit tests run in the scope job) —
src/**, Project.toml, test/runtests.jl, test/Project.toml, .github/workflows/CI.yml, anything else all all

The arguments of test/runtests.jl are one product, environments × problems: two files ext/JuMPModels/a.jl and ext/OptimalControlModels/b.jl give JuMP OptimalControl comparison problem=a,b, slightly more than needed, accepted.

core always runs (it does today).

2. Labels add, they never restrict

  • test <M> backend, test comparison: add the environment to the automatic scope, with all its problems.
  • test all problems: every problem, in the environments of the automatic scope and of the labels.
  • test everything (new label): every environment and every problem.
  • test convergence: unchanged (adds the convergence report).
  • run ci github-runner: unchanged (the CI runs only with it).

test/README.md, the comments of CI.yml and the job summary say this.

Labels or a comment command?

Kept: labels. They are already wired (adding or removing one reruns the CI), only maintainers can set them (protects the runners), and they stay visible on the pull request. A comment command (/test JuMP all) needs an issue_comment workflow, a permission check of its author, the head commit of the pull request, and its state is lost in the thread; checkboxes in the description rerun the CI on every edit. With an automatic scope, labels become rare.

Out of scope: the GPU environment (self-hosted runners, its own labels), the crons (they test everything of their scope), #344 (the full CI matrix).

Done when

  • tools/ci/scope.py computes environments × problems from the changed files with the rules above; unit tests in tools/ci/test/test_scope.py for every row of the table and for the fallbacks, each shown to fail on a broken rule.
  • Labels add to the automatic scope; test everything exists and runs everything.
  • The job summary shows, for each file, the environments and problems it adds.
  • Shown on this pull request: its own scope (a change of tools/ci/ and docs gives core), then with test JuMP backend and with test everything.
  • test/README.md and the comments of CI.yml up to date.

Activity

  1. added
    devInternal change, no user-facing impact
    on Nov 7, 2026
  2. ocots commented on Nov 7, 2026

    @ocots
    MemberAuthor

    Taken by an agent; plan to follow, then a draft PR (ct-conventions v0.10.0, ct-agents v0.4.0).

  3. ocots commented on Nov 7, 2026

    @ocots
    MemberAuthor

    Written by Claude (Claude Code) at the request of ocots.

    Two corrections to the description, found while writing the plan:

  4. ocots commented on Nov 7, 2026

    @ocots
    MemberAuthor

    Done in #406, ready for review. The label test everything is created.

  5. added a commit that references this issue on Nov 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

devInternal change, no user-facing impact

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions