Repository navigation
Make CI actually run the test suite (currently 15 of 2,280 tests) #113
Description
Part of #108 · Phase 1 · Depends on #109, #110
Problem
CI runs 15 of 2,280 tests (~0.7%).
| Workflow | What it runs | Notes |
|---|---|---|
tests.yml (push/PR) |
pytest tests/unit/ -m "not real_world" -n 2 |
mypy step is continue-on-error: true |
fast_ci.yml (PR) |
pytest tests/unit/ -m "not real_world and not slow" |
failing since 2025-09-03 |
real_world_tests.yml |
schedule/dispatch only | whole job continue-on-error: true; failed 60/60 recent runs |
real-world-tests.yml |
all three jobs gated if: false |
"Disabled - requires actual cluster access" |
tests/unit/ is 5 files, 391 LOC, 15 tests out of 240 files / 89,950 LOC / 2,280 tests.
~99.3% of the test suite never executes in CI — which is exactly why all 127 failures documented in the sibling issue are invisible, and why fast_ci.yml could fail for a year unnoticed.
There are also two near-duplicate workflow files, real_world_tests.yml and real-world-tests.yml, which is its own source of confusion.
Acceptance criteria
- CI executes the full non-billable suite on every PR (target >95% of collected tests)
- All
continue-on-error: trueremoved — including the mypy step intests.yml - No workflow job is gated
if: false; either it runs, or the file is deleted - The duplicate
real_world_tests.yml/real-world-tests.ymlpair is consolidated to one - Real-world tests run on a schedule against a substrate that actually works (see the HF Jobs issue) rather than against unreachable institutional hosts
- A red CI is a blocking signal: branch protection requires the test job to pass
Note on sequencing
Turning CI on before fixing the 127 failures will make master permanently red. Land the failure-fix issue first, or gate this behind a temporary allowlist that shrinks to zero — but do not solve it by re-adding continue-on-error.
Verification
gh run list --workflow=tests.yml --limit 1 --json conclusion
# and confirm the run's log reports ~1900 tests collected, not 15
Activity
Issue #110: Restrict black pin to py>=3.9 and run linting once
Fix: give the test matrix a bound that fits the work it now does
Fix three fabricated-result bugs, close the RCE, and overhaul the doc…
jeremymanning commentedon Aug 19, 2026
Resolved
CI runs the whole non-billable suite.
.github/workflows/tests.yml:81:Not
tests/unit/. That command collects 1258 tests on the current tree, against the 15 this issue reported.The steps can also fail now:
--exit-zeroon flake8 andcontinue-on-error: trueon mypy are both gone, so the lint and type steps are load-bearing rather than decorative.Recent runs on this branch:
Testssuccess across ubuntu 3.10/3.11/3.12, macos 3.11/3.12 and windows 3.11/3.12;Fast CIsuccess.Two related things fixed since:
real_world_tests.ymlworkflow — which had failed all 94 of its scheduled runs since 2025-08-24 without ever passing, because it referencedGCP_CREDENTIALSandAZURE_CREDENTIALS, secrets that were never created — was deleted in0e3490e. Onlyreal-world-tests.ymlremains.scripts/check_docs_examples.pynow runs in CI too (Fix all broken README/docs examples and unify the version across 4 locations #124).One thing I could not verify: whether branch protection on
masteractually requires the test job.gh api repos/ContextLab/clustrix/branches/master/protectionwould settle it. A green workflow that is not a required check can still be merged past.