Skip to content

Add a weekly newest-dependencies canary - #3316

Closed
maxisbey wants to merge 5 commits into
mainfrom
ci/dependency-canary
Closed

Add a weekly newest-dependencies canary#3316
maxisbey wants to merge 5 commits into
mainfrom
ci/dependency-canary

Conversation

@maxisbey

Copy link
Copy Markdown
Contributor

Adds a scheduled "dependency canary": once a week, re-resolve the runtime dependencies of mcp[cli,rich] to the newest versions our specifiers allow, run the test suite against them, and keep a single tracking issue in sync with the result. Also moves CI to uv 0.11.33, which the canary's resolution step relies on.

Motivation and Context

pyproject.toml deliberately carries floors only, so anyone running pip install mcp gets the newest release of every dependency the day it ships. PR CI never sees that combination: it tests uv.lock (locked) and the floors (lowest-direct). Looking back over the repo's history there have been roughly six "a new upstream release broke the SDK" events a year (click 8.2, pydantic 2.11/2.12/2.13, starlette 0.52 typing, httpx 1.0 pre-releases, pytest 9.1, pyjwt 2.11, ...), with a median of 11 days before anyone here noticed and a third of them reported by users first.

We have tried two other shapes already. A real highest leg on every PR (#1609) was removed in #1869 after a half-uploaded ruff release turned every open PR red — an upstream release should not be able to do that. The weekly uv lock --upgrade PR that replaced it (#1874) produced four PRs, none merged (CI did not even trigger on the bot's pushes), and was deleted in #2919 in favour of Dependabot, whose PRs tend to get closed in bulk. The common thread is that a PR nobody particularly wants to merge is not an alert.

What demonstrably works elsewhere is a scheduled run that files one deduplicated issue and closes it again when things go green (FastMCP's upgrade checks close each incident within days; xarray, dask and scikit-learn run the same pattern). Scheduled runs that only go red, or only ping Slack, rot: pydantic's dependencies-check workflow has failed 100+ runs in a row unnoticed, and logfire's weekly job sat red for 16 weeks.

What the workflow does

.github/workflows/dependency-canary.yml, Mondays 05:23 UTC plus workflow_dispatch:

  1. resolve (ubuntu): compute the runtime closure of mcp[cli,rich] from the lock (39 packages today), strip the dependency groups uv sync does not install (translate, codegen — otherwise anthropic would hold pydantic below a new major forever), and uv lock -P <each> --exclude-newer-package <each>=<now − 24h>. Everything outside the closure keeps its locked version as a preference (it moves only if a floated dependency forces it, and the report tags such rows tooling), so a pytest or ruff release cannot masquerade as an SDK break; releases younger than a day are invisible, which makes the ci: replace highest resolution with locked in test matrix #1869 partial-upload class inexpressible. It also downloads the uv.lock the last green scheduled run uploaded and diffs against it, so the report can show only what moved since then (usually one to three packages — that table is the suspect list). The resolved uv.lock, the literal commands used, and the logs are uploaded as an artifact; the lock is the exact repro.
  2. test on ubuntu 3.10, ubuntu 3.14 (+ the examples smoke tests and an informational pyright src/mcp), windows 3.14: uv sync --frozen from that lock, run pytest, re-run failures serially (flakes drop out), and if they persist re-run once more with DeprecationWarning/PendingDeprecationWarning/FutureWarning demoted. Each cell ends up as pass, flaky, warnings-only, error or install-failed.
  3. report: check every planned cell actually reported (the matrix is defined once and handed to both the test job and the report; a cell that timed out or lost its runner counts as no result, never as a pass), assemble one Markdown report (status, what changed since last green, per-cell failures, runtime deps held below latest by someone else's cap, full diff vs uv.lock, the literal commands to reproduce under uvx uv@<same version>, a short runbook) and sync the tracking issue: open it (labels dependency-canary + dependencies, assigned to @maxisbey and @Kludex, P0 only when every cell hard-fails) / overwrite the body and leave a one-line comment while it stays red / comment and close when green. Runs where the canary itself had a problem (a cell without a result, uv lock failing for non-resolution reasons, a cancelled test job) are a separate incomplete class that only comments on an open incident and never rewrites or closes it. A human-closed issue is never reopened; the next red run opens a fresh one linking the previous. Only scheduled runs (or a dispatch with file-issue: true) touch issues; prerelease: true dispatches are investigative and never do.

Everything is gh CLI plus two small scripts under scripts/ci/; no third-party actions, permissions: {} at the top with issues: write only on the report job, which installs nothing.

What it deliberately does not do

  • Open a PR adding a ceiling. A cap protects nobody until it is released, and even then both pip and uv route around a retroactive cap by picking the newest uncapped mcp whenever anything else in the environment wants the new version — same crash, older mcp, no diagnostic. The bot also cannot tell which package (or interaction) is at fault, and a cap PR is trivially green, which is how caps accumulate. The issue's "What to do" section keeps "cap at the breaking version + same-day patch release + removal issue" as the documented exception; a maintainer decides.
  • Run daily or test pre-releases on the schedule. Weekly matches how fast we have historically acted, and logfire went daily→weekly for noise reasons. Four of the nine past incidents did have a pre-release out weeks earlier (all pydantic), but pydantic already runs this repo's suite against their main daily, so a scheduled pre-release lane is left as a possible follow-up rather than day-one noise. workflow_dispatch with prerelease: true covers ad-hoc checks.
  • Refresh uv.lock. That is a separate concern (and a separate PR); the canary reports against whatever the lock is.

uv 0.11.33

Combining -P <closure> with an upload-time cutoff only keeps the non-upgraded packages at their locked versions from uv 0.10 (astral-sh/uv#17721); on 0.9.5 the cutoff would silently float everything. Rather than run the canary on a different uv than CI, the second commit moves every workflow to 0.11.33 (0.10 and 0.11 have no breaking changes touching frozen syncs, lowest-direct, or lock --check).

How Has This Been Tested?

  • The three scripts were exercised locally: the lock diff against a real before/after pair of locks, the cell script against planted failures covering all four classifications (hard failure, deprecation-only at test time and at import/collection time, fails-once flake), and the report script in report-only mode across fixture runs (all pass; one cell missing; one cell error; every cell failing → P0; warnings plus a missing cell; uv lock failure; resolve-job infrastructure failure; cancelled test job; nothing changed since last green).
  • The workflow ran end to end on this branch via a temporary push trigger (removed again in the last commit): run 31947729778 and, after the review revisions, run 31949021363. The resolve job floated exactly the runtime closure to current PyPI (starlette 1.6.0, uvicorn 0.52.3, pydantic 2.13.4, pyjwt 2.13.0, rich 15, typer 0.27.1, cryptography 50, ...) while pytest/ruff/pyright/coverage stayed at their locked versions, all three cells passed (5600-odd tests each, plus the examples smoke tests on ubuntu 3.14), and the report job rendered the would-be issue body into the run summary without touching issues.
  • zizmor --persona regular and actionlint are clean on the new workflow.

Newest-allowed is green today, so merging this should not open an issue on the first Monday.

Breaking Changes

None; CI only.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update

Checklist

  • I have read the MCP Documentation
  • My code follows the repository's style guidelines
  • New and existing tests pass locally
  • I have added appropriate error handling
  • I have added or updated documentation as needed

Additional context

  • Known blind spots, also listed in the workflow header: Python 3.11–3.13 and macOS are not run; a runtime dep capped by a dev dependency (logfire → opentelemetry-sdk → opentelemetry-api) cannot reach its newest release, and the report's "Not tested at their newest release" section says so each run.
  • Whoever last edits the cron: line also gets GitHub's own failure e-mail for red scheduled runs; everyone else only sees the issue.
  • Possible follow-ups: move translate/codegen out of the project resolution (PEP 723 script metadata) so the runner no longer has to strip them; a scheduled --prerelease-package <direct deps>=allow lane reporting to a separate, never-P0 issue; pointing DEPENDENCY_POLICY.md (Publish versioning, roadmap, and dependency policies for v2 #3215) at the issue runbook.

AI Disclaimer

Users installing `mcp` get the newest release of every dependency the day
it ships, but PR CI only tests uv.lock and the floors, so an upstream
release that breaks the SDK is currently noticed by users first (median
11 days across past incidents). Running "highest" on every PR was tried
and removed (#1869) because a half-uploaded release turns unrelated PRs
red; the weekly lock-bump PR that replaced it (#1874) never got merged.

This adds a scheduled workflow instead. Every Monday it re-resolves the
runtime closure of mcp[cli,rich] to the newest versions the specifiers
allow (test tooling stays at uv.lock, releases younger than a day are
ignored), runs the suite on ubuntu 3.10/3.14 and windows 3.14, and keeps
a single tracking issue in sync: opened and assigned when newest-allowed
breaks, refreshed while it stays broken, closed once it passes again.
Each cell re-runs failures serially and once more with deprecation
warnings demoted, so the issue says whether users are actually broken or
a dependency merely deprecated something. The report lists what changed
since the last green run, what is held below latest by someone else's
cap, and the exact command to reproduce the resolution.

It never runs on pull requests and never opens a PR adding a ceiling;
the issue carries the runbook and a maintainer decides.

No-Verification-Needed: CI-only change (workflow + scripts/ci); scripts exercised locally, workflow verified by a branch run
The canary relies on `--exclude-newer` leaving locked versions in place
for packages it does not explicitly upgrade, which uv only guarantees
from 0.10 (astral-sh/uv#17721), and on relative cutoffs. Move every
workflow to the same pin so the canary and PR CI cannot disagree about
resolution semantics. 0.10/0.11 carry no breaking changes that touch how
this repo uses uv (frozen syncs, lowest-direct resolution, lock --check).

No-Verification-Needed: CI configuration only; exercised by PR CI itself
The workflow ran green end to end on the branch (run 31947729778), so
remove the push trigger that exercised it. Also read pyright's error
count rather than its last output line for the informational summary,
and silence its new-version nag.

No-Verification-Needed: CI-only change (workflow trigger + report cosmetics)
Review feedback, taken as one structural change rather than patches:
the report classified whatever artifacts happened to arrive, so a cell
that timed out or lost its runner simply vanished (two passing cells
read as green and would have closed a live incident), a cancelled run
could still write to the issue, and any resolve-job failure was
reported as "cannot be resolved". Now the matrix is defined once and
published by the resolve job, the report checks every planned cell
against `needs.test.result`, a cell writes a provisional status before
doing anything, `uv lock` records whether it was the thing that failed,
and incomplete runs are their own class that comments on an open
incident instead of rewriting or closing it. Artifacts carry their own
top-level directory and are merged on download, so the layout no longer
depends on how many of them exist.

Smaller corrections from the same review: per-package cutoffs instead
of a global --exclude-newer (a freshly bumped exact pin elsewhere could
otherwise fail the resolution); the since-last-green diff uses the lock
the last green run actually uploaded rather than re-resolving today's
tree at an old cutoff; the issue's reproduce block is the literal
commands the job ran (group strip included) under `uvx uv@<version>`;
per-cell output is size-bounded at the source and the full report goes
to the step summary before any truncation; the flake/deprecation re-runs
key on whether pytest recorded failures rather than on its exit code;
titles say "nothing changed since last green" when that is the case;
bash runs with pipefail everywhere; the group-strip generator follows
include-group and default-groups = "all"; wording says tooling is
preferred at uv.lock, not frozen. The temporary branch trigger is back
for one more end-to-end run.

No-Verification-Needed: CI-only change (workflow + scripts/ci); scripts exercised locally, workflow verified by a branch run
Second end-to-end branch run (31949021363) is green with the revised layout.

No-Verification-Needed: CI-only change (workflow trigger)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant