test: ratchet mutation score over enumerated decision kernels

[devin-ai-integration]

Summary

Adds a mutation-score ratchet over an enumerated set of decision kernels, so "are these tests
load-bearing?" gets a mechanical answer instead of a coverage percentage. Closes #1415 (umbrella
#1412, Track A).

Scope is a registry, not a config file: scripts/mutation/modules.ts owns the five modules
(src/kernel/errors.ts, src/daemon/ref-frame.ts, src/commands/interaction/runtime/settle.ts,
src/utils/scroll-edge-state.ts, src/selectors/), and everything else derives from it —
stryker.config.json's mutate globs are asserted equal to mutateGlobs(), PR-affected selection
maps changed files through moduleForFile, and the weekly shard matrix is asserted equal to
ALL_MODULE_IDS. Modules that spawn subprocesses or wait real time stay out by construction.

The ratchet, not Stryker, owns pass/fail (thresholds.break: null), and it is deliberately dormant:

evaluateRatchet(scores, baseline, provenance)
  → verdict per module: new | improved | held | regressed | provenance-drift
  → failed = baseline.gating && anyRegressed        // reports, but does not fail, before graduation

applyRun(...)                                       // --update only
  → keeps the high-water score on regression        // a bad week cannot lower the bar
  → rebases (never "regresses") on provenance drift // tool/config change ≠ weaker tests
  → stableRuns++ only for a comparable FULL sweep; gating = stableRuns >= requiredStableRuns (2)

Each baseline entry carries the strykerVersion and a content hash of stryker.config.json, which is
what makes drift separable from test strength. Committed starting point (full sweep, Stryker 9.6.1):
daemon-ref-frame 100%, selectors 70.56%, interaction-settle 68.95%, kernel-errors 55.19%,
scroll-edge-state 28.75% — 2,054 mutants total. A failing check prints the surviving mutants
(file:line, mutator, replacement) plus the command to reproduce.

Two things the diff won't explain on its own:

• The weekly sweep is sharded one job per module and merged with --report-dir before ratcheting.
  Measured locally, selectors alone is ~1,280 mutants / ~7 min on 8 cores; a single job would sit on
  the 30-minute acceptance budget on a 2-core-class runner. Sharding keeps every job far under it and
  the ratchet still sees one full sweep.
• The test suite Stryker replays is derived, not listed. Stryker reruns the suite per mutant, so
  scripts/mutation/test-scope.ts asks Vitest for the related tests (vitest related over the
  mutated files — the same delegation pnpm check:affected uses) and drops three groups it cannot
  run: the subprocess-stub tests (real time), non-src/ conformance gates, and thread-hostile tests
  (Stryker's vitest runner hard-codes pool: 'threads', where process.chdir throws and a
  node:worker_threads worker inside a worker raises uncaught MessagePort errors that kill the
  runner). Both exclusions are derived from imports, so they cannot rot, and neither can hide a
  survivor — a mutant only an excluded test could kill is reported as surviving.

A kernel's tests own the module too: KERNEL_MODULES.tests lists them, so a diff that only weakens
src/kernel/__tests__/errors.test.ts still runs that kernel's mutants on the PR. The list is verified
rather than trusted — modules.test.ts walks the import graph and fails if a listed test never reaches
the kernel, and separately fails if a mirrored test file of a kernel source is not listed.
(src/utils/__tests__/errors.test.ts moved to src/kernel/__tests__/errors.test.ts so it mirrors its
source per the topology rule.)

Every run also writes the standard scheduled-lane artifact envelope (#1430) via
scripts/lib/lane-envelope.ts — schemaVersion, commit SHA, tool.stryker, configHash, seed
(explicit null: enumerated input, not randomized), duration, result, per-module scores — uploaded by
both workflows and folded into the weekly step summary, so a lane going dark or a tool bump is visible
without reading logs. It sits in scripts/lib/ so the other scheduled lanes can adopt it.

Also extracts the duplicated parseArgs + --help boilerplate shared by this script and
scripts/check-affected/run.ts into scripts/lib/cli-args.ts.

Validation

pnpm format:check, pnpm lint, pnpm typecheck, pnpm fallow clean; unit suite green (516 files,
4,554 tests). pnpm mutation:test (29 cases) covers the acceptance criteria directly: a lowered score
fails once gating is on, the failure lists surviving mutants, the same lowering only reports before
graduation, regressions keep the high-water baseline, graduation needs two consecutive full sweeps,
affected PR runs never advance it, and provenance drift is not a test-strength regression. Real
Stryker runs produced the committed baseline (per-module, merged) — that is the sweep's own evidence.

Not verified here: pnpm build and the Node integration smokes fail on this machine on pristine
origin/main too (tsdown cannot import unrun; Node 22.14 has no default .ts stripping), so
those are preexisting environment failures rather than something this PR touches. The weekly
workflow's own wall clock is a projection from local timings until its first scheduled (or manually
dispatched) run.

26 files touched; scope stayed inside the mutation lane plus the small shared-arg extraction.

Link to Devin session: https://app.devin.ai/sessions/97c343f089bb49cf820c2569653f3b35
Requested by: @thymikee

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[github-actions]

PR Preview Action v1.8.1
🚀 View preview at https://callstack.github.io/agent-device/pr-preview/pr-1441/
Built to branch gh-pages at 2026-07-27 18:18 UTC. Preview will be ready when the GitHub Pages deployment is complete.

[github-actions]

Size Report

Metric	Base	Current	Diff
JS raw	1.87 MB	1.87 MB	0 B
JS gzip	598.6 kB	598.6 kB	0 B
npm tarball	714.1 kB	714.2 kB	+105 B
npm unpacked	2.50 MB	2.50 MB	+859 B

Startup median (7 runs, lower is better):

Scenario	Base	Current	Diff
CLI --version	28.5 ms	28.2 ms	-0.3 ms
CLI --help	61.1 ms	59.3 ms	-1.7 ms

Top changed chunks: no changes in the largest emitted chunks.

[thymikee]

Review: changes requested — three P1 blockers.

1. Restore package.json bench:help-conformance to node scripts/help-conformance-bench.mjs. The new node scripts/mutation scripts/help-conformance-bench.mjs invokes a directory instead of the benchmark entrypoint.
2. Make affected mutation selection cover tests for all four exact-file kernels, both in moduleForFile() and the PR workflow paths. Current tests live outside those exact source paths (for example src/utils/__tests__/errors.test.ts, src/daemon/__tests__/ref-frame.test.ts, and src/commands/interaction/runtime/settle.test.ts), so weakening them skips the intended PR mutation ratchet.
3. Add obs: scheduled-lane health, freshness telemetry, and standard artifact envelope #1430’s standard artifact envelope and freshness integration for this new weekly lane. The weekly workflow currently uploads raw reports/proposed baseline only; it lacks the required schema version, commit SHA, tool/config provenance, duration, and result envelope.

All current checks are green, but the PR-side mutation job only ran its self-tests, reported no affected decision kernels, and ran no mutants. That is not readiness evidence for the new lane.

[devin-ai-integration]

All three P1s fixed in a19412b (branch also rebased onto latest main).

1. bench:help-conformance is back to node scripts/help-conformance-bench.mjs. A sed that added scripts/mutation to the format paths also hit that script — good catch.

2. Kernel tests now own their module. KERNEL_MODULES gained a tests field, moduleForFile() matches it, and the PR path filter lists the same files (workflow.test.ts asserts the parity). Two supporting changes:

• src/utils/__tests__/errors.test.ts → src/kernel/__tests__/errors.test.ts, so it mirrors src/kernel/errors.ts per the topology rule. A new test asserts that any mirrored test file of a kernel source selects that module, so a future mirror cannot be silently unlisted.
• Listed tests are verified, not asserted: modules.test.ts walks the import graph and fails if a listed test never reaches the kernel's sources. That is what validated daemon/handlers/__tests__/interaction-settle.test.ts (reaches settle.ts transitively), and it is why scroll-edge-state lists its gesture/dispatch callers — it has no mirrored test at all, which is also why it scores lowest (28.75%).

affectedModules(['src/kernel/__tests__/errors.test.ts', 'src/commands/interaction/runtime/settle.test.ts']) now returns ['kernel-errors', 'interaction-settle'].

3. Envelope. Both lanes write and upload .tmp/mutation/lane-envelope.json via scripts/lib/lane-envelope.ts: schemaVersion, commit SHA, tool.stryker, configHash, seed (explicit null — the input is enumerated, not randomized), startedAt/finishedAt/durationMs, result, plus per-module score/status and the graduation counters. The weekly job also folds it into the step summary. It lives in scripts/lib/ rather than the mutation tree so #1430 can adopt it for the other scheduled lanes instead of re-deriving field names.

On readiness: agreed that the PR job's "no affected kernels" run proved nothing. This push touches the kernel registry, so the job runs real mutants on it. Locally pnpm mutation:run --modules daemon-ref-frame kills 55/55 in 1m05s, and the merged five-shard sweep (2,054 mutants) is the committed baseline. The <30-minute weekly claim is still a projection from local shard timings — I can workflow_dispatch the weekly lane once this is on main and post the real wall clock.

[devin-ai-integration]

Readiness evidence from this push's PR job (all 23 checks green): the affected lane selected kernel-errors from the registry diff and ran real mutants on an ubuntu runner — 183 mutants, 101 killed, 55.19%, held against the committed baseline, Stryker 7m16s / 557s lane total. So the ubuntu-runner score reproduces the locally-recorded baseline exactly, which is the cross-machine determinism the ratchet depends on.

That also gives a real per-shard data point for the weekly budget: 183 mutants ≈ 7m16s here, and the largest shard (selectors, 1,277 mutants) is the one the sharding exists for. I'll dispatch the weekly lane after merge and post the actual per-shard wall clock rather than leaving the <30-min claim as a projection.

[thymikee]

Re-review at head a19412b: the bench:help-conformance script is fixed and checks are green, but two P1 blockers remain. First, affected test ownership is still a hand-picked list rather than derived/comprehensive: for example src/tests/daemon-error.test.ts directly exercises normalizeError from src/kernel/errors.ts but is absent from both the kernel-errors test ownership and workflow paths. Weakening that test skips the affected mutation run, and the current parity tests cannot detect omissions. Second, issue 1412 requires issue 1430 health/freshness monitoring to land with this new scheduled lane. The envelope is only written after report parsing and ratchet evaluation, so setup/Stryker/report failures can still produce no envelope, and there is no derived cadence monitor to detect a dark lane. Fix ownership derivation/completeness and land the shared health path before readiness.

[devin-ai-integration]

Both remaining blockers addressed at 8c1dbe794.

1. Ownership is derived, not hand-picked. The registry's tests field is gone. scripts/mutation/ownership.ts computes ownership from the static import graph: a test owns every kernel its imports reach, transitively. Your example lands automatically —

ownersOf('src/__tests__/daemon-error.test.ts') -> ['kernel-errors']   # via src/daemon.ts

and ownership.test.ts asserts completeness rather than parity with a list: it enumerates the whole derived map (ownedTestFiles), recomputes reachability independently for every owning file, and requires each module to have owners. There is no list left to omit an entry from.

The honest consequence: reachability is a superset of kill-power, so the derived owner sets are large (kernel-errors 469 test files, the others ~320 each). Any src test change therefore selects several kernels. I took that over any narrowing heuristic, since a narrowing is exactly the false negative you flagged. To keep it affordable, the PR lane is now select → per-module shard matrix (--list-affected) → merged ratchet, so wall clock is one module rather than their sum, and the path filter is deliberately wide (src/**/*.test.ts) with the decision left to select. workflow.test.ts now asserts that width instead of enumerating test paths.

2. Envelope on every exit path + a cadence monitor. run.ts reads provenance/baseline before doing any work and writes the envelope from a finally, carrying the stage it died in (select / stryker / report / ratchet / complete) and the error message. Covered by an integration test that runs the real CLI against a missing report and asserts result: fail, stage: report, plus intact tool/config provenance.

For the dark-lane half I landed the shared path rather than deferring to #1430: scripts/lane-health/ + a daily lane-health.yml. The lane list is derived from .github/workflows/ (so this PR's own weekly lane is watched without being listed), cadence comes from each lane's own cron, and a lane is reported dark after two missed cadences, failing after two consecutive non-success scheduled runs (cancellations excluded), or never-run once it has been declared longer than two cadences without firing. Unhealthy lanes ping one tracking issue; the job never fails, because a second red lane nobody watches isn't an alert.

Run against this repo just now, it immediately found a real one:

| `replays-nightly.yml`         | failing | two consecutive scheduled runs concluded failure, failure |
| `mutation-weekly.yml`         | fresh   | declared recently, awaiting its first run                 |
| `conformance-differential.yml`| fresh   | last scheduled run 2026-07-27T06:12:51Z                  |

Gates green locally: format:check, lint, typecheck, fallow, check:production-exports, pnpm mutation:test (33 cases), pnpm lane-health:test (7 cases). Rebased on latest main. Still outstanding by construction: the <30-min weekly figure remains a projection from per-shard timings until the first real scheduled sweep, which I'll dispatch and report after merge.

[thymikee]

Re-reviewed at da4bf16. The derived ownership change closes the prior hand-picked-test omission, including daemon-error.test.ts, and the current GitHub checks plus the focused ownership/workflow/envelope/health tests are green.

Two blockers remain:

• [P1] Reject incomplete shard sets before writing the aggregate envelope. readShardedReports only requires at least one mutation.json, while summarizeReport creates a zero score for every requested module absent from the downloaded reports. While gating is false, that becomes a non-failing regression, so the ratchet can write a complete/pass aggregate envelope even though a matrix shard failed. Validate that every requested module has a real report/nonzero mutant coverage, or otherwise propagate the shard result, and cover a partial-shard failure.

• [P1] Extend the envelope boundary to genuine early failures. Argument parsing and provenance/baseline reads occur before the try/finally, so malformed or missing configuration still exits without an envelope. The weekly workflow also runs mutation:test before invoking the runner, so a self-test failure produces no aggregate lane envelope. Ensure these pre-state and pre-run failures create and upload a fail envelope, with tests that exercise them.

Residual risk, not a blocker: issue #1415 requires the weekly lane to complete in under 30 minutes, but the first real weekly run has not happened and shard timeouts remain 45 minutes. Please capture the first scheduled or manual weekly timing as follow-up evidence.

[devin-ai-integration]

Both P1s fixed in 0218d2174.

[P1] Incomplete shard sets no longer pass. The aggregate now asserts, after scoring, that every requested module actually reported mutants — a module scored from no report is a dead shard, not a 0% score:

Incomplete shard set from .tmp/mutation/shards: no mutants for daemon-ref-frame.
A shard job failed or its artifact is absent; the aggregate is not a sweep.

That throws before evaluateRatchet, so nothing can write a complete/pass envelope or advance stableRuns off a partial matrix, independent of gating. Covered by a partial-shard test that writes one shard, requests two modules, and asserts exit≠0 plus a fail envelope at stage ratchet.

[P1] The envelope boundary now covers pre-state and pre-run failures. Argument parsing, readProvenance(), and readBaseline() moved inside the try/finally (initial stage is setup, provenance defaults to unknown instead of the envelope being skipped, and the commit read tolerates failure), so a bad --modules, an unreadable stryker.config.json, or a malformed baseline all emit a fail envelope naming the error.

For steps that run outside the runner — the weekly pnpm mutation:test, the affected select job, the artifact download — there's now --fail-envelope <reason>, invoked from if: failure() in both workflows and uploaded like any other envelope. It writes only when no envelope exists, so a genuine ratchet verdict is never clobbered by the generic recovery step. Tests cover the malformed invocation, the --fail-envelope path, and the no-clobber rule; a workflow assertion pins both lanes to having a recovery step.

Gates: format:check, lint, typecheck, check:fallow, 39 mutation self-tests green.

On the 30-minute budget — agreed, still unproven. The 45-minute shard timeout is deliberate headroom so a slow runner reports instead of being killed; I'll dispatch the weekly lane after merge and post real per-shard wall clock as follow-up evidence.

[thymikee]

Re-review confirms the missing-shard and early setup-envelope blockers are fixed and tested. One lane-result bug remains: the weekly ratchet can write a pass envelope, then cp/git checkout can fail; the if: failure() fallback calls --fail-envelope, but onlyIfAbsent preserves any existing envelope, so the failed scheduled job is published as passing. Preserve an existing failure, but upgrade an existing pass to fail for a later workflow failure, and add a pass-then-post-step-failure regression.

All head checks are green and the focused suite passes; the kernel-errors shard took ~10m38, but no full Mutation Weekly run exists, so the <30m budget remains unproven residual risk.

[devin-ai-integration]

Good catch on the pass-then-post-step-failure hole — fixed in f6fbba1c7.

--fail-envelope is now asymmetric: it preserves an existing envelope only when that envelope already reports fail (the specific reason beats the generic recovery one), and it overwrites an existing pass, because a pass written before a later step failed is exactly the lie the envelope exists to prevent. Absent or unreadable envelopes are written as before.

The regression test does the real sequence: a passing --report-dir ratchet writes a pass envelope, then --fail-envelope "the baseline copy step failed" runs, and the envelope must end up result: fail carrying that reason — paired with the existing preserve-the-first-failure test on the other side. 40 mutation self-tests, format:check/lint/typecheck/check:fallow green.

Agreed on the residual risk: <30m stays unproven until a real Mutation Weekly run exists. The ~10m38 kernel-errors shard is the only datapoint; selectors is the long pole (~1,280 mutants). I'll dispatch the lane after merge and post per-shard wall clock.