Skip to content

fix(mutate): run each test process in a private copy of the module - #69

Merged
aledbf merged 2 commits into
mainfrom
fix/mutate-isolate-mutants
Oct 26, 2026
Merged

aledbf merged 2 commits into
mainfrom
fix/mutate-isolate-mutants

Conversation

@aledbf

@aledbf aledbf commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Problem

mutate ran every test process with its working directory set to the package's real source directory: schema binaries (Ask), each per-test coverage run (Cover), and go test for edits built alone (Run). Edits asked concurrently shared that directory, so files one edit's tests wrote were read by another's tests.

In github.com/devcontainers/cli, -j 4 reported a different set of extendImageWithFeatures edits as SURVIVED on each run. Every one of them was refused at -j 1, and also when applied by hand. Dockerfile-with-features, Dockerfile-with-features.dockerignore and devcontainer-features.builtin.env were left in the checkout after the run.

Fix

  • The run's budget of test processes (slots) is now a pool of Trees, one per -j. A tree is a private copy of the module, made lazily the first time its slot is used, and reset to the source's state after every test process.
  • What gets copied: what git sees, i.e. tracked files plus files that are untracked but not ignored. Tests still read testdata/ and ../../docs/... as they would in place. Directories git ignores (node_modules, build output) and nested repositories are symlinked rather than copied, so they stay readable. Ignored files are left out. Outside a git work tree, every file under the module root is copied.
  • Building: packages are still built from the real checkout. Run now builds with go test -c and runs the binary in the copy, because go test always runs a binary in its package's directory. It still prints go test's ok/FAIL <pkg> line per package.
  • Reporting: anything the tests left behind, changed or removed is reported in the run's output and in Tally.Wrote. The source tree is never the working directory.

Why this design: a per-worker copy is paid for once per worker, not once per edit. Resetting costs one directory walk comparing size, mode and mtime. A copy of only the package directory would break the common ../.. reads.

Behavior changes:

  • Alone-built edits are built with -vet=off, matching schema builds.
  • Process cost labels for alone runs are split into build and run.
  • Ask, Run and Cover take a tree or tree slots and return an error when the copy cannot be made or reset.

Tests

  • TestATestThatWritesInItsDirectoryWritesInACopyOfItsOwn (RunAll) and TestAnEditBuiltAloneIsAskedInACopyOfItsOwn (Run) use a package whose test leaves note in its directory and reads testdata/ and ../doc.txt. When Tree.in is made to hand back the real directory (the old behavior), both edits that stop Save writing are reported SURVIVED and p/note is left in the source. With the fix, both edits are refused and the source stays clean.
  • TestATreeIsTheModuleAsGitSeesItPutBackAfterEachRun covers how a copy is made and reset under git.
  • TestAnEditItsOwnTestsLetThroughIsAskedOfEachImporterInTurn checks that an edit the package's own tests let through is still asked of each importer in turn.

Gates

  • task test: pass, coverage 80.5% (floor 60%)
  • task lint: 0 issues, and every go-tools gate reports OK
  • task mutate: 144 refused, 0 survived, 14 did not build
  • task site: builds
  • UPDATE_GOLDEN is not needed: go test ./cmd/mutate/ passes unchanged.

Follow-up: a temporary directory per test process (49c8d8f)

Checking the PR against devcontainer-cli at -j 4 still turned up a false survivor on some runs (feature_install.go:700, then :757). The cause is that all test processes share os.TempDir(). The code under test creates <tmp>/devcontainercli-<user>/container-features/<version>-<unix ms>. Two coverage runs started in the same millisecond broke each other, so both tests were marked as failing with nothing broken and asked about no edit, and the edits only they reached survived.

Each tree now gives its process an empty TMPDIR, which is removed after the process.

  • Regression test: TestEachTestProcessHasATemporaryDirectoryOfItsOwn. Two tests each claim a file in os.TempDir() with O_EXCL and leave it there. Before the fix, Half's unchecked edit is refused because of another run's claim, and the files leak into the temp dir mutate was given. Both checks pass with the fix.
  • Consumer check: devcontainer-cli, with feature_install.go as the diff, -j 4, run 5 times in a row. Every run had 206 refused, 0 survived, no "fails with nothing broken" coverage notes, a clean worktree, and no new directories in the real /tmp.
  • Gates: task test passes (80.5%), task lint reports 0 issues with every gate OK, task mutate reports 143 refused and 0 survived, and task site builds.

🤖 Generated with Claude Code

aledbf and others added 2 commits October 25, 2026 21:54
Every test process ran with its working directory in the package's
source directory: a schema binary (Ask), each coverage run (Cover) and
go test for an edit built alone (Run). Edits asked side by side shared
that directory, so a file one edit's tests wrote was read by another's:
in github.com/devcontainers/cli, -j 4 let a different set of
extendImageWithFeatures edits through on each run, every one refused at
-j 1, and Dockerfile-with-features and its siblings were left in the
checkout afterwards.

The run's budget of test processes is now a pool of trees, one per -j.
A tree is a copy of the module as git sees it (tracked and not ignored
files, with ignored directories and nested repositories linked), made
the first time a slot is used and put back as the source is after every
process. Tests are still built where the sources are; Run builds with
go test -c and runs the binary in the copy, since go test always runs
it in the package's directory. Whatever the tests left, changed or
removed is named in the run's report (Tally.Wrote).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mutate runs the same package's tests in many processes at once, which
go test never does, and they all shared os.TempDir(). In
github.com/devcontainers/cli the code under test makes
<tmp>/devcontainercli-<user>/container-features/<version>-<unix ms>:
two coverage runs started in the same millisecond broke each other,
both tests were marked as failing with nothing broken and asked about
no edit, and a different edit only they reached survived at -j 4 on
some runs.

Each tree now hands the process it runs an empty TMPDIR under it,
removed after the process, beside the copy of the module.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@aledbf
aledbf merged commit 5fc0868 into main Oct 26, 2026
3 checks passed
@aledbf
aledbf deleted the fix/mutate-isolate-mutants branch October 26, 2026 01:18
aledbf added a commit to spin-stack/devcontainer-cli that referenced this pull request Oct 26, 2026
go-tools 5fc0868 runs each mutate test process in a private copy of the
module with a temporary directory of its own (spin-stack/go-tools#69).
Here, mutants of internal/cli wrote Dockerfile-with-features and a
timestamped cache folder into shared places, so `mutate -j 4` reported
survivors that a serial run refused, a different set each run. The pin
also brings testquality's check for tests that sleep a second or more;
it finds none here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant