Repository navigation
fix(mutate): run each test process in a private copy of the module - #69
Merged
Merged
Conversation
Every test process ran with its working directory in the package's source directory: a schema binary (Ask), each coverage run (Cover) and go test for an edit built alone (Run). Edits asked side by side shared that directory, so a file one edit's tests wrote was read by another's: in github.com/devcontainers/cli, -j 4 let a different set of extendImageWithFeatures edits through on each run, every one refused at -j 1, and Dockerfile-with-features and its siblings were left in the checkout afterwards. The run's budget of test processes is now a pool of trees, one per -j. A tree is a copy of the module as git sees it (tracked and not ignored files, with ignored directories and nested repositories linked), made the first time a slot is used and put back as the source is after every process. Tests are still built where the sources are; Run builds with go test -c and runs the binary in the copy, since go test always runs it in the package's directory. Whatever the tests left, changed or removed is named in the run's report (Tally.Wrote). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mutate runs the same package's tests in many processes at once, which go test never does, and they all shared os.TempDir(). In github.com/devcontainers/cli the code under test makes <tmp>/devcontainercli-<user>/container-features/<version>-<unix ms>: two coverage runs started in the same millisecond broke each other, both tests were marked as failing with nothing broken and asked about no edit, and a different edit only they reached survived at -j 4 on some runs. Each tree now hands the process it runs an empty TMPDIR under it, removed after the process, beside the copy of the module. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
aledbf
added a commit
to spin-stack/devcontainer-cli
that referenced
this pull request
Oct 26, 2026
go-tools 5fc0868 runs each mutate test process in a private copy of the module with a temporary directory of its own (spin-stack/go-tools#69). Here, mutants of internal/cli wrote Dockerfile-with-features and a timestamped cache folder into shared places, so `mutate -j 4` reported survivors that a serial run refused, a different set each run. The pin also brings testquality's check for tests that sleep a second or more; it finds none here. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
mutateran every test process with its working directory set to the package's real source directory: schema binaries (Ask), each per-test coverage run (Cover), andgo testfor edits built alone (Run). Edits asked concurrently shared that directory, so files one edit's tests wrote were read by another's tests.In
github.com/devcontainers/cli,-j 4reported a different set ofextendImageWithFeaturesedits as SURVIVED on each run. Every one of them was refused at-j 1, and also when applied by hand.Dockerfile-with-features,Dockerfile-with-features.dockerignoreanddevcontainer-features.builtin.envwere left in the checkout after the run.Fix
slots) is now a pool ofTrees, one per-j. A tree is a private copy of the module, made lazily the first time its slot is used, and reset to the source's state after every test process.testdata/and../../docs/...as they would in place. Directories git ignores (node_modules, build output) and nested repositories are symlinked rather than copied, so they stay readable. Ignored files are left out. Outside a git work tree, every file under the module root is copied.Runnow builds withgo test -cand runs the binary in the copy, becausego testalways runs a binary in its package's directory. It still prints go test'sok/FAIL <pkg>line per package.Tally.Wrote. The source tree is never the working directory.Why this design: a per-worker copy is paid for once per worker, not once per edit. Resetting costs one directory walk comparing size, mode and mtime. A copy of only the package directory would break the common
../..reads.Behavior changes:
-vet=off, matching schema builds.buildandrun.Ask,RunandCovertake a tree or tree slots and return an error when the copy cannot be made or reset.Tests
TestATestThatWritesInItsDirectoryWritesInACopyOfItsOwn(RunAll) andTestAnEditBuiltAloneIsAskedInACopyOfItsOwn(Run) use a package whose test leavesnotein its directory and readstestdata/and../doc.txt. WhenTree.inis made to hand back the real directory (the old behavior), both edits that stopSavewriting are reported SURVIVED andp/noteis left in the source. With the fix, both edits are refused and the source stays clean.TestATreeIsTheModuleAsGitSeesItPutBackAfterEachRuncovers how a copy is made and reset under git.TestAnEditItsOwnTestsLetThroughIsAskedOfEachImporterInTurnchecks that an edit the package's own tests let through is still asked of each importer in turn.Gates
task test: pass, coverage 80.5% (floor 60%)task lint: 0 issues, and every go-tools gate reports OKtask mutate: 144 refused, 0 survived, 14 did not buildtask site: buildsUPDATE_GOLDENis not needed:go test ./cmd/mutate/passes unchanged.Follow-up: a temporary directory per test process (49c8d8f)
Checking the PR against devcontainer-cli at
-j 4still turned up a false survivor on some runs (feature_install.go:700, then:757). The cause is that all test processes shareos.TempDir(). The code under test creates<tmp>/devcontainercli-<user>/container-features/<version>-<unix ms>. Two coverage runs started in the same millisecond broke each other, so both tests were marked as failing with nothing broken and asked about no edit, and the edits only they reached survived.Each tree now gives its process an empty
TMPDIR, which is removed after the process.TestEachTestProcessHasATemporaryDirectoryOfItsOwn. Two tests each claim a file inos.TempDir()withO_EXCLand leave it there. Before the fix, Half's unchecked edit is refused because of another run's claim, and the files leak into the temp dir mutate was given. Both checks pass with the fix.feature_install.goas the diff,-j 4, run 5 times in a row. Every run had 206 refused, 0 survived, no "fails with nothing broken" coverage notes, a clean worktree, and no new directories in the real/tmp.task testpasses (80.5%),task lintreports 0 issues with every gate OK,task mutatereports 143 refused and 0 survived, andtask sitebuilds.🤖 Generated with Claude Code