From 020a97dd64b704484da515abce72ace85086c1b9 Mon Sep 17 00:00:00 2001 From: japabu Date: Sat, 3 Oct 2026 11:49:42 +0200 Subject: [PATCH 1/6] A build reads its fork checkout, and only its process's start moves it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `sysroot::fork_checkout` made a linked worktree's `rust/` and moved one behind its pin, and every build called it: the guest suite's workers reach `toolchain::ensure` on their own threads, so the first run of a new worktree, and the first after a merge moved the gitlink, ran `git worktree add` or `git checkout --detach` on one checkout from every worker at once. Measured at ec7de748c with `cargo test --test toyos-build -- screen_`, three workers, on 14 cores: - a new worktree's first run, at load average 74, exited 1 with 2 of 3 red: two workers each ran `git worktree add`, one died on `fatal: '…/rust' already exists`, and the third read the checkout the winner was still writing and died in `llvm::refuse_uncommitted_bootstrap` on every file of `src/bootstrap` as `D`; - with the checkout one commit behind its pin, at load average 23, it exited 1 with 2 of 3 red on `Unable to create '…/worktrees/rust2/index.lock': File exists`. The move is now `sysroot::make_fork_checkout`, which a process calls once where it starts, before it has a second thread: `cargo run` beside the primary's `ensure_submodules`, the harness before its first build, and the licence gate. `fork_checkout` is what a build calls, and it moves nothing: a checkout that is not at the pin or ahead of it is refused by name, which is what a build sees when the pin moves under a run. So no worker is a mover and there is nothing to lock. `a_build_reads_the_fork_checkout_and_never_moves_it` starts twelve builds at once against a checkout that is not made and against one behind its pin: each is refused and the checkout stands as it was, and all twelve read the one `make_fork_checkout` made and the one it moved. With `fork_checkout` calling `make_fork_checkout` again it exits 101, each of the twelve running `git worktree add`. Closes issues/build/a-suites-first-run-after-the-fork-pin-moves-races-its-own-checkout.md and issues/build/a-worktrees-first-wide-run-reds-on-the-fork-checkout-it-is-still-making.md, two records of this one defect. The `git submodule update --init library/backtrace` arm issues/build/the-fork-checkout-runs-git-submodule-in-a-linked-worktree.md records is the same code under the mover's new name. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_013UDZQ6fSKw14e4w2TKTRfm --- ...e-fork-pin-moves-races-its-own-checkout.md | 40 ----- ...on-the-fork-checkout-it-is-still-making.md | 24 --- ...runs-git-submodule-in-a-linked-worktree.md | 8 +- src/licence.rs | 1 + src/main.rs | 2 + src/sysroot.rs | 149 ++++++++++++++---- tests/toyos.rs | 4 + 7 files changed, 126 insertions(+), 102 deletions(-) delete mode 100644 issues/build/a-suites-first-run-after-the-fork-pin-moves-races-its-own-checkout.md delete mode 100644 issues/build/a-worktrees-first-wide-run-reds-on-the-fork-checkout-it-is-still-making.md diff --git a/issues/build/a-suites-first-run-after-the-fork-pin-moves-races-its-own-checkout.md b/issues/build/a-suites-first-run-after-the-fork-pin-moves-races-its-own-checkout.md deleted file mode 100644 index 19139ad49dd..00000000000 --- a/issues/build/a-suites-first-run-after-the-fork-pin-moves-races-its-own-checkout.md +++ /dev/null @@ -1,40 +0,0 @@ ---- -status: open -kind: tooling -opened: 2026-10-03 ---- - -# A suite's first run after the fork pin moves races its own checkout - -`sysroot::fork_checkout` (`src/sysroot.rs`) moves a linked worktree's `rust/` -to the commit the tree pins when the checkout is behind it, with -`git checkout --detach`. The guest suite boots its tests on twelve threads and -each reaches `fork_checkout` on its own, so the first run after a merge that -moved the gitlink runs that checkout in every thread at once. The same unlocked -function reds a new worktree's first run while `git worktree add` is still -writing: -`issues/build/a-worktrees-first-wide-run-reds-on-the-fork-checkout-it-is-still-making.md`. - -## Measured - -A worktree at `bf28c1e38` whose `rust/` stood at `c4c65e3e8` under a pin of -`95960d6c2`, after a merge of `origin/main`: `cargo test --test toyos-build` -exited 1 with 18 of 26 red in 177 s. Every red is one of two sentences. One is -`git ["checkout", "--detach", "-q", "95960d6c2…"] in …/rust failed`, under -git's `Unable to create '…/modules/rust/worktrees/rust1/index.lock': File -exists`. The other is `…/rust is at c4c65e3e8… with uncommitted work`, from a -thread that read the checkout's status while another thread was moving it. The -checkout ended at the pin, and the next run of the same command did not repeat -either. - -## Owner - -The toolchain stage of -`issues/build/the-tooling-is-a-review-prompt-and-three-workflows.md`. - -## What would close it - -One mover: the checkout is moved under a lock every thread of the process -takes, or once before the suite starts its workers, and -`cargo test --test toyos-build` on a worktree whose checkout is behind its pin -moves it and boots its tests. diff --git a/issues/build/a-worktrees-first-wide-run-reds-on-the-fork-checkout-it-is-still-making.md b/issues/build/a-worktrees-first-wide-run-reds-on-the-fork-checkout-it-is-still-making.md deleted file mode 100644 index b3a0dac0624..00000000000 --- a/issues/build/a-worktrees-first-wide-run-reds-on-the-fork-checkout-it-is-still-making.md +++ /dev/null @@ -1,24 +0,0 @@ ---- -status: open -kind: tooling -opened: 2026-10-02 ---- - -# A worktree's first wide run reds on the fork checkout it is still making - -`sysroot::fork_checkout` makes a new worktree's `rust/` with `git worktree add` -under no lock a second thread of the same process waits on: it tests -`rust/.git`, which exists from the first moment of the checkout, and every -other worker then takes the checkout as made while git is still writing its -files. - -The first `cargo test --test toyos-build -- screen_` in a new worktree, three -wide, on a host at load average 35 of 14 cores: one worker printed `Making -…/rust a fork checkout`, and within 0.4 s the other two panicked in -`llvm::refuse_uncommitted_bootstrap` (`src/llvm.rs`) on `rust/src/bootstrap -holds changes no commit does`, listing every file of `src/bootstrap` as `D`. -`screen_panic_muted` and `screen_fatal_halt_composited` red with that -sentence, the run exited 1, and the same command passed once the checkout -existed. - -**Exit**: the first guest run of a new worktree is green at any width. diff --git a/issues/build/the-fork-checkout-runs-git-submodule-in-a-linked-worktree.md b/issues/build/the-fork-checkout-runs-git-submodule-in-a-linked-worktree.md index a37a9a3e683..206f6cc1fb2 100644 --- a/issues/build/the-fork-checkout-runs-git-submodule-in-a-linked-worktree.md +++ b/issues/build/the-fork-checkout-runs-git-submodule-in-a-linked-worktree.md @@ -6,7 +6,7 @@ opened: 2026-09-30 # The fork checkout runs `git submodule` in a linked worktree -`sysroot::fork_checkout` makes a worktree's `rust/` a linked worktree of the +`sysroot::make_fork_checkout` makes a worktree's `rust/` a linked worktree of the primary's `rust` (`git worktree add --detach`). When the primary's `library/backtrace` lacks the commit the fork pins, it runs `git submodule update --init library/backtrace` inside that linked worktree, @@ -20,10 +20,10 @@ exist; this arm is the same command one level down and is unmeasured. `ensure_submodule` (`src/lib.rs`) runs `git submodule update --init library/backtrace` in the same fork checkout, from `sysroot::build_std` and `compiler::build_in_fork`, whenever that checkout's `library/backtrace` is -empty or gone: what a first `fork_checkout` leaves when it stops after adding +empty or gone: what a first `make_fork_checkout` leaves when it stops after adding the fork's worktree and before adding `library/backtrace`. -Bootstrap does the same for `src/llvm-project`, which `fork_checkout` leaves an +Bootstrap does the same for `src/llvm-project`, which `make_fork_checkout` leaves an empty directory. When the checkout's LLVM key is in no store, `llvm::build_in_fork` runs `x build src/llvm-project/llvm`, and bootstrap's `update_submodule` (fork `aca5f527`, @@ -33,7 +33,7 @@ sync` and `git submodule update --init --recursive --depth=1 src/llvm-project` in the linked worktree. This arm is read from the code and unmeasured. Bootstrap returns at `actual_hash == checked_out_hash` instead once `src/llvm-project` is a `git worktree add --detach` at the gitlink from a -repository that holds the commit, as `fork_checkout` makes `library/backtrace`; +repository that holds the commit, as `make_fork_checkout` makes `library/backtrace`; one added from a sparse clone inherits its patterns until `git sparse-checkout disable` runs in it. diff --git a/src/licence.rs b/src/licence.rs index baf8c6ef70a..f5d9d28ab5c 100644 --- a/src/licence.rs +++ b/src/licence.rs @@ -1174,6 +1174,7 @@ fn metadata( /// whose `rust/` was never initialised — a CI runner's — fetches that commit /// alone. fn std_library(root: &Path) -> Result { + crate::sysroot::make_fork_checkout(root); let fork = crate::sysroot::fork_checkout(root); if !fork.join("library/Cargo.toml").exists() { crate::ensure_shallow_fork(root)?; diff --git a/src/main.rs b/src/main.rs index fb9ccd2ed8a..0f9826314fa 100644 --- a/src/main.rs +++ b/src/main.rs @@ -205,6 +205,8 @@ fn main() { if matches!(toyos_build::toolchain::owner(&root), toyos_build::toolchain::Owner::Us) { toyos_build::ensure_submodules(&root); } + // There it is a fork checkout, and this is the process's one move of it. + toyos_build::sysroot::make_fork_checkout(&root); // Toolchain included: `build` holds the build lock across both, so no other // agent's clean or bootstrap can land between the two. diff --git a/src/sysroot.rs b/src/sysroot.rs index ca973dbfe65..a0a40a7f68d 100644 --- a/src/sysroot.rs +++ b/src/sysroot.rs @@ -20,8 +20,10 @@ //! moved ([`Identity`]). Each build refuses dep-info that says otherwise. //! //! **Each worktree builds std in its own fork checkout.** The primary builds in its `rust/`; -//! a linked worktree in its own `rust/`, made on first need as a git worktree of -//! the primary's fork repository at the commit this tree pins ([`fork_checkout`]). +//! a linked worktree in its own `rust/`, made where its process starts as a git +//! worktree of the primary's fork repository at the commit this tree pins +//! ([`make_fork_checkout`]), and read by every build after that +//! ([`fork_checkout`]). //! `library/std` names `toyos-abi` and `toyos` as `../../../`, so each //! checkout's std compiles against its own worktree's ABI with nothing //! rewritten. The build is bootstrap's stage-0 local rebuild: the compiler the @@ -494,28 +496,63 @@ fn pinned_fork(root: &Path) -> String { } } -/// The fork checkout `root`'s std is built in. +/// The commit the fork checkout at `fork` holds, where a tree pinning `pinned` +/// does not build its std there: it is neither that commit nor ahead of it. +fn off_pin(fork: &Path, pinned: &str) -> Option { + let head = git_out(fork, &["rev-parse", "HEAD"]).trim().to_string(); + let ahead = Command::new("git") + .args(["merge-base", "--is-ancestor", pinned, &head]) + .current_dir(fork) + .status() + .is_ok_and(|s| s.success()); + (head != pinned && !ahead).then_some(head) +} + +/// The fork checkout `root`'s std is built in: the primary's own `rust/`, or a +/// linked worktree's as [`make_fork_checkout`] left it. Every worker of a +/// process builds at once, so a build moves no checkout: one that is not at the +/// commit this tree pins or ahead of it is refused by name. +pub fn fork_checkout(root: &Path) -> PathBuf { + let fork = root.join("rust"); + match toolchain::owner(root) { + Owner::Us => {} + Owner::Installed => panic!("an installed toolchain has no fork checkout to build std in"), + Owner::Elsewhere(_) => { + let pinned = pinned_fork(root); + assert!( + fork.join(".git").exists() && off_pin(&fork, &pinned).is_none(), + "{} is not a fork checkout at {pinned}, which this tree pins, or ahead of it, and \ + a build moves none: `sysroot::make_fork_checkout` does, where a process starts. \ + One that changed under this run is moved by the next.", + fork.display(), + ); + } + } + fork +} + +/// Make a linked worktree's fork checkout the one its tree pins. The primary's +/// is its own `rust/`, and an installed toolchain has none. /// -/// The primary's is its own `rust/`. A linked worktree's `rust/` starts as the -/// empty stub `git worktree add` leaves; it is made here, the first time it is -/// needed, as a git worktree of the primary's fork repository at the commit -/// this tree pins, sharing its objects — and `library/backtrace` the same way -/// from the primary's, or by git's own clone where the primary does not hold -/// that commit. +/// A linked worktree's `rust/` starts as the empty stub `git worktree add` +/// leaves; it is made here as a git worktree of the primary's fork repository +/// at the commit this tree pins, sharing its objects — and `library/backtrace` +/// the same way from the primary's, or by git's own clone where the primary +/// does not hold that commit. /// -/// A checkout that exists is used as it stands, which is where an agent edits +/// A checkout that exists is left as it stands, which is where an agent edits /// the fork; one whose `HEAD` is neither the pinned commit nor ahead of it is -/// moved there itself, fetching the commit from the primary's repository first -/// if the checkout does not already hold it, unless the checkout has local -/// changes, in which case it is refused by name rather than moved out from -/// under whoever made them. -pub fn fork_checkout(root: &Path) -> PathBuf { +/// moved there, fetching the commit from the primary's repository first if the +/// checkout does not already hold it, unless the checkout has local changes, in +/// which case it is refused by name rather than moved out from under whoever +/// made them. +/// +/// **Its caller is the checkout's one mover**: a process calls it where it +/// starts, before a second thread of it builds. Two movers of one checkout +/// corrupt it, and nothing here keeps a second out. +pub fn make_fork_checkout(root: &Path) { + let Owner::Elsewhere(primary) = toolchain::owner(root) else { return }; let fork = root.join("rust"); - let primary = match toolchain::owner(root) { - Owner::Us => return fork, - Owner::Installed => panic!("an installed toolchain has no fork checkout to build std in"), - Owner::Elsewhere(primary) => primary, - }; let pinned = pinned_fork(root); if !fork.join(".git").exists() { let stub = fs::read_dir(&fork).map_or(0, |d| d.count()); @@ -544,18 +581,9 @@ pub fn fork_checkout(root: &Path) -> PathBuf { } else { git_run(&fork, &["submodule", "update", "--init", "library/backtrace"]); } - return fork; - } - let head = git_out(&fork, &["rev-parse", "HEAD"]); - let head = head.trim(); - let ahead = Command::new("git") - .args(["merge-base", "--is-ancestor", &pinned, head]) - .current_dir(&fork) - .status() - .is_ok_and(|s| s.success()); - if head == pinned || ahead { - return fork; + return; } + let Some(head) = off_pin(&fork, &pinned) else { return }; let dirty = git_out(&fork, &["status", "--porcelain", "--ignore-submodules=none"]); assert!( dirty.is_empty(), @@ -574,7 +602,6 @@ pub fn fork_checkout(root: &Path) -> PathBuf { } git_run(&fork, &["checkout", "--detach", "-q", &pinned]); eprintln!("{} was at {head}, behind this tree's pin {pinned}: checked it out", fork.display()); - fork } /// What a sysroot's [`SOURCES`] says: its key, the fork checkout its std was @@ -1432,6 +1459,7 @@ mod tests { let (primary, linked, c1, c2) = two_pins(&base); let before = git(&primary.join("rust"), &["status", "--porcelain"]); + make_fork_checkout(&linked); let fork = fork_checkout(&linked); assert_eq!(fork, linked.join("rust")); @@ -1450,22 +1478,75 @@ mod tests { // Work on the fork in the worktree's own checkout is what it builds. write(&fork.join("library/std/src/lib.rs"), "pub fn c() {}\n"); git(&fork, &["commit", "-qam", "C3, the agent's own"]); + make_fork_checkout(&linked); assert_eq!(fork_checkout(&linked), fork); // A clean checkout behind what the tree pins is moved to the pin itself. git(&fork, &["checkout", "-q", &c1]); - assert_eq!(fork_checkout(&linked), fork); + make_fork_checkout(&linked); assert_eq!(git(&fork, &["rev-parse", "HEAD"]), c2, "a checkout behind its pin was not moved to it"); // One with local changes is never moved out from under whoever made them. git(&fork, &["checkout", "-q", &c1]); write(&fork.join("library/std/src/lib.rs"), "pub fn uncommitted() {}\n"); - let refused = std::panic::catch_unwind(|| fork_checkout(&linked)) + let refused = std::panic::catch_unwind(|| make_fork_checkout(&linked)) .expect_err("a fork checkout with local changes was moved out from under them"); let message = refused.downcast::().expect("a formatted refusal"); assert!(message.contains(&c1) && message.contains(&c2), "{message}"); } + /// What each of twelve builds starting at once in `linked` read as its fork + /// checkout, or was refused with. + fn twelve_builds_read(linked: &Path) -> Vec> { + let start = std::sync::Barrier::new(12); + std::thread::scope(|builds| { + let reads: Vec<_> = (0..12) + .map(|_| { + builds.spawn(|| { + start.wait(); + std::panic::catch_unwind(|| fork_checkout(linked)) + .map_err(|refusal| *refusal.downcast::().expect("a formatted refusal")) + }) + }) + .collect(); + reads.into_iter().map(|read| read.join().expect("a build's refusal is caught")).collect() + }) + } + + /// **A build reads its fork checkout and never moves it**: twelve at once + /// neither make one that is not there nor move one behind its pin — each is + /// refused by name and the checkout stands as it was — and all twelve read + /// the one [`make_fork_checkout`] made, and the one it moved. + #[test] + fn a_build_reads_the_fork_checkout_and_never_moves_it() { + let base = TempDir::new("fork-builds"); + let (_primary, linked, c1, c2) = two_pins(&base); + let fork = linked.join("rust"); + let head = || git(&fork, &["rev-parse", "HEAD"]); + let refused = || { + for read in twelve_builds_read(&linked) { + let said = read.expect_err("a build took a fork checkout its tree does not pin"); + assert!(said.contains(&c2) && said.contains("`sysroot::make_fork_checkout`"), "{said}"); + } + }; + let read = || assert_eq!(twelve_builds_read(&linked), vec![Ok(fork.clone()); 12]); + + refused(); + assert!(!fork.join(".git").exists(), "a build made the fork checkout"); + + make_fork_checkout(&linked); + read(); + assert_eq!(head(), c2); + + git(&fork, &["checkout", "-q", &c1]); + refused(); + assert_eq!(head(), c1, "a build moved the fork checkout"); + + make_fork_checkout(&linked); + read(); + assert_eq!(head(), c2); + } + /// The primary's compiler under `base`: `rustc` and `rust-lld`, and the C /// toolchain `src/clang.rs` provisions beside them if `clang`; no cargo. fn primary_compiler(base: &Path, clang: bool) -> Compiler { diff --git a/tests/toyos.rs b/tests/toyos.rs index 097e00fe3f9..4791f577acf 100644 --- a/tests/toyos.rs +++ b/tests/toyos.rs @@ -4000,6 +4000,10 @@ fn main() { } }; + // Before anything is built and before a worker exists: the process's one + // move of its fork checkout. + toyos_build::sysroot::make_fork_checkout(&compile::repo_root()); + if let Some(mode) = parsed.metal { let (c_bins, rust_bins) = build_shared_bins(); let selected: Vec<(&str, &'static metal::Metal)> = METAL From e38dbc314db342c49ff47e894a09ee3a387456f3 Mon Sep 17 00:00:00 2001 From: japabu Date: Sat, 3 Oct 2026 11:52:09 +0200 Subject: [PATCH 2/6] issues: two processes starting in one worktree are two movers of its fork checkout MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit One process is one mover now; two are still two. Measured with the mover called where a process starts (020a97dd6), from an empty `rust/` stub at load average 36 of 14 cores: two `cargo test --test toyos-build` started together, one on `screen_panic_muted` and one on `screen_fatal_halt_composited`, both printed `Making …/rust a fork checkout`, and they exited 101 and 0. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_013UDZQ6fSKw14e4w2TKTRfm --- ...ree-are-two-movers-of-its-fork-checkout.md | 31 +++++++++++++++++++ 1 file changed, 31 insertions(+) create mode 100644 issues/build/two-processes-starting-in-one-worktree-are-two-movers-of-its-fork-checkout.md diff --git a/issues/build/two-processes-starting-in-one-worktree-are-two-movers-of-its-fork-checkout.md b/issues/build/two-processes-starting-in-one-worktree-are-two-movers-of-its-fork-checkout.md new file mode 100644 index 00000000000..0b92518df1b --- /dev/null +++ b/issues/build/two-processes-starting-in-one-worktree-are-two-movers-of-its-fork-checkout.md @@ -0,0 +1,31 @@ +--- +status: open +kind: tooling +opened: 2026-10-03 +--- + +# Two processes starting in one worktree are two movers of its fork checkout + +`sysroot::make_fork_checkout` (`src/sysroot.rs`) makes a linked worktree's +`rust/`, or moves it to the commit the tree pins, once where a process starts, +so one process is one mover. Two build-system processes that start in one +worktree while its checkout is not made or is behind its pin are two, under no +lock: both run `git worktree add` or `git checkout --detach` there. + +## Measured + +In a worktree whose `rust/` was the empty stub, on a host at load average 36 +of 14 cores, `cargo test --test toyos-build -- screen_panic_muted` and +`cargo test --test toyos-build -- screen_fatal_halt_composited` were started +together. Both printed `Making …/rust a fork checkout`. The first exited 101 on +git's `fatal: '…/rust' already exists`; the second made the checkout and exited +0. + +## Owner + +The toolchain item of +`issues/build/the-tooling-is-a-review-prompt-and-three-workflows.md`, which +makes the build system's locks unnecessary rather than writing them down. + +**Exit**: two build-system processes started together in a new worktree both +exit 0. From a8dcbccb280038c0811c5195616fb1170cdb6d9a Mon Sep 17 00:00:00 2001 From: japabu Date: Sat, 3 Oct 2026 12:55:26 +0200 Subject: [PATCH 3/6] The fork checkout is made and moved under its worktree's lock, one act_if each MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The first round moved the checkout once where a process starts and had every build refuse one off its pin. The review of #683 found that this left two processes as two movers (measured there: exits 101 and 0), let a process arriving inside the `git worktree add` take a checkout still being written, and made a plain `--list` make a 61,561-file checkout, while the tree already held the mechanism for a decide-then-act defect on state a worktree owns: `Held::act_if(Scope::Worktree, …)`, which `build::invalidate_stale` uses, on the `lock` `sysroot::ensure` already receives. So `fork_checkout` is main's again and takes that lock: the making is one `act_if` and the move to the pin another. `flock` is per open file description, so twelve workers' `Held`s exclude each other as two processes do. `licence::std_library` takes the worktree's lock shared for the call, as a build does. The three `make_fork_checkout` call sites, the refusal in `fork_checkout`, the issue file the first round added and its four renames in the submodule issue are gone. The twelve-thread test of readers goes. The test that replaces it starts twelve builds at once on a checkout not made and on one behind its pin; each returns it whole and at the pin. The ahead-of-pin case now asserts the commit. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_013UDZQ6fSKw14e4w2TKTRfm --- ...runs-git-submodule-in-a-linked-worktree.md | 8 +- ...ree-are-two-movers-of-its-fork-checkout.md | 31 -- src/buildlock.rs | 9 +- src/licence.rs | 4 +- src/main.rs | 2 - src/sysroot.rs | 277 ++++++++---------- tests/toyos.rs | 4 - 7 files changed, 131 insertions(+), 204 deletions(-) delete mode 100644 issues/build/two-processes-starting-in-one-worktree-are-two-movers-of-its-fork-checkout.md diff --git a/issues/build/the-fork-checkout-runs-git-submodule-in-a-linked-worktree.md b/issues/build/the-fork-checkout-runs-git-submodule-in-a-linked-worktree.md index 206f6cc1fb2..a37a9a3e683 100644 --- a/issues/build/the-fork-checkout-runs-git-submodule-in-a-linked-worktree.md +++ b/issues/build/the-fork-checkout-runs-git-submodule-in-a-linked-worktree.md @@ -6,7 +6,7 @@ opened: 2026-09-30 # The fork checkout runs `git submodule` in a linked worktree -`sysroot::make_fork_checkout` makes a worktree's `rust/` a linked worktree of the +`sysroot::fork_checkout` makes a worktree's `rust/` a linked worktree of the primary's `rust` (`git worktree add --detach`). When the primary's `library/backtrace` lacks the commit the fork pins, it runs `git submodule update --init library/backtrace` inside that linked worktree, @@ -20,10 +20,10 @@ exist; this arm is the same command one level down and is unmeasured. `ensure_submodule` (`src/lib.rs`) runs `git submodule update --init library/backtrace` in the same fork checkout, from `sysroot::build_std` and `compiler::build_in_fork`, whenever that checkout's `library/backtrace` is -empty or gone: what a first `make_fork_checkout` leaves when it stops after adding +empty or gone: what a first `fork_checkout` leaves when it stops after adding the fork's worktree and before adding `library/backtrace`. -Bootstrap does the same for `src/llvm-project`, which `make_fork_checkout` leaves an +Bootstrap does the same for `src/llvm-project`, which `fork_checkout` leaves an empty directory. When the checkout's LLVM key is in no store, `llvm::build_in_fork` runs `x build src/llvm-project/llvm`, and bootstrap's `update_submodule` (fork `aca5f527`, @@ -33,7 +33,7 @@ sync` and `git submodule update --init --recursive --depth=1 src/llvm-project` in the linked worktree. This arm is read from the code and unmeasured. Bootstrap returns at `actual_hash == checked_out_hash` instead once `src/llvm-project` is a `git worktree add --detach` at the gitlink from a -repository that holds the commit, as `make_fork_checkout` makes `library/backtrace`; +repository that holds the commit, as `fork_checkout` makes `library/backtrace`; one added from a sparse clone inherits its patterns until `git sparse-checkout disable` runs in it. diff --git a/issues/build/two-processes-starting-in-one-worktree-are-two-movers-of-its-fork-checkout.md b/issues/build/two-processes-starting-in-one-worktree-are-two-movers-of-its-fork-checkout.md deleted file mode 100644 index 0b92518df1b..00000000000 --- a/issues/build/two-processes-starting-in-one-worktree-are-two-movers-of-its-fork-checkout.md +++ /dev/null @@ -1,31 +0,0 @@ ---- -status: open -kind: tooling -opened: 2026-10-03 ---- - -# Two processes starting in one worktree are two movers of its fork checkout - -`sysroot::make_fork_checkout` (`src/sysroot.rs`) makes a linked worktree's -`rust/`, or moves it to the commit the tree pins, once where a process starts, -so one process is one mover. Two build-system processes that start in one -worktree while its checkout is not made or is behind its pin are two, under no -lock: both run `git worktree add` or `git checkout --detach` there. - -## Measured - -In a worktree whose `rust/` was the empty stub, on a host at load average 36 -of 14 cores, `cargo test --test toyos-build -- screen_panic_muted` and -`cargo test --test toyos-build -- screen_fatal_halt_composited` were started -together. Both printed `Making …/rust a fork checkout`. The first exited 101 on -git's `fatal: '…/rust' already exists`; the second made the checkout and exited -0. - -## Owner - -The toolchain item of -`issues/build/the-tooling-is-a-review-prompt-and-three-workflows.md`, which -makes the build system's locks unnecessary rather than writing them down. - -**Exit**: two build-system processes started together in a new worktree both -exit 0. diff --git a/src/buildlock.rs b/src/buildlock.rs index cdf7ed00ec0..d4a95e7af6f 100644 --- a/src/buildlock.rs +++ b/src/buildlock.rs @@ -15,8 +15,8 @@ //! - **shared** — "I am building against the state as it stands". Any number //! at once. //! - **exclusive** — "I am replacing it": the rust bootstrap, this worktree's -//! std build, the `cargo clean`s. One at a time, and never while a build -//! holds the shared mode. +//! std build, the `cargo clean`s, the making or moving of its fork checkout. +//! One at a time, and never while a build holds the shared mode. //! //! And two [`Scope`]s: a crate target directory is shared by the builds in one //! worktree, while the primary's `rust/build` — the compiler every sysroot is @@ -94,8 +94,9 @@ pub enum Scope { /// State every worktree shares: the primary's `rust/` build tree — the /// compiler every sysroot is made with — and the machine-global rustup link. Global, - /// State this worktree alone owns — its crate target directories. Two - /// worktrees cleaning their own have nothing to say to each other. + /// State this worktree alone owns — its crate target directories and its + /// fork checkout. Two worktrees cleaning their own have nothing to say to + /// each other. Worktree, } diff --git a/src/licence.rs b/src/licence.rs index f5d9d28ab5c..cc0b165562c 100644 --- a/src/licence.rs +++ b/src/licence.rs @@ -1174,8 +1174,8 @@ fn metadata( /// whose `rust/` was never initialised — a CI runner's — fetches that commit /// alone. fn std_library(root: &Path) -> Result { - crate::sysroot::make_fork_checkout(root); - let fork = crate::sysroot::fork_checkout(root); + let mut lock = crate::buildlock::shared(root, "the licences of what ships"); + let fork = crate::sysroot::fork_checkout(root, &mut lock); if !fork.join("library/Cargo.toml").exists() { crate::ensure_shallow_fork(root)?; } diff --git a/src/main.rs b/src/main.rs index 0f9826314fa..fb9ccd2ed8a 100644 --- a/src/main.rs +++ b/src/main.rs @@ -205,8 +205,6 @@ fn main() { if matches!(toyos_build::toolchain::owner(&root), toyos_build::toolchain::Owner::Us) { toyos_build::ensure_submodules(&root); } - // There it is a fork checkout, and this is the process's one move of it. - toyos_build::sysroot::make_fork_checkout(&root); // Toolchain included: `build` holds the build lock across both, so no other // agent's clean or bootstrap can land between the two. diff --git a/src/sysroot.rs b/src/sysroot.rs index a0a40a7f68d..98f9e30eb2c 100644 --- a/src/sysroot.rs +++ b/src/sysroot.rs @@ -20,10 +20,8 @@ //! moved ([`Identity`]). Each build refuses dep-info that says otherwise. //! //! **Each worktree builds std in its own fork checkout.** The primary builds in its `rust/`; -//! a linked worktree in its own `rust/`, made where its process starts as a git -//! worktree of the primary's fork repository at the commit this tree pins -//! ([`make_fork_checkout`]), and read by every build after that -//! ([`fork_checkout`]). +//! a linked worktree in its own `rust/`, made on first need as a git worktree of +//! the primary's fork repository at the commit this tree pins ([`fork_checkout`]). //! `library/std` names `toyos-abi` and `toyos` as `../../../`, so each //! checkout's std compiles against its own worktree's ABI with nothing //! rewritten. The build is bootstrap's stage-0 local rebuild: the compiler the @@ -53,7 +51,7 @@ use std::process::Command; use sha2::{Digest, Sha256}; use crate::arch::Arch; -use crate::buildlock::{self, Guard, Held, Keyed}; +use crate::buildlock::{self, Guard, Held, Keyed, Scope}; use crate::compiler::{self, Compiler}; use crate::identity; use crate::keystore::{self, Key}; @@ -496,112 +494,101 @@ fn pinned_fork(root: &Path) -> String { } } -/// The commit the fork checkout at `fork` holds, where a tree pinning `pinned` -/// does not build its std there: it is neither that commit nor ahead of it. -fn off_pin(fork: &Path, pinned: &str) -> Option { - let head = git_out(fork, &["rev-parse", "HEAD"]).trim().to_string(); - let ahead = Command::new("git") - .args(["merge-base", "--is-ancestor", pinned, &head]) - .current_dir(fork) - .status() - .is_ok_and(|s| s.success()); - (head != pinned && !ahead).then_some(head) -} - -/// The fork checkout `root`'s std is built in: the primary's own `rust/`, or a -/// linked worktree's as [`make_fork_checkout`] left it. Every worker of a -/// process builds at once, so a build moves no checkout: one that is not at the -/// commit this tree pins or ahead of it is refused by name. -pub fn fork_checkout(root: &Path) -> PathBuf { - let fork = root.join("rust"); - match toolchain::owner(root) { - Owner::Us => {} - Owner::Installed => panic!("an installed toolchain has no fork checkout to build std in"), - Owner::Elsewhere(_) => { - let pinned = pinned_fork(root); - assert!( - fork.join(".git").exists() && off_pin(&fork, &pinned).is_none(), - "{} is not a fork checkout at {pinned}, which this tree pins, or ahead of it, and \ - a build moves none: `sysroot::make_fork_checkout` does, where a process starts. \ - One that changed under this run is moved by the next.", - fork.display(), - ); - } - } - fork -} - -/// Make a linked worktree's fork checkout the one its tree pins. The primary's -/// is its own `rust/`, and an installed toolchain has none. +/// The fork checkout `root`'s std is built in. /// -/// A linked worktree's `rust/` starts as the empty stub `git worktree add` -/// leaves; it is made here as a git worktree of the primary's fork repository -/// at the commit this tree pins, sharing its objects — and `library/backtrace` -/// the same way from the primary's, or by git's own clone where the primary -/// does not hold that commit. +/// The primary's is its own `rust/`. A linked worktree's `rust/` starts as the +/// empty stub `git worktree add` leaves; it is made here, the first time it is +/// needed, as a git worktree of the primary's fork repository at the commit +/// this tree pins, sharing its objects — and `library/backtrace` the same way +/// from the primary's, or by git's own clone where the primary does not hold +/// that commit. /// -/// A checkout that exists is left as it stands, which is where an agent edits +/// A checkout that exists is used as it stands, which is where an agent edits /// the fork; one whose `HEAD` is neither the pinned commit nor ahead of it is -/// moved there, fetching the commit from the primary's repository first if the -/// checkout does not already hold it, unless the checkout has local changes, in -/// which case it is refused by name rather than moved out from under whoever -/// made them. +/// moved there itself, fetching the commit from the primary's repository first +/// if the checkout does not already hold it, unless the checkout has local +/// changes, in which case it is refused by name rather than moved out from +/// under whoever made them. /// -/// **Its caller is the checkout's one mover**: a process calls it where it -/// starts, before a second thread of it builds. Two movers of one checkout -/// corrupt it, and nothing here keeps a second out. -pub fn make_fork_checkout(root: &Path) { - let Owner::Elsewhere(primary) = toolchain::owner(root) else { return }; +/// Every build in a worktree asks this at once, so the making and the move are +/// each decided and done under the worktree's lock held exclusively +/// ([`Held::act_if`]): one build does it, and none reads a checkout another is +/// still writing. +pub fn fork_checkout(root: &Path, lock: &mut Held) -> PathBuf { let fork = root.join("rust"); + let primary = match toolchain::owner(root) { + Owner::Us => return fork, + Owner::Installed => panic!("an installed toolchain has no fork checkout to build std in"), + Owner::Elsewhere(primary) => primary, + }; let pinned = pinned_fork(root); - if !fork.join(".git").exists() { - let stub = fs::read_dir(&fork).map_or(0, |d| d.count()); - assert!( - stub == 0, - "{} is neither a fork checkout nor the empty stub a worktree starts with", - fork.display() - ); - let _ = fs::remove_dir(&fork); - eprintln!("Making {} a fork checkout at {pinned} (a git worktree of the primary's)", fork.display()); - git_run(&primary.join("rust"), &["worktree", "add", "--detach", path_str(&fork), &pinned]); - let backtrace = git_out(&fork, &["ls-tree", "HEAD", "library/backtrace"]); - let commit = backtrace.split_whitespace().nth(2).unwrap_or_else(|| { - panic!("{} pins no library/backtrace: {backtrace:?}", fork.display()) - }); - let theirs = primary.join("rust/library/backtrace"); - let held = Command::new("git") - .args(["cat-file", "-e", &format!("{commit}^{{commit}}")]) - .current_dir(&theirs) - .status() - .is_ok_and(|s| s.success()); - let at = fork.join("library/backtrace"); - if held { - let _ = fs::remove_dir(&at); - git_run(&theirs, &["worktree", "add", "--detach", path_str(&at), commit]); - } else { - git_run(&fork, &["submodule", "update", "--init", "library/backtrace"]); - } - return; - } - let Some(head) = off_pin(&fork, &pinned) else { return }; - let dirty = git_out(&fork, &["status", "--porcelain", "--ignore-submodules=none"]); - assert!( - dirty.is_empty(), - "{} is at {head} with uncommitted work, and this tree pins the fork at {pinned}, which \ - that is not ahead of: a build here would compile a std this tree does not name, and \ - moving the checkout would lose that work.\n{dirty}", - fork.display(), + lock.act_if( + Scope::Worktree, + "make the fork checkout", + || (!fork.join(".git").exists()).then_some(()), + |()| { + let stub = fs::read_dir(&fork).map_or(0, |d| d.count()); + assert!( + stub == 0, + "{} is neither a fork checkout nor the empty stub a worktree starts with", + fork.display() + ); + let _ = fs::remove_dir(&fork); + eprintln!("Making {} a fork checkout at {pinned} (a git worktree of the primary's)", fork.display()); + git_run(&primary.join("rust"), &["worktree", "add", "--detach", path_str(&fork), &pinned]); + let backtrace = git_out(&fork, &["ls-tree", "HEAD", "library/backtrace"]); + let commit = backtrace.split_whitespace().nth(2).unwrap_or_else(|| { + panic!("{} pins no library/backtrace: {backtrace:?}", fork.display()) + }); + let theirs = primary.join("rust/library/backtrace"); + let held = Command::new("git") + .args(["cat-file", "-e", &format!("{commit}^{{commit}}")]) + .current_dir(&theirs) + .status() + .is_ok_and(|s| s.success()); + let at = fork.join("library/backtrace"); + if held { + let _ = fs::remove_dir(&at); + git_run(&theirs, &["worktree", "add", "--detach", path_str(&at), commit]); + } else { + git_run(&fork, &["submodule", "update", "--init", "library/backtrace"]); + } + }, ); - let held = Command::new("git") - .args(["cat-file", "-e", &format!("{pinned}^{{commit}}")]) - .current_dir(&fork) - .status() - .is_ok_and(|s| s.success()); - if !held { - git_run(&fork, &["fetch", path_str(&primary.join("rust")), &pinned]); - } - git_run(&fork, &["checkout", "--detach", "-q", &pinned]); - eprintln!("{} was at {head}, behind this tree's pin {pinned}: checked it out", fork.display()); + lock.act_if( + Scope::Worktree, + "move the fork checkout to its pin", + || { + let head = git_out(&fork, &["rev-parse", "HEAD"]).trim().to_string(); + let ahead = Command::new("git") + .args(["merge-base", "--is-ancestor", &pinned, &head]) + .current_dir(&fork) + .status() + .is_ok_and(|s| s.success()); + (head != pinned && !ahead).then_some(head) + }, + |head| { + let dirty = git_out(&fork, &["status", "--porcelain", "--ignore-submodules=none"]); + assert!( + dirty.is_empty(), + "{} is at {head} with uncommitted work, and this tree pins the fork at {pinned}, which \ + that is not ahead of: a build here would compile a std this tree does not name, and \ + moving the checkout would lose that work.\n{dirty}", + fork.display(), + ); + let held = Command::new("git") + .args(["cat-file", "-e", &format!("{pinned}^{{commit}}")]) + .current_dir(&fork) + .status() + .is_ok_and(|s| s.success()); + if !held { + git_run(&fork, &["fetch", path_str(&primary.join("rust")), &pinned]); + } + git_run(&fork, &["checkout", "--detach", "-q", &pinned]); + eprintln!("{} was at {head}, behind this tree's pin {pinned}: checked it out", fork.display()); + }, + ); + fork } /// What a sysroot's [`SOURCES`] says: its key, the fork checkout its std was @@ -645,7 +632,7 @@ fn held(root: &Path, key: &Key, dir: &Path, make: impl FnMut()) -> Guard { /// The sysroot this worktree's sources name, made if nobody has made it, and /// held in use for as long as the returned value lives. pub fn ensure(root: &Path, rust_dir: &Path, lock: &mut Held) -> Sysroot { - let fork = fork_checkout(root); + let fork = fork_checkout(root, lock); let compiler = compiler::resolve(root, rust_dir, &fork, lock); let keys = Keys::of(root, &compiler, &fork); let dir = sysroots_dir(rust_dir).join(&keys.sysroot); @@ -1436,6 +1423,7 @@ mod tests { fs::create_dir_all(&primary).unwrap(); git(&primary, &["init", "-q"]); write(&primary.join("toyos-abi/src/lib.rs"), "pub struct A;\n"); + write(&primary.join(".gitignore"), ".build-locks/\n"); git(&primary, &["submodule", "add", "-q", fork.to_str().unwrap(), "rust"]); git(&primary.join("rust"), &["checkout", "-q", &c1]); git(&primary, &["add", "-A"]); @@ -1458,9 +1446,9 @@ mod tests { let base = TempDir::new("fork-pins"); let (primary, linked, c1, c2) = two_pins(&base); let before = git(&primary.join("rust"), &["status", "--porcelain"]); + let mut lock = buildlock::shared(&linked, "a build"); - make_fork_checkout(&linked); - let fork = fork_checkout(&linked); + let fork = fork_checkout(&linked, &mut lock); assert_eq!(fork, linked.join("rust")); assert_eq!(git(&fork, &["rev-parse", "HEAD"]), c2); @@ -1478,73 +1466,48 @@ mod tests { // Work on the fork in the worktree's own checkout is what it builds. write(&fork.join("library/std/src/lib.rs"), "pub fn c() {}\n"); git(&fork, &["commit", "-qam", "C3, the agent's own"]); - make_fork_checkout(&linked); - assert_eq!(fork_checkout(&linked), fork); + let c3 = git(&fork, &["rev-parse", "HEAD"]); + assert_eq!(fork_checkout(&linked, &mut lock), fork); + assert_eq!(git(&fork, &["rev-parse", "HEAD"]), c3, "a checkout ahead of its pin was moved"); // A clean checkout behind what the tree pins is moved to the pin itself. git(&fork, &["checkout", "-q", &c1]); - make_fork_checkout(&linked); + assert_eq!(fork_checkout(&linked, &mut lock), fork); assert_eq!(git(&fork, &["rev-parse", "HEAD"]), c2, "a checkout behind its pin was not moved to it"); // One with local changes is never moved out from under whoever made them. git(&fork, &["checkout", "-q", &c1]); write(&fork.join("library/std/src/lib.rs"), "pub fn uncommitted() {}\n"); - let refused = std::panic::catch_unwind(|| make_fork_checkout(&linked)) - .expect_err("a fork checkout with local changes was moved out from under them"); - let message = refused.downcast::().expect("a formatted refusal"); + let message = refusal(|| drop(fork_checkout(&linked, &mut lock))); assert!(message.contains(&c1) && message.contains(&c2), "{message}"); } - /// What each of twelve builds starting at once in `linked` read as its fork - /// checkout, or was refused with. - fn twelve_builds_read(linked: &Path) -> Vec> { - let start = std::sync::Barrier::new(12); - std::thread::scope(|builds| { - let reads: Vec<_> = (0..12) - .map(|_| { - builds.spawn(|| { - start.wait(); - std::panic::catch_unwind(|| fork_checkout(linked)) - .map_err(|refusal| *refusal.downcast::().expect("a formatted refusal")) - }) - }) - .collect(); - reads.into_iter().map(|read| read.join().expect("a build's refusal is caught")).collect() - }) - } - - /// **A build reads its fork checkout and never moves it**: twelve at once - /// neither make one that is not there nor move one behind its pin — each is - /// refused by name and the checkout stands as it was — and all twelve read - /// the one [`make_fork_checkout`] made, and the one it moved. + /// **Twelve builds starting at once in a worktree make its fork checkout + /// once, and move one behind its pin once**: each holds the worktree's lock + /// as a process of its own does, and each returns the checkout whole and at + /// the pin. #[test] - fn a_build_reads_the_fork_checkout_and_never_moves_it() { + fn twelve_builds_at_once_make_and_move_one_fork_checkout() { let base = TempDir::new("fork-builds"); let (_primary, linked, c1, c2) = two_pins(&base); let fork = linked.join("rust"); - let head = || git(&fork, &["rev-parse", "HEAD"]); - let refused = || { - for read in twelve_builds_read(&linked) { - let said = read.expect_err("a build took a fork checkout its tree does not pin"); - assert!(said.contains(&c2) && said.contains("`sysroot::make_fork_checkout`"), "{said}"); - } + let twelve_build = || { + let start = std::sync::Barrier::new(12); + std::thread::scope(|builds| { + for _ in 0..12 { + builds.spawn(|| { + let mut lock = buildlock::shared(&linked, "a build"); + start.wait(); + assert_eq!(fork_checkout(&linked, &mut lock), fork); + assert_eq!(git(&fork, &["rev-parse", "HEAD"]), c2); + assert!(fork.join("library/backtrace/lib.rs").is_file(), "a build read a checkout still being made"); + }); + } + }); }; - let read = || assert_eq!(twelve_builds_read(&linked), vec![Ok(fork.clone()); 12]); - - refused(); - assert!(!fork.join(".git").exists(), "a build made the fork checkout"); - - make_fork_checkout(&linked); - read(); - assert_eq!(head(), c2); - + twelve_build(); git(&fork, &["checkout", "-q", &c1]); - refused(); - assert_eq!(head(), c1, "a build moved the fork checkout"); - - make_fork_checkout(&linked); - read(); - assert_eq!(head(), c2); + twelve_build(); } /// The primary's compiler under `base`: `rustc` and `rust-lld`, and the C diff --git a/tests/toyos.rs b/tests/toyos.rs index 4791f577acf..097e00fe3f9 100644 --- a/tests/toyos.rs +++ b/tests/toyos.rs @@ -4000,10 +4000,6 @@ fn main() { } }; - // Before anything is built and before a worker exists: the process's one - // move of its fork checkout. - toyos_build::sysroot::make_fork_checkout(&compile::repo_root()); - if let Some(mode) = parsed.metal { let (c_bins, rust_bins) = build_shared_bins(); let selected: Vec<(&str, &'static metal::Metal)> = METAL From b497e67cac082e88d6a33a298cc3a264a0fab53b Mon Sep 17 00:00:00 2001 From: japabu Date: Sat, 3 Oct 2026 12:59:34 +0200 Subject: [PATCH 4/6] No submodule is initialised in a linked worktree, and a killed maker's checkout has a record `licence::std_library` sends a fork checkout without `library/Cargo.toml` to `ensure_shallow_fork`, which is a CI runner's path. Under the worktree's lock a checkout another process is still making no longer gets there: the reader waits. One a killed `git worktree add` left still did. Measured in a fixture at a8dcbccb2: `fork_checkout` returned the remains, `git submodule--helper update --init --depth=1 -- rust` ran in the linked worktree, and afterwards `git status` in the fixture primary's `rust/` exits 128 on `cannot chdir to '../../../../../../linked/rust'`. `ensure_shallow_fork` now refuses a linked worktree by name, and `a_linked_worktree_initialises_no_submodule` holds it: on what a killed maker leaves, the refusal is returned and git in the primary's fork still runs. That a build takes such remains as made is the base's too and stays true: issues/build/a-fork-checkout-a-killed-git-worktree-add-left-is-taken-as-made.md. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_013UDZQ6fSKw14e4w2TKTRfm --- ...-git-worktree-add-left-is-taken-as-made.md | 38 +++++++++++++++++++ src/lib.rs | 10 ++++- src/sysroot.rs | 18 +++++++++ 3 files changed, 65 insertions(+), 1 deletion(-) create mode 100644 issues/build/a-fork-checkout-a-killed-git-worktree-add-left-is-taken-as-made.md diff --git a/issues/build/a-fork-checkout-a-killed-git-worktree-add-left-is-taken-as-made.md b/issues/build/a-fork-checkout-a-killed-git-worktree-add-left-is-taken-as-made.md new file mode 100644 index 00000000000..3a6def9351b --- /dev/null +++ b/issues/build/a-fork-checkout-a-killed-git-worktree-add-left-is-taken-as-made.md @@ -0,0 +1,38 @@ +--- +status: open +kind: tooling +opened: 2026-10-03 +--- + +# A fork checkout a killed `git worktree add` left is taken as made + +`sysroot::fork_checkout` (`src/sysroot.rs`) takes a linked worktree's `rust/` as +made where `rust/.git` exists and `HEAD` is the pinned commit or ahead of it. A +`git worktree add` killed while it writes its files leaves both, so every build +after it is handed a checkout with files missing. Git records the state itself: +the worktree stays `locked` with the reason `initializing`, beside a stale +`index.lock`. + +## Measured + +In a fixture whose fork checks its last file, `x.py`, out through a filter that +waits, `git worktree add --detach` was killed with SIGKILL once `rust/.git`, +`HEAD` and the files before `x.py` were written. `fork_checkout` returned what +it left: `rust/.git`, `HEAD` at the pin, no `x.py`, `locked` reading +`initializing`. `ensure_shallow_fork` refuses it by name, so the licence gate +reds on it. + +In a fixture fork of 30,004 files the same kill left 253 of them. Afterwards +`git worktree add` at that path exits 128 on `already exists`, +`git worktree remove --force` exits 128 on the lock, and +`git worktree remove --force --force` exits 255 on `Directory not empty` and +unregisters the worktree. + +## Owner + +The toolchain item of +`issues/build/the-tooling-is-a-review-prompt-and-three-workflows.md`, whose +stores are published by an atomic rename. + +**Exit**: a build that finds a fork checkout git records as `locked` makes it +again or refuses it by name. diff --git a/src/lib.rs b/src/lib.rs index a5ef9ce72e3..83d3029567a 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -120,11 +120,19 @@ pub fn ensure_submodules(repo_dir: &Path) { /// `rust/` at the commit this tree pins and with no history, where a CI /// runner's checkout has none: what reading the fork's sources and building -/// the toolchain from them need. +/// the toolchain from them need. Refused in a linked worktree, whose `rust/` is +/// `sysroot::fork_checkout`'s to make: `git submodule` there rewrites the +/// `core.worktree` of the primary's fork. pub fn ensure_shallow_fork(root: &Path) -> Result<(), String> { if root.join("rust/x.py").exists() { return Ok(()); } + if let toolchain::Owner::Elsewhere(_) = toolchain::owner(root) { + return Err(format!( + "{} is a linked worktree's fork checkout and is not whole: no submodule is initialised there", + root.join("rust").display() + )); + } let status = Command::new("git") .args(["submodule", "update", "--init", "--depth", "1", "rust"]) .current_dir(root) diff --git a/src/sysroot.rs b/src/sysroot.rs index 98f9e30eb2c..86615d31b0f 100644 --- a/src/sysroot.rs +++ b/src/sysroot.rs @@ -1510,6 +1510,24 @@ mod tests { twelve_build(); } + /// **No submodule is initialised in a linked worktree**: where its `rust/` + /// holds `.git` and no file, as a killed `git worktree add` leaves it, + /// [`crate::ensure_shallow_fork`] refuses, and git in the primary's fork + /// still runs. + #[test] + fn a_linked_worktree_initialises_no_submodule() { + let base = TempDir::new("fork-shallow"); + let (primary, linked, c1, c2) = two_pins(&base); + let fork = linked.join("rust"); + fs::remove_dir(&fork).unwrap(); + git(&primary.join("rust"), &["worktree", "add", "-q", "--detach", "--no-checkout", path_str(&fork), &c2]); + + let refused = crate::ensure_shallow_fork(&linked); + + assert_eq!(git(&primary.join("rust"), &["rev-parse", "HEAD"]), c1); + assert!(refused.as_ref().is_err_and(|why| why.contains("linked worktree")), "{refused:?}"); + } + /// The primary's compiler under `base`: `rustc` and `rust-lld`, and the C /// toolchain `src/clang.rs` provisions beside them if `clang`; no cargo. fn primary_compiler(base: &Path, clang: bool) -> Compiler { From cfd4931a6a0e31b3403ed0a60682efe052c95d34 Mon Sep 17 00:00:00 2001 From: japabu Date: Sat, 3 Oct 2026 13:05:09 +0200 Subject: [PATCH 5/6] The submodule test's doc says the state it arranges `git worktree add --no-checkout` leaves a checkout with `.git` and no file; a killed one leaves some. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_013UDZQ6fSKw14e4w2TKTRfm --- src/sysroot.rs | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/src/sysroot.rs b/src/sysroot.rs index 86615d31b0f..73b5104fe45 100644 --- a/src/sysroot.rs +++ b/src/sysroot.rs @@ -1511,9 +1511,8 @@ mod tests { } /// **No submodule is initialised in a linked worktree**: where its `rust/` - /// holds `.git` and no file, as a killed `git worktree add` leaves it, - /// [`crate::ensure_shallow_fork`] refuses, and git in the primary's fork - /// still runs. + /// is a fork checkout that is not whole, [`crate::ensure_shallow_fork`] + /// refuses, and git in the primary's fork still runs. #[test] fn a_linked_worktree_initialises_no_submodule() { let base = TempDir::new("fork-shallow"); From c402baf6c82a7837abbf24a60b457f1ba95ebbf5 Mon Sep 17 00:00:00 2001 From: japabu Date: Sat, 3 Oct 2026 13:44:40 +0200 Subject: [PATCH 6/6] The shallow-fork test names ensure_shallow_fork, and the killed-maker issue says what the licence gate reaches Round 2's review of cfd4931a6 on #683, its named changes. `fork_checkout` and `ensure_submodule` still run `git submodule` in a linked worktree's fork checkout, so "no submodule is initialised in a linked worktree" is true of `ensure_shallow_fork` and not of the tree. The test is renamed to `ensure_shallow_fork_initialises_no_submodule_in_a_linked_worktree`, and the clause of its doc that claimed it of the tree is deleted. The new issue said the licence gate reds on what a killed `git worktree add` left. `licence::std_library` reaches `ensure_shallow_fork` only where the checkout has no `library/Cargo.toml`, which the fixture's fork never has and which the 30,004-file run's kill left among its 253 files; past it the gate reads what `library/` holds. The issue now says that. It also recorded one kill of two. The other is the builder's alone, with its `git worktree add` still writing and the worktree's lock freed: the issue names it as read from the code and unmeasured. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_013UDZQ6fSKw14e4w2TKTRfm --- ...-git-worktree-add-left-is-taken-as-made.md | 31 ++++++++++++++----- src/sysroot.rs | 8 ++--- 2 files changed, 27 insertions(+), 12 deletions(-) diff --git a/issues/build/a-fork-checkout-a-killed-git-worktree-add-left-is-taken-as-made.md b/issues/build/a-fork-checkout-a-killed-git-worktree-add-left-is-taken-as-made.md index 3a6def9351b..533165a4474 100644 --- a/issues/build/a-fork-checkout-a-killed-git-worktree-add-left-is-taken-as-made.md +++ b/issues/build/a-fork-checkout-a-killed-git-worktree-add-left-is-taken-as-made.md @@ -19,14 +19,29 @@ In a fixture whose fork checks its last file, `x.py`, out through a filter that waits, `git worktree add --detach` was killed with SIGKILL once `rust/.git`, `HEAD` and the files before `x.py` were written. `fork_checkout` returned what it left: `rust/.git`, `HEAD` at the pin, no `x.py`, `locked` reading -`initializing`. `ensure_shallow_fork` refuses it by name, so the licence gate -reds on it. - -In a fixture fork of 30,004 files the same kill left 253 of them. Afterwards -`git worktree add` at that path exits 128 on `already exists`, -`git worktree remove --force` exits 128 on the lock, and -`git worktree remove --force --force` exits 255 on `Directory not empty` and -unregisters the worktree. +`initializing`. `ensure_shallow_fork` refuses it by name. The licence gate +reaches that refusal only for a kill before `library/Cargo.toml` is written, +a file this fixture's fork never has: `licence::std_library` calls +`ensure_shallow_fork` only without it, and past it the gate reads what +`library/` holds and runs no `git submodule`. + +In a fixture fork of 30,004 files the same kill left 253 of them, +`library/Cargo.toml` among them. Afterwards `git worktree add` at that path +exits 128 on `already exists`, `git worktree remove --force` exits 128 on the +lock, and `git worktree remove --force --force` exits 255 on +`Directory not empty` and unregisters the worktree. + +## Read, not measured + +Both measurements kill git. The kill `src/buildlock.rs`'s header calls routine +is the builder's alone. Then the kernel frees the worktree's lock at once, and +`git worktree add` goes on writing: it is the builder's child and holds no +descriptor of the lock, `git_run` being `Command::status` and the lock file +opened close-on-exec. A build that starts then finds `rust/.git`, `HEAD` at the +pin and the lock free. When that git ends, the checkout has no +`library/backtrace`, the state +`issues/build/the-fork-checkout-runs-git-submodule-in-a-linked-worktree.md` +records of a `fork_checkout` that stopped before adding it. ## Owner diff --git a/src/sysroot.rs b/src/sysroot.rs index 73b5104fe45..b1e608fc4e3 100644 --- a/src/sysroot.rs +++ b/src/sysroot.rs @@ -1510,11 +1510,11 @@ mod tests { twelve_build(); } - /// **No submodule is initialised in a linked worktree**: where its `rust/` - /// is a fork checkout that is not whole, [`crate::ensure_shallow_fork`] - /// refuses, and git in the primary's fork still runs. + /// Where a linked worktree's `rust/` is a fork checkout that is not whole, + /// [`crate::ensure_shallow_fork`] refuses, and git in the primary's fork + /// still runs. #[test] - fn a_linked_worktree_initialises_no_submodule() { + fn ensure_shallow_fork_initialises_no_submodule_in_a_linked_worktree() { let base = TempDir::new("fork-shallow"); let (primary, linked, c1, c2) = two_pins(&base); let fork = linked.join("rust");