From 80f4c946da6434a3d8f3d236caa82e37236e0106 Mon Sep 17 00:00:00 2001 From: japabu Date: Thu, 1 Oct 2026 12:38:25 +0200 Subject: [PATCH 01/30] ci: the guest suite is a pull request's `guest` check, on this tree's toolchain ci.yml gains two jobs on every non-draft pull request and in the merge queue: - `toolchain`, `cargo run -- --ci toolchain` on bare ubuntu-24.04, as the nightly's `build` runs it: a lookup when the tree hashes what an earlier run published, a bootstrap of hours when the branch moved the trees the tag hashes (src/release.rs). A runner's toolchain is the release its tree's tag names, and nothing else can install one, so a gate that only installed would red every pull request from the landing that moved main's tag until a nightly published it. - `guest`, `cargo run -- --ci guest` in the nightly's pinned debian:sid container with `/dev/kvm`, restoring the guest cache the nightly's `tcg` writes. It needs `toolchain` and runs whatever that concluded, unless the run was cancelled: GitHub reads a skipped required check as green, so a failed toolchain must reach `guest` as a red install, not skip it. The nightly keeps what the gate does not cover: `tcg` (x86-64 decoded by TCG, and the guest cache's one writer), `build` (the SDK alias on main), `host` and portability. Its `guest` job goes; `tcg` takes the steps it shared through anchors. Two gates in src/ci.rs replace prose: `guest` needs `toolchain` and is not skipped past it, and ci.yml's `guest` and the nightly's `tcg` name one image digest and one cache path list, since a restore whose paths are not its writer's restores nothing, silently. CLAUDE.md, the implementer's and the orchestrator's prompts say the guest suite runs in CI's `guest` check and agents never run QEMU locally; the host-tool rows name ci.yml's jobs beside the nightly's. src/release.rs's "the nightly's `build` job is what publishes one" stays: release.rs is hashed into the toolchain's tag, so any edit to it costs a toolchain bootstrap. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L --- .claude/agents/implementer.md | 2 +- .claude/agents/orchestrator.md | 6 +- .github/qemu-version | 4 +- .github/workflows/ci.yml | 107 +++++++++++++++++- .github/workflows/nightly.yml | 63 ++++------- CLAUDE.md | 2 +- ...d-runs-host-tools-outside-rust-and-qemu.md | 16 +-- src/ci.rs | 80 +++++++++---- tests/common/lane.rs | 7 +- 9 files changed, 207 insertions(+), 80 deletions(-) diff --git a/.claude/agents/implementer.md b/.claude/agents/implementer.md index a4d1bb91c6a..48eae3b60b9 100644 --- a/.claude/agents/implementer.md +++ b/.claude/agents/implementer.md @@ -20,7 +20,7 @@ Where hardware or anything uncertain is involved, take the cheap measurement bef guess. Then build, then test before anyone reviews: - Host tests only: `cargo run -- --ci host`, and `cargo run -- --build-only` at most for the - image. Never a guest test or any other `cargo run`: the orchestrator runs every guest test. + image. Never a guest test or any other `cargo run`: the guest suite runs in CI's `guest` check. - A result is the command's own exit code: ` > 2>&1; echo EXIT=$?`. A grepped `test result` line is not one, and a gate you did not run is a gate you do not claim. - Long commands run in the background with output to a file under the job scratchpad the brief diff --git a/.claude/agents/orchestrator.md b/.claude/agents/orchestrator.md index 6eb84aae517..ef6ea3ea4a5 100644 --- a/.claude/agents/orchestrator.md +++ b/.claude/agents/orchestrator.md @@ -33,8 +33,8 @@ scaffolding, deleted once its question is answered. Every task gets a fresh agent with an explicit model matched to the judgment in it: the strongest for drivers, security boundaries and reviews of them, and a mid tier for mechanical fixes from an exact list. A resumed agent only ever finishes its own interrupted task. A finished agent's report -is acted on before the next agent is dispatched: its guest runs queued, its review spawned, or its -fix round sent. When the permission check refuses an agent, ask the owner and never route around it. +is acted on before the next agent is dispatched: its review spawned, or its fix round sent. When +the permission check refuses an agent, ask the owner and never route around it. A brief is the fence: what to build, where it may touch, the worktree and branch, the scratchpad for its logs, and the two checks expected of high-risk code. The role files carry the standing rules, so a brief carries only the task. @@ -68,7 +68,7 @@ its script path under the job directory, and revert any mutation a killed run le left mid-flash or mid-boot is power-cycled by the owner and comes back to Ubuntu: BootNext is one-shot. -A mutation loop, guest or metal, starts on a clean worktree at the head under review and leaves it +A metal mutation loop starts on a clean worktree at the head under review and leaves it clean: `git apply --check`, `git apply`, the tests by name, `git apply -R`. None runs while an agent edits that worktree. A queue script passes only flags `src/testargs.rs` declares: any other word becomes the run's filter, and a one-test run reports as a pass. diff --git a/.github/qemu-version b/.github/qemu-version index bece0dd15d2..5e6cc8ab4e3 100644 --- a/.github/qemu-version +++ b/.github/qemu-version @@ -3,6 +3,6 @@ # build system's prerequisite check. The version decides test outcomes — 8.2.2 # and 11.0.3 were measured disagreeing about the same tree — so this is a # declaration the guest image is held to, never a fact read off it. The image -# digest in nightly.yml is chosen to satisfy this line; changing either is a -# deliberate act: it says the instrument moved. +# digest ci.yml and nightly.yml share is chosen to satisfy this line; changing +# either is a deliberate act: it says the instrument moved. 11.1.1 diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 4f348c1b995..2f5a8b536c6 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -1,8 +1,8 @@ name: ci -# A pull request and the merge queue: the host tests, and no guest. Each step -# is `cargo run -- --ci ` (src/ci.rs), which runs the same on a dev host; -# nightly.yml boots the guests. +# A pull request and the merge queue: the host tests, and the guest suite on +# this tree's toolchain. Each step is `cargo run -- --ci ` (src/ci.rs), +# which runs the same on a dev host. on: pull_request: @@ -41,3 +41,104 @@ jobs: restore-keys: host- - run: cargo run -- --ci host + + # Publishes this tree's toolchain if nobody has: a lookup when the tree hashes + # what main's does (src/release.rs), a bootstrap of hours when it moved + # those trees. Bare `ubuntu-24.04`, not a container: its glibc is the + # release's floor. + toolchain: + if: github.event_name == 'merge_group' || github.event.pull_request.draft == false + runs-on: ubuntu-24.04 + timeout-minutes: 350 + permissions: + contents: write + steps: + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + with: + fetch-depth: 0 + + - name: disk, QEMU and CMake + run: | + sudo rm -rf /usr/local/lib/android /usr/share/dotnet /opt/ghc \ + /usr/local/share/boost /usr/local/.ghcup + sudo apt-get update -qq + sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq qemu-system-x86 zstd \ + cmake=3.28.3-1build7 + + - env: + GH_TOKEN: ${{ github.token }} + run: cargo run -- --ci toolchain + + # A required check that is skipped reads as green, so this runs whatever + # `toolchain` concluded: a toolchain it did not publish reds the install. + guest: + needs: toolchain + if: ${{ !cancelled() && (github.event_name == 'merge_group' || github.event.pull_request.draft == false) }} + runs-on: ubuntu-24.04 + # A wedge guard, not a budget. + timeout-minutes: 60 + # The digest is the instrument's one pin, and nightly.yml's `tcg` names the + # same: a dated image names the snapshot archive it was built from, and + # `deps` installs QEMU from that archive. The node ships + # `crw-rw---- root:kvm` and root opens it. + container: + image: debian:sid-20260918@sha256:a2aa46262453eba3f464d8b1c7a8c31db85eb15af180ae34dd400615d7208547 + options: --device=/dev/kvm + env: + GH_TOKEN: ${{ github.token }} + steps: + # Before the checkout, which wants git. Three attempts, because the + # archive is fixed and the network to it is not. + - name: deps + run: | + snap=$(sed -n 's|^# \(http://snapshot\.debian\.org/archive/debian/[0-9]*T[0-9]*Z\)$|\1|p' \ + /etc/apt/sources.list.d/debian.sources) + test -n "$snap" + echo "deb $snap sid main" > /etc/apt/sources.list + rm /etc/apt/sources.list.d/debian.sources + for attempt in 1 2 3; do + apt-get -o Acquire::Check-Valid-Until=false update -qq > /tmp/apt.log 2>&1 \ + && DEBIAN_FRONTEND=noninteractive apt-get install -y -qq git curl ca-certificates \ + zstd build-essential qemu-system-x86 ovmf-generic qemu-system-arm \ + qemu-efi-aarch64 >> /tmp/apt.log 2>&1 \ + && break + [ "$attempt" = 3 ] && { cat /tmp/apt.log; exit 1; } + sleep 20 + done + # `actions/checkout` sets this only in a config it discards. + git config --global --add safe.directory "$GITHUB_WORKSPACE" + curl --proto '=https' --tlsv1.2 -sSf -o "$RUNNER_TEMP/rustup-init.sh" https://sh.rustup.rs + sh "$RUNNER_TEMP/rustup-init.sh" -y --profile minimal --default-toolchain stable + echo "$HOME/.cargo/bin" >> "$GITHUB_PATH" + + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + + # nightly.yml's `tcg` is the one writer; the paths are the cache's + # version, so they are the writer's list. + - uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 + with: + path: | + ~/.cargo/registry/index + ~/.cargo/registry/cache + ~/.cargo/git/db + target + kernel/target + bootloader/target + userland/target + tests/target + tests/toyos-rust-tests/*/target + key: guest-${{ github.run_id }} + restore-keys: guest- + + - run: cargo run -- --ci guest + + # Every boot's 16550 log: what a guest that died early still leaves. + - name: serial logs + if: failure() + continue-on-error: true + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 + with: + name: serial-${{ github.job }} + path: target/red-run-serial/**/uart-*.log + if-no-files-found: warn + retention-days: 7 diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index 3b7808ad94f..4025bbe17d6 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -1,7 +1,8 @@ name: nightly -# Everything that boots a guest, the host gate again to write the cache the -# merge queue restores, and portability. +# What the pull request's gate does not cover: the guest suite under TCG, the +# SDK alias on main, the host gate again to write the cache the merge queue +# restores, and portability. # Every job's logic is `cargo run -- --ci ` (src/ci.rs); this file says # where each one runs. @@ -73,24 +74,25 @@ jobs: GH_TOKEN: ${{ github.token }} run: cargo run -- --ci toolchain - guest: + # The guest suite with no `/dev/kvm`, so the only lane that decodes the paths + # a KVM host's CPU never does; and the guest cache's one writer, since what + # it builds does not depend on the accelerator. + tcg: needs: build runs-on: ubuntu-24.04 # A wedge guard, not a budget. timeout-minutes: 60 - # The digest is the instrument's one pin: a dated image names the snapshot - # archive it was built from, and `deps` installs QEMU from that archive. - # The node ships `crw-rw---- root:kvm` and root opens it. - container: &kvm - image: &image debian:sid-20260918@sha256:a2aa46262453eba3f464d8b1c7a8c31db85eb15af180ae34dd400615d7208547 - options: --device=/dev/kvm + # ci.yml's `guest` names the same digest, the instrument's one pin: a dated + # image names the snapshot archive it was built from, and `deps` installs + # QEMU from that archive. + container: + image: debian:sid-20260918@sha256:a2aa46262453eba3f464d8b1c7a8c31db85eb15af180ae34dd400615d7208547 env: GH_TOKEN: ${{ github.token }} steps: # Before the checkout, which wants git. Three attempts, because the # archive is fixed and the network to it is not. - - &deps - name: deps + - name: deps run: | snap=$(sed -n 's|^# \(http://snapshot\.debian\.org/archive/debian/[0-9]*T[0-9]*Z\)$|\1|p' \ /etc/apt/sources.list.d/debian.sources) @@ -115,8 +117,9 @@ jobs: - &checkout uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 - - &guest-cache - uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 + # ci.yml's `guest` restores what this saves, and the paths are the + # cache's version, so its list is this one. + - uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 with: path: &guest-paths | ~/.cargo/registry/index @@ -133,9 +136,15 @@ jobs: - run: cargo run -- --ci guest + # After the test: what is worth keeping is a tree that built and booted. + - if: github.ref == 'refs/heads/main' + uses: actions/cache/save@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 + with: + path: *guest-paths + key: guest-${{ github.run_id }} + # Every boot's 16550 log: what a guest that died early still leaves. - - &serial - name: serial logs + - name: serial logs if: failure() continue-on-error: true uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 @@ -145,30 +154,6 @@ jobs: if-no-files-found: warn retention-days: 7 - # The guest suite again with no `/dev/kvm`, so the only lane that decodes the - # paths a KVM host's CPU never does; and the guest cache's one writer, since - # what it builds does not depend on the accelerator. - tcg: - needs: build - runs-on: ubuntu-24.04 - timeout-minutes: 60 - container: - image: *image - env: - GH_TOKEN: ${{ github.token }} - steps: - - *deps - - *checkout - - *guest-cache - - run: cargo run -- --ci guest - # After the test: what is worth keeping is a tree that built and booted. - - if: github.ref == 'refs/heads/main' - uses: actions/cache/save@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 - with: - path: *guest-paths - key: guest-${{ github.run_id }} - - *serial - # `cargo run -- --build-only` from a fresh machine. `sid` as it stands, # image and archive both, and no cache — a fresh machine is the premise. portability-linux: diff --git a/CLAUDE.md b/CLAUDE.md index 0b910305b39..8ba6baca279 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -72,7 +72,7 @@ The bar is not yet the tree: `.claude/agents/reviewer.md`, "Arrivals", says wher The testing rules live where they are enforced: the PR gate and the nightly in `.github/workflows/`. Operationally: - `cargo run` builds everything (toolchain, kernel, bootloader, userland, image) and launches QEMU; `--build-only` skips the launch. `cargo test` runs the QEMU harness; `cargo run -- --ci host` runs every host suite, as the PR gate's required `host` check does. -- **Agents never run QEMU.** An agent verifies with host tests and builds the image at most; the orchestrator runs every guest test, one suite at a time. +- **Agents never run QEMU locally.** An agent verifies with host tests and builds the image at most; the guest suite runs in CI's `guest` check. - **Both produce large output**: run them in the background and read the output file — `[N characters truncated]` means data was lost. A full boot is under a second; incremental builds finish in seconds. - **Leave the machine as you found it.** The development machine is shared: every agent stops what it started, removes the worktrees and scratch build output it no longer needs, and never leaves an emulator, a build or a watcher running. diff --git a/issues/build/the-build-runs-host-tools-outside-rust-and-qemu.md b/issues/build/the-build-runs-host-tools-outside-rust-and-qemu.md index b00a133dd5f..b487cc9045f 100644 --- a/issues/build/the-build-runs-host-tools-outside-rust-and-qemu.md +++ b/issues/build/the-build-runs-host-tools-outside-rust-and-qemu.md @@ -18,18 +18,18 @@ arrives and is not one. M4 and M5 are stages of `issues/build/toyos-builds-itsel | CMake | rustc's bootstrap, for LLVM and clang; `src/llvm.rs`, for the LLVM's key; `src/libcxx.rs`, for the C++ runtime in every sysroot build | admitted: no Rust tool does the job, LLVM, clang and LLD are described in CMake, and upstream's only other descriptions are a GN overlay it does not support and a Bazel one | M5 runs it in the guest | | `sh` running LLVM's `config.guess`, and the POSIX tools and `cc` it runs | LLVM's CMake, whenever this host builds an LLVM, and the C++ runtime's, in every sysroot build, ask it the host's triple, unconditionally (`get_host_triple` in `rust/src/llvm-project/llvm/cmake/modules/GetHostTriple.cmake`, which runs `sh` by name) | refused: a Rust tool does the shell's part, brush 0.4.0: on the development host (macOS, arm64) `config.guess` printed `/bin/sh`'s triple under it, `arm64-apple-darwin27.0.0`, exit 0 each. The script runs `sed`, `uname`, `mktemp`, `grep`, `rm`, `rmdir` and `cc` there under either shell, and that `cc` is the `cc` rows'. Five of the other six are refused, uutils' doing each: under brush with sed 0.2.0, grep 0.2.0 and coreutils 0.12.0's `mktemp`, `rm` and `rmdir`, and nothing else on `PATH` but the host's `uname` and `cc`, it printed that triple, exit 0. `uname` is admitted: coreutils 0.12.0's answers `-p` with `unknown` where macOS's answers `arm`, and `config.guess` reads that as PowerPC, `powerpc-apple-darwin27.0.0`, exit 0 | CMake finds brush as its `sh`, uutils' `sed`, `grep`, `mktemp`, `rm` and `rmdir`, and a Rust `uname` that answers `-p` as the host's does; or M5 runs it in the guest | | `git` for worktrees, submodules, checkouts, fixtures and rustc's bootstrap | adds, removes and prunes worktrees (`src/worktree.rs`, `src/sysroot.rs`); updates submodules (`src/lib.rs`, `src/sysroot.rs`, `src/licence.rs`, `src/release.rs`); fetches the fork from the primary's and checks it out (`src/sysroot.rs`); fast-forwards the primary (`src/sync.rs`); makes the tests' fixture repositories; runs inside rustc's bootstrap | admitted: no Rust tool does the job, gitoxide 0.85 adds, removes and prunes no worktree, updates no submodule, stages, resets and pushes nothing, checks out only a fresh clone and fetches a local path by spawning `git`; a fixture must be what `git` makes, and bootstrap runs `git` itself | M4 runs it in the guest | -| `git` for reads, a config write, a commit's paths written out, and clones and fetches over HTTPS | `rev-parse`, `show-ref`, `for-each-ref`, `rev-list`, `log`, `branch --contains`, `merge-base`, `ls-tree`, `ls-files`, `cat-file`, `config --get-regexp`, `worktree list`, `status`, `diff`, `ls-remote` and `grep`, in the build system and its tests; `config --global --add safe.directory` in the nightly's containers; `checkout -- ` through an index of its own, which writes the C++ runtime's sources out of the LLVM commit into the stored LLVM (`src/llvm.rs`); `src/sync.rs`'s fetch of `origin`; every workflow's checkout | refused: a Rust tool does it, gitoxide 0.85, which reads refs, objects, the index, config, worktrees and status, adds a value to a config file and writes it (gix-config 0.58's `File::section_mut_or_create_new`, `SectionMut::push`, `File::write_to`), walks history, diffs, and lists, fetches and clones a remote over HTTPS; `grep` is a search of the files its index names; and gitoxide's CLI 0.59 (gix 0.88) wrote the runtimes' sources of LLVM `849da7d6` into an empty directory, each path's tree through `gix rev parse`, `gix index from-tree` and `gix free index checkout-exclusive`, exit 0 each: the 18759 files `git` writes there, byte for byte and mode for mode | those are gitoxide's | -| `cc`, `c++` and `ar` on a Linux host, `build-essential` on the nightly's runners | rustc links every host binary through `cc`; `cc` and `c++` compile LLVM, clang, LLD and `rustc_llvm` (`src/llvm.rs` names both to bootstrap); `ar` archives what `cc::Build` compiles | admitted: no Rust tool compiles C or C++, or takes rustc's host link | M5: no host in the loop | +| `git` for reads, a config write, a commit's paths written out, and clones and fetches over HTTPS | `rev-parse`, `show-ref`, `for-each-ref`, `rev-list`, `log`, `branch --contains`, `merge-base`, `ls-tree`, `ls-files`, `cat-file`, `config --get-regexp`, `worktree list`, `status`, `diff`, `ls-remote` and `grep`, in the build system and its tests; `config --global --add safe.directory` in CI's containers; `checkout -- ` through an index of its own, which writes the C++ runtime's sources out of the LLVM commit into the stored LLVM (`src/llvm.rs`); `src/sync.rs`'s fetch of `origin`; every workflow's checkout | refused: a Rust tool does it, gitoxide 0.85, which reads refs, objects, the index, config, worktrees and status, adds a value to a config file and writes it (gix-config 0.58's `File::section_mut_or_create_new`, `SectionMut::push`, `File::write_to`), walks history, diffs, and lists, fetches and clones a remote over HTTPS; `grep` is a search of the files its index names; and gitoxide's CLI 0.59 (gix 0.88) wrote the runtimes' sources of LLVM `849da7d6` into an empty directory, each path's tree through `gix rev parse`, `gix index from-tree` and `gix free index checkout-exclusive`, exit 0 each: the 18759 files `git` writes there, byte for byte and mode for mode | those are gitoxide's | +| `cc`, `c++` and `ar` on a Linux host, `build-essential` in CI's containers | rustc links every host binary through `cc`; `cc` and `c++` compile LLVM, clang, LLD and `rustc_llvm` (`src/llvm.rs` names both to bootstrap); `ar` archives what `cc::Build` compiles | admitted: no Rust tool compiles C or C++, or takes rustc's host link | M5: no host in the loop | | the toolchain's own `clang`, `llvm-ar`, `rust-lld` and `llvm-config`, built from `ToyOSOrg/llvm-project` | rustc links every guest binary with `rust-lld`; `clang` compiles the C corpus (`tests/common/compile.rs`) and, with `llvm-ar`, doomgeneric through `cc::Build` (`src/clang.rs`); rustc's bootstrap asks `llvm-config` how to link LLVM | admitted: our fork's C++, which ToyOS can one day build and run; no Rust tool compiles C, `cc::Build` archives with an `ar`, bootstrap reads LLVM through `llvm-config`, and `CLAUDE.md` links everything with `rust-lld` | M5: no host in the loop | -| `ovmf-generic`, `qemu-efi-aarch64` | the x86-64 and AArch64 UEFI firmware of the nightly's guest containers (`src/firmware.rs`), packaged by Debian apart from QEMU | admitted: QEMU's own firmware, and no Rust firmware does its job | the instrument's QEMU carries its own firmware | -| `ca-certificates` | the trust store `git` and `curl` verify against in the nightly's containers | admitted: data both of them need | goes when neither runs there | +| `ovmf-generic`, `qemu-efi-aarch64` | the x86-64 and AArch64 UEFI firmware of CI's guest containers (`src/firmware.rs`), packaged by Debian apart from QEMU | admitted: QEMU's own firmware, and no Rust firmware does its job | the instrument's QEMU carries its own firmware | +| `ca-certificates` | the trust store `git` and `curl` verify against in CI's containers | admitted: data both of them need | goes when neither runs there | | `env` | the two portability jobs run `cargo run -- --build-only` under it, with `GITHUB_ACTIONS` and `CI` unset | refused: a Rust tool does it, the build system, which already runs that build with both removed (`src/release.rs`) | the build system removes both itself | | the T14's Ubuntu and every tool `src/metal.rs` runs on it over `ssh` | the metal loop, on the T14 and never on a development host | outside the rule: recovery equipment on a test machine, not the build's host | they leave with Ubuntu (`issues/boot-media/the-machine-updates-itself-without-ubuntu.md`) | | `sh` running `rust/x`, and Python running `x.py` and `bootstrap.py` | every toolchain build (`src/toolchain.rs`) | refused: a Rust tool does it, upstream's bootstrap binary, which builds with stable cargo, fetches its own stage0 (`rust/src/bootstrap/src/core/download.rs`) and needs no Python | `src/toolchain.rs` runs the bootstrap binary | | `curl` in rustc's bootstrap | fetches the stage0 `rust/src/stage0` pins, for a compiler or LLVM build whose build directory lacks it, whichever bootstrap runs | refused: a Rust tool does it, rustup installs the dated beta the pin names, and bootstrap takes a stage0 through `build.rustc` and `build.cargo`, as `src/sysroot.rs` hands it one | no toolchain build fetches with `curl` | | `curl` in `src/release.rs` and `src/sdkversion.rs` | the toolchain release's lookup and download and the crates.io index, on CI runners only | refused: a Rust tool does it, `ureq`, which `userland/doom/build.rs` already fetches with | those fetches are Rust's | | `tar` and `zstd` | `src/release.rs` packs and unpacks the toolchain release, on CI runners only | refused: a Rust tool does it, the `tar` crate `userland/doom/build.rs` already unpacks with, and a zstd crate | both are done in Rust, in-process | -| `gh` | `src/release.rs` asks whether a toolchain release exists, creates it and moves the `sdk-` alias, on the nightly's `build` runner | refused: not C or C++ source, it is Go | `src/release.rs` speaks GitHub's REST API itself | +| `gh` | `src/release.rs` asks whether a toolchain release exists, creates it and moves the `sdk-` alias, on the `toolchain` and nightly `build` runners | refused: not C or C++ source, it is Go | `src/release.rs` speaks GitHub's REST API itself | | `ssh` | `src/metal.rs` reaches the T14's Ubuntu with it, only in the metal loop | refused: a Rust tool does it, the repository's own russh client `crate::build::ssh_client_host`, which `src/metaltalk.rs` already drives | `src/metal.rs` drives that client, or Ubuntu leaves the loop | | `cc`, `c++`, `ar` and `xcrun` on a macOS host, Apple's Command Line Tools | what the Linux row's tools do, and rustc asks `xcrun` for the SDK on every host link that names no `SDKROOT` (`rust/compiler/rustc_codegen_ssa/src/back/apple.rs`), as `src/llvm.rs` does for the LLVM's key | refused: one host OS alone | M5: no host in the loop | | `newfs_msdos` and `hdiutil` | `toyos-fat32`'s host tests format and mount their fixtures with them (`toyos-fat32/tests/common/mod.rs`), which keeps the `host` job on `macos-latest` | refused: one host OS alone | `issues/filesystem/fat32-suite-needs-macos-binaries.md` | @@ -41,9 +41,9 @@ arrives and is not one. M4 and M5 are stages of `issues/build/toyos-builds-itsel | `sync` | the README's Linux flashing steps flush the stick with it | refused: a Rust tool does it, the build system can flush what it writes | `issues/build/the-owners-flash-script-runs-diskutil.md` | | `sudo` on macOS | `diag/flash.sh` and the README's macOS flashing steps run `dd` under it | admitted: no Rust tool raises a process to root on macOS; sudo-rs "is targeted for FreeBSD and Linux-based operating systems only" (its README at `89bae8a`) | goes with both flashes by hand | | `sudo` on Linux | the README's Linux flashing steps run `dd` under it | refused: a Rust tool does it, sudo-rs | the README's Linux steps run sudo-rs | -| `sh` running rustup's `rustup-init.sh` | the nightly's three rustup installs, fetched with `curl` | refused: a Rust tool does it, rustup's own `rustup-init` binary | a job installs rustup without it | -| `nightly.yml`, job `build`, step "disk, QEMU and CMake" | `sudo rm -rf` of five preinstalled SDK directories, then `sudo apt-get update` and `install` | refused: shell of our own | each step is one command | -| `nightly.yml`, step `deps`, which jobs `guest` and `tcg` share | the snapshot archive read out of `debian.sources` with `sed`, checked with `test` and written back with `echo` and `rm`, a three-attempt `apt-get` loop with `sleep` and `cat`, `git config`, and rustup through `curl`, `sh` and `echo` | refused: shell of our own | each step is one command | +| `sh` running rustup's `rustup-init.sh` | CI's four rustup installs, fetched with `curl` | refused: a Rust tool does it, rustup's own `rustup-init` binary | a job installs rustup without it | +| `ci.yml`'s `toolchain` and `nightly.yml`'s `build`, step "disk, QEMU and CMake" | `sudo rm -rf` of five preinstalled SDK directories, then `sudo apt-get update` and `install` | refused: shell of our own | each step is one command | +| `ci.yml`'s `guest` and `nightly.yml`'s `tcg`, step `deps`, the same in each | the snapshot archive read out of `debian.sources` with `sed`, checked with `test` and written back with `echo` and `rm`, a three-attempt `apt-get` loop with `sleep` and `cat`, `git config`, and rustup through `curl`, `sh` and `echo` | refused: shell of our own | each step is one command | | `nightly.yml`, job `portability-linux`, step "deps" | the same loop, `git config`, and rustup the same way | refused: shell of our own | each step is one command | | `nightly.yml`, job `portability-macos`, the rustup step | `curl`, `sh rustup-init.sh`, and `echo` into `$GITHUB_PATH` | refused: shell of our own | each step is one command | | `umask 077 && cat > ` | `src/metal.rs` stages the sudoers rule on the T14 with it | refused: shell of our own | Ubuntu leaves the metal loop | diff --git a/src/ci.rs b/src/ci.rs index c7e6ca72395..80a276ece64 100644 --- a/src/ci.rs +++ b/src/ci.rs @@ -3,11 +3,13 @@ //! verdict. //! //! `.github/workflows/` is three files. `ci.yml` runs on a pull request and in -//! the merge queue and boots no guest: [`Job::Host`] runs as `host`. Every -//! test that boots no guest is in [`Job::Host`], so a merge is gated on all of -//! them. `nightly.yml` runs everything that boots a guest, `host` again to -//! write the cache the merge queue restores, and portability. `publish.yml` -//! puts a landing's crates on crates.io. +//! the merge queue: [`Job::Host`] as `host`, [`Job::Toolchain`] as +//! `toolchain`, and [`Job::Guest`] as `guest`, its x86-64 guests on KVM. Every +//! test that boots no guest is in [`Job::Host`] and every guest test in +//! [`Job::Guest`], so a merge is gated on all of them. `nightly.yml` runs the +//! guest suite again under TCG, the toolchain again to move the SDK alias on +//! main, `host` again to write the cache the merge queue restores, and +//! portability. `publish.yml` puts a landing's crates on crates.io. //! //! A host job runs every step and reds if any failed; a guest job stops at the //! first failure among the instrument, the toolchain and the suite, because @@ -34,8 +36,8 @@ const USAGE: &str = "cargo run -- --ci , where is one of: host every host test: the build system, the harness's own checks, the host workspace, the licences of what ships, clippy, the model controls, userland and the SDK (ci.yml, nightly) - toolchain publish this tree's toolchain if nobody has (nightly) - guest the guest suite (nightly) + toolchain publish this tree's toolchain if nobody has (ci.yml, nightly) + guest the guest suite (ci.yml, nightly) publish put main's SDK crates on crates.io (publish.yml)"; #[derive(Debug, PartialEq, Eq)] @@ -391,10 +393,9 @@ fn run_control(root: &Path, control: &Control) -> Result { judge_control(control, green, &log) } -/// The merge queue's whole gate, and the nightly's host lane: every test that -/// runs on the host and boots no guest. The build system's own tests, every -/// member of the host workspace, clippy with warnings denied, the concurrency -/// models' negative controls, every userland crate with a host test +/// Every test that runs on the host and boots no guest. The build system's own +/// tests, every member of the host workspace, clippy with warnings denied, the +/// concurrency models' negative controls, every userland crate with a host test /// ([`crate::userlandhost`], which also reds on a userland test none of them /// runs), and the SDK. /// @@ -403,10 +404,10 @@ fn run_control(root: &Path, control: &Control) -> Result { /// that writes scratch past a `toyos_tmpdir::TempDir`, or holds one past its /// end, is a test that fills the host's disk one run at a time. /// -/// Clippy needs none of the ToyOS toolchain the nightly alone builds — the -/// kernel and the bootloader lint against every architecture's bare targets -/// ([`crate::clippy::BARE_TARGETS`]), which any rustup installs, and userland carries no -/// clippy shape (`src/clippy.rs`). Userland and the SDK are tested against the +/// Clippy needs none of the ToyOS toolchain — the kernel and the bootloader +/// lint against every architecture's bare targets ([`crate::clippy::BARE_TARGETS`]), +/// which any rustup installs, and userland carries no clippy shape +/// (`src/clippy.rs`). Userland and the SDK are tested against the /// host triple for the same reason. fn host(root: &Path) -> Vec { let tmp = toyos_tmpdir::TempDir::new("ci-host"); @@ -848,12 +849,53 @@ mod tests { assert!(at_tip("", tip).is_err()); } + fn workflow(name: &str) -> String { + std::fs::read_to_string(repo_root().join(".github/workflows").join(name)) + .unwrap_or_else(|e| panic!("{name}: {e}")) + } + + /// The lines of job `name` in `text`, empty if it has none. + fn job<'a>(text: &'a str, name: &str) -> Vec<&'a str> { + let head = format!(" {name}:"); + text.lines() + .skip_while(|l| *l != head) + .skip(1) + .take_while(|l| l.is_empty() || l.starts_with(" ")) + .collect() + } + + /// `host` and `guest` are the required checks. A skipped job reads as green + /// to one, so `guest` runs whatever `toolchain` concluded. #[test] - fn the_required_check_is_a_job_on_every_pull_request() { - let text = std::fs::read_to_string(repo_root().join(".github/workflows/ci.yml")) - .expect("ci.yml is readable"); + fn the_required_checks_are_jobs_on_every_pull_request() { + let text = workflow("ci.yml"); assert!(text.contains("\n pull_request:\n") && text.contains("\n merge_group:")); - assert!(text.contains("\n host:"), "ci.yml runs no job `host`"); + assert!(!job(&text, "host").is_empty(), "ci.yml runs no job `host`"); + let guest = job(&text, "guest").join("\n"); + assert!(guest.contains("needs: toolchain") && guest.contains("!cancelled()"), "{guest}"); + } + + /// ci.yml's `guest` and the nightly's `tcg` boot one instrument and share + /// one cache: the image's digest pins QEMU and its firmware, and a restore + /// whose paths are not its writer's restores nothing, in silence. + #[test] + fn the_guest_lanes_share_an_instrument_and_a_cache() { + let lane = |file: &str, name: &str| { + let text = workflow(file); + let lines = job(&text, name); + let image = lines.iter().find_map(|l| l.trim_start().strip_prefix("image: ")); + let paths: Vec<&str> = lines + .iter() + .skip_while(|l| !l.trim_start().starts_with("path:")) + .skip(1) + .take_while(|l| !l.trim_start().starts_with("key:")) + .map(|l| l.trim()) + .collect(); + (image.map(str::to_string), paths.join("\n")) + }; + let (pr, nightly) = (lane("ci.yml", "guest"), lane("nightly.yml", "tcg")); + assert!(pr.0.as_deref().is_some_and(|i| i.contains("@sha256:")) && !pr.1.is_empty(), "{pr:?}"); + assert_eq!(pr, nightly); } /// Every workflow's `pull_request:` trigger names `main` alone, and none diff --git a/tests/common/lane.rs b/tests/common/lane.rs index 526a0669856..09844eb52d6 100644 --- a/tests/common/lane.rs +++ b/tests/common/lane.rs @@ -66,10 +66,9 @@ static RUN: OnceLock = OnceLock::new(); /// it, and it is gone when the run is, green or red (`toyos_tmpdir` is the /// policy, and what reclaims the directory of a run that was killed). /// -/// A red run's serial logs are the parts of it read afterwards — by an agent -/// and by the nightly's artifact — so they are copied to a directory of their -/// own under [`RED_RUN_SERIAL`] first: megabytes, where the images are -/// gigabytes. Named for this run's own root — unique across every process a +/// A red run's serial logs are the parts of it read afterwards, as CI's +/// artifact, so they are copied to a directory of their own under +/// [`RED_RUN_SERIAL`] first: megabytes, where the images are gigabytes. Named for this run's own root — unique across every process a /// shared `$TMPDIR` ever holds — so two red runs of one worktree never share, /// and neither overwrites, a destination. /// From da7c7c0edd2e9536f1d78f4abf474f976dba291a Mon Sep 17 00:00:00 2001 From: japabu Date: Thu, 1 Oct 2026 15:17:56 +0200 Subject: [PATCH 02/30] issues: the two reds CI's guest check found at main's tree The first runs of the 21-test suite on a GitHub runner went 4 passed and 17 failed, on PR #671's `guest` check (run 36863809437) and on main's own nightly `guest` lane at 06788146b (run 36843762360), with the same reds in both: - every `virt_*` boot takes a Synchronous Exception inside the kernel image before the kernel prints anything, under Debian's AAVMF 2026.05-2, where the dev host's QEMU-bundled edk2-stable202408 boots them green; - `nested_nmi_is_loud` under KVM: the unlocked nested-NMI report and cpu1's "joining scheduler" record interleave byte by byte on the 16550, so "NESTED NMI" never appears whole. Neither is this branch's to fix; each is filed with its exit. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L --- ...es-at-the-kernels-entry-on-cis-firmware.md | 37 +++++++++++++++++++ ...erleaves-with-another-cpus-console-line.md | 27 ++++++++++++++ 2 files changed, 64 insertions(+) create mode 100644 issues/kernel/every-aarch64-guest-dies-at-the-kernels-entry-on-cis-firmware.md create mode 100644 issues/kernel/the-nested-nmi-report-interleaves-with-another-cpus-console-line.md diff --git a/issues/kernel/every-aarch64-guest-dies-at-the-kernels-entry-on-cis-firmware.md b/issues/kernel/every-aarch64-guest-dies-at-the-kernels-entry-on-cis-firmware.md new file mode 100644 index 00000000000..f2dc2ea2efb --- /dev/null +++ b/issues/kernel/every-aarch64-guest-dies-at-the-kernels-entry-on-cis-firmware.md @@ -0,0 +1,37 @@ +--- +status: open +kind: defect +opened: 2026-10-01 +--- + +# Every AArch64 guest dies at the kernel's entry on CI's firmware + +All sixteen `virt_*` tests red in CI. Each boot's console ends the same way: +`Loader log: the kernel handoff begins`, then the firmware's +`Synchronous Exception at 0x00000000BC33EB20`. The PC differs per kernel build, +but always falls inside the kernel image the loader placed at `0xbc200000`. That +happens at EL1 entry (`Profile::Virt`) and at EL2 entry (`VirtEl2`), on one CPU +and on eight. The kernel prints nothing first, so the tests time out waiting for +their first marker. + +**Evidence**, identical in two runs on toolchain +`toolchain-linux-x86_64-48dd24f826263d6c`: +- Main's nightly `guest` lane at `06788146b`, run 36843762360, job 110374194368. +- PR #671's `guest` check, on its merge onto `59052827f`, run 36863809437, job + 110375742604. + +Both ran QEMU 11.1.1 under TCG `-cpu max` on an AMD EPYC 9V45 with 4 cores. The +firmware was Debian's `AAVMF_CODE.no-secboot.fd`, "version 2026.05-2". Both +logged `test result: FAILED. 4 passed, 17 failed`. The other red is +`issues/kernel/the-nested-nmi-report-interleaves-with-another-cpus-console-line.md`. + +The same sixteen tests pass on the dev host, in a whole-suite run of #670's +branch (`test result: ok. 21 passed`). That host ran QEMU 11.1.1 from +Homebrew, under the same TCG `-cpu max` for `VirtEl2`. Two parts of the +instrument differ: +- The firmware: the dev host runs QEMU's bundled `edk2-stable202408-prebuilt.qemu.org`. +- The toolchain: the dev host builds its own, and CI installs the published release. + +`.github/qemu-version` pins neither of the two. + +**Exit:** the sixteen `virt_*` tests are green in CI's `guest` check. diff --git a/issues/kernel/the-nested-nmi-report-interleaves-with-another-cpus-console-line.md b/issues/kernel/the-nested-nmi-report-interleaves-with-another-cpus-console-line.md new file mode 100644 index 00000000000..7d04edfd279 --- /dev/null +++ b/issues/kernel/the-nested-nmi-report-interleaves-with-another-cpus-console-line.md @@ -0,0 +1,27 @@ +--- +status: open +kind: defect +opened: 2026-10-01 +--- + +# The nested-NMI report interleaves with another CPU's console line + +`nested_nmi` (`kernel/src/arch/x86_64/idt/nmi.rs`) writes its report through +`serial::panic_raw`, which takes no lock. When cpu1 is writing its own record at +the same moment, the two lines interleave byte by byte on the 16550. The cpu1 +line was `[kernel 0.385 cpu1] CPU 1: joining scheduler`, and the 16550 carried: + + [[kenrnmel i0.38]5 cpNu1E] CSPUT 1E: Djo inNiMngI s choednule r + +`NESTED NMI` is never whole on the console, so `nested_nmi_is_loud` times out +waiting for it, and the machine halts with its report unreadable. + +**Evidence:** red under KVM in two runs, with byte-identical interleaving: +- Main's nightly `guest` lane at `06788146b`, run 36843762360, job 110374194368. +- PR #671's `guest` check, on its merge onto `59052827f`, run 36863809437, job + 110375742604. + +It is green on the dev host under TCG, in a whole-suite run of #670's branch. + +**Exit:** `nested_nmi_is_loud` is green in CI's KVM `guest` check, and a report +written while another CPU is writing a record reads whole. From 2a0e045c4783db88abc9b559a999fdc294d5d717 Mon Sep 17 00:00:00 2001 From: japabu Date: Thu, 1 Oct 2026 15:18:25 +0200 Subject: [PATCH 03/30] issues: the nested-NMI interleave is green under TCG on a runner too Main's nightly `tcg` lane at 06788146b (job 110374194382) passed `nested_nmi_is_loud` on the same container with no `/dev/kvm`, and its sixteen `virt_*` reds are the KVM lane's. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L --- ...ed-nmi-report-interleaves-with-another-cpus-console-line.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/issues/kernel/the-nested-nmi-report-interleaves-with-another-cpus-console-line.md b/issues/kernel/the-nested-nmi-report-interleaves-with-another-cpus-console-line.md index 7d04edfd279..b5b68dcdbe4 100644 --- a/issues/kernel/the-nested-nmi-report-interleaves-with-another-cpus-console-line.md +++ b/issues/kernel/the-nested-nmi-report-interleaves-with-another-cpus-console-line.md @@ -21,7 +21,8 @@ waiting for it, and the machine halts with its report unreadable. - PR #671's `guest` check, on its merge onto `59052827f`, run 36863809437, job 110375742604. -It is green on the dev host under TCG, in a whole-suite run of #670's branch. +It is green under TCG: on the dev host in a whole-suite run of #670's branch, +and on a runner in main's nightly `tcg` lane at `06788146b` (job 110374194382). **Exit:** `nested_nmi_is_loud` is green in CI's KVM `guest` check, and a report written while another CPU is writing a record reads whole. From 90a989e209efae57e70958f459f0fa4cdd751cda Mon Sep 17 00:00:00 2001 From: japabu Date: Thu, 1 Oct 2026 17:03:51 +0200 Subject: [PATCH 04/30] Review round 2: only main publishes, CI installs only what a digest and a vouched commit stand behind, and one definition per lane Security (B1). No job a pull request, the merge queue or the nightly runs holds a token that writes: ci.yml and nightly.yml give their jobs `contents: read` and `actions: read` at the top, and the two workflows they call ask for nothing of their own. The one `contents: write` in any workflow is publish.yml's `release`, and `cargo run -- --ci release` refuses by name before it reads anything unless it runs as publish.yml on main, pushed or dispatched. The nightly's `build`, which published from any branch it was dispatched on, is gone: runs 36709239346 and 36600425263 published wt/toyos-castore's and wt/toyos-notiers' toolchains that way. CI no longer installs a release. toolchain.yml uploads each toolchain it bootstraps whole (upload-artifact v7, `archive: false`), so GitHub's recorded SHA-256 of the artifact is the tarball's own. `release::install` takes the newest build of its tree's tag made by main's publisher; failing that, it takes the newest made by a run of a commit its tree vouches for: its first-parent chain, and the head each merge on that chain took in. It refuses any other, and any download whose bytes hash to anything but GitHub's digest. An artifact is rewritten by nobody. The tag now hashes the build system that builds the toolchain: every module src/toolchain.rs and src/release.rs reach through `crate::`, 18 files, held to the sources by a test that recomputes the closure. Over main's last 100 first-parent landings that moves the tag on 32 where the old trees moved it on 20. The three `SOURCE` constants that existed only for the tag go. Timing (B2). Main publishes on every push (publish.yml's `toolchain` and `release`), not at 03:00. A tree no build answers for bootstraps in its own run's `toolchain` job and publishes nothing: 2h19m to 3h08m in the nightly `build` jobs that bootstrapped since 2026-09-29. Its merge group and every tree in main's window before main's own build lands install that pull request head's build, so the queue never bootstraps unless main moved the toolchain's inputs after the head's last run. One definition (B3). guest.yml is the guest lane, with KVM and the cache save as inputs; toolchain.yml is the toolchain job. ci.yml, nightly.yml and publish.yml call them. The shared-digest test and the cross-file comments go. B4: the guest lanes' gate holds `guest`'s `if:` whole, as `!cancelled()` around `host`'s condition, so the `needs.toolchain.result` mutation reds it. B5: CLAUDE.md and implementer.md say the plain suite runs in CI's `guest` check and the orchestrator runs every guest mutation; orchestrator.md is main's again. The two issue files this branch filed go: #675 and #676, batched with it, close them. The release-tag and install-digest issues close here; the release asset's remaining mutability is filed. The REMOVEd prose is deleted. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L --- .claude/agents/implementer.md | 3 +- .claude/agents/orchestrator.md | 6 +- .github/qemu-version | 4 +- .github/workflows/ci.yml | 107 +-- .github/workflows/guest.yml | 92 +++ .github/workflows/nightly.yml | 122 +-- .github/workflows/publish.yml | 43 +- .github/workflows/toolchain.yml | 44 ++ CLAUDE.md | 2 +- ...t-is-whatever-its-last-writer-put-there.md | 32 + ...d-runs-host-tools-outside-rust-and-qemu.md | 16 +- ...-build-system-that-builds-the-toolchain.md | 13 - ...-unpacks-an-asset-no-digest-vouches-for.md | 25 - ...es-at-the-kernels-entry-on-cis-firmware.md | 37 - ...are-one-rep-movsb-or-stosb-on-every-cpu.md | 6 +- ...s-linux-on-every-machine-toyos-supports.md | 4 +- ...erleaves-with-another-cpus-console-line.md | 28 - ...es-not-model-an-affected-cpus-l1tf-line.md | 2 +- src/ci.rs | 123 +-- src/clang.rs | 3 - src/libcxx.rs | 3 - src/n2.rs | 3 - src/release.rs | 725 ++++++++++++++---- src/toolchain.rs | 9 +- tests/common/lane.rs | 10 +- 25 files changed, 895 insertions(+), 567 deletions(-) create mode 100644 .github/workflows/guest.yml create mode 100644 .github/workflows/toolchain.yml create mode 100644 issues/build/a-toolchain-releases-asset-is-whatever-its-last-writer-put-there.md delete mode 100644 issues/build/the-release-tag-hashes-none-of-the-build-system-that-builds-the-toolchain.md delete mode 100644 issues/build/the-toolchain-install-unpacks-an-asset-no-digest-vouches-for.md delete mode 100644 issues/kernel/every-aarch64-guest-dies-at-the-kernels-entry-on-cis-firmware.md delete mode 100644 issues/kernel/the-nested-nmi-report-interleaves-with-another-cpus-console-line.md diff --git a/.claude/agents/implementer.md b/.claude/agents/implementer.md index 48eae3b60b9..d3925207903 100644 --- a/.claude/agents/implementer.md +++ b/.claude/agents/implementer.md @@ -20,7 +20,8 @@ Where hardware or anything uncertain is involved, take the cheap measurement bef guess. Then build, then test before anyone reviews: - Host tests only: `cargo run -- --ci host`, and `cargo run -- --build-only` at most for the - image. Never a guest test or any other `cargo run`: the guest suite runs in CI's `guest` check. + image. Never a guest test or any other `cargo run`: the plain suite runs in CI's `guest` check, + and the orchestrator runs every guest mutation. - A result is the command's own exit code: ` > 2>&1; echo EXIT=$?`. A grepped `test result` line is not one, and a gate you did not run is a gate you do not claim. - Long commands run in the background with output to a file under the job scratchpad the brief diff --git a/.claude/agents/orchestrator.md b/.claude/agents/orchestrator.md index ef6ea3ea4a5..6eb84aae517 100644 --- a/.claude/agents/orchestrator.md +++ b/.claude/agents/orchestrator.md @@ -33,8 +33,8 @@ scaffolding, deleted once its question is answered. Every task gets a fresh agent with an explicit model matched to the judgment in it: the strongest for drivers, security boundaries and reviews of them, and a mid tier for mechanical fixes from an exact list. A resumed agent only ever finishes its own interrupted task. A finished agent's report -is acted on before the next agent is dispatched: its review spawned, or its fix round sent. When -the permission check refuses an agent, ask the owner and never route around it. +is acted on before the next agent is dispatched: its guest runs queued, its review spawned, or its +fix round sent. When the permission check refuses an agent, ask the owner and never route around it. A brief is the fence: what to build, where it may touch, the worktree and branch, the scratchpad for its logs, and the two checks expected of high-risk code. The role files carry the standing rules, so a brief carries only the task. @@ -68,7 +68,7 @@ its script path under the job directory, and revert any mutation a killed run le left mid-flash or mid-boot is power-cycled by the owner and comes back to Ubuntu: BootNext is one-shot. -A metal mutation loop starts on a clean worktree at the head under review and leaves it +A mutation loop, guest or metal, starts on a clean worktree at the head under review and leaves it clean: `git apply --check`, `git apply`, the tests by name, `git apply -R`. None runs while an agent edits that worktree. A queue script passes only flags `src/testargs.rs` declares: any other word becomes the run's filter, and a one-test run reports as a pass. diff --git a/.github/qemu-version b/.github/qemu-version index 5e6cc8ab4e3..b4fc096a925 100644 --- a/.github/qemu-version +++ b/.github/qemu-version @@ -3,6 +3,6 @@ # build system's prerequisite check. The version decides test outcomes — 8.2.2 # and 11.0.3 were measured disagreeing about the same tree — so this is a # declaration the guest image is held to, never a fact read off it. The image -# digest ci.yml and nightly.yml share is chosen to satisfy this line; changing -# either is a deliberate act: it says the instrument moved. +# digest is chosen to satisfy this line; changing either is a deliberate act: +# it says the instrument moved. 11.1.1 diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 61a63e2a49a..d72f914e076 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -1,9 +1,5 @@ name: ci -# A pull request and the merge queue: the host tests, and the guest suite on -# this tree's toolchain. Each step is `cargo run -- --ci ` (src/ci.rs), -# which runs the same on a dev host. - on: pull_request: branches: [main] @@ -14,6 +10,11 @@ concurrency: group: ci-${{ github.ref }} cancel-in-progress: true +# No job here writes: a toolchain release is main's publisher's alone. +permissions: + contents: read + actions: read + jobs: host: if: github.event_name == 'merge_group' || github.event.pull_request.draft == false @@ -40,103 +41,15 @@ jobs: - run: cargo run -- --ci host - # Publishes this tree's toolchain if nobody has: a lookup when the tree hashes - # what main's does (src/release.rs), a bootstrap of hours when it moved - # those trees. Bare `ubuntu-24.04`, not a container: its glibc is the - # release's floor. toolchain: if: github.event_name == 'merge_group' || github.event.pull_request.draft == false - runs-on: ubuntu-24.04 - timeout-minutes: 350 - permissions: - contents: write - steps: - - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 - with: - fetch-depth: 0 - - - name: disk, QEMU and CMake - run: | - sudo rm -rf /usr/local/lib/android /usr/share/dotnet /opt/ghc \ - /usr/local/share/boost /usr/local/.ghcup - sudo apt-get update -qq - sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq qemu-system-x86 zstd \ - cmake=3.28.3-1build7 - - - env: - GH_TOKEN: ${{ github.token }} - run: cargo run -- --ci toolchain + uses: ./.github/workflows/toolchain.yml # A required check that is skipped reads as green, so this runs whatever - # `toolchain` concluded: a toolchain it did not publish reds the install. + # `toolchain` concluded: a tree no build answers for reds the install. guest: needs: toolchain if: ${{ !cancelled() && (github.event_name == 'merge_group' || github.event.pull_request.draft == false) }} - runs-on: ubuntu-24.04 - # A wedge guard, not a budget. - timeout-minutes: 60 - # The digest is the instrument's one pin, and nightly.yml's `tcg` names the - # same: a dated image names the snapshot archive it was built from, and - # `deps` installs QEMU from that archive. The node ships - # `crw-rw---- root:kvm` and root opens it. - container: - image: debian:sid-20260918@sha256:a2aa46262453eba3f464d8b1c7a8c31db85eb15af180ae34dd400615d7208547 - options: --device=/dev/kvm - env: - GH_TOKEN: ${{ github.token }} - steps: - # Before the checkout, which wants git. Three attempts, because the - # archive is fixed and the network to it is not. - - name: deps - run: | - snap=$(sed -n 's|^# \(http://snapshot\.debian\.org/archive/debian/[0-9]*T[0-9]*Z\)$|\1|p' \ - /etc/apt/sources.list.d/debian.sources) - test -n "$snap" - echo "deb $snap sid main" > /etc/apt/sources.list - rm /etc/apt/sources.list.d/debian.sources - for attempt in 1 2 3; do - apt-get -o Acquire::Check-Valid-Until=false update -qq > /tmp/apt.log 2>&1 \ - && DEBIAN_FRONTEND=noninteractive apt-get install -y -qq git curl ca-certificates \ - zstd build-essential qemu-system-x86 ovmf-generic qemu-system-arm \ - qemu-efi-aarch64 >> /tmp/apt.log 2>&1 \ - && break - [ "$attempt" = 3 ] && { cat /tmp/apt.log; exit 1; } - sleep 20 - done - # `actions/checkout` sets this only in a config it discards. - git config --global --add safe.directory "$GITHUB_WORKSPACE" - curl --proto '=https' --tlsv1.2 -sSf -o "$RUNNER_TEMP/rustup-init.sh" https://sh.rustup.rs - sh "$RUNNER_TEMP/rustup-init.sh" -y --profile minimal --default-toolchain stable - echo "$HOME/.cargo/bin" >> "$GITHUB_PATH" - - - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 - - # nightly.yml's `tcg` is the one writer; the paths are the cache's - # version, so they are the writer's list. - - uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 - with: - path: | - ~/.cargo/registry/index - ~/.cargo/registry/cache - ~/.cargo/git/db - target - kernel/target - bootloader/target - userland/target - tests/target - tests/toyos-rust-tests/*/target - key: guest-${{ github.run_id }} - restore-keys: guest- - - - run: cargo run -- --ci guest - - # Every boot's 16550 log: what a guest that died early still leaves. - - name: serial logs - if: failure() - continue-on-error: true - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 - with: - name: serial-${{ github.job }} - path: target/red-run-serial/**/uart-*.log - if-no-files-found: warn - retention-days: 7 + uses: ./.github/workflows/guest.yml + with: + kvm: true diff --git a/.github/workflows/guest.yml b/.github/workflows/guest.yml new file mode 100644 index 00000000000..cf8a064a3a6 --- /dev/null +++ b/.github/workflows/guest.yml @@ -0,0 +1,92 @@ +name: guest + +on: + workflow_call: + inputs: + # Without it every guest is emulated: the only lane that decodes the paths + # a KVM host's CPU never does. + kvm: + type: boolean + required: true + # The guest cache's one writer is the lane that asks. + save-cache: + type: boolean + default: false + +jobs: + suite: + runs-on: ubuntu-24.04 + # A wedge guard, not a budget. + timeout-minutes: 60 + # The digest is the instrument's one pin: a dated image names the snapshot + # archive it was built from, and `deps` installs QEMU from that archive. + # The node ships `crw-rw---- root:kvm` and root opens it. + container: + image: debian:sid-20260918@sha256:a2aa46262453eba3f464d8b1c7a8c31db85eb15af180ae34dd400615d7208547 + options: ${{ inputs.kvm && '--device=/dev/kvm' || '' }} + env: + GH_TOKEN: ${{ github.token }} + steps: + # Before the checkout, which wants git. Three attempts, because the + # archive is fixed and the network to it is not. + - name: deps + run: | + snap=$(sed -n 's|^# \(http://snapshot\.debian\.org/archive/debian/[0-9]*T[0-9]*Z\)$|\1|p' \ + /etc/apt/sources.list.d/debian.sources) + test -n "$snap" + echo "deb $snap sid main" > /etc/apt/sources.list + rm /etc/apt/sources.list.d/debian.sources + for attempt in 1 2 3; do + apt-get -o Acquire::Check-Valid-Until=false update -qq > /tmp/apt.log 2>&1 \ + && DEBIAN_FRONTEND=noninteractive apt-get install -y -qq git curl ca-certificates \ + zstd build-essential qemu-system-x86 ovmf-generic qemu-system-arm \ + qemu-efi-aarch64 >> /tmp/apt.log 2>&1 \ + && break + [ "$attempt" = 3 ] && { cat /tmp/apt.log; exit 1; } + sleep 20 + done + # `actions/checkout` sets this only in a config it discards. + git config --global --add safe.directory "$GITHUB_WORKSPACE" + curl --proto '=https' --tlsv1.2 -sSf -o "$RUNNER_TEMP/rustup-init.sh" https://sh.rustup.rs + sh "$RUNNER_TEMP/rustup-init.sh" -y --profile minimal --default-toolchain stable + echo "$HOME/.cargo/bin" >> "$GITHUB_PATH" + + # The whole history: which builds this tree installs is read off it. + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + with: + fetch-depth: 0 + + - uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 + with: + path: &guest-paths | + ~/.cargo/registry/index + ~/.cargo/registry/cache + ~/.cargo/git/db + target + kernel/target + bootloader/target + userland/target + tests/target + tests/toyos-rust-tests/*/target + key: guest-${{ github.run_id }} + restore-keys: guest- + + - run: cargo run -- --ci guest + + # After the test: what is worth keeping is a tree that built and booted. + - if: inputs.save-cache + uses: actions/cache/save@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 + with: + path: *guest-paths + key: guest-${{ github.run_id }} + + # Every boot's 16550 log: what a guest that died early still leaves. + - name: serial logs + if: failure() + continue-on-error: true + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 + with: + name: serial-${{ github.job }} + path: target/red-run-serial/**/uart-*.log + if-no-files-found: warn + retention-days: 7 diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index 9720c2012c1..2059bf1497f 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -1,8 +1,5 @@ name: nightly -# What the pull request's gate does not cover: the guest suite under TCG, the -# SDK alias on main, the host gate again to write the cache the merge queue -# restores, and portability. # Every job's logic is `cargo run -- --ci ` (src/ci.rs); this file says # where each one runs. @@ -11,11 +8,15 @@ on: - cron: '0 3 * * *' workflow_dispatch: -# Never cancelled: `build` may be an hour into a bootstrap. +# Never cancelled: `toolchain` may be an hour into a bootstrap. concurrency: group: nightly-${{ github.ref }} cancel-in-progress: false +permissions: + contents: read + actions: read + jobs: host: runs-on: ubuntu-24.04 @@ -49,110 +50,16 @@ jobs: path: *host-paths key: host-linux-${{ github.run_id }} - # Publishes this tree's toolchain if nobody has, and on main moves the SDK - # alias onto it. Bare `ubuntu-24.04`, not a container: its glibc is the - # release's floor. - build: - runs-on: ubuntu-24.04 - timeout-minutes: 350 - permissions: - contents: write - steps: - - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 - with: - fetch-depth: 0 - - - name: disk, QEMU and CMake - run: | - sudo rm -rf /usr/local/lib/android /usr/share/dotnet /opt/ghc \ - /usr/local/share/boost /usr/local/.ghcup - sudo apt-get update -qq - sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq qemu-system-x86 zstd \ - cmake=3.28.3-1build7 - - - env: - GH_TOKEN: ${{ github.token }} - run: cargo run -- --ci toolchain + toolchain: + uses: ./.github/workflows/toolchain.yml - # The guest suite with no `/dev/kvm`, so the only lane that decodes the paths - # a KVM host's CPU never does; and the guest cache's one writer, since what - # it builds does not depend on the accelerator. tcg: - needs: build - runs-on: ubuntu-24.04 - # A wedge guard, not a budget. - timeout-minutes: 60 - # ci.yml's `guest` names the same digest, the instrument's one pin: a dated - # image names the snapshot archive it was built from, and `deps` installs - # QEMU from that archive. - container: - image: debian:sid-20260918@sha256:a2aa46262453eba3f464d8b1c7a8c31db85eb15af180ae34dd400615d7208547 - env: - GH_TOKEN: ${{ github.token }} - steps: - # Before the checkout, which wants git. Three attempts, because the - # archive is fixed and the network to it is not. - - name: deps - run: | - snap=$(sed -n 's|^# \(http://snapshot\.debian\.org/archive/debian/[0-9]*T[0-9]*Z\)$|\1|p' \ - /etc/apt/sources.list.d/debian.sources) - test -n "$snap" - echo "deb $snap sid main" > /etc/apt/sources.list - rm /etc/apt/sources.list.d/debian.sources - for attempt in 1 2 3; do - apt-get -o Acquire::Check-Valid-Until=false update -qq > /tmp/apt.log 2>&1 \ - && DEBIAN_FRONTEND=noninteractive apt-get install -y -qq git curl ca-certificates \ - zstd build-essential qemu-system-x86 ovmf-generic qemu-system-arm \ - qemu-efi-aarch64 >> /tmp/apt.log 2>&1 \ - && break - [ "$attempt" = 3 ] && { cat /tmp/apt.log; exit 1; } - sleep 20 - done - # `actions/checkout` sets this only in a config it discards. - git config --global --add safe.directory "$GITHUB_WORKSPACE" - curl --proto '=https' --tlsv1.2 -sSf -o "$RUNNER_TEMP/rustup-init.sh" https://sh.rustup.rs - sh "$RUNNER_TEMP/rustup-init.sh" -y --profile minimal --default-toolchain stable - echo "$HOME/.cargo/bin" >> "$GITHUB_PATH" - - - &checkout - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 - - # ci.yml's `guest` restores what this saves, and the paths are the - # cache's version, so its list is this one. - - uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 - with: - path: &guest-paths | - ~/.cargo/registry/index - ~/.cargo/registry/cache - ~/.cargo/git/db - target - kernel/target - bootloader/target - userland/target - tests/target - tests/toyos-rust-tests/*/target - key: guest-${{ github.run_id }} - restore-keys: guest- - - - run: cargo run -- --ci guest - - # After the test: what is worth keeping is a tree that built and booted. - - if: github.ref == 'refs/heads/main' - uses: actions/cache/save@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 - with: - path: *guest-paths - key: guest-${{ github.run_id }} - - # Every boot's 16550 log: what a guest that died early still leaves. - - name: serial logs - if: failure() - continue-on-error: true - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 - with: - name: serial-${{ github.job }} - path: target/red-run-serial/**/uart-*.log - if-no-files-found: warn - retention-days: 7 + needs: toolchain + if: ${{ !cancelled() }} + uses: ./.github/workflows/guest.yml + with: + kvm: false + save-cache: ${{ github.ref == 'refs/heads/main' }} # `cargo run -- --build-only` from a fresh machine. `sid` as it stands, # image and archive both, and no cache — a fresh machine is the premise. @@ -177,7 +84,8 @@ jobs: sh "$RUNNER_TEMP/rustup-init.sh" -y --profile minimal --default-toolchain stable echo "$HOME/.cargo/bin" >> "$GITHUB_PATH" - - *checkout + - &checkout + uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 - run: env -u GITHUB_ACTIONS -u CI cargo run -- --build-only diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 1c2e8f9e066..48b251a25bb 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -9,12 +9,6 @@ on: branches: [main] workflow_dispatch: {} -# Never two publishers, and never cancel one: a cancelled `cargo publish` may -# have already taken the version. -concurrency: - group: publish - cancel-in-progress: false - permissions: contents: read # crates.io trusted publishing: the job trades its OIDC token for a @@ -25,6 +19,11 @@ jobs: publish: runs-on: ubuntu-latest timeout-minutes: 30 + # Never two publishers, and never cancel one: a cancelled `cargo publish` + # may have already taken the version. + concurrency: + group: publish + cancel-in-progress: false steps: # No submodules: `cargo publish` walks the repository's vcs state, and an # initialised but empty `rust/` breaks that walk. @@ -36,3 +35,35 @@ jobs: - env: CARGO_REGISTRY_TOKEN: ${{ steps.auth.outputs.token }} run: cargo run -- --ci publish + + # Main's own build of its toolchain, which only main's publisher's runs make + # for main. One at a time: a second would bootstrap the same tag again. + toolchain: + concurrency: + group: publish-toolchain + cancel-in-progress: false + permissions: + contents: read + actions: read + uses: ./.github/workflows/toolchain.yml + + # Main's build put up as the release a consumer outside CI installs, and the + # SDK alias moved onto it once crates.io holds this tree's crates: the one job + # any workflow gives a token that writes this repository. + release: + needs: [publish, toolchain] + if: ${{ !cancelled() }} + runs-on: ubuntu-24.04 + timeout-minutes: 30 + concurrency: + group: publish-release + cancel-in-progress: false + permissions: + contents: write + actions: read + steps: + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + + - env: + GH_TOKEN: ${{ github.token }} + run: cargo run -- --ci release diff --git a/.github/workflows/toolchain.yml b/.github/workflows/toolchain.yml new file mode 100644 index 00000000000..55f2d8ffa77 --- /dev/null +++ b/.github/workflows/toolchain.yml @@ -0,0 +1,44 @@ +name: toolchain + +# A tree's toolchain, bootstrapped only where no build answers for its tag, and +# kept as this run's artifact (src/release.rs). Nothing here publishes, and the +# job holds only the token its caller gives it. + +on: + workflow_call: + +jobs: + build: + # Bare, not a container: its glibc is a build's floor. + runs-on: ubuntu-24.04 + timeout-minutes: 350 + env: + GH_TOKEN: ${{ github.token }} + steps: + # The whole history: which builds answer for this tree is read off it. + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + with: + fetch-depth: 0 + + - id: find + run: cargo run -- --ci toolchain + + # What a bootstrap needs, and only when there is one to do. + - name: disk, QEMU and CMake + if: steps.find.outputs.bootstrap == 'true' + run: | + sudo rm -rf /usr/local/lib/android /usr/share/dotnet /opt/ghc \ + /usr/local/share/boost /usr/local/.ghcup + sudo apt-get update -qq + sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq qemu-system-x86 zstd \ + cmake=3.28.3-1build7 + + - run: cargo run -- --ci bootstrap + + # Whole, not zipped: GitHub's digest of the artifact is then the + # tarball's own, which an install holds its download to. + - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 + with: + path: target/toolchain/*.tar.zst + archive: false + if-no-files-found: ignore diff --git a/CLAUDE.md b/CLAUDE.md index 8ba6baca279..695a095b6f6 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -72,7 +72,7 @@ The bar is not yet the tree: `.claude/agents/reviewer.md`, "Arrivals", says wher The testing rules live where they are enforced: the PR gate and the nightly in `.github/workflows/`. Operationally: - `cargo run` builds everything (toolchain, kernel, bootloader, userland, image) and launches QEMU; `--build-only` skips the launch. `cargo test` runs the QEMU harness; `cargo run -- --ci host` runs every host suite, as the PR gate's required `host` check does. -- **Agents never run QEMU locally.** An agent verifies with host tests and builds the image at most; the guest suite runs in CI's `guest` check. +- **Agents never run QEMU.** An agent verifies with host tests and builds the image at most; the plain suite runs in CI's `guest` check, and the orchestrator runs every patched guest run a negative control needs, one suite at a time. - **Both produce large output**: run them in the background and read the output file — `[N characters truncated]` means data was lost. A full boot is under a second; incremental builds finish in seconds. - **Leave the machine as you found it.** The development machine is shared: every agent stops what it started, removes the worktrees and scratch build output it no longer needs, and never leaves an emulator, a build or a watcher running. diff --git a/issues/build/a-toolchain-releases-asset-is-whatever-its-last-writer-put-there.md b/issues/build/a-toolchain-releases-asset-is-whatever-its-last-writer-put-there.md new file mode 100644 index 00000000000..2d481126f3a --- /dev/null +++ b/issues/build/a-toolchain-releases-asset-is-whatever-its-last-writer-put-there.md @@ -0,0 +1,32 @@ +--- +status: open +kind: tooling +opened: 2026-10-01 +--- + +# A toolchain release's asset is whatever its last writer put there + +CI installs no release. It installs a build whose bytes hash to the digest +GitHub recorded at upload, made by main's publisher or by a run of a commit its +tree vouches for (`src/release.rs`). The release that main's publisher puts up +is what a consumer outside CI installs, and so is the SDK alias that names it. +The release notes' install steps take the asset as it is served. + +A branch's workflows decide their own token's permissions. A workflow on any +branch of this repository can ask for `contents: write` and then replace that +asset, or create the release for a tag main has not yet published. A pull +request's `toolchain` job held such a token in run 36863809437; its log reads +`Contents: write`. Two nightly runs dispatched on branches built and published +their branches' toolchains under the nightly's write token: +`wt/toyos-castore` published `toolchain-linux-x86_64-688e609acf5a65c4` (run +36709239346), and `wt/toyos-notiers` published +`toolchain-linux-x86_64-92f146618d6687d8` (run 36600425263). Every toolchain release is mutable (`immutable: +false`). `publish.yml` puts main's build back only on main's next push. + +Owner: the release module (`src/release.rs`). + +**Exit**: a consumer outside CI installs only the bytes main's publisher +recorded, and its install refuses any other bytes by name. That holds when a +published release can no longer change (with the SDK alias made a new release +per move, never one moved), or when the release notes' install checks the +digest that main's publisher's artifact carries. diff --git a/issues/build/the-build-runs-host-tools-outside-rust-and-qemu.md b/issues/build/the-build-runs-host-tools-outside-rust-and-qemu.md index 6f70584873f..1b9b23b721b 100644 --- a/issues/build/the-build-runs-host-tools-outside-rust-and-qemu.md +++ b/issues/build/the-build-runs-host-tools-outside-rust-and-qemu.md @@ -21,15 +21,15 @@ arrives and is not one. M4 and M5 are stages of `issues/build/toyos-builds-itsel | `git` for reads, a config write, a commit's paths written out, and clones and fetches over HTTPS | `rev-parse`, `show-ref`, `for-each-ref`, `rev-list`, `log`, `branch --contains`, `merge-base`, `ls-tree`, `ls-files`, `cat-file`, `config --get-regexp`, `worktree list`, `status`, `diff`, `ls-remote` and `grep`, in the build system and its tests; `config --global --add safe.directory` in CI's containers; `checkout -- ` through an index of its own, which writes the C++ runtime's sources out of the LLVM commit into the stored LLVM (`src/llvm.rs`); `src/sync.rs`'s fetch of `origin`; every workflow's checkout | refused: a Rust tool does it, gitoxide 0.85, which reads refs, objects, the index, config, worktrees and status, adds a value to a config file and writes it (gix-config 0.58's `File::section_mut_or_create_new`, `SectionMut::push`, `File::write_to`), walks history, diffs, and lists, fetches and clones a remote over HTTPS; `grep` is a search of the files its index names; and gitoxide's CLI 0.59 (gix 0.88) wrote the runtimes' sources of LLVM `849da7d6` into an empty directory, each path's tree through `gix rev parse`, `gix index from-tree` and `gix free index checkout-exclusive`, exit 0 each: the 18759 files `git` writes there, byte for byte and mode for mode | those are gitoxide's | | `cc`, `c++` and `ar` on a Linux host, `build-essential` in CI's containers | rustc links every host binary through `cc`; `cc` and `c++` compile LLVM, clang, LLD and `rustc_llvm` (`src/llvm.rs` names both to bootstrap); `ar` archives what `cc::Build` compiles | admitted: no Rust tool compiles C or C++, or takes rustc's host link | M5: no host in the loop | | the toolchain's own `clang`, `llvm-ar`, `rust-lld` and `llvm-config`, built from `ToyOSOrg/llvm-project` | rustc links every guest binary with `rust-lld`; `clang` compiles the C corpus (`tests/common/compile.rs`) and, with `llvm-ar`, doomgeneric through `cc::Build` (`src/clang.rs`); rustc's bootstrap asks `llvm-config` how to link LLVM | admitted: our fork's C++, which ToyOS can one day build and run; no Rust tool compiles C, `cc::Build` archives with an `ar`, bootstrap reads LLVM through `llvm-config`, and `CLAUDE.md` links everything with `rust-lld` | M5: no host in the loop | -| `ovmf-generic`, `qemu-efi-aarch64` | the x86-64 and AArch64 UEFI firmware of CI's guest containers (`src/firmware.rs`), packaged by Debian apart from QEMU | admitted: QEMU's own firmware, and no Rust firmware does its job | the instrument's QEMU carries its own firmware | -| `ca-certificates` | the trust store `git` and `curl` verify against in CI's containers | admitted: data both of them need | goes when neither runs there | +| `ovmf-generic`, `qemu-efi-aarch64` | the x86-64 and AArch64 UEFI firmware of the nightly's guest containers (`src/firmware.rs`), packaged by Debian apart from QEMU | admitted: QEMU's own firmware, and no Rust firmware does its job | the instrument's QEMU carries its own firmware | +| `ca-certificates` | the trust store `git` and `curl` verify against in the nightly's containers | admitted: data both of them need | goes when neither runs there | | `env` | the two portability jobs run `cargo run -- --build-only` under it, with `GITHUB_ACTIONS` and `CI` unset | refused: a Rust tool does it, the build system, which already runs that build with both removed (`src/release.rs`) | the build system removes both itself | | the T14's Ubuntu and every tool `src/metal.rs` runs on it over `ssh` | the metal loop, on the T14 and never on a development host | outside the rule: recovery equipment on a test machine, not the build's host | they leave with Ubuntu (`issues/boot-media/the-machine-updates-itself-without-ubuntu.md`) | | `sh` running `rust/x`, and Python running `x.py` and `bootstrap.py` | every toolchain build (`src/toolchain.rs`) | refused: a Rust tool does it, upstream's bootstrap binary, which builds with stable cargo, fetches its own stage0 (`rust/src/bootstrap/src/core/download.rs`) and needs no Python | `src/toolchain.rs` runs the bootstrap binary | | `curl` in rustc's bootstrap | fetches the stage0 `rust/src/stage0` pins, for a compiler or LLVM build whose build directory lacks it, whichever bootstrap runs | refused: a Rust tool does it, rustup installs the dated beta the pin names, and bootstrap takes a stage0 through `build.rustc` and `build.cargo`, as `src/sysroot.rs` hands it one | no toolchain build fetches with `curl` | -| `curl` in `src/release.rs` and `src/sdkversion.rs` | the toolchain release's lookup and download and the crates.io index, on CI runners only | refused: a Rust tool does it, `ureq`, which `userland/doom/build.rs` already fetches with | those fetches are Rust's | -| `tar` and `zstd` | `src/release.rs` packs and unpacks the toolchain release, on CI runners only | refused: a Rust tool does it, the `tar` crate `userland/doom/build.rs` already unpacks with, and a zstd crate | both are done in Rust, in-process | -| `gh` | `src/release.rs` asks whether a toolchain release exists, creates it and moves the `sdk-` alias, on the `toolchain` and nightly `build` runners | refused: not C or C++ source, it is Go | `src/release.rs` speaks GitHub's REST API itself | +| `curl` in `src/release.rs` and `src/sdkversion.rs` | GitHub's API: the lookup and download of a toolchain build and the lookup of its release; and the crates.io index; on CI runners only | refused: a Rust tool does it, `ureq`, which `userland/doom/build.rs` already fetches with | those fetches are Rust's | +| `tar` and `zstd` | `src/release.rs` packs and unpacks a toolchain build, on CI runners only | refused: a Rust tool does it, the `tar` crate `userland/doom/build.rs` already unpacks with, and a zstd crate | both are done in Rust, in-process | +| `gh` | `src/release.rs` creates a toolchain release or replaces its asset, and moves the `sdk-` alias, on `publish.yml`'s `release` runner, on main alone | refused: not C or C++ source, it is Go | `src/release.rs` speaks GitHub's REST API itself | | `ssh` | `src/metal.rs` reaches the T14's Ubuntu with it, only in the metal loop | refused: a Rust tool does it, the repository's own russh client `crate::build::ssh_client_host`, which `src/metaltalk.rs` already drives | `src/metal.rs` drives that client, or Ubuntu leaves the loop | | `cc`, `c++`, `ar` and `xcrun` on a macOS host, Apple's Command Line Tools | what the Linux row's tools do, and rustc asks `xcrun` for the SDK on every host link that names no `SDKROOT` (`rust/compiler/rustc_codegen_ssa/src/back/apple.rs`), as `src/llvm.rs` does for the LLVM's key | refused: one host OS alone | M5: no host in the loop | | `diag/flash.sh` | the owner's flash of a stick by hand: `bash`, and the `stat`, `seq`, `tr`, `grep`, `cut` and `sync` it strings together | refused: shell of our own | `issues/build/the-owners-flash-script-runs-diskutil.md` | @@ -40,9 +40,9 @@ arrives and is not one. M4 and M5 are stages of `issues/build/toyos-builds-itsel | `sync` | the README's Linux flashing steps flush the stick with it | refused: a Rust tool does it, the build system can flush what it writes | `issues/build/the-owners-flash-script-runs-diskutil.md` | | `sudo` on macOS | `diag/flash.sh` and the README's macOS flashing steps run `dd` under it | admitted: no Rust tool raises a process to root on macOS; sudo-rs "is targeted for FreeBSD and Linux-based operating systems only" (its README at `89bae8a`) | goes with both flashes by hand | | `sudo` on Linux | the README's Linux flashing steps run `dd` under it | refused: a Rust tool does it, sudo-rs | the README's Linux steps run sudo-rs | -| `sh` running rustup's `rustup-init.sh` | CI's four rustup installs, fetched with `curl` | refused: a Rust tool does it, rustup's own `rustup-init` binary | a job installs rustup without it | -| `ci.yml`'s `toolchain` and `nightly.yml`'s `build`, step "disk, QEMU and CMake" | `sudo rm -rf` of five preinstalled SDK directories, then `sudo apt-get update` and `install` | refused: shell of our own | each step is one command | -| `ci.yml`'s `guest` and `nightly.yml`'s `tcg`, step `deps`, the same in each | the snapshot archive read out of `debian.sources` with `sed`, checked with `test` and written back with `echo` and `rm`, a three-attempt `apt-get` loop with `sleep` and `cat`, `git config`, and rustup through `curl`, `sh` and `echo` | refused: shell of our own | each step is one command | +| `sh` running rustup's `rustup-init.sh` | CI's rustup installs, fetched with `curl` | refused: a Rust tool does it, rustup's own `rustup-init` binary | a job installs rustup without it | +| `toolchain.yml`, step "disk, QEMU and CMake" | `sudo rm -rf` of five preinstalled SDK directories, then `sudo apt-get update` and `install` | refused: shell of our own | each step is one command | +| `guest.yml`, step `deps` | the snapshot archive read out of `debian.sources` with `sed`, checked with `test` and written back with `echo` and `rm`, a three-attempt `apt-get` loop with `sleep` and `cat`, `git config`, and rustup through `curl`, `sh` and `echo` | refused: shell of our own | each step is one command | | `nightly.yml`, job `portability-linux`, step "deps" | the same loop, `git config`, and rustup the same way | refused: shell of our own | each step is one command | | `nightly.yml`, job `portability-macos`, the rustup step | `curl`, `sh rustup-init.sh`, and `echo` into `$GITHUB_PATH` | refused: shell of our own | each step is one command | | `umask 077 && cat > ` | `src/metal.rs` stages the sudoers rule on the T14 with it | refused: shell of our own | Ubuntu leaves the metal loop | diff --git a/issues/build/the-release-tag-hashes-none-of-the-build-system-that-builds-the-toolchain.md b/issues/build/the-release-tag-hashes-none-of-the-build-system-that-builds-the-toolchain.md deleted file mode 100644 index a526dfc0049..00000000000 --- a/issues/build/the-release-tag-hashes-none-of-the-build-system-that-builds-the-toolchain.md +++ /dev/null @@ -1,13 +0,0 @@ ---- -status: open -kind: tooling -opened: 2026-09-27 ---- - -# The release tag hashes none of the build system that builds the toolchain - -`src/toolchain.rs`'s `write_config`, `src/sysroot.rs` and `src/libc.rs` decide -the tarball's bytes and are not in `release::trees()`, so a change to them keeps -the old tag and CI installs a toolchain its tree does not describe. - -**Exit**: a commit to each moves the tag, shown by `src/release.rs`'s tests. diff --git a/issues/build/the-toolchain-install-unpacks-an-asset-no-digest-vouches-for.md b/issues/build/the-toolchain-install-unpacks-an-asset-no-digest-vouches-for.md deleted file mode 100644 index 3d2f5285f42..00000000000 --- a/issues/build/the-toolchain-install-unpacks-an-asset-no-digest-vouches-for.md +++ /dev/null @@ -1,25 +0,0 @@ ---- -status: open -kind: tooling -opened: 2026-09-28 ---- - -# The toolchain install unpacks an asset no digest vouches for - -`release::install` (`src/release.rs`) downloads the `toyos-toolchain.tar.zst` -asset of the release its tree's tag names, unpacks it into `rust/build` and -links it as rustup's `toyos`, and checks nothing about its bytes: the tag is -the hash of the tarball's inputs (`trees`), not of the tarball, and the -`toyos-sysroot-witness` an installed toolchain is held to comes inside it. -Every guest job compiles and boots with what it installs. - -A job holding this repository's `contents: write` token can replace a -release asset, so any code such a job runs can replace the toolchain every -later job installs. The nightly's `build` job holds that token while it -bootstraps the toolchain. - -Owner: the release module (`src/release.rs`). - -**Exit condition.** `install` unpacks only an asset whose SHA-256 is one that -no job holding a write token can rewrite — committed to the tree it installs -for — and refuses any other by name. diff --git a/issues/kernel/every-aarch64-guest-dies-at-the-kernels-entry-on-cis-firmware.md b/issues/kernel/every-aarch64-guest-dies-at-the-kernels-entry-on-cis-firmware.md deleted file mode 100644 index f2dc2ea2efb..00000000000 --- a/issues/kernel/every-aarch64-guest-dies-at-the-kernels-entry-on-cis-firmware.md +++ /dev/null @@ -1,37 +0,0 @@ ---- -status: open -kind: defect -opened: 2026-10-01 ---- - -# Every AArch64 guest dies at the kernel's entry on CI's firmware - -All sixteen `virt_*` tests red in CI. Each boot's console ends the same way: -`Loader log: the kernel handoff begins`, then the firmware's -`Synchronous Exception at 0x00000000BC33EB20`. The PC differs per kernel build, -but always falls inside the kernel image the loader placed at `0xbc200000`. That -happens at EL1 entry (`Profile::Virt`) and at EL2 entry (`VirtEl2`), on one CPU -and on eight. The kernel prints nothing first, so the tests time out waiting for -their first marker. - -**Evidence**, identical in two runs on toolchain -`toolchain-linux-x86_64-48dd24f826263d6c`: -- Main's nightly `guest` lane at `06788146b`, run 36843762360, job 110374194368. -- PR #671's `guest` check, on its merge onto `59052827f`, run 36863809437, job - 110375742604. - -Both ran QEMU 11.1.1 under TCG `-cpu max` on an AMD EPYC 9V45 with 4 cores. The -firmware was Debian's `AAVMF_CODE.no-secboot.fd`, "version 2026.05-2". Both -logged `test result: FAILED. 4 passed, 17 failed`. The other red is -`issues/kernel/the-nested-nmi-report-interleaves-with-another-cpus-console-line.md`. - -The same sixteen tests pass on the dev host, in a whole-suite run of #670's -branch (`test result: ok. 21 passed`). That host ran QEMU 11.1.1 from -Homebrew, under the same TCG `-cpu max` for `VirtEl2`. Two parts of the -instrument differ: -- The firmware: the dev host runs QEMU's bundled `edk2-stable202408-prebuilt.qemu.org`. -- The toolchain: the dev host builds its own, and CI installs the published release. - -`.github/qemu-version` pins neither of the two. - -**Exit:** the sixteen `virt_*` tests are green in CI's `guest` check. diff --git a/issues/kernel/kernel-forward-copies-and-fills-are-one-rep-movsb-or-stosb-on-every-cpu.md b/issues/kernel/kernel-forward-copies-and-fills-are-one-rep-movsb-or-stosb-on-every-cpu.md index ae7e1d78368..0fc1863ff7b 100644 --- a/issues/kernel/kernel-forward-copies-and-fills-are-one-rep-movsb-or-stosb-on-every-cpu.md +++ b/issues/kernel/kernel-forward-copies-and-fills-are-one-rep-movsb-or-stosb-on-every-cpu.md @@ -15,11 +15,7 @@ either way. The kernel links the sysroot's `compiler_builtins`, which no kernel build flag recompiles: the feature is `rustflags = ["-Ctarget-feature=+ermsb"]` under `[target.x86_64-unknown-none]` in the std build's `bootstrap.toml` (`std_config`, `src/sysroot.rs:522`; bootstrap's `core/config/toml/target.rs:41`), -and it moves the sysroot key (`RECIPE`, `src/sysroot.rs:65`), which reaches -CI's installed toolchain only once the release tag's `trees()` -(`src/release.rs:25-31`) hashes `src/sysroot.rs`, the work of -`issues/build/the-release-tag-hashes-none-of-the-build-system-that-builds-the-toolchain.md`, -which lands first. One path on every +and it moves the sysroot key (`RECIPE`, `src/sysroot.rs:65`). One path on every CPU: `rep movsb` is correct without ERMS, CPUID.(7,0):EBX bit 9, and no CPU is refused. Zen 2 lacks ERMS (a Ryzen 9 PRO 3900, family 0x17, reads EBX 0x219C91A9: InstLatx64 ddff8a92, diff --git a/issues/kernel/the-kernel-is-at-least-as-secure-as-linux-on-every-machine-toyos-supports.md b/issues/kernel/the-kernel-is-at-least-as-secure-as-linux-on-every-machine-toyos-supports.md index 46ee0798305..0e1168d8bf5 100644 --- a/issues/kernel/the-kernel-is-at-least-as-secure-as-linux-on-every-machine-toyos-supports.md +++ b/issues/kernel/the-kernel-is-at-least-as-secure-as-linux-on-every-machine-toyos-supports.md @@ -8,8 +8,8 @@ opened: 2026-09-29 Parity with Linux at `Ubuntu-6.8.0-142.142`, pinned by the first issue below, is the floor on every CPU ToyOS supports, and the kernel also takes every -security feature such a CPU offers. The proving machines are the T14 and the -nightly's AMD EPYC KVM guests; the PR gate's TCG model proves wiring only. A +security feature such a CPU offers. The proving machines are the T14 and +AMD EPYC KVM guests; the PR gate's TCG model proves wiring only. A probe is a `boot-actuators` arm or a `test-actuators` `SYS_DEBUG` action. **Exit**: every issue below is closed, in the order listed. diff --git a/issues/kernel/the-nested-nmi-report-interleaves-with-another-cpus-console-line.md b/issues/kernel/the-nested-nmi-report-interleaves-with-another-cpus-console-line.md deleted file mode 100644 index b5b68dcdbe4..00000000000 --- a/issues/kernel/the-nested-nmi-report-interleaves-with-another-cpus-console-line.md +++ /dev/null @@ -1,28 +0,0 @@ ---- -status: open -kind: defect -opened: 2026-10-01 ---- - -# The nested-NMI report interleaves with another CPU's console line - -`nested_nmi` (`kernel/src/arch/x86_64/idt/nmi.rs`) writes its report through -`serial::panic_raw`, which takes no lock. When cpu1 is writing its own record at -the same moment, the two lines interleave byte by byte on the 16550. The cpu1 -line was `[kernel 0.385 cpu1] CPU 1: joining scheduler`, and the 16550 carried: - - [[kenrnmel i0.38]5 cpNu1E] CSPUT 1E: Djo inNiMngI s choednule r - -`NESTED NMI` is never whole on the console, so `nested_nmi_is_loud` times out -waiting for it, and the machine halts with its report unreadable. - -**Evidence:** red under KVM in two runs, with byte-identical interleaving: -- Main's nightly `guest` lane at `06788146b`, run 36843762360, job 110374194368. -- PR #671's `guest` check, on its merge onto `59052827f`, run 36863809437, job - 110375742604. - -It is green under TCG: on the dev host in a whole-suite run of #670's branch, -and on a runner in main's nightly `tcg` lane at `06788146b` (job 110374194382). - -**Exit:** `nested_nmi_is_loud` is green in CI's KVM `guest` check, and a report -written while another CPU is writing a record reads whole. diff --git a/issues/kernel/the-speculation-decision-does-not-model-an-affected-cpus-l1tf-line.md b/issues/kernel/the-speculation-decision-does-not-model-an-affected-cpus-l1tf-line.md index 1fa0f6fc125..596202191f5 100644 --- a/issues/kernel/the-speculation-decision-does-not-model-an-affected-cpus-l1tf-line.md +++ b/issues/kernel/the-speculation-decision-does-not-model-an-affected-cpus-l1tf-line.md @@ -25,7 +25,7 @@ second has no counterpart in a kernel that runs no VM, so parity needs a ruling on which `VMX:` state ToyOS is held to. No machine ToyOS is tested on reaches it: the T14 has `RDCL_NO`, and the TCG -model and the nightly's KVM runners are AMD. +model and the KVM runners are AMD. **Exit**: the line is decided from the memory map and CPUID and the owner's `VMX:` ruling, and held by a fixture captured under the pinned Linux on such a diff --git a/src/ci.rs b/src/ci.rs index 88cdcd0449d..c79b0b15cbf 100644 --- a/src/ci.rs +++ b/src/ci.rs @@ -2,15 +2,6 @@ //! checkout, a cache and one line, and this host runs the same job to the same //! verdict. //! -//! `.github/workflows/` is three files. `ci.yml` runs on a pull request and in -//! the merge queue: [`Job::Host`] as `host`, [`Job::Toolchain`] as -//! `toolchain`, and [`Job::Guest`] as `guest`, its x86-64 guests on KVM. Every -//! test that boots no guest is in [`Job::Host`] and every guest test in -//! [`Job::Guest`], so a merge is gated on all of them. `nightly.yml` runs the -//! guest suite again under TCG, the toolchain again to move the SDK alias on -//! main, `host` again to write the cache the merge queue restores, and -//! portability. `publish.yml` puts a landing's crates on crates.io. -//! //! A host job runs every step and reds if any failed; a guest job stops at the //! first failure among the instrument, the toolchain and the suite, because //! what follows a wrong instrument or a missing toolchain measures nothing — @@ -36,15 +27,19 @@ const USAGE: &str = "cargo run -- --ci , where is one of: host every host test: the build system, the harness's own checks, the host workspace, the licences of what ships, clippy, the model controls, userland and the SDK (ci.yml, nightly) - toolchain publish this tree's toolchain if nobody has (ci.yml, nightly) - guest the guest suite (ci.yml, nightly) + toolchain whether a build of this tree's toolchain answers for it + bootstrap build this tree's toolchain, unless a build answers for it + guest the guest suite, on the build that answers for this tree + release put main's build up as its toolchain release: main's publisher alone publish put main's SDK crates on crates.io (publish.yml)"; #[derive(Debug, PartialEq, Eq)] enum Job { Host, Toolchain, + Bootstrap, Guest, + Release, Publish, } @@ -52,7 +47,9 @@ fn parse(words: &[String]) -> Result { let job = match words.first().map(String::as_str) { Some("host") => Job::Host, Some("toolchain") => Job::Toolchain, + Some("bootstrap") => Job::Bootstrap, Some("guest") => Job::Guest, + Some("release") => Job::Release, Some("publish") => Job::Publish, Some(other) => return Err(format!("no CI job is called {other:?}")), None => return Err("which job?".to_string()), @@ -70,8 +67,10 @@ pub fn dispatch(root: &Path, args: &[String]) { }); let steps = match &job { Job::Host => host(root), - Job::Toolchain => vec![step("the toolchain release", || release::ensure_published(root))], + Job::Toolchain => vec![step("a build of this tree's toolchain", || release::toolchain(root))], + Job::Bootstrap => vec![step("this tree's toolchain", || release::bootstrap(root))], Job::Guest => guest(root, &suite_args(&["--jobs", "1"])), + Job::Release => vec![step("main's toolchain release", || release::release(root))], Job::Publish => vec![step("the SDK crates on crates.io", || publish(root))], }; let failed: Vec<&Step> = steps.iter().filter(|s| s.verdict.is_err()).collect(); @@ -857,6 +856,7 @@ mod tests { fn a_job_is_named_and_takes_nothing_after_it() { assert_eq!(parse(&words("host")), Ok(Job::Host)); assert_eq!(parse(&words("guest")), Ok(Job::Guest)); + assert_eq!(parse(&words("release")), Ok(Job::Release)); assert!(parse(&words("guest 3/12")).is_err()); assert!(parse(&words("tcg")).is_err()); assert!(parse(&words("host extra")).is_err()); @@ -948,38 +948,69 @@ mod tests { .collect() } - /// `host` and `guest` are the required checks. A skipped job reads as green - /// to one, so `guest` runs whatever `toolchain` concluded. + /// The value of a job's own `:` line. + fn field(job: &[&str], key: &str) -> Option { + let line = format!(" {key}: "); + job.iter().find_map(|l| l.strip_prefix(&line)).map(str::to_string) + } + + /// A skipped job reads as green to a required check, so `guest` runs on + /// every pull request and in the merge queue whatever `toolchain` + /// concluded: its condition is `host`'s and `!cancelled()`, and nothing + /// more. The nightly's `tcg` runs the same way. #[test] - fn the_required_checks_are_jobs_on_every_pull_request() { - let text = workflow("ci.yml"); - assert!(text.contains("\n pull_request:\n") && text.contains("\n merge_group:")); - assert!(!job(&text, "host").is_empty(), "ci.yml runs no job `host`"); - let guest = job(&text, "guest").join("\n"); - assert!(guest.contains("needs: toolchain") && guest.contains("!cancelled()"), "{guest}"); + fn the_guest_lanes_run_whatever_the_toolchain_concluded() { + let ci = workflow("ci.yml"); + assert!(ci.contains("\n pull_request:\n") && ci.contains("\n merge_group:")); + let host = field(&job(&ci, "host"), "if").expect("ci.yml's `host` has a condition"); + let guest = job(&ci, "guest"); + assert_eq!(field(&guest, "if"), Some(format!("${{{{ !cancelled() && ({host}) }}}}"))); + assert_eq!(field(&guest, "needs").as_deref(), Some("toolchain")); + assert_eq!(field(&guest, "uses").as_deref(), Some("./.github/workflows/guest.yml")); + assert_eq!(field(&job(&ci, "toolchain"), "uses").as_deref(), Some("./.github/workflows/toolchain.yml")); + let nightly = workflow("nightly.yml"); + let tcg = job(&nightly, "tcg"); + assert_eq!(field(&tcg, "if").as_deref(), Some("${{ !cancelled() }}")); + assert_eq!(field(&tcg, "needs").as_deref(), Some("toolchain")); + assert_eq!(field(&tcg, "uses").as_deref(), Some("./.github/workflows/guest.yml")); } - /// ci.yml's `guest` and the nightly's `tcg` boot one instrument and share - /// one cache: the image's digest pins QEMU and its firmware, and a restore - /// whose paths are not its writer's restores nothing, in silence. + /// No job a pull request, the merge queue or the nightly runs holds a token + /// that writes this repository: `ci.yml` and `nightly.yml` give their jobs + /// read alone, the workflows they call ask for nothing of their own, and + /// the one `contents: write` is `publish.yml`'s `release`, which main alone + /// triggers and which alone runs `--ci release`. #[test] - fn the_guest_lanes_share_an_instrument_and_a_cache() { - let lane = |file: &str, name: &str| { - let text = workflow(file); - let lines = job(&text, name); - let image = lines.iter().find_map(|l| l.trim_start().strip_prefix("image: ")); - let paths: Vec<&str> = lines - .iter() - .skip_while(|l| !l.trim_start().starts_with("path:")) - .skip(1) - .take_while(|l| !l.trim_start().starts_with("key:")) - .map(|l| l.trim()) - .collect(); - (image.map(str::to_string), paths.join("\n")) + fn only_mains_publisher_holds_a_write_token() { + let writes = |text: &str| -> Vec { + let grants = |l: &&str| l.trim_end().ends_with(": write") || l.contains("write-all"); + text.lines().filter(grants).map(|l| l.trim().to_string()).collect() }; - let (pr, nightly) = (lane("ci.yml", "guest"), lane("nightly.yml", "tcg")); - assert!(pr.0.as_deref().is_some_and(|i| i.contains("@sha256:")) && !pr.1.is_empty(), "{pr:?}"); - assert_eq!(pr, nightly); + for name in ["ci.yml", "nightly.yml"] { + let text = workflow(name); + assert!(text.contains("\npermissions:\n contents: read\n actions: read\n\n"), "{name}"); + assert!(writes(&text).is_empty() && !text.contains("--ci release"), "{name}: {:?}", writes(&text)); + } + for name in ["guest.yml", "toolchain.yml"] { + let text = workflow(name); + assert!(!text.contains("permissions:") && !text.contains("--ci release"), "{name}"); + } + let publish = workflow("publish.yml"); + assert!(publish.contains("\non:\n push:\n branches: [main]\n workflow_dispatch: {}\n"), "{publish}"); + assert_eq!(writes(&publish), ["id-token: write", "contents: write"]); + let release = job(&publish, "release"); + assert!(release.contains(&" contents: write"), "{release:?}"); + assert!(release.contains(&" run: cargo run -- --ci release"), "{release:?}"); + assert_eq!(publish.matches("--ci release").count(), 1); + } + + /// `toolchain.yml` uploads whole what `--ci bootstrap` leaves: a path that + /// missed it would upload nothing, and say nothing. + #[test] + fn the_toolchain_job_uploads_what_bootstrap_keeps() { + let text = workflow("toolchain.yml"); + let upload = format!(" path: {}/*.tar.zst\n archive: false\n", release::KEPT); + assert!(text.contains(&upload), "{text}"); } /// Every workflow's `pull_request:` trigger names `main` alone, and none @@ -1004,11 +1035,12 @@ mod tests { ); } } - assert_eq!(seen, 3, "ci.yml, nightly.yml and publish.yml"); + assert_eq!(seen, 5, "ci.yml, nightly.yml and publish.yml, and guest.yml and toolchain.yml"); } /// Exactly one job writes each cache, on the nightly, so what a pull request /// restores is one run's tree and never a race between two writers. + /// `guest.yml` saves only when its caller asks, and only the nightly asks. #[test] fn each_cache_has_one_writer() { let dir = repo_root().join(".github/workflows"); @@ -1017,6 +1049,7 @@ mod tests { let text = std::fs::read_to_string(entry.path()).expect("a readable workflow"); let name = entry.file_name().to_string_lossy().into_owned(); assert!(!text.contains("actions/cache@"), "{name}: the combined action saves too"); + assert!(name == "nightly.yml" || name == "guest.yml" || !text.contains("save-cache"), "{name}"); let lines: Vec<&str> = text.lines().collect(); for (at, line) in lines.iter().enumerate() { if line.contains("actions/cache/save@") { @@ -1024,16 +1057,20 @@ mod tests { .iter() .find_map(|l| l.trim_start().strip_prefix("key: ")) .expect("a save names its key"); - writers.push((name.clone(), key.split('$').next().unwrap_or("").to_string())); + let asked = lines[at - 1].trim() == "- if: inputs.save-cache"; + writers.push((name.clone(), key.split('$').next().unwrap_or("").to_string(), asked)); } } } assert!(!writers.is_empty(), "no job writes a cache, so every restore is cold"); writers.sort(); - let mut prefixes: Vec<&String> = writers.iter().map(|(_, p)| p).collect(); + let mut prefixes: Vec<&String> = writers.iter().map(|(_, p, _)| p).collect(); prefixes.dedup(); assert_eq!(prefixes.len(), writers.len(), "a cache with two writers: {writers:?}"); - assert!(writers.iter().all(|(f, _)| f == "nightly.yml"), "{writers:?}"); + assert!( + writers.iter().all(|(f, _, asked)| f == "nightly.yml" || (f == "guest.yml" && *asked)), + "{writers:?}" + ); } #[test] diff --git a/src/clang.rs b/src/clang.rs index 8473447f5e3..ac2f5f11629 100644 --- a/src/clang.rs +++ b/src/clang.rs @@ -26,9 +26,6 @@ use crate::arch::Arch; use crate::sysroot::clone_tree; use crate::toolchain::host_triple; -/// This file, which the release tag hashes. -pub(crate) const SOURCE: &str = file!(); - /// The LLVM every host compiler links, in every `bootstrap.toml` that builds /// one: built from `src/llvm-project` — the fork that knows the ToyOS target — /// with clang beside it, for the host and the two architectures ToyOS runs on. diff --git a/src/libcxx.rs b/src/libcxx.rs index 0612a136ee7..09c9769dc5d 100644 --- a/src/libcxx.rs +++ b/src/libcxx.rs @@ -15,9 +15,6 @@ use std::process::Command; use crate::arch::Arch; use crate::clang::CSysroot; -/// This file, which the release tag hashes. -pub(crate) const SOURCE: &str = file!(); - /// What of `src/llvm-project` the runtimes' build reads: the runtimes, the CMake /// modules they share with LLVM, and LLVM's libc, whose number parsing libc++ /// compiles in. diff --git a/src/n2.rs b/src/n2.rs index 8ce65fc3092..0fde6dd2728 100644 --- a/src/n2.rs +++ b/src/n2.rs @@ -5,9 +5,6 @@ use std::path::{Path, PathBuf}; use std::process::Command; -/// This file, which the release tag hashes. -pub(crate) const SOURCE: &str = file!(); - /// cargo's install of n2 but for its `--root`: without its default jemalloc, /// which is C. pub(crate) const N2: [&str; 7] = [ diff --git a/src/release.rs b/src/release.rs index eb567406976..4a132652550 100644 --- a/src/release.rs +++ b/src/release.rs @@ -1,43 +1,91 @@ -//! The toolchain release: the tag a tree's toolchain is published under, the -//! tarball it ships as, and how a runner installs one. +//! The toolchain a tree builds with: the tag that names it, the builds CI keeps +//! of it, and the release main publishes. //! -//! **The tag is the content hash of [`trees`].** A tree -//! whose toolchain somebody already built finds it published; a tree that moved -//! any of them asks for a tag nobody has, and `cargo run -- --ci toolchain` -//! builds it. Publishing is idempotent because the tag *is* the content. +//! **The tag is the content hash of [`trees`]**: the sources a toolchain is +//! built from and the build system's modules that build it ([`BUILDERS`]). A +//! tree that moved none of them names a toolchain somebody already built; a tree +//! that moved any names one nobody has. //! -//! The release is `x86_64-unknown-linux-gnu`'s and is built on a GitHub-hosted -//! `ubuntu-24.04`; any other host is refused rather than publishing a tarball +//! **CI installs a build only where it came from vouches for it, and only as the +//! bytes it was.** `toolchain.yml` bootstraps a tag no build answers for +//! ([`bootstrap`]) and keeps it as its run's artifact, uploaded whole; GitHub +//! records the SHA-256 of what was uploaded, and nothing rewrites an artifact. +//! [`install`] takes the newest build of its tree's tag that main's publisher +//! made and, failing that, the newest a run of a commit its tree vouches for made +//! ([`vouched`]); it refuses every other, and any download whose bytes hash to +//! anything but GitHub's digest. +//! +//! **Only main publishes** ([`release`]): `publish.yml` on main puts main's own +//! build up as the release a consumer outside CI installs, and moves the SDK +//! alias onto it. Run anywhere else, that job is refused before it reads +//! anything. +//! +//! A build is `x86_64-unknown-linux-gnu`'s and is made on a GitHub-hosted +//! `ubuntu-24.04`; any other host is refused rather than building a tarball //! nobody can install. A dev host never installs one: its build system //! bootstraps from `rust/` as always. +use std::collections::HashSet; use std::fs; +use std::io::Write; use std::path::{Path, PathBuf}; use std::process::{Command, Stdio}; +use serde_json::Value; use sha2::{Digest, Sha256}; use toyos_tmpdir::TempDir; use crate::toolchain::HOSTED_ARCH; -/// What the tag hashes, as `git rev-parse HEAD:` names them. The last is -/// this file. +/// The build system's modules that build and pack the toolchain: every module +/// `src/toolchain.rs` and this file name through `crate::`, and every module +/// those name, so the tag moves with how the toolchain is built as well as with +/// what it is built from. +pub(crate) const BUILDERS: [&str; 18] = [ + "src/arch.rs", + "src/buildlock.rs", + "src/clang.rs", + "src/compiler.rs", + "src/flags.rs", + "src/gitfixture.rs", + "src/identity.rs", + "src/keystore.rs", + "src/libc.rs", + "src/libcxx.rs", + "src/llvm.rs", + "src/n2.rs", + "src/release.rs", + "src/sdkversion.rs", + "src/sync.rs", + "src/sysroot.rs", + "src/toolchain.rs", + "src/worktree.rs", +]; + +/// What the tag hashes, as `git rev-parse HEAD:` names them. fn trees() -> Vec<&'static str> { std::iter::once("rust") .chain(crate::sysroot::SYSROOT_SOURCES) .chain(crate::sysroot::SYSROOT_MANIFESTS) - .chain([crate::clang::SOURCE, crate::libcxx::SOURCE, crate::n2::SOURCE, file!()]) + .chain(BUILDERS) .collect() } -/// The one asset a release carries. +/// The release's one asset, under the name a consumer's install fetches. const ASSET: &str = "toyos-toolchain.tar.zst"; -/// The triple the release's host half runs on. +/// Where [`bootstrap`] leaves the build `toolchain.yml` uploads. +pub(crate) const KEPT: &str = "target/toolchain"; + +/// Main's publisher: the one workflow whose builds every tree takes, and the one +/// [`release`] runs under. +const PUBLISHER: &str = ".github/workflows/publish.yml"; + +/// The triple a build's host half runs on. const HOST: &str = "x86_64-unknown-linux-gnu"; /// The oldest glibc a consumer needs: `ubuntu-24.04`'s. A build naming a newer -/// one is refused rather than published. +/// one is refused. const GLIBC_FLOOR: (u32, u32) = (2, 39); /// `toolchain-linux-x86_64-<16 hex>`: the first 16 hex digits of the SHA-256 of @@ -66,41 +114,199 @@ fn on_runner() -> bool { std::env::var("GITHUB_ACTIONS").is_ok_and(|v| v == "true") } -/// The API URL of `tag`'s asset, or `None` while it has none: `gh release -/// create` makes the release before it uploads, so a tag that exists is not yet -/// an installable toolchain. `curl` and not `gh`, because the guest containers -/// carry no `gh`. -fn asset_url(tag: &str) -> Result, String> { - let repo = std::env::var("GITHUB_REPOSITORY").unwrap_or_else(|_| "ToyOSOrg/ToyOS".into()); - let mut curl = Command::new("curl"); - curl.args(["-sSL", &format!("https://api.github.com/repos/{repo}/releases/tags/{tag}")]); - if let Ok(token) = std::env::var("GH_TOKEN") { - curl.args(["-H", &format!("Authorization: Bearer {token}")]); +fn repo() -> String { + std::env::var("GITHUB_REPOSITORY").unwrap_or_else(|_| "ToyOSOrg/ToyOS".into()) +} + +/// Whether this job is main's publisher — [`PUBLISHER`] on main, pushed or +/// dispatched — as the runner names its workflow and event; refused by name if +/// it is not. +fn publisher(workflow: Option<&str>, event: Option<&str>, repo: &str) -> Result<(), String> { + let mains = format!("{repo}/{PUBLISHER}@refs/heads/main"); + match (workflow, event) { + (Some(workflow), Some("push" | "workflow_dispatch")) if workflow == mains => Ok(()), + (workflow, event) => Err(format!( + "only {mains}, pushed or dispatched, publishes a toolchain, and this job is {} on {}", + workflow.unwrap_or("no workflow"), + event.unwrap_or("no event") + )), } - let out = curl.output().map_err(|e| format!("curl: {e}"))?; +} + +fn this_job_publishes() -> bool { + let var = |name| std::env::var(name).ok(); + publisher(var("GITHUB_WORKFLOW_REF").as_deref(), var("GITHUB_EVENT_NAME").as_deref(), &repo()).is_ok() +} + +/// GitHub's answer to `GET https://api.github.com/`, or `None` when there +/// is no such thing. +fn api(path: &str) -> Result, String> { + let token = std::env::var("GH_TOKEN").map_err(|_| "GH_TOKEN is unset".to_string())?; + let url = format!("https://api.github.com/{path}"); + let out = Command::new("curl") + .args(["-sSL", "--retry", "3", "--retry-all-errors", "-w", "\n%{http_code}"]) + .args(["-H", &format!("Authorization: Bearer {token}")]) + .args(["-H", "Accept: application/vnd.github+json", &url]) + .output() + .map_err(|e| format!("curl: {e}"))?; if !out.status.success() { - return Err(format!("curl asked for {tag} and failed: {}", out.status)); + return Err(format!("curl {url} exited {}: {}", out.status, String::from_utf8_lossy(&out.stderr).trim())); + } + let text = String::from_utf8_lossy(&out.stdout).into_owned(); + let (body, status) = text.rsplit_once('\n').unwrap_or(("", text.as_str())); + match status { + "200" => serde_json::from_str(body).map(Some).map_err(|e| format!("{url} answered no JSON: {e}")), + "404" => Ok(None), + status => Err(format!("{url} answered {status}: {body}")), } - let release: serde_json::Value = serde_json::from_slice(&out.stdout) - .map_err(|e| format!("GitHub's answer about {tag} is not JSON: {e}"))?; - Ok(named_asset(&release)) } -/// The `url` of [`ASSET`] in a release's JSON, if it carries one. -fn named_asset(release: &serde_json::Value) -> Option { - release["assets"] - .as_array()? +/// A toolchain build a run kept: its artifact, the digest GitHub recorded of its +/// bytes, and the run that uploaded it. +#[derive(Clone, Debug, PartialEq)] +struct Build { + artifact: u64, + /// `sha256:`. + digest: String, + run: u64, + /// The commit its run was for: the one pushed or queued, or a pull + /// request's head. + head: String, + /// Whether that run was main's publisher's. + mains: bool, +} + +impl Build { + fn provenance(&self) -> String { + let whose = if self.mains { "main's publisher" } else { "a commit this tree vouches for" }; + format!("artifact {} of run {}, {whose}, at {}", self.artifact, self.run, self.head) + } +} + +/// The unexpired artifacts in GitHub's `listing`, newest first, each with the +/// branch its run was on. An artifact with no digest to hold its bytes to is +/// none of them. +fn listed(listing: &Value) -> Vec<(Build, String)> { + let artifacts = listing["artifacts"].as_array().map_or(&[][..], Vec::as_slice); + artifacts .iter() - .find(|a| a["name"] == ASSET) - .and_then(|a| a["url"].as_str()) + .filter(|a| a["expired"] == false) + .filter_map(|a| { + let run = &a["workflow_run"]; + let build = Build { + artifact: a["id"].as_u64()?, + digest: a["digest"].as_str()?.to_string(), + run: run["id"].as_u64()?, + head: run["head_sha"].as_str()?.to_string(), + mains: false, + }; + Some((build, run["head_branch"].as_str()?.to_string())) + }) + .collect() +} + +/// Whether GitHub's `run` is main's publisher's: [`PUBLISHER`] on main, pushed +/// or dispatched, in `repo` and from it. +fn is_mains(run: &Value, repo: &str) -> bool { + run["path"] == PUBLISHER + && run["head_branch"] == "main" + && matches!(run["event"].as_str(), Some("push" | "workflow_dispatch")) + && run["repository"]["full_name"] == repo + && run["head_repository"]["full_name"] == repo +} + +/// Which of `builds`, newest first, a tree installs: the newest main's publisher +/// made, else — unless only main's will do — the newest a run of a commit in +/// `vouched` made. +fn choose<'a>(builds: &'a [Build], vouched: &HashSet, only_mains: bool) -> Option<&'a Build> { + builds.iter().find(|b| b.mains).or_else(|| { + if only_mains { + None + } else { + builds.iter().find(|b| vouched.contains(&b.head)) + } + }) +} + +/// The commits whose runs' builds this tree takes beside main's publisher's: +/// each commit on its first-parent chain, and the head each merge on that chain +/// took in — main's commits, each head main merged after its review, and a pull +/// request's own head — but no commit a branch passed on its way to the head +/// that was merged. +fn vouched(root: &Path) -> Result, String> { + Ok(merged_heads(&crate::sync::git(root, &["rev-list", "--first-parent", "--parents", "HEAD"])?)) +} + +/// [`vouched`] read off `git rev-list --first-parent --parents`: each line's +/// commit, and each parent it has but its first. +fn merged_heads(rev_list: &str) -> HashSet { + rev_list + .lines() + .flat_map(|line| { + let mut words = line.split_whitespace(); + let commit = words.next(); + words.next(); + commit.into_iter().chain(words) + }) .map(str::to_string) + .collect() +} + +/// The build of `tag` this tree installs ([`choose`]), if a run kept one. +fn find(root: &Path, tag: &str, only_mains: bool) -> Result, String> { + let repo = repo(); + let listing = api(&format!("repos/{repo}/actions/artifacts?name={tag}.tar.zst&per_page=100"))? + .ok_or_else(|| format!("{repo} lists no artifacts"))?; + let mut builds = Vec::new(); + for (mut build, branch) in listed(&listing) { + if branch == "main" { + let run = api(&format!("repos/{repo}/actions/runs/{}", build.run))?; + build.mains = run.is_some_and(|run| is_mains(&run, &repo)); + } + builds.push(build); + } + let vouched = if only_mains { HashSet::new() } else { vouched(root)? }; + Ok(choose(&builds, &vouched, only_mains).cloned()) +} + +/// `cargo run -- --ci toolchain`: whether a build answers for this tree's +/// toolchain, told to the job's next steps as `bootstrap` in `$GITHUB_OUTPUT`. +/// Main's publisher takes only its own. +pub fn toolchain(root: &Path) -> Result { + let output = std::env::var("GITHUB_OUTPUT") + .map_err(|_| "not a runner: a dev host builds its own toolchain with `cargo run`".to_string())?; + let tag = tag(root)?; + let found = find(root, &tag, this_job_publishes())?; + fs::OpenOptions::new() + .append(true) + .open(&output) + .and_then(|mut file| writeln!(file, "bootstrap={}", found.is_none())) + .map_err(|e| format!("{output}: {e}"))?; + Ok(match found { + Some(build) => format!("{tag}: {}", build.provenance()), + None => format!("{tag}: no build answers for it, so this job bootstraps one"), + }) } -fn sleep(seconds: u64) { - std::thread::sleep(std::time::Duration::from_secs(seconds)); +/// `cargo run -- --ci bootstrap`: this tree's toolchain built and left in +/// [`KEPT`] for its run to keep, unless a build already answers for it. +pub fn bootstrap(root: &Path) -> Result { + if !on_runner() { + return Err("not a runner: a dev host builds its own toolchain with `cargo run`".into()); + } + let tag = tag(root)?; + if let Some(build) = find(root, &tag, this_job_publishes())? { + return Ok(format!("{tag}: {}, so nothing is built", build.provenance())); + } + let kept = root.join(KEPT); + fs::create_dir_all(&kept).map_err(|e| format!("{}: {e}", kept.display()))?; + let tarball = kept.join(format!("{tag}.tar.zst")); + build(root, &tag, &tarball)?; + Ok(format!("{tag} built into {}", tarball.display())) } -/// Install this tree's published toolchain as rustup's `toyos`, on a runner. +/// Install the build of this tree's toolchain it takes ([`find`]) as rustup's +/// `toyos`, on a runner, once its bytes are the ones GitHub recorded. /// /// Off a runner this says so and does nothing: the build system owns the dev /// host's toolchain. @@ -108,55 +314,65 @@ pub fn install(root: &Path) -> Result { if !on_runner() { return Ok("not a runner: the build system uses this checkout's own toolchain".into()); } - if std::env::var("GH_TOKEN").is_err() { - return Err("GH_TOKEN is unset, and the release download is authenticated".into()); - } let tag = tag(root)?; - let mut url = None; - for _ in 0..10 { - url = asset_url(&tag)?; - if url.is_some() { - break; - } - println!("{tag} carries no {ASSET} yet; asking again in 15 s"); - sleep(15); - } - let url = url.ok_or_else(|| { + let build = find(root, &tag, false)?.ok_or_else(|| { format!( - "{tag} carries no {ASSET}, so there is nothing to install: the nightly's `build` \ - job is what publishes one" + "no build of {tag} answers for this tree: main's publisher kept none, and no run of a \ + commit this tree vouches for kept one. `toolchain.yml` bootstraps it ahead of this job" ) })?; - let token = std::env::var("GH_TOKEN").expect("checked above"); let staging = TempDir::new("toolchain-install"); let tarball = staging.join(ASSET); + fetch(&build, &tarball)?; let into = root.join("rust/build"); fs::create_dir_all(&into).map_err(|e| format!("{}: {e}", into.display()))?; - // The retry is on the transfer and the unpack together: a truncated body is - // a `zstd` failure, not a `curl` one. + unpack(&tarball, &into)?; + let stage2 = into.join(format!("{HOST}/stage2")); + run(Command::new("rustup").args(["toolchain", "link", "toyos"]).arg(&stage2))?; + run(Command::new(stage2.join("bin/rustc")).arg("-vV"))?; + Ok(format!("installed {tag} as `toyos`: {}", build.provenance())) +} + +/// `build`'s bytes at `to`, held to the digest GitHub recorded of them. Three +/// transfers at most: a body cut short is a wrong digest, not a curl failure. +fn fetch(build: &Build, to: &Path) -> Result<(), String> { let mut last = String::new(); for attempt in 1..=3 { - let fetched = Command::new("curl") - .args(["-sSL", "--retry", "3", "--retry-all-errors", "--retry-delay", "5"]) - .args(["-H", &format!("Authorization: Bearer {token}")]) - .args(["-H", "Accept: application/octet-stream", &url, "-o"]) - .arg(&tarball) - .status() - .map_err(|e| format!("curl: {e}"))?; - match fetched.success().then(|| unpack(&tarball, &into)) { - Some(Ok(())) => { - let stage2 = into.join(format!("{HOST}/stage2")); - run(Command::new("rustup").args(["toolchain", "link", "toyos"]).arg(&stage2))?; - run(Command::new(stage2.join("bin/rustc")).arg("-vV"))?; - return Ok(format!("installed {tag} as `toyos`")); - } - Some(Err(e)) => last = e, - None => last = format!("curl exited {fetched}"), + match download(build.artifact, to).and_then(|()| verify(to, &build.digest)) { + Ok(()) => return Ok(()), + Err(why) => last = why, } - println!("toolchain download attempt {attempt} failed: {last}"); - sleep(10); + println!("toolchain download attempt {attempt}: {last}"); + } + Err(format!("artifact {} did not arrive as GitHub recorded it, in three attempts: {last}", build.artifact)) +} + +/// `artifact`'s bytes into `to`. GitHub answers with a redirect to storage that +/// carries its own credential, and curl sends the token to no other host. +fn download(artifact: u64, to: &Path) -> Result<(), String> { + let token = std::env::var("GH_TOKEN").map_err(|_| "GH_TOKEN is unset".to_string())?; + let url = format!("https://api.github.com/repos/{}/actions/artifacts/{artifact}/zip", repo()); + let status = Command::new("curl") + .args(["-sSfL", "--retry", "3", "--retry-all-errors", "-o"]) + .arg(to) + .args(["-H", &format!("Authorization: Bearer {token}"), &url]) + .status() + .map_err(|e| format!("curl: {e}"))?; + status.success().then_some(()).ok_or_else(|| format!("curl {url} exited {status}")) +} + +/// Whether `path` hashes to `digest`, GitHub's `sha256:`. +fn verify(path: &Path, digest: &str) -> Result<(), String> { + let want = digest.strip_prefix("sha256:").ok_or_else(|| format!("{digest:?} is not a SHA-256 digest"))?; + let mut file = fs::File::open(path).map_err(|e| format!("{}: {e}", path.display()))?; + let mut hasher = Sha256::new(); + std::io::copy(&mut file, &mut hasher).map_err(|e| format!("{}: {e}", path.display()))?; + let got: String = hasher.finalize().iter().map(|b| format!("{b:02x}")).collect(); + if got == want { + Ok(()) + } else { + Err(format!("{} hashes to {got}, and GitHub recorded {want}", path.display())) } - Err(format!("the toolchain did not download and unpack in three attempts: {last}")) } /// `zstd -dc | tar -C -x`. @@ -183,48 +399,70 @@ fn run(cmd: &mut Command) -> Result<(), String> { status.success().then_some(()).ok_or_else(|| format!("{cmd:?} exited {status}")) } -/// Whether `gh` says `tag` carries [`ASSET`]. -fn published(root: &Path, tag: &str) -> bool { - Command::new("gh") - .args(["release", "view", tag, "--json", "assets", "--jq", ".assets[].name"]) - .current_dir(root) - .output() - .is_ok_and(|o| String::from_utf8_lossy(&o.stdout).lines().any(|l| l == ASSET)) +/// `cargo run -- --ci release`: main's own build of its toolchain put up as the +/// release a consumer outside CI installs, and the `sdk-` alias moved +/// onto it. Refused before anything is read unless this job is main's +/// publisher. +pub fn release(root: &Path) -> Result { + let var = |name| std::env::var(name).ok(); + publisher(var("GITHUB_WORKFLOW_REF").as_deref(), var("GITHUB_EVENT_NAME").as_deref(), &repo())?; + let tag = tag(root)?; + let build = find(root, &tag, true)? + .ok_or_else(|| format!("main's publisher kept no build of {tag}: this run's `toolchain` job makes it"))?; + let tmp = TempDir::new("toolchain-release"); + let tarball = tmp.join(ASSET); + fetch(&build, &tarball)?; + let manifest = manifest(root, &tag, &build.head)?; + fs::write(tmp.join("notes.md"), notes(root, &tag, &manifest)?).map_err(|e| e.to_string())?; + let put = put_up(root, &tag, &build.digest, &tarball, &tmp.join("notes.md"))?; + Ok(format!("{put}; {}", alias(root, &manifest, &tmp)?)) } -/// `cargo run -- --ci toolchain`: make sure this tree's toolchain is published, -/// building it if nobody has; on `main`, also move the `sdk-` alias a -/// consumer pins onto it. -pub fn ensure_published(root: &Path) -> Result { - let tag = tag(root)?; - println!("this tree's toolchain: {tag}"); +/// `tag`'s release, made to carry exactly the bytes `digest` names: created if +/// there is none, its asset replaced if another writer's is there, and then held +/// to the digest GitHub records of what it carries. +fn put_up(root: &Path, tag: &str, digest: &str, tarball: &Path, notes: &Path) -> Result { + let path = format!("repos/{}/releases/tags/{tag}", repo()); + let carried = |release: Option| -> Option { + let assets = release?["assets"].as_array()?.clone(); + assets.iter().find(|a| a["name"] == ASSET)?["digest"].as_str().map(str::to_string) + }; + let said = match api(&path)? { + None => { + run(Command::new("gh") + .args(["release", "create", tag, "--title", tag, "--notes-file"]) + .arg(notes) + .arg(tarball) + .current_dir(root))?; + "published" + } + Some(release) if carried(Some(release.clone())).as_deref() == Some(digest) => { + return Ok(format!("{tag} already carries main's build")); + } + Some(_) => { + run(Command::new("gh").args(["release", "upload", tag, "--clobber"]).arg(tarball).current_dir(root))?; + "had another writer's asset, now main's build" + } + }; + let now = carried(api(&path)?); + if now.as_deref() != Some(digest) { + return Err(format!("{tag} carries {now:?} after the upload, and main's build is {digest}")); + } + Ok(format!("{tag} {said}")) +} + +/// Bootstrap this tree's toolchain, hold it to the glibc floor, and pack it into +/// `tarball`. +fn build(root: &Path, tag: &str, tarball: &Path) -> Result<(), String> { if !(cfg!(target_os = "linux") && crate::arch::Arch::HOST == Some(crate::arch::Arch::X86_64)) { return Err(format!( - "the release is {HOST}'s and this host is not one; a tarball built here would \ - install nowhere" + "a build is {HOST}'s and this host is not one; a tarball built here would install \ + nowhere" )); } - let tmp = TempDir::new("toolchain-publish"); - let manifest = manifest(root, &tag)?; - let notes = notes(root, &tag, &manifest)?; - fs::write(tmp.join("TOOLCHAIN"), &manifest).map_err(|e| e.to_string())?; - fs::write(tmp.join("notes.md"), ¬es).map_err(|e| e.to_string())?; + let toyos = std::env::var("GITHUB_SHA").or_else(|_| crate::sync::git(root, &["rev-parse", "HEAD"]))?; + let manifest = manifest(root, tag, &toyos)?; println!("{manifest}"); - - let mut said = if published(root, &tag) { - format!("{tag} is already published") - } else { - build(root, &tag, &tmp)?; - format!("{tag} built and published") - }; - if std::env::var("GITHUB_REF").is_ok_and(|r| r == "refs/heads/main") { - said.push_str(&format!("; {}", alias(root, &manifest, &tmp)?)); - } - Ok(said) -} - -/// Bootstrap, check the glibc floor, package, publish, and wait for the asset. -fn build(root: &Path, tag: &str, tmp: &Path) -> Result<(), String> { run(Command::new("git").args(["submodule", "update", "--init", "rust"]).current_dir(root))?; // Bootstrap takes `HEAD^1` as the upstream commit whose artifacts to fetch // when it sees GitHub Actions; in this fork that is our own merge, which @@ -247,17 +485,16 @@ fn build(root: &Path, tag: &str, tmp: &Path) -> Result<(), String> { let need = shipped_glibc(&stage2)?; if need > GLIBC_FLOOR { return Err(format!( - "the host half needs GLIBC_{}.{} and the release states {}.{}: build it on the \ - oldest supported glibc, or move GLIBC_FLOOR deliberately", + "the host half needs GLIBC_{}.{} and a build states {}.{}: build it on the oldest \ + supported glibc, or move GLIBC_FLOOR deliberately", need.0, need.1, GLIBC_FLOOR.0, GLIBC_FLOOR.1 )); } - fs::copy(tmp.join("TOOLCHAIN"), build.join("TOOLCHAIN")).map_err(|e| e.to_string())?; + fs::write(build.join("TOOLCHAIN"), &manifest).map_err(|e| e.to_string())?; // `lib/rustlib/` and the sysroot's `bin/cargo` are links into this // runner's own toolchain; `Owner::Installed` recreates both. GNU tar's // `--transform` renames the sysroot to the path an installer links. - let tarball = tmp.join(ASSET); let mut tar = Command::new("tar") .arg("-C") .arg(&build) @@ -270,37 +507,19 @@ fn build(root: &Path, tag: &str, tmp: &Path) -> Result<(), String> { .spawn() .map_err(|e| format!("tar: {e}"))?; let stream = tar.stdout.take().expect("piped"); - let zstd = Command::new("zstd").args(["-T0", "-3", "-f", "-o"]).arg(&tarball).stdin(stream).status(); + let zstd = Command::new("zstd").args(["-T0", "-3", "-f", "-o"]).arg(tarball).stdin(stream).status(); let tar = tar.wait().map_err(|e| format!("tar: {e}"))?; let zstd = zstd.map_err(|e| format!("zstd: {e}"))?; if !(tar.success() && zstd.success()) { return Err(format!("packaging: tar exited {tar}, zstd {zstd}")); } - - let created = Command::new("gh") - .args(["release", "create", tag, "--title", tag, "--notes-file"]) - .arg(tmp.join("notes.md")) - .arg(&tarball) - .current_dir(root) - .status() - .map_err(|e| format!("gh: {e}"))?; - if !created.success() { - println!("`gh release create {tag}` refused; another run may have published it first"); - } - for _ in 0..20 { - if published(root, tag) { - return Ok(()); - } - println!("{tag} carries no {ASSET} yet; waiting"); - sleep(15); - } - Err(format!("{tag} carries no {ASSET}, so nothing can install this toolchain")) + Ok(()) } /// Every `GLIBC_x.y` the shipped host binaries and libraries name, as the /// newest: `rustc` and its libraries, and the `rust-lld`, clang and LLVM tools /// beside them. A byte scan: it can only over-report, so its failure is a -/// refused publish. +/// refused build. fn shipped_glibc(stage2: &Path) -> Result<(u32, u32), String> { let mut files: Vec = Vec::new(); let tools = stage2.join(format!("lib/rustlib/{HOST}/bin")); @@ -348,14 +567,13 @@ fn glibc_named(bytes: &[u8]) -> (u32, u32) { newest } -/// `TOOLCHAIN`: the pin a consumer writes down, and what it gets. Inside the -/// tarball, and the alias release's own asset. -fn manifest(root: &Path, tag: &str) -> Result { - let head = |rev: &str| crate::sync::git(root, &["rev-parse", rev]); - let toyos = std::env::var("GITHUB_SHA").or_else(|_| head("HEAD"))?; +/// `TOOLCHAIN`: the pin a consumer writes down, and what it gets, `toyos` being +/// the commit that built it. Inside the tarball, and the alias release's own +/// asset. +fn manifest(root: &Path, tag: &str, toyos: &str) -> Result { Ok(format!( "toolchain {tag}\ntoyos {toyos}\nrust {}\nhost {HOST}\nglibc {}.{}\n", - head("HEAD:rust")?, + crate::sync::git(root, &["rev-parse", "HEAD:rust"])?, GLIBC_FLOOR.0, GLIBC_FLOOR.1 )) @@ -363,8 +581,7 @@ fn manifest(root: &Path, tag: &str) -> Result { /// The release notes: how to install it, what glibc it needs. fn notes(root: &Path, tag: &str, manifest: &str) -> Result { - let repo = std::env::var("GITHUB_REPOSITORY").unwrap_or_else(|_| "ToyOSOrg/ToyOS".into()); - let url = format!("https://github.com/{repo}/releases/download/{tag}/{ASSET}"); + let url = format!("https://github.com/{}/releases/download/{tag}/{ASSET}", repo()); let (major, minor) = GLIBC_FLOOR; let userland = fs::read_to_string(root.join("userland/Cargo.toml")).map_err(|e| e.to_string())?; let rwh = userland @@ -438,11 +655,12 @@ fn alias(root: &Path, manifest: &str, tmp: &Path) -> Result { #[cfg(test)] mod tests { use super::*; + use std::collections::BTreeSet; /// The packaging is one of the trees its own tag hashes. #[test] fn the_tag_hashes_this_file() { - assert_eq!(trees().last(), Some(&file!())); + assert!(trees().contains(&file!())); for tree in trees() { assert!( Path::new(env!("CARGO_MANIFEST_DIR")).join(tree).exists(), @@ -451,7 +669,55 @@ mod tests { } } - fn git(dir: &Path, args: &[&str]) { + /// Every module `text` names as `crate::`, alone or in a + /// `crate::{…}` group. + fn crate_modules(text: &str) -> Vec { + let ident = |name: &str| -> String { + name.trim().chars().take_while(|c| c.is_ascii_alphanumeric() || *c == '_').collect() + }; + let mut found = Vec::new(); + for (at, _) in text.match_indices("crate::") { + let rest = &text[at + "crate::".len()..]; + match rest.strip_prefix('{') { + Some(group) => found.extend(group.split('}').next().unwrap_or("").split(',').map(ident)), + None => found.push(ident(rest)), + } + } + found.retain(|name| !name.is_empty()); + found + } + + /// [`BUILDERS`] is every module `src/toolchain.rs` and this file reach through + /// `crate::`: a module the toolchain's build starts calling is one more the + /// tag hashes, and this is what says so. + #[test] + fn the_tag_hashes_every_module_that_builds_the_toolchain() { + let here = Path::new(env!("CARGO_MANIFEST_DIR")); + let mut reached = BTreeSet::new(); + let mut todo = vec!["src/toolchain.rs".to_string(), file!().to_string()]; + while let Some(file) = todo.pop() { + if !reached.insert(file.clone()) { + continue; + } + let text = fs::read_to_string(here.join(&file)).unwrap_or_else(|e| panic!("{file}: {e}")); + for module in crate_modules(&text) { + let path = format!("src/{module}.rs"); + if here.join(&path).is_file() { + todo.push(path); + } + } + } + let declared: BTreeSet = BUILDERS.iter().map(|s| s.to_string()).collect(); + assert_eq!(reached, declared); + } + + #[test] + fn a_group_import_names_each_of_its_modules() { + let text = "use crate::{flags, release::tag, sync};\nlet x = crate::arch::Arch::HOST;"; + assert_eq!(crate_modules(text), ["flags", "release", "sync", "arch"]); + } + + fn git(dir: &Path, args: &[&str]) -> String { let out = Command::new("git") .args(["-c", "commit.gpgsign=false", "-c", "user.email=t@t", "-c", "user.name=t"]) .args(["-c", "init.defaultBranch=main"]) @@ -461,12 +727,13 @@ mod tests { .output() .expect("run git"); assert!(out.status.success(), "git {args:?}: {}", String::from_utf8_lossy(&out.stderr)); + String::from_utf8_lossy(&out.stdout).trim().to_string() } - /// A commit to the C and C++ toolchain's declarations or headers, or to the - /// n2 it is built under, moves the tag. + /// A commit to the C and C++ toolchain's declarations or headers, to the n2 + /// it is built under, or to any module that builds it moves the tag. #[test] - fn the_tag_moves_with_the_c_toolchain() { + fn the_tag_moves_with_what_the_toolchain_is_built_from_and_by() { let repo = TempDir::new("release-tag"); let here = Path::new(env!("CARGO_MANIFEST_DIR")); let write = |path: &str, text: &str| { @@ -477,7 +744,9 @@ mod tests { git(&repo, &["init", "-q"]); for tree in trees() { match tree { - "rust" => git(&repo, &["update-index", "--add", "--cacheinfo", "160000,1111111111111111111111111111111111111111,rust"]), + "rust" => { + git(&repo, &["update-index", "--add", "--cacheinfo", "160000,1111111111111111111111111111111111111111,rust"]); + } file if here.join(file).is_file() => write(file, &fs::read_to_string(here.join(file)).unwrap()), dir => write(&format!("{dir}/placeholder"), "x"), } @@ -487,33 +756,35 @@ mod tests { git(&repo, &["commit", "-qm", "the tree"]); let mut before = tag(&repo).unwrap(); - let clang = fs::read_to_string(here.join(crate::clang::SOURCE)).unwrap(); + let clang = fs::read_to_string(here.join("src/clang.rs")).unwrap(); let tools = r#"const TOOLS: [&str; 3] = ["llvm-ar", "clang", "ld.lld"];"#; let objdump = r#"const TOOLS: [&str; 4] = ["llvm-ar", "clang", "ld.lld", "llvm-objdump"];"#; let targets = r#"targets = \"AArch64;X86\""#; let riscv = r#"targets = \"AArch64;RISCV;X86\""#; assert!(clang.contains(tools) && clang.contains(targets), "src/clang.rs no longer declares what this mutates"); let with_objdump = clang.replace(tools, objdump); - let cxx = fs::read_to_string(here.join(crate::libcxx::SOURCE)).unwrap(); + let cxx = fs::read_to_string(here.join("src/libcxx.rs")).unwrap(); let (no_fs, fs_on) = (r#"("LIBCXX_ENABLE_FILESYSTEM", "OFF")"#, r#"("LIBCXX_ENABLE_FILESYSTEM", "ON")"#); assert!(cxx.contains(no_fs), "src/libcxx.rs no longer declares what this mutates"); - let n2 = fs::read_to_string(here.join(crate::n2::SOURCE)).unwrap(); + let n2 = fs::read_to_string(here.join("src/n2.rs")).unwrap(); let pin = crate::n2::N2[crate::n2::N2.len() - 1]; assert!(n2.contains(pin), "src/n2.rs no longer declares what this mutates"); - let mutations = [ - (crate::clang::SOURCE, with_objdump.clone()), - (crate::clang::SOURCE, with_objdump.replace(targets, riscv)), - (crate::libcxx::SOURCE, cxx.replace(no_fs, fs_on)), - (crate::n2::SOURCE, n2.replace(pin, &"0".repeat(pin.len()))), - ("userland/libc/include/placeholder", "y".to_string()), - ]; - for (path, text) in mutations { + let mut moves = |path: &str, text: String| { write(path, &text); git(&repo, &["add", "-A"]); git(&repo, &["commit", "-qm", "a mutation"]); let after = tag(&repo).unwrap(); assert_ne!(after, before, "a commit to {path} kept the tag"); before = after; + }; + moves("src/clang.rs", with_objdump.clone()); + moves("src/clang.rs", with_objdump.replace(targets, riscv)); + moves("src/libcxx.rs", cxx.replace(no_fs, fs_on)); + moves("src/n2.rs", n2.replace(pin, &"0".repeat(pin.len()))); + moves("userland/libc/include/placeholder", "y".to_string()); + for builder in BUILDERS { + let text = fs::read_to_string(repo.join(builder)).unwrap(); + moves(builder, format!("{text}\nconst MOVED: () = ();\n")); } } @@ -538,16 +809,130 @@ mod tests { assert_eq!(glibc_named(b"no version here"), (0, 0)); } + const REPO: &str = "ToyOSOrg/ToyOS"; + + /// The negative control on the publisher: a pull request's job, the merge + /// queue's, the nightly's, and main's publisher dispatched on a branch or + /// run by any other event are each refused, by name. + #[test] + fn only_mains_publisher_publishes() { + let mains = format!("{REPO}/.github/workflows/publish.yml@refs/heads/main"); + assert!(publisher(Some(&mains), Some("push"), REPO).is_ok()); + assert!(publisher(Some(&mains), Some("workflow_dispatch"), REPO).is_ok()); + let refused = [ + (format!("{REPO}/.github/workflows/ci.yml@refs/pull/671/merge"), "pull_request"), + (format!("{REPO}/.github/workflows/ci.yml@refs/heads/gh-readonly-queue/main/pr-671-59052827f"), "merge_group"), + (format!("{REPO}/.github/workflows/nightly.yml@refs/heads/main"), "schedule"), + (format!("{REPO}/.github/workflows/publish.yml@refs/heads/wt/toyos-guestci"), "workflow_dispatch"), + (mains.clone(), "pull_request_target"), + ("Fork/ToyOS/.github/workflows/publish.yml@refs/heads/main".to_string(), "push"), + ]; + for (workflow, event) in refused { + let why = publisher(Some(&workflow), Some(event), REPO).expect_err(&workflow); + assert!(why.contains(&workflow) && why.contains(event), "{why}"); + } + assert!(publisher(None, None, REPO).is_err()); + } + + fn run_json(path: &str, branch: &str, event: &str, head_repo: &str) -> Value { + serde_json::json!({ + "path": path, "head_branch": branch, "event": event, + "repository": { "full_name": REPO }, "head_repository": { "full_name": head_repo }, + }) + } + + #[test] + fn mains_builds_are_publish_yml_on_main_and_nothing_else() { + let publish = ".github/workflows/publish.yml"; + assert!(is_mains(&run_json(publish, "main", "push", REPO), REPO)); + assert!(is_mains(&run_json(publish, "main", "workflow_dispatch", REPO), REPO)); + assert!(!is_mains(&run_json(".github/workflows/nightly.yml", "main", "schedule", REPO), REPO)); + assert!(!is_mains(&run_json(".github/workflows/ci.yml", "main", "pull_request", REPO), REPO)); + assert!(!is_mains(&run_json(publish, "wt/toyos-guestci", "workflow_dispatch", REPO), REPO)); + assert!(!is_mains(&run_json(publish, "main", "pull_request", "Fork/ToyOS"), REPO)); + } + + fn artifact(id: u64, digest: Value, run: u64, head: &str, branch: &str, expired: bool) -> Value { + serde_json::json!({ + "id": id, "digest": digest, "expired": expired, + "workflow_run": { "id": run, "head_sha": head, "head_branch": branch }, + }) + } + + /// GitHub's list, read: an expired artifact and one with no digest are not + /// builds. + #[test] + fn a_build_is_an_unexpired_artifact_with_a_digest() { + let listing = serde_json::json!({ "artifacts": [ + artifact(1, "sha256:aa".into(), 10, "h1", "main", false), + artifact(3, "sha256:cc".into(), 12, "h3", "wt/y", true), + artifact(4, Value::Null, 13, "h4", "wt/z", false), + artifact(5, "sha256:ee".into(), 14, "h5", "wt/q", false), + ]}); + let got: Vec<(u64, String)> = listed(&listing).into_iter().map(|(b, branch)| (b.artifact, branch)).collect(); + assert_eq!(got, [(1, "main".to_string()), (5, "wt/q".to_string())]); + } + + fn kept(artifact: u64, head: &str, mains: bool) -> Build { + Build { artifact, digest: format!("sha256:{artifact}"), run: artifact, head: head.into(), mains } + } + + /// The negative control on what a tree installs: main's publisher's build + /// over any newer one, a build a vouched commit's run made after it, and + /// never one from a commit the tree does not vouch for; and main's publisher + /// takes nothing but its own. + #[test] + fn a_tree_installs_mains_build_else_one_its_history_vouches_for() { + let vouched: HashSet = ["landed".to_string(), "own".to_string()].into(); + let builds = [kept(4, "passed-through", false), kept(3, "own", false), kept(2, "main", true)]; + assert_eq!(choose(&builds, &vouched, false), Some(&builds[2])); + assert_eq!(choose(&builds, &vouched, true), Some(&builds[2])); + let unpublished = [kept(4, "passed-through", false), kept(3, "own", false), kept(1, "landed", false)]; + assert_eq!(choose(&unpublished, &vouched, false), Some(&unpublished[1])); + assert_eq!(choose(&unpublished, &vouched, true), None); + assert_eq!(choose(&[kept(4, "passed-through", false)], &vouched, false), None); + } + + /// A tree vouches for its first-parent chain and the head each merge on it + /// took in, and for no commit a branch passed through before its head. + #[test] + fn a_tree_vouches_for_its_first_parent_chain_and_the_heads_it_merged() { + let repo = TempDir::new("release-vouched"); + let commit = |message: &str| { + git(&repo, &["commit", "-q", "--allow-empty", "-m", message]); + git(&repo, &["rev-parse", "HEAD"]) + }; + git(&repo, &["init", "-q"]); + let m0 = commit("m0"); + git(&repo, &["switch", "-q", "-c", "landed"]); + let p0 = commit("p0"); + let p1 = commit("p1"); + git(&repo, &["switch", "-q", "main"]); + let m1 = commit("m1"); + git(&repo, &["merge", "-q", "--no-ff", "-m", "m2", "landed"]); + let m2 = git(&repo, &["rev-parse", "HEAD"]); + git(&repo, &["switch", "-q", "-c", "own", &m1]); + let q0 = commit("q0"); + let q1 = commit("q1"); + git(&repo, &["switch", "-q", "--detach", &m2]); + git(&repo, &["merge", "-q", "--no-ff", "-m", "the pull request's merge", "own"]); + let head = git(&repo, &["rev-parse", "HEAD"]); + let got = vouched(&repo).unwrap(); + let want: HashSet = [head, m2, m1, m0, p1, q1].into(); + assert_eq!(got, want, "p0 {p0} and q0 {q0} are what a branch passed through"); + } + + /// The negative control on the download: bytes that hash to anything but + /// GitHub's digest are refused, and so is a digest that is not SHA-256's. #[test] - fn an_asset_is_found_by_name_and_a_release_without_it_has_none() { - let with: serde_json::Value = serde_json::from_str(&format!( - r#"{{"assets":[{{"name":"other","url":"u1"}},{{"name":"{ASSET}","url":"u2"}}]}}"# - )) - .unwrap(); - assert_eq!(named_asset(&with).as_deref(), Some("u2")); - let without: serde_json::Value = serde_json::from_str(r#"{"assets":[]}"#).unwrap(); - assert_eq!(named_asset(&without), None); - let missing: serde_json::Value = serde_json::from_str(r#"{"message":"Not Found"}"#).unwrap(); - assert_eq!(named_asset(&missing), None); + fn a_download_is_held_to_the_digest_github_recorded() { + let dir = TempDir::new("release-verify"); + let file = dir.join("toolchain.tar.zst"); + fs::write(&file, b"").unwrap(); + let empty = format!("sha256:{}", sha256_hex(b"")); + assert!(verify(&file, &empty).is_ok()); + fs::write(&file, b"another writer's").unwrap(); + assert!(verify(&file, &empty).unwrap_err().contains("hashes to")); + assert!(verify(&file, &sha256_hex(b"another writer's")).unwrap_err().contains("is not a SHA-256 digest")); } } diff --git a/src/toolchain.rs b/src/toolchain.rs index 0126f827d76..01897e9d957 100644 --- a/src/toolchain.rs +++ b/src/toolchain.rs @@ -586,9 +586,9 @@ pub fn ensure(root: &Path, lock: &mut buildlock::Held) -> Sysroot { /// it is the one this tree needs, and say what to do when it is not. /// /// No amount of source here can rebuild a sysroot without `rust/`, so there is -/// nothing to decide and the answer is always to publish a toolchain built from -/// these sources. Its std fork is pinned by the release tag, which is a function -/// of `rust` (`src/release.rs`). +/// nothing to decide and the answer is always the toolchain built from these +/// sources: the one the release tag names, which hashes every source and every +/// module it is built from (`src/release.rs`). fn check_installed_toolchain(root: &Path, rust_dir: &Path) { let stage2 = stage2(rust_dir); let linked = rustup_link(); @@ -621,7 +621,8 @@ fn check_installed_toolchain(root: &Path, rust_dir: &Path) { recorded.as_deref() == Some(want.as_str()), "this checkout and the installed toolchain at {} disagree about {}, so a build \ here would link its kernel against another tree's struct layouts.\n\ - Publish a toolchain built from these sources and install that one instead.", + Install the build this tree's release tag names; if that is the one installed, \ + the tag hashes less than the toolchain is built from (`src/release.rs`).", stage2.display(), differing_trees(recorded.as_deref(), &want), ); diff --git a/tests/common/lane.rs b/tests/common/lane.rs index 09844eb52d6..23001849676 100644 --- a/tests/common/lane.rs +++ b/tests/common/lane.rs @@ -66,11 +66,11 @@ static RUN: OnceLock = OnceLock::new(); /// it, and it is gone when the run is, green or red (`toyos_tmpdir` is the /// policy, and what reclaims the directory of a run that was killed). /// -/// A red run's serial logs are the parts of it read afterwards, as CI's -/// artifact, so they are copied to a directory of their own under -/// [`RED_RUN_SERIAL`] first: megabytes, where the images are gigabytes. Named for this run's own root — unique across every process a -/// shared `$TMPDIR` ever holds — so two red runs of one worktree never share, -/// and neither overwrites, a destination. +/// A red run's serial logs are the parts of it read afterwards, so they are +/// copied to a directory of their own under [`RED_RUN_SERIAL`] first: +/// megabytes, where the images are gigabytes. Named for this run's own root — +/// unique across every process a shared `$TMPDIR` ever holds — so two red runs +/// of one worktree never share, and neither overwrites, a destination. /// /// [`Run::exit`] is the one way out of the suite with a status; returning from /// `main` drops this as green, and unwinding out of it as red. From 88b548799ba332a6ead5914971a4a7c680478a80 Mon Sep 17 00:00:00 2001 From: japabu Date: Thu, 1 Oct 2026 17:13:44 +0200 Subject: [PATCH 05/30] release: the publisher's refusal is tested at the release job's own entry `release_as` takes the workflow and event the runner names, so the test that refuses a pull request's, the merge queue's, the nightly's and a branch dispatch's job calls the release job itself rather than the check it starts with: deleting the check now reds a test, where before only a run of `cargo run -- --ci release` under a pull request's environment showed it. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L --- src/release.rs | 35 ++++++++++++++++++++++------------- 1 file changed, 22 insertions(+), 13 deletions(-) diff --git a/src/release.rs b/src/release.rs index 4a132652550..e4c36aea7a5 100644 --- a/src/release.rs +++ b/src/release.rs @@ -405,7 +405,12 @@ fn run(cmd: &mut Command) -> Result<(), String> { /// publisher. pub fn release(root: &Path) -> Result { let var = |name| std::env::var(name).ok(); - publisher(var("GITHUB_WORKFLOW_REF").as_deref(), var("GITHUB_EVENT_NAME").as_deref(), &repo())?; + release_as(root, var("GITHUB_WORKFLOW_REF").as_deref(), var("GITHUB_EVENT_NAME").as_deref()) +} + +/// [`release`], run as the job the runner names by its workflow and event. +fn release_as(root: &Path, workflow: Option<&str>, event: Option<&str>) -> Result { + publisher(workflow, event, &repo())?; let tag = tag(root)?; let build = find(root, &tag, true)? .ok_or_else(|| format!("main's publisher kept no build of {tag}: this run's `toolchain` job makes it"))?; @@ -811,27 +816,31 @@ mod tests { const REPO: &str = "ToyOSOrg/ToyOS"; - /// The negative control on the publisher: a pull request's job, the merge - /// queue's, the nightly's, and main's publisher dispatched on a branch or - /// run by any other event are each refused, by name. + /// The negative control on the publisher: the release job run as a pull + /// request's job, the merge queue's, the nightly's, or main's publisher + /// dispatched on a branch or run by any other event is refused by name + /// before it reads anything, and so is another repository's publisher. #[test] fn only_mains_publisher_publishes() { let mains = format!("{REPO}/.github/workflows/publish.yml@refs/heads/main"); assert!(publisher(Some(&mains), Some("push"), REPO).is_ok()); assert!(publisher(Some(&mains), Some("workflow_dispatch"), REPO).is_ok()); + let fork = "Fork/ToyOS/.github/workflows/publish.yml@refs/heads/main"; + assert!(publisher(Some(fork), Some("push"), REPO).unwrap_err().contains(fork)); + let root = Path::new(env!("CARGO_MANIFEST_DIR")); + let repo = repo(); let refused = [ - (format!("{REPO}/.github/workflows/ci.yml@refs/pull/671/merge"), "pull_request"), - (format!("{REPO}/.github/workflows/ci.yml@refs/heads/gh-readonly-queue/main/pr-671-59052827f"), "merge_group"), - (format!("{REPO}/.github/workflows/nightly.yml@refs/heads/main"), "schedule"), - (format!("{REPO}/.github/workflows/publish.yml@refs/heads/wt/toyos-guestci"), "workflow_dispatch"), - (mains.clone(), "pull_request_target"), - ("Fork/ToyOS/.github/workflows/publish.yml@refs/heads/main".to_string(), "push"), + (format!("{repo}/.github/workflows/ci.yml@refs/pull/671/merge"), "pull_request"), + (format!("{repo}/.github/workflows/ci.yml@refs/heads/gh-readonly-queue/main/pr-671-59052827f"), "merge_group"), + (format!("{repo}/.github/workflows/nightly.yml@refs/heads/main"), "schedule"), + (format!("{repo}/.github/workflows/publish.yml@refs/heads/wt/toyos-guestci"), "workflow_dispatch"), + (format!("{repo}/.github/workflows/publish.yml@refs/heads/main"), "pull_request_target"), ]; for (workflow, event) in refused { - let why = publisher(Some(&workflow), Some(event), REPO).expect_err(&workflow); - assert!(why.contains(&workflow) && why.contains(event), "{why}"); + let why = release_as(root, Some(&workflow), Some(event)).expect_err(&workflow); + assert!(why.starts_with("only ") && why.contains(&workflow) && why.contains(event), "{why}"); } - assert!(publisher(None, None, REPO).is_err()); + assert!(release_as(root, None, None).unwrap_err().starts_with("only ")); } fn run_json(path: &str, branch: &str, event: &str, head_repo: &str) -> Value { From 9a393422157a682eda5e088f64dd2379f32eeca6 Mon Sep 17 00:00:00 2001 From: japabu Date: Thu, 1 Oct 2026 17:15:14 +0200 Subject: [PATCH 06/30] release: the tag test moves each builder from its source, so a tag that drops one reds on its assertion It read each builder back from the fixture, which holds only what `trees()` names, so a tag that stopped hashing the builders panicked on a missing file instead of reporting the module whose commit kept the tag. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L --- src/release.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/release.rs b/src/release.rs index e4c36aea7a5..d2461ce51bc 100644 --- a/src/release.rs +++ b/src/release.rs @@ -788,7 +788,7 @@ mod tests { moves("src/n2.rs", n2.replace(pin, &"0".repeat(pin.len()))); moves("userland/libc/include/placeholder", "y".to_string()); for builder in BUILDERS { - let text = fs::read_to_string(repo.join(builder)).unwrap(); + let text = fs::read_to_string(here.join(builder)).unwrap(); moves(builder, format!("{text}\nconst MOVED: () = ();\n")); } } From 1077064939fe6b4e9bc7e10df829f4ccd97b99ef Mon Sep 17 00:00:00 2001 From: japabu Date: Thu, 1 Oct 2026 19:02:51 +0200 Subject: [PATCH 07/30] Toolchain stores keep what builds read: a lean LLVM, no hosted rustc unless shipped, no rustdoc or rustc debuginfo Round 3's ruling makes every store fit GitHub's 10 GB before CI carries them. Measured on the Linux release asset toolchain-linux-x86_64-48dd24f826263d6c and on this host's stores, compressed as actions/cache v4.3.0 stores an entry (tar, then zstdmt at level 3, as its logs print). a. The LLVM store keeps what builds read of bootstrap's install (`llvm::keep`): llvm-config, clang and llvm-ar, and llvm-objcopy on an Apple host; LLVM's headers; every library llvm-config names, since a compiler links LLVM through it and it refuses to name an absent one; and clang's resource headers. Not LLVM's other tools, clang's libraries and headers, nor CMake's package files. RECIPE moves to 4, so every LLVM key moves. Every compiler build says `llvm-tools = false`, because bootstrap copies its fourteen LLVM tools from llvm-config's bindir and would fail; `clang::provision` now copies llvm-ar, and on an Apple host llvm-objcopy as rust-objcopy, which rustc runs to strip a Darwin binary (rustc_codegen_ssa/src/back/link.rs) and which build scripts read here (the atime of every sysroot's rust-objcopy on this host is past its mtime). This host's LLVM 1425e623e612b348 is 741,348,616 B; what `keep` takes of it, staged by hand with the same selection, is 121,952,968 B. b. The primary builds the ToyOS-hosted rustc only for a build whose config ships it (`ensure`'s `hosted_rustc`). system.toml says no, and build.rs's `shipped` refuses every config that says yes, so no build makes it today, and the release no longer packs it. A compiler rebuild removes the hosted rustc of the compiler it replaced. The primary's bootstrap builds the host alone, as a worktree's compiler already does: every sysroot builds its own guest libraries and replaced the bootstrap's. c. The sysroot's 602,893,995 B was its compiler, 387,515,558 B, and the guest libraries, 215,377,065 B. Of the compiler: rustc's driver, 139,196,011 B alone, of which its line-table debuginfo is 64,174,215 B (stripped by llvm-objcopy --strip-debug, 75,021,796 B); 99,649,817 B of LLVM tools no build runs; rustdoc, 13,738,023 B, which no build runs (the toolchain builds no doc-test). Of the guest libraries, 163,254,188 B is metadata, which every crate compiled for those targets reads: upstream stable's own core metadata is 64,091,289 B raw for aarch64-unknown-none-softfloat, against the fork's 67,346,795 B for x86_64-unknown-none. Their rlibs' debuginfo is 7,222,696 B (38,817,617 B with it, 31,594,921 B without), which the linker reads, so it stays. So the primary builds `compiler/rustc library` and no rustdoc, and every compiler build says `debuginfo-level-rustc = 0`. The compiler RECIPE moves to 6. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L --- src/build.rs | 8 +- src/clang.rs | 62 ++++++++---- src/compiler.rs | 4 +- src/llvm.rs | 145 ++++++++++++++++++++++---- src/release.rs | 11 +- src/sysroot.rs | 3 +- src/toolchain.rs | 218 +++++++++++++++++++++++++++------------- tests/common/compile.rs | 2 +- 8 files changed, 325 insertions(+), 128 deletions(-) diff --git a/src/build.rs b/src/build.rs index 81493de21c5..835cadf6422 100644 --- a/src/build.rs +++ b/src/build.rs @@ -1718,10 +1718,10 @@ fn shipped_parts(root: &Path, boot: &Boot, plan: &Plan) -> (Vec, Vec, Ve // Held until the last staged artifact has been read back, so no clean of // this worktree's crate targets can land inside this build. let mut lock = buildlock::shared(root, "build"); - let sysroot = toolchain::ensure(root, &mut lock); + let config = parse_config(&boot.config); + let sysroot = toolchain::ensure(root, &mut lock, config.hosted_rustc); let env = GuestEnv::new(&sysroot); - let config = parse_config(&boot.config); invalidate_stale(root, &mut lock, &env.identity, &config_targets(root, &config)); @@ -1940,7 +1940,7 @@ pub fn build_test_parts( // back after the userland build, and a clean landing in between is the // same defect as one landing mid-compile. let mut lock = buildlock::shared(root, "test image"); - let sysroot = crate::toolchain::ensure(root, &mut lock); + let sysroot = crate::toolchain::ensure(root, &mut lock, config.hosted_rustc); let env = GuestEnv::new(&sysroot); invalidate_stale(root, &mut lock, &env.identity, &config_targets(root, &config)); @@ -2148,7 +2148,7 @@ struct TestBuild { impl TestBuild { fn begin(root: &Path, arch: Arch, what: &str, stale_targets: &[(PathBuf, Clean)]) -> Self { let mut lock = buildlock::shared(root, what); - let sysroot = crate::toolchain::ensure(root, &mut lock); + let sysroot = crate::toolchain::ensure(root, &mut lock, false); let env = GuestEnv::new(&sysroot); invalidate_stale(root, &mut lock, &env.identity, stale_targets); let artifact = buildlock::artifact(root); diff --git a/src/clang.rs b/src/clang.rs index ac2f5f11629..ca7207a8e1e 100644 --- a/src/clang.rs +++ b/src/clang.rs @@ -3,17 +3,20 @@ //! //! **Beside `rust-lld`, in `lib/rustlib//bin/`**, which every copy of a //! toolchain directory — a compiler of a worktree's own, a sysroot, the -//! published release — carries whole. Bootstrap puts LLVM's tools there -//! already, `llvm-ar` — the archiver `cc` builds a C library with, where the -//! host's may not index ELF at all — among them. This adds the rest: +//! published release — carries whole. Bootstrap copies none of LLVM's tools +//! there (`llvm-tools = false`); this puts the ones a build runs: //! //! - `clang`, the driver whose ToyOS toolchain (`src/llvm-project`'s //! `clang/lib/Driver/ToolChains/ToyOS.cpp`) names `ld.lld`, the sysroot and //! `-ltoyos_c`; +//! - `llvm-ar`, the archiver `cc` builds a C library with, where the host's may +//! not index ELF at all; //! - `ld.lld`, a link to `rust-lld` — the same LLD, which takes its flavour from //! the name it is run by — found by clang in its own directory; //! - `../lib/clang//include`, clang's own headers (`stddef.h`, -//! `stdarg.h`), which it looks for relative to itself. +//! `stdarg.h`), which it looks for relative to itself; +//! - on an Apple host, `rust-objcopy`, LLVM's `llvm-objcopy`, which rustc runs +//! from here to strip a Darwin binary. //! //! Bootstrap removes `stage2` on every assemble, so these are put back after //! every build that makes one, and a toolchain directory without all of it is @@ -37,6 +40,14 @@ pub(crate) const LLVM_CONFIG: &str = "download-ci-llvm = false\n\ /// What a toolchain directory's `bin` must hold for C. const TOOLS: [&str; 3] = ["llvm-ar", "clang", "ld.lld"]; +/// What an Apple host's toolchain directory's `bin` must hold besides. +const APPLE_STRIP: &str = "rust-objcopy"; + +/// [`TOOLS`], and on an Apple host [`APPLE_STRIP`]. +fn tools() -> impl Iterator { + TOOLS.into_iter().chain(host_triple().ends_with("apple-darwin").then_some(APPLE_STRIP)) +} + /// `lib/rustlib//bin` of `toolchain`, where `rust-lld` is. fn bin(toolchain: &Path) -> PathBuf { toolchain.join("lib/rustlib").join(host_triple()).join("bin") @@ -94,8 +105,7 @@ fn resource_parent(toolchain: &Path) -> PathBuf { /// What of the C toolchain `toolchain` lacks, by name. fn absent(toolchain: &Path) -> Vec { let bin = bin(toolchain); - let mut gone: Vec = TOOLS - .iter() + let mut gone: Vec = tools() .map(|name| bin.join(name)) .filter(|path| !path.exists()) .map(|path| path.display().to_string()) @@ -133,7 +143,7 @@ pub(crate) fn assert_present(toolchain: &Path) { } /// The one version directory under an LLVM build's `lib/clang`. -fn resource_version(llvm: &Path) -> PathBuf { +pub(crate) fn resource_version(llvm: &Path) -> PathBuf { let parent = llvm.join("lib/clang"); let versions: Vec = fs::read_dir(&parent) .unwrap_or_else(|e| panic!("read {}: {e} — was LLVM built with clang = true?", parent.display())) @@ -150,12 +160,13 @@ fn resource_version(llvm: &Path) -> PathBuf { /// its rustc links. pub(crate) fn provision(stage2: &Path, llvm: &Path) { let bin = bin(stage2); - let from = llvm.join("bin/clang"); - let to = bin.join("clang"); - let _ = fs::remove_file(&to); - // `fs::copy` follows `clang`'s link to `clang-`, and clones where - // the filesystem can. - fs::copy(&from, &to).unwrap_or_else(|e| panic!("copy {} -> {}: {e}", from.display(), to.display())); + let apple = host_triple().ends_with("apple-darwin").then_some((crate::llvm::APPLE_TOOL, APPLE_STRIP)); + for (tool, name) in [("clang", "clang"), ("llvm-ar", "llvm-ar")].into_iter().chain(apple) { + let (from, to) = (llvm.join("bin").join(tool), bin.join(name)); + let _ = fs::remove_file(&to); + // `fs::copy` clones where the filesystem can. + fs::copy(&from, &to).unwrap_or_else(|e| panic!("copy {} -> {}: {e}", from.display(), to.display())); + } let lld = bin.join("ld.lld"); let _ = fs::remove_file(&lld); std::os::unix::fs::symlink("rust-lld", &lld) @@ -184,26 +195,32 @@ mod tests { let llvm = base.join("llvm"); write(&llvm.join("bin/clang-22"), "the clang"); std::os::unix::fs::symlink("clang-22", llvm.join("bin/clang")).unwrap(); + write(&llvm.join("bin/llvm-ar"), "the archiver"); + write(&llvm.join("bin/llvm-objcopy"), "the objcopy"); write(&llvm.join("lib/clang/22/include/stddef.h"), "typedef long ptrdiff_t;"); llvm } /// **A toolchain without its C compiler is refused by name**, and one - /// provisioned from an LLVM carries the binary behind `clang`'s link, an - /// `ld.lld` that is `rust-lld`, and clang's headers where clang looks. + /// provisioned from an LLVM carries the binary behind `clang`'s link, that + /// LLVM's archiver, an `ld.lld` that is `rust-lld`, clang's headers where + /// clang looks, and on an Apple host the `rust-objcopy` rustc strips with. #[test] fn a_toolchain_carries_the_clang_of_its_llvm_or_is_refused() { let base = TempDir::new("clang"); let llvm = llvm(&base); - // What bootstrap's own assemble leaves beside `rust-lld`. + let apple = host_triple().ends_with("apple-darwin"); + // What bootstrap's own assemble leaves: `rust-lld` and no LLVM tool. let stage2 = base.join("stage2"); write(&bin(&stage2).join("rust-lld"), "lld"); - write(&bin(&stage2).join("llvm-ar"), "the archiver"); assert!(defect(&stage2).is_some()); let refused = std::panic::catch_unwind(|| assert_present(&stage2)).expect_err("no clang, and not refused"); let said = refused.downcast_ref::().expect("a formatted refusal"); - assert!(said.contains("clang") && said.contains("ld.lld") && said.contains("include"), "{said}"); + for named in ["clang", "llvm-ar", "ld.lld", "include"] { + assert!(said.contains(named), "{named}: {said}"); + } + assert_eq!(said.contains(APPLE_STRIP), apple, "{said}"); provision(&stage2, &llvm); assert_eq!(defect(&stage2), None); @@ -212,6 +229,7 @@ mod tests { assert_eq!(fs::read_link(bin(&stage2).join("ld.lld")).unwrap(), Path::new("rust-lld")); assert_eq!(fs::read_to_string(bin(&stage2).join("ld.lld")).unwrap(), "lld"); assert_eq!(fs::read_to_string(bin(&stage2).join("llvm-ar")).unwrap(), "the archiver"); + assert_eq!(fs::read_to_string(bin(&stage2).join(APPLE_STRIP)).ok().as_deref(), apple.then_some("the objcopy")); let stddef = resource_parent(&stage2).join("22/include/stddef.h"); assert_eq!(fs::read_to_string(stddef).unwrap(), "typedef long ptrdiff_t;"); @@ -222,7 +240,11 @@ mod tests { assert!(!resource_parent(&stage2).join("22").exists(), "the old headers stayed beside the new"); assert_eq!(fs::read_to_string(resource_parent(&stage2).join("23/include/stddef.h")).unwrap(), "v23"); - fs::remove_file(bin(&stage2).join("llvm-ar")).unwrap(); - assert!(defect(&stage2).is_some(), "a missing llvm-ar went unnoticed"); + let lost = if apple { [APPLE_STRIP, "llvm-ar"].as_slice() } else { ["llvm-ar"].as_slice() }; + for tool in lost { + provision(&stage2, &llvm); + fs::remove_file(bin(&stage2).join(tool)).unwrap(); + assert!(defect(&stage2).is_some_and(|d| d.contains(tool)), "a missing {tool} went unnoticed"); + } } } diff --git a/src/compiler.rs b/src/compiler.rs index 5f2e9ad511c..12b09435791 100644 --- a/src/compiler.rs +++ b/src/compiler.rs @@ -53,7 +53,7 @@ use crate::toolchain::{self, host_triple}; /// What changes how a key's sources become a compiler and is none of them: the /// build below. Moving it moves every key. -const RECIPE: &str = "bootstrap stage 2 of compiler/rustc and library, profile compiler, host only, with rust-lld, host linker pinned, LLVM, clang and LLD from the host's LLVM; 5"; +const RECIPE: &str = "bootstrap stage 2 of compiler/rustc and library, profile compiler, host only, with rust-lld, host linker pinned, LLVM, clang and LLD from the host's LLVM, no LLVM tool copied, rustc without debuginfo; 6"; /// What a compiler's key is the identity of, in its fork checkout. const KEYED: [&str; 4] = ["compiler", "src/tools", "src/stage0", "Cargo.lock"]; @@ -360,6 +360,7 @@ target = ["{host}"] [rust] incremental = true lld = true +{lean} [target.{host}] {pin} @@ -367,6 +368,7 @@ lld = true "#, build_dir = build_dir.display(), llvm = crate::clang::LLVM_CONFIG, + lean = toolchain::LEAN, pin = toolchain::HOST_LINKER_PIN, external = crate::llvm::host_lines(llvm), ) diff --git a/src/llvm.rs b/src/llvm.rs index 6b505d942c9..7829e197996 100644 --- a/src/llvm.rs +++ b/src/llvm.rs @@ -7,12 +7,12 @@ //! the committed tree of its `src/bootstrap` (one holding what no commit does is //! refused), the bootstrap configuration below, [`RECIPE`], and the tools the //! host builds it with ([`host_tools`]). `rust/build/llvm//` in the primary -//! is bootstrap's install of that LLVM and its clang, with its LLD in `bin/` -//! beside `llvm-config` and in `src/` the runtimes' sources the C++ runtime is -//! built from (`src/libcxx.rs`) as its commit holds them, made by whichever -//! build first needs it ([`resolve`]), and stored only when it was built from -//! what the key names. Once its [`SOURCE`] file exists it is read-only, its -//! directories as well as its files. +//! is what builds read of bootstrap's install of that LLVM and its clang +//! ([`keep`]), with its LLD in `bin/` beside `llvm-config` and in `src/` the +//! runtimes' sources the C++ runtime is built from (`src/libcxx.rs`) as its +//! commit holds them, made by whichever build first needs it ([`resolve`]), and +//! stored only when it was built from what the key names. Once its [`SOURCE`] +//! file exists it is read-only, its directories as well as its files. //! Every compiler build, the primary's and a worktree's own, names it as the //! host's `llvm-config` with `llvm-has-rust-patches`, so bootstrap builds no //! LLVM and takes LLD from beside it as `rust-lld`; `clang::provision` copies its @@ -48,8 +48,10 @@ use crate::toolchain::{self, host_triple}; /// What changes how a key's sources become an LLVM and is none of the other /// parts: the build's targets and what is kept of it. Moving it moves every key. -const RECIPE: &str = "bootstrap build of src/llvm-project/llvm and src/llvm-project/lld; the install's bin, \ - include and lib, and lld in bin, and the runtimes' sources in src, read-only; 3"; +const RECIPE: &str = "bootstrap build of src/llvm-project/llvm and src/llvm-project/lld; of the install, \ + llvm-config, clang and llvm-ar in bin, and llvm-objcopy on an Apple host, LLVM's headers, \ + every library llvm-config names and clang's resource headers; lld in bin, and the \ + runtimes' sources in src, read-only; 4"; /// What of the caller's environment the LLVM build, and every tool its key /// asks, sees: @@ -85,12 +87,18 @@ const NO_HOST_LIBRARIES: [&str; 12] = [ "LLVM_ENABLE_Z3_SOLVER", ]; -/// What of bootstrap's install an LLVM keeps: `build/` beside them is CMake's -/// tree, which nothing reads once the install is made. -const KEPT: [&str; 3] = ["bin", "include", "lib"]; +/// The tools of an LLVM's `bin` a build runs: bootstrap asks `llvm-config` how +/// to link LLVM and takes `lld` as `rust-lld`; `clang::provision` copies `clang` +/// and `llvm-ar`, and on an Apple host [`APPLE_TOOL`]. +const TOOLS: [&str; 4] = ["llvm-config", "lld", "clang", "llvm-ar"]; -/// What a compiler build and `clang::provision` read of an LLVM. -const TOOLS: [&str; 4] = ["bin/llvm-config", "bin/lld", "bin/clang", "bin/llvm-ar"]; +/// What an Apple host's toolchain carries as `rust-objcopy`, which rustc runs to +/// strip a Darwin binary (`compiler/rustc_codegen_ssa/src/back/link.rs`). +pub(crate) const APPLE_TOOL: &str = "llvm-objcopy"; + +/// LLVM's headers: the compiler's LLVM wrapper compiles against them, where +/// `llvm-config --cxxflags` names the install's `include`. +const HEADERS: [&str; 2] = ["include/llvm", "include/llvm-c"]; /// The file a finished LLVM carries last, naming its key. A directory without /// it is a build that did not finish. @@ -226,13 +234,22 @@ fn held_with(root: &Path, rust_dir: &Path, fork: &Path, build: impl Fn(&Path) -> Llvm { dir, _using: using } } +/// [`TOOLS`], and on an Apple host [`APPLE_TOOL`]. +fn tools() -> impl Iterator { + TOOLS.into_iter().chain(host_triple().ends_with("apple-darwin").then_some(APPLE_TOOL)) +} + /// Why `dir` is not a finished LLVM, if it is not. fn defect(dir: &Path) -> Option { if !dir.join(SOURCE).is_file() { return Some(format!("{} carries no {SOURCE}", dir.display())); } - let kept = KEPT.iter().map(|k| dir.join(k)).chain(crate::libcxx::SOURCES.iter().map(|s| dir.join("src").join(s))); - let tools = TOOLS.iter().map(|t| dir.join(t)).filter(|p| !p.is_file()); + let kept = HEADERS + .iter() + .chain(&["lib/clang"]) + .map(|k| dir.join(k)) + .chain(crate::libcxx::SOURCES.iter().map(|s| dir.join("src").join(s))); + let tools = tools().map(|t| dir.join("bin").join(t)).filter(|p| !p.is_file()); let gone: Vec = kept.filter(|p| !p.is_dir()).chain(tools).map(|p| p.display().to_string()).collect(); (!gone.is_empty()).then(|| format!("{} carries no {}", dir.display(), gone.join(", "))) } @@ -260,9 +277,7 @@ fn place(fork: &Path, key: &Key, dir: &Path, build: &impl Fn(&Path) -> PathBuf) if partial.exists() { keystore::remove(&partial); } - for part in KEPT { - clone_tree(&built.join(&host).join("llvm").join(part), &partial.join(part)); - } + keep(&built.join(&host).join("llvm"), &partial); let lld = built.join(&host).join("lld/bin/lld"); fs::copy(&lld, partial.join("bin/lld")) .unwrap_or_else(|e| panic!("copy {} -> {}: {e}", lld.display(), partial.join("bin/lld").display())); @@ -297,6 +312,45 @@ fn place(fork: &Path, key: &Key, dir: &Path, build: &impl Fn(&Path) -> PathBuf) fs::remove_dir_all(&built).unwrap_or_else(|e| panic!("remove {}: {e}", built.display())); } +/// Copy into `to` what builds read of the LLVM installed at `install`: [`tools`] +/// but `lld`, which is LLD's own build's, each a file whatever link it is +/// installed as; [`HEADERS`]; every library its `llvm-config` names, since a +/// compiler links LLVM through it and it refuses to name one that is absent; +/// and clang's resource headers. +fn keep(install: &Path, to: &Path) { + let bin = to.join("bin"); + fs::create_dir_all(&bin).unwrap_or_else(|e| panic!("create {}: {e}", bin.display())); + for tool in tools().filter(|tool| *tool != "lld") { + let from = install.join("bin").join(tool); + fs::copy(&from, bin.join(tool)).unwrap_or_else(|e| panic!("copy {} -> {}: {e}", from.display(), bin.display())); + } + for headers in HEADERS { + clone_tree(&install.join(headers), &to.join(headers)); + } + let lib = to.join("lib"); + fs::create_dir_all(&lib).unwrap_or_else(|e| panic!("create {}: {e}", lib.display())); + for library in libraries(install) { + let name = library.file_name().unwrap_or_else(|| panic!("{} names no file", library.display())); + fs::copy(&library, lib.join(name)).unwrap_or_else(|e| panic!("copy {} -> {}: {e}", library.display(), lib.display())); + } + let resource = crate::clang::resource_version(install); + let version = resource.file_name().unwrap_or_else(|| panic!("{} names no version", resource.display())); + clone_tree(&resource.join("include"), &lib.join("clang").join(version).join("include")); +} + +/// Every library the `llvm-config` of the LLVM installed at `install` names. +fn libraries(install: &Path) -> Vec { + let config = install.join("bin/llvm-config"); + let mut command = Command::new(&config); + command.args(["--link-static", "--libfiles"]); + clear(&mut command); + let out = command.output().unwrap_or_else(|e| panic!("run {}: {e}", config.display())); + assert!(out.status.success(), "{command:?} failed: {}", String::from_utf8_lossy(&out.stderr)); + let named: Vec = String::from_utf8_lossy(&out.stdout).split_whitespace().map(PathBuf::from).collect(); + assert!(!named.is_empty(), "{command:?} named no library"); + named +} + /// Write `paths` as `commit` holds them, from the repository at `checkout`, /// under `dest`: through an index of their own and with no sparse pattern, so /// nothing the checkout holds beside the commit, tracked, ignored or left out, @@ -460,15 +514,27 @@ mod tests { /// Bootstrap's stand-in: what its LLVM and LLD builds leave in the build /// directory, CMake's tree among them. fn fake_build(fork: &Path) -> PathBuf { + use std::os::unix::fs::PermissionsExt; let built = fork.join("build/toyos-llvm"); let _ = fs::remove_dir_all(&built); let install = built.join(host_triple()).join("llvm"); - for tool in ["llvm-config", "clang-22", "llvm-ar"] { + for tool in ["clang-22", "llvm-ar", "llvm-objcopy", "opt"] { write(&install.join("bin").join(tool), &format!("the {tool}")); } std::os::unix::fs::symlink("clang-22", install.join("bin/clang")).unwrap(); + // What a real `llvm-config --libfiles` answers: the component libraries, + // never clang's, nor one no component is. + let named = ["libLLVMCore.a", "libLLVMSupport.a"].map(|lib| install.join("lib").join(lib).display().to_string()); + let config = install.join("bin/llvm-config"); + write(&config, &format!("#!/bin/sh\necho {}\n", named.join(" "))); + fs::set_permissions(&config, fs::Permissions::from_mode(0o755)).unwrap(); + for lib in ["libLLVMCore.a", "libLLVMSupport.a", "libLLVMTableGen.a", "libclangBasic.a"] { + write(&install.join("lib").join(lib), lib); + } write(&install.join("include/llvm/Config/llvm-config.h"), "#define LLVM_VERSION_MAJOR 22"); - write(&install.join("lib/libLLVMCore.a"), "core"); + write(&install.join("include/llvm-c/Core.h"), "LLVMContextRef LLVMContextCreate(void);"); + write(&install.join("include/clang/Basic/Version.h"), "#define CLANG_VERSION 22"); + write(&install.join("lib/cmake/llvm/LLVMConfig.cmake"), "set(LLVM_PACKAGE_VERSION 22)"); write(&install.join("lib/clang/22/include/stddef.h"), "typedef long ptrdiff_t;"); write(&install.join("build/CMakeCache.txt"), "the build tree"); write(&built.join(host_triple()).join("lld/bin/lld"), "the lld"); @@ -544,7 +610,6 @@ mod tests { assert_eq!(makes.get(), 1); assert_eq!(defect(&la.dir), None); assert_eq!(fs::read_to_string(la.dir.join("bin/lld")).unwrap(), "the lld"); - assert_eq!(fs::read_link(la.dir.join("bin/clang")).unwrap(), Path::new("clang-22")); assert!(la.dir.join("lib/clang/22/include/stddef.h").is_file()); assert_eq!(fs::read_to_string(la.dir.join("src/libcxx/CMakeLists.txt")).unwrap(), "the libcxx of A", "the runtimes' sources are not the commit's"); assert!(!la.dir.join("build").exists(), "CMake's tree was kept"); @@ -559,6 +624,44 @@ mod tests { assert_eq!(snapshot(&store(&rust_dir)), before, "an LLVM was written after it was whole"); } + /// **An LLVM keeps what builds read of the install and nothing else**: the + /// tools a build runs, `clang` as the file its link names; LLVM's headers; + /// every library `llvm-config` names, and no other; clang's resource + /// headers. No other tool, clang's headers and libraries, nor CMake's + /// package files. + #[test] + fn an_llvm_keeps_what_builds_read_and_nothing_else() { + let scratch = Scratch::new("llvm-kept"); + let (_primary, rust_dir, [_same, a, _b]) = estate_built(&scratch); + let dir = choose(&a, &rust_dir, &a.join("rust"), fake_build).dir; + let apple = host_triple().ends_with("apple-darwin"); + let mut files: Vec = snapshot(&dir) + .into_iter() + .map(|(path, _)| path.strip_prefix(&dir).unwrap().display().to_string()) + .filter(|path| !path.starts_with("src/")) + .collect(); + files.sort(); + let mut want = vec![ + "SOURCE", + "bin/clang", + "bin/lld", + "bin/llvm-ar", + "bin/llvm-config", + "include/llvm-c/Core.h", + "include/llvm/Config/llvm-config.h", + "lib/clang/22/include/stddef.h", + "lib/libLLVMCore.a", + "lib/libLLVMSupport.a", + ]; + if apple { + want.push("bin/llvm-objcopy"); + } + want.sort(); + assert_eq!(files, want); + assert_eq!(fs::read_to_string(dir.join("bin/clang")).unwrap(), "the clang-22"); + assert!(!fs::symlink_metadata(dir.join("bin/clang")).unwrap().file_type().is_symlink(), "clang is the link, not the file"); + } + /// **A placed LLVM cannot be written**, through its own path or through a /// link bootstrap makes to one of its files, and nothing in it can be /// removed, replaced or added. diff --git a/src/release.rs b/src/release.rs index d2461ce51bc..d95839c53c4 100644 --- a/src/release.rs +++ b/src/release.rs @@ -35,8 +35,6 @@ use serde_json::Value; use sha2::{Digest, Sha256}; use toyos_tmpdir::TempDir; -use crate::toolchain::HOSTED_ARCH; - /// The build system's modules that build and pack the toolchain: every module /// `src/toolchain.rs` and this file name through `crate::`, and every module /// those name, so the tag moves with how the toolchain is built as well as with @@ -497,16 +495,15 @@ fn build(root: &Path, tag: &str, tarball: &Path) -> Result<(), String> { } fs::write(build.join("TOOLCHAIN"), &manifest).map_err(|e| e.to_string())?; - // `lib/rustlib/` and the sysroot's `bin/cargo` are links into this - // runner's own toolchain; `Owner::Installed` recreates both. GNU tar's - // `--transform` renames the sysroot to the path an installer links. + // The sysroot's `bin/cargo` is a link into this runner's own toolchain; + // `Owner::Installed` recreates it. GNU tar's `--transform` renames the + // sysroot to the path an installer links. let mut tar = Command::new("tar") .arg("-C") .arg(&build) - .arg(format!("--exclude={}/stage2/lib/rustlib/{HOST}", HOSTED_ARCH.userland())) .arg(format!("--exclude={sysroot}/bin/cargo")) .arg(format!("--transform=s,^{sysroot},{HOST}/stage2,")) - .args(["-c", &sysroot, &format!("{}/stage2", HOSTED_ARCH.userland())]) + .args(["-c", &sysroot]) .args(["toyos-sysroot-witness", "TOOLCHAIN"]) .stdout(Stdio::piped()) .spawn() diff --git a/src/sysroot.rs b/src/sysroot.rs index 868929b31bd..4475eb4e9f0 100644 --- a/src/sysroot.rs +++ b/src/sysroot.rs @@ -1323,9 +1323,8 @@ mod tests { write(&compiler.stage2.join("bin/rustc"), "rustc"); let lld = toolchain::rust_lld(&compiler.stage2); write(&lld, "lld"); - write(&lld.with_file_name("llvm-ar"), "llvm-ar"); if clang { - for tool in ["clang", "ld.lld"] { + for tool in ["clang", "llvm-ar", "ld.lld", "rust-objcopy"] { write(&lld.with_file_name(tool), tool); } write(&lld.parent().unwrap().parent().unwrap().join("lib/clang/22/include/stddef.h"), "stddef"); diff --git a/src/toolchain.rs b/src/toolchain.rs index 01897e9d957..8a5fa80df4f 100644 --- a/src/toolchain.rs +++ b/src/toolchain.rs @@ -121,6 +121,31 @@ pub(crate) fn stage2(rust_dir: &Path) -> PathBuf { rust_dir.join(format!("build/{}/stage2", host_triple())) } +/// The ToyOS-hosted rustc's toolchain directory, beside the primary's compiler. +fn hosted_stage2(rust_dir: &Path) -> PathBuf { + rust_dir.join(format!("build/{}/stage2", HOSTED_ARCH.userland())) +} + +/// Whether the primary builds the hosted rustc: a build whose config ships it +/// `asked`, and `rustc` is not there or `stamp`, which says it is this +/// compiler's, is not. +fn hosted_rustc_owed(asked: bool, stamp: &Path, rustc: &Path) -> bool { + asked && (!stamp.exists() || !rustc.exists()) +} + +/// Remove the hosted rustc a compiler rebuild left stale, and its `stamp`: no +/// build reads one until a config that ships it asks, and that build makes it +/// anew. +fn forget_hosted_rustc(rust_dir: &Path, stamp: &Path) { + let gone = |path: &Path, removed: std::io::Result<()>| match removed { + Err(e) if e.kind() != std::io::ErrorKind::NotFound => panic!("remove {}: {e}", path.display()), + _ => {} + }; + gone(stamp, fs::remove_file(stamp)); + let stale = hosted_stage2(rust_dir); + gone(&stale, fs::remove_dir_all(&stale)); +} + /// Every `toyos-abi`/`toyos` source file a std build under `dep_info` actually /// compiled, read out of cargo's dep-info rather than out of what was asked for. fn std_toyos_sources(dep_info: &Path) -> Vec { @@ -468,7 +493,9 @@ fn bootstrap(current: bool, toolchain_exists: bool) -> Option { } /// Ensure the toolchain is up to date, and return the sysroot this checkout's -/// sources name — made if nobody has made it (`src/sysroot.rs`). +/// sources name — made if nobody has made it (`src/sysroot.rs`). The primary +/// builds the ToyOS-hosted rustc only for a build whose config ships it +/// (`hosted_rustc`). /// /// Every step decides under the caller's shared lock and acts under the /// exclusive one, so the common answer — nothing to do — costs no @@ -490,7 +517,7 @@ fn bootstrap(current: bool, toolchain_exists: bool) -> Option { /// `stage1-std//dist/deps` while another's `rustc` creates a temp file /// inside it, and the loser dies compiling `core` with `couldn't create a temp /// dir: No such file or directory`. -pub fn ensure(root: &Path, lock: &mut buildlock::Held) -> Sysroot { +pub fn ensure(root: &Path, lock: &mut buildlock::Held, hosted_rustc: bool) -> Sysroot { let rust_dir = rust_dir(root); let stamps_dir = root.join("target/stamps"); fs::create_dir_all(&stamps_dir).ok(); @@ -537,22 +564,22 @@ pub fn ensure(root: &Path, lock: &mut buildlock::Held) -> Sysroot { |kind| { eprintln!("Building full toolchain (this takes a while on first run)..."); let llvm = crate::llvm::resolve(root, &rust_dir, &rust_dir); - rebuild_compiler(&rust_dir, &llvm.dir, || full_bootstrap(root, &rust_dir, &llvm.dir)); + rebuild_compiler(&rust_dir, &llvm.dir, || full_bootstrap(&rust_dir, &llvm.dir)); if kind.invalidate_hosted { - let _ = fs::remove_file(&hosted_stamp); + forget_hosted_rustc(&rust_dir, &hosted_stamp); } }, ); - let hosted_rustc = rust_dir.join(format!("build/{}/stage2/bin/rustc", HOSTED_ARCH.userland())); + let hosted = hosted_stage2(&rust_dir).join("bin/rustc"); lock.act_if( Scope::Global, "build the ToyOS-hosted rustc", - || (!hosted_stamp.exists() || !hosted_rustc.exists()).then_some(()), + || hosted_rustc_owed(hosted_rustc, &hosted_stamp, &hosted).then_some(()), |()| { let llvm = crate::llvm::resolve(root, &rust_dir, &rust_dir); reassemble(&rust_dir, &llvm.dir, || build_hosted_rustc(&rust_dir, &llvm.dir)); - assert!(hosted_rustc.exists(), "Failed to build hosted rustc"); + assert!(hosted.exists(), "Failed to build hosted rustc"); fs::write(&hosted_stamp, "").unwrap(); }, ); @@ -748,9 +775,9 @@ impl Drop for Restore { /// Where a compile error starts in an `x build` log, if there is one. /// -/// Both bootstrap callers let a non-zero `x build` through when the artifacts -/// they need are on disk, because rustdoc for ToyOS does not link and never -/// has. That allowance used to be *anything at all*, as long as a `rustc` from +/// The hosted rustc's build lets a non-zero `x build` through when the +/// artifacts it needs are on disk, because rustdoc for ToyOS does not link and +/// never has. That allowance used to be *anything at all*, as long as a `rustc` from /// some earlier build was still there — so run `31370078581` compiled std with /// `error[E0433]`, took the allowance, and died 83 seconds and 260 lines later /// at a missing file. The reported failure was the consequence. @@ -787,12 +814,15 @@ fn tolerated_failure(log: &[String], what: &str) { ); } -fn full_bootstrap(root: &Path, rust_dir: &Path, llvm: &Path) { +/// What the primary's compiler build builds: rustc and the host's libraries, +/// which build scripts and proc macros link. No rustdoc: no build runs one. +const COMPILER_BUILD: [&str; 7] = ["build", "--stage", "2", "--warnings", "warn", "compiler/rustc", "library"]; + +fn full_bootstrap(rust_dir: &Path, llvm: &Path) { // Ensure library/backtrace is checked out — std depends on it. // Other rust submodules (llvm, docs, cargo) are handled by bootstrap on demand. crate::ensure_submodule(rust_dir, "library/backtrace"); - // Write bootstrap.toml — ToyOS as target only, not host (fast rebuilds) let host = host_triple(); write_config(rust_dir, &host, false, llvm); @@ -805,28 +835,9 @@ fn full_bootstrap(root: &Path, rust_dir: &Path, llvm: &Path) { } } - let build = ["build", "--stage", "2", "--warnings", "warn"]; - let (ok, log) = x_build(rust_dir, &build, "the toolchain"); - - if !ok { - refuse_on_compile_error(&log, "the toolchain"); - // rustdoc for ToyOS may fail to link; rustc may not be missing. - let stage2 = rust_dir.join(format!("build/{host}/stage2")); - assert!( - stage2.join("bin/rustc").exists(), - "the toolchain build failed and {} is not there.\n\ - Nothing in its output was a compile error, so this is a link or a bootstrap \ - failure — the last lines above are the whole of what it said.", - stage2.join("bin/rustc").display() - ); - tolerated_failure(&log, "the toolchain build"); - } - for arch in Arch::ALL { - assert_std_built_from( - root, - &rust_dir.join(format!("build/{host}/stage1-std/{}", arch.userland())), - ); - } + let (ok, log) = x_build(rust_dir, &COMPILER_BUILD, "the toolchain"); + refuse_on_compile_error(&log, "the toolchain"); + assert!(ok, "the toolchain build failed, and nothing in its output was a compile error"); } fn build_hosted_rustc(rust_dir: &Path, llvm: &Path) { @@ -862,11 +873,7 @@ fn build_hosted_rustc(rust_dir: &Path, llvm: &Path) { // (`write_config` says why), so the host-only build runs once more to put // it back: everything it would compile is already built. write_config(rust_dir, &host, false, llvm); - let (ok, log) = x_build( - rust_dir, - &["build", "--stage", "2", "--warnings", "warn"], - "the toolchain, reassembled", - ); + let (ok, log) = x_build(rust_dir, &COMPILER_BUILD, "the toolchain, reassembled"); refuse_on_compile_error(&log, "the toolchain, reassembled"); assert!( rust_lld(&stage2(rust_dir)).is_file(), @@ -894,41 +901,25 @@ fn build_hosted_rustc(rust_dir: &Path, llvm: &Path) { /// otherwise ties it to `lld` for `x86_64-unknown-linux-gnu`, and a host rustc /// whose build environment flips with the config is rebuilt by each of those /// two builds. +/// +/// The host-only toolchain builds no guest target's libraries: every sysroot +/// builds its own (`src/sysroot.rs`). fn write_config(rust_dir: &Path, host: &str, with_hosted_rustc: bool, llvm: &Path) { let host_line = if with_hosted_rustc { format!("host = [\"{host}\", \"{}\"]", HOSTED_ARCH.userland()) } else { format!("host = [\"{host}\"]") }; + let (guests, userland) = if with_hosted_rustc { + (GUEST_TARGETS.map(GuestTarget::triple).to_vec(), hosted_targets(llvm)) + } else { + (Vec::new(), String::new()) + }; let targets = std::iter::once(host) - .chain(GUEST_TARGETS.map(GuestTarget::triple)) + .chain(guests) .map(|t| format!("\"{t}\"")) .collect::>() .join(", "); - let userland: String = Arch::ALL - .iter() - .map(|arch| { - let linker = format!("linker = \"{}\"", llvm.join("bin/lld").display()); - let hosted = if with_hosted_rustc && *arch == HOSTED_ARCH { - // Cranelift because no LLVM is built for a ToyOS host yet, and - // only for that reason: the hosted rustc carries LLVM once clang - // and libc++ run on ToyOS, and Cranelift is not where the - // compiler that builds ToyOS goes. - // - // The archiver is that LLVM's too: the compiler's crates carry - // C built for this target (blake3's assembly), and a host `ar` - // that indexes only its own object format, as macOS's does, - // leaves those ELF members out of the index lld pulls from. - format!( - "\nar = \"{}\"\ncodegen-backends = [\"cranelift\"]", - llvm.join("bin/llvm-ar").display(), - ) - } else { - String::new() - }; - format!("[target.{}]\n{linker}{hosted}\nrpath = false\n\n", arch.userland()) - }) - .collect(); let config = format!( r#"change-id = "ignore" profile = "compiler" @@ -943,6 +934,7 @@ target = [{targets}] [rust] incremental = true lld = {lld} +{LEAN} [target.{host}] {HOST_LINKER_PIN} @@ -956,10 +948,46 @@ lld = {lld} fs::write(rust_dir.join("bootstrap.toml"), config).unwrap(); } +/// The `[target]` sections of ToyOS userland in the hosted rustc's build: each +/// links with the LLD of the LLVM at `llvm`, and the one the hosted rustc runs +/// on builds it. +fn hosted_targets(llvm: &Path) -> String { + Arch::ALL + .iter() + .map(|arch| { + let linker = format!("linker = \"{}\"", llvm.join("bin/lld").display()); + let hosted = if *arch == HOSTED_ARCH { + // Cranelift because no LLVM is built for a ToyOS host yet, and + // only for that reason: the hosted rustc carries LLVM once clang + // and libc++ run on ToyOS, and Cranelift is not where the + // compiler that builds ToyOS goes. + // + // The archiver is that LLVM's too: the compiler's crates carry + // C built for this target (blake3's assembly), and a host `ar` + // that indexes only its own object format, as macOS's does, + // leaves those ELF members out of the index lld pulls from. + format!( + "\nar = \"{}\"\ncodegen-backends = [\"cranelift\"]", + llvm.join("bin/llvm-ar").display(), + ) + } else { + String::new() + }; + format!("[target.{}]\n{linker}{hosted}\nrpath = false\n\n", arch.userland()) + }) + .collect() +} + /// What the host rustc links its own binaries with, held to one answer in every /// `bootstrap.toml` that builds a host compiler: [`write_config`] says why. pub(crate) const HOST_LINKER_PIN: &str = "default-linker-linux-override = \"off\""; +/// The `[rust]` options every `bootstrap.toml` that builds a compiler shares +/// beyond its profile's: no LLVM tool copied into the compiler's sysroot, since +/// `clang::provision` puts there the ones a build runs; and no debuginfo in +/// rustc, which no build reads. +pub(crate) const LEAN: &str = "llvm-tools = false\ndebuginfo-level-rustc = 0"; + /// The linker every guest target names, as the toolchain at `toolchain` carries /// it: `lib/rustlib//bin/rust-lld`, where rustc itself looks for it. pub fn rust_lld(toolchain: &Path) -> PathBuf { @@ -1137,17 +1165,23 @@ mod tests { assert!(said.contains("clang") && !said.contains("rust-lld,"), "the refusal names clang alone: {said}"); } - /// Every build links the host's LLVM, and every guest links through its - /// LLD, named by path. + /// Every build links the host's LLVM, copies none of its tools and leaves + /// rustc without debuginfo; the host-only one builds no guest target, and + /// in the hosted rustc's each guest links through that LLVM's LLD, named + /// by path. #[test] fn every_build_links_the_host_s_llvm_and_names_its_lld_by_path() { let rust_dir = TempDir::new("lld-config"); let llvm = rust_dir.join("build/llvm/k"); let lld = format!("linker = \"{}\"", llvm.join("bin/lld").display()); - for (hosted, lld_flag) in [(true, "lld = false"), (false, "lld = true")] { + let lean = "\n[rust]\nincremental = true\nlld = "; + for (hosted, lld_flag) in [(true, "false"), (false, "true")] { write_config(&rust_dir, "h", hosted, &llvm); let config = fs::read_to_string(rust_dir.join("bootstrap.toml")).unwrap(); - assert!(config.contains(lld_flag) && config.contains(&lld) && !config.contains("\"rust-lld\""), "{config}"); + assert!(config.contains(&format!("{lean}{lld_flag}\n{LEAN}\n")), "{config}"); + assert_eq!(config.contains(&lld), hosted, "{config}"); + assert!(!config.contains("\"rust-lld\""), "{config}"); + assert_eq!(config.contains("\ntarget = [\"h\"]\n"), !hosted, "{config}"); let host = format!( "[target.h]\ndefault-linker-linux-override = \"off\"\nllvm-config = \"{}/bin/llvm-config\"\nllvm-has-rust-patches = true\n", llvm.display() @@ -1178,7 +1212,13 @@ mod tests { /// The LLVM `clang::provision` reads, in `rust_dir`'s store. fn store_llvm(rust_dir: &Path) -> PathBuf { let llvm = rust_dir.join("build/llvm/k"); - for (file, text) in [("bin/clang", "clang"), ("lib/clang/22/include/stddef.h", "stddef")] { + let files = [ + ("bin/clang", "clang"), + ("bin/llvm-ar", "llvm-ar"), + ("bin/llvm-objcopy", "llvm-objcopy"), + ("lib/clang/22/include/stddef.h", "stddef"), + ]; + for (file, text) in files { fs::create_dir_all(llvm.join(file).parent().unwrap()).unwrap(); fs::write(llvm.join(file), text).unwrap(); } @@ -1186,13 +1226,13 @@ mod tests { } /// What bootstrap leaves in `rust_dir`: `stage2` made again, with `rustc` - /// and the LLVM tools it assembles, and neither cargo nor clang; and the - /// hosted rustc's sysroot without the host target. + /// and `rust-lld` and none of cargo, clang or an LLVM tool; and the hosted + /// rustc's sysroot without the host target. fn bootstrapped(rust_dir: &Path) { let stage2 = stage2(rust_dir); let _ = fs::remove_dir_all(&stage2); let lld = rust_lld(&stage2); - for file in [stage2.join("bin/rustc"), lld.clone(), lld.with_file_name("llvm-ar")] { + for file in [stage2.join("bin/rustc"), lld.clone()] { fs::create_dir_all(file.parent().unwrap()).unwrap(); fs::write(file, b"").unwrap(); } @@ -1264,6 +1304,40 @@ mod tests { assert!(own.iter().all(|dir| dir.join("bin/tool").is_file()), "a stopped bootstrap took the LLVM its compiler linked"); } + /// **The hosted rustc is built only for a build whose config ships it**, + /// and then only when the one this compiler built is not there. + #[test] + fn the_hosted_rustc_is_built_only_when_a_build_ships_it() { + let rust_dir = TempDir::new("hosted-owed"); + let stamp = rust_dir.join("stamps/hosted-rustc.stamp"); + let rustc = hosted_stage2(&rust_dir).join("bin/rustc"); + assert!(!hosted_rustc_owed(false, &stamp, &rustc), "a build that ships no hosted rustc built one"); + assert!(hosted_rustc_owed(true, &stamp, &rustc)); + for file in [&stamp, &rustc] { + fs::create_dir_all(file.parent().unwrap()).unwrap(); + fs::write(file, "").unwrap(); + } + assert!(!hosted_rustc_owed(true, &stamp, &rustc), "a hosted rustc this compiler built was built again"); + fs::remove_file(&stamp).unwrap(); + assert!(hosted_rustc_owed(true, &stamp, &rustc), "a hosted rustc of another compiler was taken"); + } + + /// **A compiler rebuild leaves no hosted rustc of the compiler it + /// replaced**, nor its stamp; with neither there, that is no error. + #[test] + fn a_rebuilt_compiler_leaves_no_hosted_rustc_of_the_one_before() { + let rust_dir = TempDir::new("hosted-stale"); + let stamp = rust_dir.join("stamps/hosted-rustc.stamp"); + let rustc = hosted_stage2(&rust_dir).join("bin/rustc"); + for file in [&stamp, &rustc] { + fs::create_dir_all(file.parent().unwrap()).unwrap(); + fs::write(file, "").unwrap(); + } + forget_hosted_rustc(&rust_dir, &stamp); + assert!(!stamp.exists() && !hosted_stage2(&rust_dir).exists(), "the old compiler's hosted rustc stayed"); + forget_hosted_rustc(&rust_dir, &stamp); + } + /// **The primary bootstraps a new compiler exactly when its `stage2` is not /// current, and otherwise only when rustup has none.** #[test] diff --git a/tests/common/compile.rs b/tests/common/compile.rs index 1ee15db5ac7..3675b5d9587 100644 --- a/tests/common/compile.rs +++ b/tests/common/compile.rs @@ -22,7 +22,7 @@ pub fn c_sysroot() -> CSysroot { static C: OnceLock = OnceLock::new(); C.get_or_init(|| { let mut lock = toyos_build::buildlock::shared(&repo_root(), "the C sysroot"); - let sysroot = toyos_build::toolchain::ensure(&repo_root(), &mut lock); + let sysroot = toyos_build::toolchain::ensure(&repo_root(), &mut lock, false); CSysroot::of(&sysroot.dir, super::qemu::SUITE_ARCH) }) .clone() From 0237a67ec13f167942a59b70f6ec7a6f2fca5b47 Mon Sep 17 00:00:00 2001 From: japabu Date: Thu, 1 Oct 2026 19:28:29 +0200 Subject: [PATCH 08/30] A lean compiler build that runs: no codegen test, and rust-objcopy on PATH on an Apple host The first build of the previous commit on this host stopped twice, and each stop is a fix here; with both, `cargo run -- --build-only` built LLVM c54c833acd75e98b, compiler a4e3d9b05e1747d8, freestanding libraries 8d5c8b9068ca7480, sysroot 3b3ed0fb252fe96d and the image, EXIT=0. - Bootstrap's sanity check demands LLVM's FileCheck beside an external llvm-config while codegen tests are on (src/bootstrap/src/core/sanity.rs:320). No build runs them, so every compiler build says `codegen-tests = false`, and the LLVM store keeps no FileCheck. - The rust workspace strips lld-wrapper (`strip = true` in its Cargo.toml), and on an Apple host rustc strips by running `rust-objcopy`, which the stage-1 sysroot carries only when bootstrap copies LLVM's tools. Stage 2's lld-wrapper failed with "unable to run `rust-objcopy`". `x_build_compiler` puts the LLVM's llvm-objcopy first on the build's PATH as rust-objcopy, on an Apple host alone. Tests: the generated configs are held to the three lean options, a worktree's compiler config too, and a fake bootstrap reports which rust-objcopy its PATH finds first. Clippy's redundant clone in a test fake is gone. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L --- src/compiler.rs | 13 ++++++++- src/toolchain.rs | 74 +++++++++++++++++++++++++++++++++++++++++------- 2 files changed, 76 insertions(+), 11 deletions(-) diff --git a/src/compiler.rs b/src/compiler.rs index 12b09435791..8d786855b91 100644 --- a/src/compiler.rs +++ b/src/compiler.rs @@ -330,7 +330,7 @@ fn build_in_fork(root: &Path, rust_dir: &Path, fork: &Path) -> PathBuf { fs::write(&config, config_text(&build_dir, &host, &llvm.dir)).unwrap_or_else(|e| panic!("write {}: {e}", config.display())); let config = config.to_str().unwrap_or_else(|| panic!("{} is not UTF-8", config.display())); let args = ["build", "--stage", "2", "--config", config, "--warnings", "warn", "compiler/rustc", "library"]; - let (ok, log) = toolchain::x_build(fork, &args, "the compiler"); + let (ok, log) = toolchain::x_build_compiler(fork, &args, "the compiler", &llvm.dir); toolchain::refuse_on_compile_error(&log, "the compiler"); assert!(ok, "the compiler build in {} failed, and nothing in its output was a compile error", fork.display()); let stage2 = build_dir.join(&host).join("stage2"); @@ -772,6 +772,17 @@ pub(crate) mod tests { assert_ne!(key(&fork), tools, "another LLVM commit did not move the key"); } + /// **A compiler of a worktree's own is built as the primary's is**: for + /// the host alone, against the host's LLVM, copying none of its tools, with + /// rustc without debuginfo and no codegen test. + #[test] + fn a_worktree_compiler_is_built_lean_against_the_host_s_llvm() { + let config = config_text(Path::new("/b"), "h", Path::new("/llvm")); + let lean = "\nlld = true\nllvm-tools = false\ndebuginfo-level-rustc = 0\ncodegen-tests = false\n"; + assert!(config.contains(lean) && config.contains("\ntarget = [\"h\"]\n"), "{config}"); + assert!(config.contains("\nllvm-config = \"/llvm/bin/llvm-config\"\n"), "{config}"); + } + /// A primary record naming another LLVM, or none, is another compiler. #[test] fn another_llvm_is_another_compiler() { diff --git a/src/toolchain.rs b/src/toolchain.rs index 8a5fa80df4f..85a2d8aefaa 100644 --- a/src/toolchain.rs +++ b/src/toolchain.rs @@ -696,6 +696,27 @@ pub(crate) fn x_build(rust_dir: &Path, args: &[&str], what: &str) -> (bool, Vec< x_build_with(rust_dir, args, what, |_| {}) } +/// [`x_build`] of a compiler that links the LLVM at `llvm`. On an Apple host +/// rustc strips a Darwin binary by running `rust-objcopy` (`rustc_codegen_ssa`'s +/// `back/link.rs`), the rust workspace strips `lld-wrapper`, and the stage-1 +/// sysroot carries no `rust-objcopy` when bootstrap copies none of LLVM's tools +/// ([`LEAN`]): the build finds that LLVM's `llvm-objcopy` by that name on `PATH`. +pub(crate) fn x_build_compiler(rust_dir: &Path, args: &[&str], what: &str, llvm: &Path) -> (bool, Vec) { + if !host_triple().ends_with("apple-darwin") { + return x_build(rust_dir, args, what); + } + let strip = toyos_tmpdir::TempDir::new("rust-objcopy"); + let objcopy = llvm.join("bin").join(crate::llvm::APPLE_TOOL); + std::os::unix::fs::symlink(&objcopy, strip.join("rust-objcopy")) + .unwrap_or_else(|e| panic!("link {} as rust-objcopy: {e}", objcopy.display())); + let caller = std::env::var_os("PATH").unwrap_or_else(|| panic!("PATH is unset, and bootstrap finds its tools on it")); + let path = std::env::join_paths(std::iter::once(strip.to_path_buf()).chain(std::env::split_paths(&caller))) + .unwrap_or_else(|e| panic!("{} cannot lead PATH: {e}", strip.display())); + x_build_with(rust_dir, args, what, |command| { + command.env("PATH", path); + }) +} + /// [`x_build`], with bootstrap's environment what `environment` makes of this /// process's. pub(crate) fn x_build_with( @@ -835,7 +856,7 @@ fn full_bootstrap(rust_dir: &Path, llvm: &Path) { } } - let (ok, log) = x_build(rust_dir, &COMPILER_BUILD, "the toolchain"); + let (ok, log) = x_build_compiler(rust_dir, &COMPILER_BUILD, "the toolchain", llvm); refuse_on_compile_error(&log, "the toolchain"); assert!(ok, "the toolchain build failed, and nothing in its output was a compile error"); } @@ -846,7 +867,7 @@ fn build_hosted_rustc(rust_dir: &Path, llvm: &Path) { write_config(rust_dir, &host, true, llvm); let (ok, log) = - x_build(rust_dir, &["build", "--stage", "2", "--warnings", "warn"], "the hosted rustc"); + x_build_compiler(rust_dir, &["build", "--stage", "2", "--warnings", "warn"], "the hosted rustc", llvm); refuse_on_compile_error(&log, "the hosted rustc"); // rustdoc for ToyOS may fail to link; rustc and librustc_driver may not. @@ -873,7 +894,7 @@ fn build_hosted_rustc(rust_dir: &Path, llvm: &Path) { // (`write_config` says why), so the host-only build runs once more to put // it back: everything it would compile is already built. write_config(rust_dir, &host, false, llvm); - let (ok, log) = x_build(rust_dir, &COMPILER_BUILD, "the toolchain, reassembled"); + let (ok, log) = x_build_compiler(rust_dir, &COMPILER_BUILD, "the toolchain, reassembled", llvm); refuse_on_compile_error(&log, "the toolchain, reassembled"); assert!( rust_lld(&stage2(rust_dir)).is_file(), @@ -984,9 +1005,10 @@ pub(crate) const HOST_LINKER_PIN: &str = "default-linker-linux-override = \"off\ /// The `[rust]` options every `bootstrap.toml` that builds a compiler shares /// beyond its profile's: no LLVM tool copied into the compiler's sysroot, since -/// `clang::provision` puts there the ones a build runs; and no debuginfo in -/// rustc, which no build reads. -pub(crate) const LEAN: &str = "llvm-tools = false\ndebuginfo-level-rustc = 0"; +/// `clang::provision` puts there the ones a build runs; no debuginfo in rustc, +/// which no build reads; and no codegen test, for which bootstrap demands +/// LLVM's `FileCheck` beside `llvm-config` (`src/bootstrap/src/core/sanity.rs`). +pub(crate) const LEAN: &str = "llvm-tools = false\ndebuginfo-level-rustc = 0\ncodegen-tests = false"; /// The linker every guest target names, as the toolchain at `toolchain` carries /// it: `lib/rustlib//bin/rust-lld`, where rustc itself looks for it. @@ -1100,6 +1122,39 @@ mod tests { panic!("re-locked both, and failed"); } + /// **A compiler build on an Apple host finds its LLVM's `llvm-objcopy` as + /// the `rust-objcopy` its stage-1 rustc strips with**, first on its `PATH`; + /// on any other host it finds none of ours. `./x` here is this test binary, + /// running [`a_fake_bootstrap_that_strips`]. + #[test] + fn a_compiler_build_finds_the_llvm_s_objcopy_where_rustc_strips_with_it() { + let fork = TempDir::new("x-build-strip"); + fs::create_dir_all(fork.join("library")).unwrap(); + for lock in ["Cargo.lock", "library/Cargo.lock"] { + fs::write(fork.join(lock), "# as committed\n").unwrap(); + } + fs::write(fork.join(FAKE), "").unwrap(); + std::os::unix::fs::symlink(std::env::current_exe().unwrap(), fork.join("x")).unwrap(); + let llvm = fork.join("llvm"); + fs::create_dir_all(llvm.join("bin")).unwrap(); + fs::write(llvm.join("bin").join(crate::llvm::APPLE_TOOL), "the llvm-objcopy").unwrap(); + + let args = ["--exact", "toolchain::tests::a_fake_bootstrap_that_strips", "--include-ignored", "--nocapture"]; + let (ok, log) = x_build_compiler(&fork, &args, "a fake bootstrap", &llvm); + assert!(ok, "the fake bootstrap did not run: {log:?}"); + let found = if host_triple().ends_with("apple-darwin") { "the llvm-objcopy" } else { "none" }; + assert!(log.iter().any(|l| *l == format!("rust-objcopy: {found}")), "{log:?}"); + } + + #[test] + #[ignore = "the bootstrap `a_compiler_build_finds_the_llvm_s_objcopy_where_rustc_strips_with_it` runs; never runs on its own"] + fn a_fake_bootstrap_that_strips() { + assert!(Path::new(FAKE).is_file(), "a_fake_bootstrap_that_strips ran outside a fake fork checkout; it is not a test"); + let path = std::env::var_os("PATH").unwrap_or_default(); + let first = std::env::split_paths(&path).map(|dir| dir.join("rust-objcopy")).find(|tool| tool.exists()); + println!("rust-objcopy: {}", first.map_or("none".to_string(), |tool| fs::read_to_string(tool).unwrap())); + } + /// **A restore that cannot write leaves the file as it found it**, and /// names what it could not write. #[test] @@ -1174,11 +1229,11 @@ mod tests { let rust_dir = TempDir::new("lld-config"); let llvm = rust_dir.join("build/llvm/k"); let lld = format!("linker = \"{}\"", llvm.join("bin/lld").display()); - let lean = "\n[rust]\nincremental = true\nlld = "; + let lean = "\nllvm-tools = false\ndebuginfo-level-rustc = 0\ncodegen-tests = false\n"; for (hosted, lld_flag) in [(true, "false"), (false, "true")] { write_config(&rust_dir, "h", hosted, &llvm); let config = fs::read_to_string(rust_dir.join("bootstrap.toml")).unwrap(); - assert!(config.contains(&format!("{lean}{lld_flag}\n{LEAN}\n")), "{config}"); + assert!(config.contains(&format!("\n[rust]\nincremental = true\nlld = {lld_flag}{lean}")), "{config}"); assert_eq!(config.contains(&lld), hosted, "{config}"); assert!(!config.contains("\"rust-lld\""), "{config}"); assert_eq!(config.contains("\ntarget = [\"h\"]\n"), !hosted, "{config}"); @@ -1231,8 +1286,7 @@ mod tests { fn bootstrapped(rust_dir: &Path) { let stage2 = stage2(rust_dir); let _ = fs::remove_dir_all(&stage2); - let lld = rust_lld(&stage2); - for file in [stage2.join("bin/rustc"), lld.clone()] { + for file in [stage2.join("bin/rustc"), rust_lld(&stage2)] { fs::create_dir_all(file.parent().unwrap()).unwrap(); fs::write(file, b"").unwrap(); } From c1564508b08d0650c4dc337cd4bcb9e654d2ea95 Mon Sep 17 00:00:00 2001 From: japabu Date: Thu, 1 Oct 2026 19:39:29 +0200 Subject: [PATCH 09/30] issues: the two reds main's guest lanes found stay until main's nightly meets their exits Round 2 deleted both because #675 and #676 were to land in one batch with this branch and close them. The owner has since ruled that #675 and #676 land on their own reviews, with a nightly run on main to confirm them, so nothing closes these two by this branch's landing. They are restored as round 2 found them, at 90a989e20^: when this branch lands, each goes only if main's nightly has met its exit, and otherwise stays, corrected. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L --- ...es-at-the-kernels-entry-on-cis-firmware.md | 37 +++++++++++++++++++ ...erleaves-with-another-cpus-console-line.md | 28 ++++++++++++++ 2 files changed, 65 insertions(+) create mode 100644 issues/kernel/every-aarch64-guest-dies-at-the-kernels-entry-on-cis-firmware.md create mode 100644 issues/kernel/the-nested-nmi-report-interleaves-with-another-cpus-console-line.md diff --git a/issues/kernel/every-aarch64-guest-dies-at-the-kernels-entry-on-cis-firmware.md b/issues/kernel/every-aarch64-guest-dies-at-the-kernels-entry-on-cis-firmware.md new file mode 100644 index 00000000000..f2dc2ea2efb --- /dev/null +++ b/issues/kernel/every-aarch64-guest-dies-at-the-kernels-entry-on-cis-firmware.md @@ -0,0 +1,37 @@ +--- +status: open +kind: defect +opened: 2026-10-01 +--- + +# Every AArch64 guest dies at the kernel's entry on CI's firmware + +All sixteen `virt_*` tests red in CI. Each boot's console ends the same way: +`Loader log: the kernel handoff begins`, then the firmware's +`Synchronous Exception at 0x00000000BC33EB20`. The PC differs per kernel build, +but always falls inside the kernel image the loader placed at `0xbc200000`. That +happens at EL1 entry (`Profile::Virt`) and at EL2 entry (`VirtEl2`), on one CPU +and on eight. The kernel prints nothing first, so the tests time out waiting for +their first marker. + +**Evidence**, identical in two runs on toolchain +`toolchain-linux-x86_64-48dd24f826263d6c`: +- Main's nightly `guest` lane at `06788146b`, run 36843762360, job 110374194368. +- PR #671's `guest` check, on its merge onto `59052827f`, run 36863809437, job + 110375742604. + +Both ran QEMU 11.1.1 under TCG `-cpu max` on an AMD EPYC 9V45 with 4 cores. The +firmware was Debian's `AAVMF_CODE.no-secboot.fd`, "version 2026.05-2". Both +logged `test result: FAILED. 4 passed, 17 failed`. The other red is +`issues/kernel/the-nested-nmi-report-interleaves-with-another-cpus-console-line.md`. + +The same sixteen tests pass on the dev host, in a whole-suite run of #670's +branch (`test result: ok. 21 passed`). That host ran QEMU 11.1.1 from +Homebrew, under the same TCG `-cpu max` for `VirtEl2`. Two parts of the +instrument differ: +- The firmware: the dev host runs QEMU's bundled `edk2-stable202408-prebuilt.qemu.org`. +- The toolchain: the dev host builds its own, and CI installs the published release. + +`.github/qemu-version` pins neither of the two. + +**Exit:** the sixteen `virt_*` tests are green in CI's `guest` check. diff --git a/issues/kernel/the-nested-nmi-report-interleaves-with-another-cpus-console-line.md b/issues/kernel/the-nested-nmi-report-interleaves-with-another-cpus-console-line.md new file mode 100644 index 00000000000..b5b68dcdbe4 --- /dev/null +++ b/issues/kernel/the-nested-nmi-report-interleaves-with-another-cpus-console-line.md @@ -0,0 +1,28 @@ +--- +status: open +kind: defect +opened: 2026-10-01 +--- + +# The nested-NMI report interleaves with another CPU's console line + +`nested_nmi` (`kernel/src/arch/x86_64/idt/nmi.rs`) writes its report through +`serial::panic_raw`, which takes no lock. When cpu1 is writing its own record at +the same moment, the two lines interleave byte by byte on the 16550. The cpu1 +line was `[kernel 0.385 cpu1] CPU 1: joining scheduler`, and the 16550 carried: + + [[kenrnmel i0.38]5 cpNu1E] CSPUT 1E: Djo inNiMngI s choednule r + +`NESTED NMI` is never whole on the console, so `nested_nmi_is_loud` times out +waiting for it, and the machine halts with its report unreadable. + +**Evidence:** red under KVM in two runs, with byte-identical interleaving: +- Main's nightly `guest` lane at `06788146b`, run 36843762360, job 110374194368. +- PR #671's `guest` check, on its merge onto `59052827f`, run 36863809437, job + 110375742604. + +It is green under TCG: on the dev host in a whole-suite run of #670's branch, +and on a runner in main's nightly `tcg` lane at `06788146b` (job 110374194382). + +**Exit:** `nested_nmi_is_loud` is green in CI's KVM `guest` check, and a report +written while another CPU is writing a record reads whole. From 6441727434a1a10209fb6e44596bfb8c5656b472 Mon Sep 17 00:00:00 2001 From: japabu Date: Thu, 1 Oct 2026 20:40:47 +0200 Subject: [PATCH 10/30] Toolchain stores are cache entries by the build system's own keys, and the guest jobs keep no target cache CI now caches the toolchain as the four stores the build system already keeps: the LLVM, the primary's compiler, the freestanding targets' libraries and the sysroot. Each one is an actions/cache entry keyed by the build system's own key. GitHub's ref scoping is the provenance. Main's push and main's nightly save into main's scope, which every ref restores. A pull request's run saves only into its own scope, and so does a merge group's. toolchain.yml: `--ci toolchain` clones `rust/` at depth 1 (the helper the licence gate already used, now shared in src/lib.rs) and writes each layer's entry and paths as step outputs. Four restore steps take exactly those. `--ci bootstrap` builds what none restored: nothing at all when the sysroot was restored, since that is all a guest job reads. It refuses a restored layer that is not whole, because a rebuild under that key could never be saved over the entry. It tells each save step `built` or `kept`, and the save's guard reads that. The disk, QEMU and CMake step goes: - the apt CMake 3.28.3 sat behind the image's own /usr/local/bin/cmake 3.31.6 on PATH; - a toolchain build boots nothing; - portability-linux builds the whole toolchain on the same runner without the cleanup (job 110308854428's run, 36843762360). guest.yml keeps no target cache and no registry cache. A cold guest job fetched its 65 crates in under two seconds (tcg job 110374194382). It restores the sysroot by the toolchain job's key, red on a miss, and `release::install` lays that store out as the installed toolchain, after holding its recorded witness to the tree's. The job holds no token. Removed: the artifact listing, `is_mains`, vouching, `choose`, the digest install, the release tag and BUILDERS. The nightly's `release` packs main's restored sysroot, one build and not two. `publish.yml` keeps main's `toolchain` and loses `release`. - The tarball is packed in-process with the tar crate and ruzstd, with sorted entries and no owner or time, so one sysroot packs to one digest. - GitHub's REST API is spoken through curl with the toyos-build user agent. gh, tar and zstd no longer run from ToyOS code. - `put` decides create, carried, upload or replace from the release's JSON. Another writer's asset is replaced. - The release runs only as nightly.yml on main, scheduled or dispatched, and only at main's tip. Keys read what their builds read: - A compiler's key reads its whole build: RECIPE, the bootstrap configuration (`compiler::config_text`, which the primary now writes too) and the tools clang::provision puts beside it. - The primary's record names that build and every KEYED source, not just `compiler/`. It stays git-based: `source` takes 325 ms against `key`'s 850 ms on the development host, and the primary asks it on every build. - The freestanding key reads the compiler's key (`Compiler::key`, its record) in place of its driver's mtime, so every key is known before any store is built. - The LLVM key names n2, the Ninja its build runs, by its pin. - The sysroot key reads libc's cargo invocations, its lockfile and userland's cargo configuration. A stage2 whose rustc runs is linked, not rebuilt, when rustup has no `toyos`, as on a runner that restored it. Bootstrap never sees GITHUB_ACTIONS or CI, from any caller (x_build_with). Gates (src/ci.rs): - only main's runs save what other refs restore; - no low-trust trigger; - no widened cache-mode; - `guest / suite` has no condition of its own; - every job that saves holds to the allow-list; - the one write token is the nightly release's, read whatever YAML spells it; - each store is restored and saved by the entry its job wrote. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L --- .github/workflows/ci.yml | 5 +- .github/workflows/guest.yml | 38 +- .github/workflows/nightly.yml | 24 +- .github/workflows/publish.yml | 24 - .github/workflows/toolchain.yml | 84 +- Cargo.lock | 37 + Cargo.toml | 4 + ...t-is-whatever-its-last-writer-put-there.md | 13 +- ...d-runs-host-tools-outside-rust-and-qemu.md | 14 +- ...se-is-compressed-at-zstds-fastest-level.md | 21 + issues/build/toyos-is-a-normal-target.md | 6 +- ...chain-releases-report-the-same-rustc-vv.md | 2 +- src/ci.rs | 368 ++++- src/clang.rs | 2 +- src/compiler.rs | 129 +- src/keystore.rs | 7 + src/lib.rs | 15 + src/libc.rs | 32 +- src/licence.rs | 7 +- src/llvm.rs | 9 +- src/release.rs | 1290 ++++++++--------- src/sysroot.rs | 107 +- src/toolchain.rs | 171 ++- 23 files changed, 1427 insertions(+), 982 deletions(-) create mode 100644 issues/build/the-toolchain-release-is-compressed-at-zstds-fastest-level.md diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index d72f914e076..fbc9fdb99f9 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -13,7 +13,6 @@ concurrency: # No job here writes: a toolchain release is main's publisher's alone. permissions: contents: read - actions: read jobs: host: @@ -46,10 +45,12 @@ jobs: uses: ./.github/workflows/toolchain.yml # A required check that is skipped reads as green, so this runs whatever - # `toolchain` concluded: a tree no build answers for reds the install. + # `toolchain` concluded. guest: needs: toolchain if: ${{ !cancelled() && (github.event_name == 'merge_group' || github.event.pull_request.draft == false) }} uses: ./.github/workflows/guest.yml with: kvm: true + sysroot-key: ${{ needs.toolchain.outputs.sysroot-key }} + sysroot-path: ${{ needs.toolchain.outputs.sysroot-path }} diff --git a/.github/workflows/guest.yml b/.github/workflows/guest.yml index cf8a064a3a6..1eb8c1cb7a5 100644 --- a/.github/workflows/guest.yml +++ b/.github/workflows/guest.yml @@ -8,10 +8,12 @@ on: kvm: type: boolean required: true - # The guest cache's one writer is the lane that asks. - save-cache: - type: boolean - default: false + sysroot-key: + type: string + required: true + sysroot-path: + type: string + required: true jobs: suite: @@ -24,8 +26,6 @@ jobs: container: image: debian:sid-20260918@sha256:a2aa46262453eba3f464d8b1c7a8c31db85eb15af180ae34dd400615d7208547 options: ${{ inputs.kvm && '--device=/dev/kvm' || '' }} - env: - GH_TOKEN: ${{ github.token }} steps: # Before the checkout, which wants git. Three attempts, because the # archive is fixed and the network to it is not. @@ -36,6 +36,7 @@ jobs: test -n "$snap" echo "deb $snap sid main" > /etc/apt/sources.list rm /etc/apt/sources.list.d/debian.sources + # `zstd`: what the sysroot's restore unpacks the toolchain job's entry with. for attempt in 1 2 3; do apt-get -o Acquire::Check-Valid-Until=false update -qq > /tmp/apt.log 2>&1 \ && DEBIAN_FRONTEND=noninteractive apt-get install -y -qq git curl ca-certificates \ @@ -51,35 +52,16 @@ jobs: sh "$RUNNER_TEMP/rustup-init.sh" -y --profile minimal --default-toolchain stable echo "$HOME/.cargo/bin" >> "$GITHUB_PATH" - # The whole history: which builds this tree installs is read off it. - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 - with: - fetch-depth: 0 - uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 with: - path: &guest-paths | - ~/.cargo/registry/index - ~/.cargo/registry/cache - ~/.cargo/git/db - target - kernel/target - bootloader/target - userland/target - tests/target - tests/toyos-rust-tests/*/target - key: guest-${{ github.run_id }} - restore-keys: guest- + path: ${{ inputs.sysroot-path }} + key: ${{ inputs.sysroot-key }} + fail-on-cache-miss: true - run: cargo run -- --ci guest - # After the test: what is worth keeping is a tree that built and booted. - - if: inputs.save-cache - uses: actions/cache/save@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 - with: - path: *guest-paths - key: guest-${{ github.run_id }} - # Every boot's 16550 log: what a guest that died early still leaves. - name: serial logs if: failure() diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index 2059bf1497f..0ee218ce736 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -8,14 +8,12 @@ on: - cron: '0 3 * * *' workflow_dispatch: -# Never cancelled: `toolchain` may be an hour into a bootstrap. concurrency: group: nightly-${{ github.ref }} cancel-in-progress: false permissions: contents: read - actions: read jobs: host: @@ -53,13 +51,33 @@ jobs: toolchain: uses: ./.github/workflows/toolchain.yml + release: + needs: toolchain + runs-on: ubuntu-24.04 + timeout-minutes: 30 + permissions: + contents: write + steps: + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + + - uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 + with: + path: ${{ needs.toolchain.outputs.sysroot-path }} + key: ${{ needs.toolchain.outputs.sysroot-key }} + fail-on-cache-miss: true + + - env: + GH_TOKEN: ${{ github.token }} + run: cargo run -- --ci release + tcg: needs: toolchain if: ${{ !cancelled() }} uses: ./.github/workflows/guest.yml with: kvm: false - save-cache: ${{ github.ref == 'refs/heads/main' }} + sysroot-key: ${{ needs.toolchain.outputs.sysroot-key }} + sysroot-path: ${{ needs.toolchain.outputs.sysroot-path }} # `cargo run -- --build-only` from a fresh machine. `sid` as it stands, # image and archive both, and no cache — a fresh machine is the premise. diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 48b251a25bb..43909793a6d 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -36,34 +36,10 @@ jobs: CARGO_REGISTRY_TOKEN: ${{ steps.auth.outputs.token }} run: cargo run -- --ci publish - # Main's own build of its toolchain, which only main's publisher's runs make - # for main. One at a time: a second would bootstrap the same tag again. toolchain: concurrency: group: publish-toolchain cancel-in-progress: false permissions: contents: read - actions: read uses: ./.github/workflows/toolchain.yml - - # Main's build put up as the release a consumer outside CI installs, and the - # SDK alias moved onto it once crates.io holds this tree's crates: the one job - # any workflow gives a token that writes this repository. - release: - needs: [publish, toolchain] - if: ${{ !cancelled() }} - runs-on: ubuntu-24.04 - timeout-minutes: 30 - concurrency: - group: publish-release - cancel-in-progress: false - permissions: - contents: write - actions: read - steps: - - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 - - - env: - GH_TOKEN: ${{ github.token }} - run: cargo run -- --ci release diff --git a/.github/workflows/toolchain.yml b/.github/workflows/toolchain.yml index 55f2d8ffa77..4048cfb62a6 100644 --- a/.github/workflows/toolchain.yml +++ b/.github/workflows/toolchain.yml @@ -1,44 +1,70 @@ name: toolchain -# A tree's toolchain, bootstrapped only where no build answers for its tag, and -# kept as this run's artifact (src/release.rs). Nothing here publishes, and the -# job holds only the token its caller gives it. - on: workflow_call: + outputs: + sysroot-key: + value: ${{ jobs.build.outputs.sysroot-key }} + sysroot-path: + value: ${{ jobs.build.outputs.sysroot-path }} jobs: build: # Bare, not a container: its glibc is a build's floor. runs-on: ubuntu-24.04 timeout-minutes: 350 - env: - GH_TOKEN: ${{ github.token }} + outputs: + sysroot-key: ${{ steps.keys.outputs.sysroot-key }} + sysroot-path: ${{ steps.keys.outputs.sysroot-path }} steps: - # The whole history: which builds answer for this tree is read off it. - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 - with: - fetch-depth: 0 - - id: find + - id: keys run: cargo run -- --ci toolchain - # What a bootstrap needs, and only when there is one to do. - - name: disk, QEMU and CMake - if: steps.find.outputs.bootstrap == 'true' - run: | - sudo rm -rf /usr/local/lib/android /usr/share/dotnet /opt/ghc \ - /usr/local/share/boost /usr/local/.ghcup - sudo apt-get update -qq - sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq qemu-system-x86 zstd \ - cmake=3.28.3-1build7 - - - run: cargo run -- --ci bootstrap - - # Whole, not zipped: GitHub's digest of the artifact is then the - # tarball's own, which an install holds its download to. - - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 - with: - path: target/toolchain/*.tar.zst - archive: false - if-no-files-found: ignore + - uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 + with: + path: ${{ steps.keys.outputs.llvm-path }} + key: ${{ steps.keys.outputs.llvm-key }} + + - uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 + with: + path: ${{ steps.keys.outputs.compiler-path }} + key: ${{ steps.keys.outputs.compiler-key }} + + - uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 + with: + path: ${{ steps.keys.outputs.freestanding-path }} + key: ${{ steps.keys.outputs.freestanding-key }} + + - uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 + with: + path: ${{ steps.keys.outputs.sysroot-path }} + key: ${{ steps.keys.outputs.sysroot-key }} + + - id: build + run: cargo run -- --ci bootstrap + + - if: steps.build.outputs.llvm == 'built' + uses: actions/cache/save@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 + with: + path: ${{ steps.keys.outputs.llvm-path }} + key: ${{ steps.keys.outputs.llvm-key }} + + - if: steps.build.outputs.compiler == 'built' + uses: actions/cache/save@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 + with: + path: ${{ steps.keys.outputs.compiler-path }} + key: ${{ steps.keys.outputs.compiler-key }} + + - if: steps.build.outputs.freestanding == 'built' + uses: actions/cache/save@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 + with: + path: ${{ steps.keys.outputs.freestanding-path }} + key: ${{ steps.keys.outputs.freestanding-key }} + + - if: steps.build.outputs.sysroot == 'built' + uses: actions/cache/save@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 + with: + path: ${{ steps.keys.outputs.sysroot-path }} + key: ${{ steps.keys.outputs.sysroot-key }} diff --git a/Cargo.lock b/Cargo.lock index cc05a45c729..8eadadd3255 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -255,6 +255,16 @@ version = "0.2.9" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "28dea519a9695b9977216879a3ebfddf92f1c08c05d984f8996aecd6ecdc811d" +[[package]] +name = "filetime" +version = "0.2.29" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5c287a33c7f0a620c38e641e7f60827713987b3c0f26e8ddc9462cc69cf75759" +dependencies = [ + "cfg-if", + "libc", +] + [[package]] name = "find-msvc-tools" version = "0.1.9" @@ -696,6 +706,15 @@ version = "1.0.22" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b39cdef0fa800fc44525c84ccb54a029961a8215f9619753635a9c0d2538d46d" +[[package]] +name = "ruzstd" +version = "0.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a252f5e20f038fe7b4ea53e073e65398d652c864cc162fc77c56c2f13717b888" +dependencies = [ + "twox-hash", +] + [[package]] name = "scoped-tls" version = "1.0.1" @@ -821,6 +840,16 @@ dependencies = [ "unicode-ident", ] +[[package]] +name = "tar" +version = "0.4.46" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3f6221d9a6003c78398e3b239969f352578258df48c8eb051caadae0015bc840" +dependencies = [ + "filetime", + "libc", +] + [[package]] name = "thiserror" version = "2.0.18" @@ -940,9 +969,11 @@ dependencies = [ "gpt", "image", "libc", + "ruzstd", "serde", "serde_json", "sha2", + "tar", "toml", "toyos-abi", "toyos-blackbox", @@ -1286,6 +1317,12 @@ version = "0.21.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2c591d83f69777866b9126b24c6dd9a18351f177e49d625920d19f989fd31cf8" +[[package]] +name = "twox-hash" +version = "2.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5283634e518fe9e82c7b20520bb4bc209009fd16c82077c802f8111ecbb0117a" + [[package]] name = "typenum" version = "1.19.0" diff --git a/Cargo.toml b/Cargo.toml index 318e42c520e..07f2acaf259 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -163,6 +163,10 @@ image = { version = "0.25", default-features = false, features = ["jpeg"] } # The digest behind `NOTICE`'s record of every committed binary file # (`src/sourcegate.rs`). sha2 = "0.10" +# The toolchain release's tarball, packed in-process (`src/release.rs`): Rust's +# own tar and zstd, where the binaries are hosts' tools. +tar = { version = "0.4.46", default-features = false } +ruzstd = "0.9.0" [dev-dependencies] # `USER_TOP`, so the harness judges a held `rsp` against the bound the kernel diff --git a/issues/build/a-toolchain-releases-asset-is-whatever-its-last-writer-put-there.md b/issues/build/a-toolchain-releases-asset-is-whatever-its-last-writer-put-there.md index 2d481126f3a..eb6fe43cc07 100644 --- a/issues/build/a-toolchain-releases-asset-is-whatever-its-last-writer-put-there.md +++ b/issues/build/a-toolchain-releases-asset-is-whatever-its-last-writer-put-there.md @@ -6,11 +6,9 @@ opened: 2026-10-01 # A toolchain release's asset is whatever its last writer put there -CI installs no release. It installs a build whose bytes hash to the digest -GitHub recorded at upload, made by main's publisher or by a run of a commit its -tree vouches for (`src/release.rs`). The release that main's publisher puts up -is what a consumer outside CI installs, and so is the SDK alias that names it. -The release notes' install steps take the asset as it is served. +The release that main's publisher puts up is what a consumer outside CI +installs, and so is the SDK alias that names it. The release notes' install +steps take the asset as it is served. A branch's workflows decide their own token's permissions. A workflow on any branch of this repository can ask for `contents: write` and then replace that @@ -21,12 +19,11 @@ their branches' toolchains under the nightly's write token: `wt/toyos-castore` published `toolchain-linux-x86_64-688e609acf5a65c4` (run 36709239346), and `wt/toyos-notiers` published `toolchain-linux-x86_64-92f146618d6687d8` (run 36600425263). Every toolchain release is mutable (`immutable: -false`). `publish.yml` puts main's build back only on main's next push. +false`). Owner: the release module (`src/release.rs`). **Exit**: a consumer outside CI installs only the bytes main's publisher recorded, and its install refuses any other bytes by name. That holds when a published release can no longer change (with the SDK alias made a new release -per move, never one moved), or when the release notes' install checks the -digest that main's publisher's artifact carries. +per move, never one moved). diff --git a/issues/build/the-build-runs-host-tools-outside-rust-and-qemu.md b/issues/build/the-build-runs-host-tools-outside-rust-and-qemu.md index 1b9b23b721b..ac1f9c74bd1 100644 --- a/issues/build/the-build-runs-host-tools-outside-rust-and-qemu.md +++ b/issues/build/the-build-runs-host-tools-outside-rust-and-qemu.md @@ -17,19 +17,18 @@ arrives and is not one. M4 and M5 are stages of `issues/build/toyos-builds-itsel | Python | LLVM's CMake, whenever this host builds an LLVM (`src/llvm.rs`); the C++ runtime's CMake, in every sysroot build (`src/libcxx.rs`), and its build, which runs `libcxx/utils/generate_iwyu_mapping.py` for a header it installs | admitted: no Rust tool does the job, LLVM's CMake requires one (`find_package(Python3 … REQUIRED)` in `rust/src/llvm-project/llvm/CMakeLists.txt`), and so does the runtimes', unconditionally (`runtimes/CMakeLists.txt`): configured as `src/libcxx.rs` does with no Python on `PATH` or in CMake's search, it found none and stopped, exit 1, and with only `python3` added it configured, exit 0 | M5 runs it in the guest | | CMake | rustc's bootstrap, for LLVM and clang; `src/llvm.rs`, for the LLVM's key; `src/libcxx.rs`, for the C++ runtime in every sysroot build | admitted: no Rust tool does the job, LLVM, clang and LLD are described in CMake, and upstream's only other descriptions are a GN overlay it does not support and a Bazel one | M5 runs it in the guest | | `sh` running LLVM's `config.guess`, and the POSIX tools and `cc` it runs | LLVM's CMake, whenever this host builds an LLVM, and the C++ runtime's, in every sysroot build, ask it the host's triple, unconditionally (`get_host_triple` in `rust/src/llvm-project/llvm/cmake/modules/GetHostTriple.cmake`, which runs `sh` by name) | refused: a Rust tool does the shell's part, brush 0.4.0: on the development host (macOS, arm64) `config.guess` printed `/bin/sh`'s triple under it, `arm64-apple-darwin27.0.0`, exit 0 each. The script runs `sed`, `uname`, `mktemp`, `grep`, `rm`, `rmdir` and `cc` there under either shell, and that `cc` is the `cc` rows'. Five of the other six are refused, uutils' doing each: under brush with sed 0.2.0, grep 0.2.0 and coreutils 0.12.0's `mktemp`, `rm` and `rmdir`, and nothing else on `PATH` but the host's `uname` and `cc`, it printed that triple, exit 0. `uname` is admitted: coreutils 0.12.0's answers `-p` with `unknown` where macOS's answers `arm`, and `config.guess` reads that as PowerPC, `powerpc-apple-darwin27.0.0`, exit 0 | CMake finds brush as its `sh`, uutils' `sed`, `grep`, `mktemp`, `rm` and `rmdir`, and a Rust `uname` that answers `-p` as the host's does; or M5 runs it in the guest | -| `git` for worktrees, submodules, checkouts, fixtures and rustc's bootstrap | adds, removes and prunes worktrees (`src/worktree.rs`, `src/sysroot.rs`); updates submodules (`src/lib.rs`, `src/sysroot.rs`, `src/licence.rs`, `src/release.rs`); fetches the fork from the primary's and checks it out (`src/sysroot.rs`); fast-forwards the primary (`src/sync.rs`); makes the tests' fixture repositories; runs inside rustc's bootstrap | admitted: no Rust tool does the job, gitoxide 0.85 adds, removes and prunes no worktree, updates no submodule, stages, resets and pushes nothing, checks out only a fresh clone and fetches a local path by spawning `git`; a fixture must be what `git` makes, and bootstrap runs `git` itself | M4 runs it in the guest | +| `git` for worktrees, submodules, checkouts, fixtures and rustc's bootstrap | adds, removes and prunes worktrees (`src/worktree.rs`, `src/sysroot.rs`); updates submodules (`src/lib.rs`, `src/sysroot.rs`); fetches the fork from the primary's and checks it out (`src/sysroot.rs`); fast-forwards the primary (`src/sync.rs`); makes the tests' fixture repositories; runs inside rustc's bootstrap | admitted: no Rust tool does the job, gitoxide 0.85 adds, removes and prunes no worktree, updates no submodule, stages, resets and pushes nothing, checks out only a fresh clone and fetches a local path by spawning `git`; a fixture must be what `git` makes, and bootstrap runs `git` itself | M4 runs it in the guest | | `git` for reads, a config write, a commit's paths written out, and clones and fetches over HTTPS | `rev-parse`, `show-ref`, `for-each-ref`, `rev-list`, `log`, `branch --contains`, `merge-base`, `ls-tree`, `ls-files`, `cat-file`, `config --get-regexp`, `worktree list`, `status`, `diff`, `ls-remote` and `grep`, in the build system and its tests; `config --global --add safe.directory` in CI's containers; `checkout -- ` through an index of its own, which writes the C++ runtime's sources out of the LLVM commit into the stored LLVM (`src/llvm.rs`); `src/sync.rs`'s fetch of `origin`; every workflow's checkout | refused: a Rust tool does it, gitoxide 0.85, which reads refs, objects, the index, config, worktrees and status, adds a value to a config file and writes it (gix-config 0.58's `File::section_mut_or_create_new`, `SectionMut::push`, `File::write_to`), walks history, diffs, and lists, fetches and clones a remote over HTTPS; `grep` is a search of the files its index names; and gitoxide's CLI 0.59 (gix 0.88) wrote the runtimes' sources of LLVM `849da7d6` into an empty directory, each path's tree through `gix rev parse`, `gix index from-tree` and `gix free index checkout-exclusive`, exit 0 each: the 18759 files `git` writes there, byte for byte and mode for mode | those are gitoxide's | | `cc`, `c++` and `ar` on a Linux host, `build-essential` in CI's containers | rustc links every host binary through `cc`; `cc` and `c++` compile LLVM, clang, LLD and `rustc_llvm` (`src/llvm.rs` names both to bootstrap); `ar` archives what `cc::Build` compiles | admitted: no Rust tool compiles C or C++, or takes rustc's host link | M5: no host in the loop | | the toolchain's own `clang`, `llvm-ar`, `rust-lld` and `llvm-config`, built from `ToyOSOrg/llvm-project` | rustc links every guest binary with `rust-lld`; `clang` compiles the C corpus (`tests/common/compile.rs`) and, with `llvm-ar`, doomgeneric through `cc::Build` (`src/clang.rs`); rustc's bootstrap asks `llvm-config` how to link LLVM | admitted: our fork's C++, which ToyOS can one day build and run; no Rust tool compiles C, `cc::Build` archives with an `ar`, bootstrap reads LLVM through `llvm-config`, and `CLAUDE.md` links everything with `rust-lld` | M5: no host in the loop | -| `ovmf-generic`, `qemu-efi-aarch64` | the x86-64 and AArch64 UEFI firmware of the nightly's guest containers (`src/firmware.rs`), packaged by Debian apart from QEMU | admitted: QEMU's own firmware, and no Rust firmware does its job | the instrument's QEMU carries its own firmware | -| `ca-certificates` | the trust store `git` and `curl` verify against in the nightly's containers | admitted: data both of them need | goes when neither runs there | -| `env` | the two portability jobs run `cargo run -- --build-only` under it, with `GITHUB_ACTIONS` and `CI` unset | refused: a Rust tool does it, the build system, which already runs that build with both removed (`src/release.rs`) | the build system removes both itself | +| `ovmf-generic`, `qemu-efi-aarch64` | the x86-64 and AArch64 UEFI firmware of CI's guest containers (`src/firmware.rs`), packaged by Debian apart from QEMU | admitted: QEMU's own firmware, and no Rust firmware does its job | the instrument's QEMU carries its own firmware | +| `ca-certificates` | the trust store `git` and `curl` verify against in CI's containers | admitted: data both of them need | goes when neither runs there | +| `env` | the two portability jobs run `cargo run -- --build-only` under it, with `GITHUB_ACTIONS` and `CI` unset | refused: a Rust tool does it, the build system, which already runs bootstrap with both removed (`src/toolchain.rs`) | the build system removes both itself | | the T14's Ubuntu and every tool `src/metal.rs` runs on it over `ssh` | the metal loop, on the T14 and never on a development host | outside the rule: recovery equipment on a test machine, not the build's host | they leave with Ubuntu (`issues/boot-media/the-machine-updates-itself-without-ubuntu.md`) | | `sh` running `rust/x`, and Python running `x.py` and `bootstrap.py` | every toolchain build (`src/toolchain.rs`) | refused: a Rust tool does it, upstream's bootstrap binary, which builds with stable cargo, fetches its own stage0 (`rust/src/bootstrap/src/core/download.rs`) and needs no Python | `src/toolchain.rs` runs the bootstrap binary | | `curl` in rustc's bootstrap | fetches the stage0 `rust/src/stage0` pins, for a compiler or LLVM build whose build directory lacks it, whichever bootstrap runs | refused: a Rust tool does it, rustup installs the dated beta the pin names, and bootstrap takes a stage0 through `build.rustc` and `build.cargo`, as `src/sysroot.rs` hands it one | no toolchain build fetches with `curl` | -| `curl` in `src/release.rs` and `src/sdkversion.rs` | GitHub's API: the lookup and download of a toolchain build and the lookup of its release; and the crates.io index; on CI runners only | refused: a Rust tool does it, `ureq`, which `userland/doom/build.rs` already fetches with | those fetches are Rust's | -| `tar` and `zstd` | `src/release.rs` packs and unpacks a toolchain build, on CI runners only | refused: a Rust tool does it, the `tar` crate `userland/doom/build.rs` already unpacks with, and a zstd crate | both are done in Rust, in-process | -| `gh` | `src/release.rs` creates a toolchain release or replaces its asset, and moves the `sdk-` alias, on `publish.yml`'s `release` runner, on main alone | refused: not C or C++ source, it is Go | `src/release.rs` speaks GitHub's REST API itself | +| `curl` in `src/release.rs` and `src/sdkversion.rs` | GitHub's API, which puts the toolchain release up and moves the `sdk-` alias on the nightly's `release` runner, on main alone; and the crates.io index; on CI runners only | refused: a Rust tool does it, `ureq`, which `userland/doom/build.rs` already fetches with | those fetches are Rust's | +| `tar` and `zstd` | `actions/cache` packs and unpacks every cache entry with them, on CI runners; `guest.yml`'s `deps` installs `zstd`, without which the guest's restore names no entry the toolchain job saved | admitted: GitHub's cache action runs them, and CI keeps no store between runs without it | CI keeps no store between runs | | `ssh` | `src/metal.rs` reaches the T14's Ubuntu with it, only in the metal loop | refused: a Rust tool does it, the repository's own russh client `crate::build::ssh_client_host`, which `src/metaltalk.rs` already drives | `src/metal.rs` drives that client, or Ubuntu leaves the loop | | `cc`, `c++`, `ar` and `xcrun` on a macOS host, Apple's Command Line Tools | what the Linux row's tools do, and rustc asks `xcrun` for the SDK on every host link that names no `SDKROOT` (`rust/compiler/rustc_codegen_ssa/src/back/apple.rs`), as `src/llvm.rs` does for the LLVM's key | refused: one host OS alone | M5: no host in the loop | | `diag/flash.sh` | the owner's flash of a stick by hand: `bash`, and the `stat`, `seq`, `tr`, `grep`, `cut` and `sync` it strings together | refused: shell of our own | `issues/build/the-owners-flash-script-runs-diskutil.md` | @@ -41,7 +40,6 @@ arrives and is not one. M4 and M5 are stages of `issues/build/toyos-builds-itsel | `sudo` on macOS | `diag/flash.sh` and the README's macOS flashing steps run `dd` under it | admitted: no Rust tool raises a process to root on macOS; sudo-rs "is targeted for FreeBSD and Linux-based operating systems only" (its README at `89bae8a`) | goes with both flashes by hand | | `sudo` on Linux | the README's Linux flashing steps run `dd` under it | refused: a Rust tool does it, sudo-rs | the README's Linux steps run sudo-rs | | `sh` running rustup's `rustup-init.sh` | CI's rustup installs, fetched with `curl` | refused: a Rust tool does it, rustup's own `rustup-init` binary | a job installs rustup without it | -| `toolchain.yml`, step "disk, QEMU and CMake" | `sudo rm -rf` of five preinstalled SDK directories, then `sudo apt-get update` and `install` | refused: shell of our own | each step is one command | | `guest.yml`, step `deps` | the snapshot archive read out of `debian.sources` with `sed`, checked with `test` and written back with `echo` and `rm`, a three-attempt `apt-get` loop with `sleep` and `cat`, `git config`, and rustup through `curl`, `sh` and `echo` | refused: shell of our own | each step is one command | | `nightly.yml`, job `portability-linux`, step "deps" | the same loop, `git config`, and rustup the same way | refused: shell of our own | each step is one command | | `nightly.yml`, job `portability-macos`, the rustup step | `curl`, `sh rustup-init.sh`, and `echo` into `$GITHUB_PATH` | refused: shell of our own | each step is one command | diff --git a/issues/build/the-toolchain-release-is-compressed-at-zstds-fastest-level.md b/issues/build/the-toolchain-release-is-compressed-at-zstds-fastest-level.md new file mode 100644 index 00000000000..1c16610758c --- /dev/null +++ b/issues/build/the-toolchain-release-is-compressed-at-zstds-fastest-level.md @@ -0,0 +1,21 @@ +--- +status: open +kind: tooling +opened: 2026-10-01 +--- + +# The toolchain release is compressed at zstd's fastest level + +`release::pack` (`src/release.rs`) compresses the release's tarball in-process +with ruzstd 0.9.0, whose encoder implements one level, `Fastest`, which its +documentation calls roughly zstd's level 1. The zstd binary that packed it +before compressed at level 3. Measured on the development host (macOS, arm64) +over one 1,369,952,256-byte tarball of sysroot `3b3ed0fb252fe96d`: ruzstd +wrote 566,444,661 bytes, the `zstd` CLI 379,391,222 at `-3` and 418,323,664 at +`-1`. Every consumer outside CI downloads the difference. + +Owner: the release module (`src/release.rs`). + +**Exit**: the release's tarball is compressed by a Rust encoder at least as +small as zstd's level 3 on the same tarball, ruzstd's `Default` level once it +is implemented. diff --git a/issues/build/toyos-is-a-normal-target.md b/issues/build/toyos-is-a-normal-target.md index e40c54efb34..99f20333fd0 100644 --- a/issues/build/toyos-is-a-normal-target.md +++ b/issues/build/toyos-is-a-normal-target.md @@ -41,11 +41,7 @@ Stages, in order: export PATH="$PATH:$PWD/$stage2/bin" cargo +toyos build --target x86_64-unknown-toyos - `toyos-ld` is in that `bin/` because rustc's ToyOS target names its linker - and finds it on `PATH`, and the release tag is the content hash of - everything the tarball's bytes depend on — the linker and the packaging - among them, so a change to either mints a release rather than reusing one - built without it. The glibc floor is 2.39 — `ubuntu-24.04`'s, the + The glibc floor is 2.39 — `ubuntu-24.04`'s, the image the host half is built on — measured over the shipped binaries and asserted at publish time, so a build on a newer machine is refused rather than published. A program that opens a window also carries a `[patch]` of diff --git a/issues/build/two-toolchain-releases-report-the-same-rustc-vv.md b/issues/build/two-toolchain-releases-report-the-same-rustc-vv.md index 7f9caaad8cc..2c2514e6a20 100644 --- a/issues/build/two-toolchain-releases-report-the-same-rustc-vv.md +++ b/issues/build/two-toolchain-releases-report-the-same-rustc-vv.md @@ -12,7 +12,7 @@ default for that channel is `omit-git-hash = true` (`rust/src/bootstrap/src/core/config/config.rs`, the `omit_git_hash` line). Every release then reports `rustc 1.99.0-dev` with byte-identical `rustc -vV` output — and the release tag hashes more than `rust` anyway: `toyos-abi/src`, -`toyos/src`, `userland/libc/src`, `toyos-ld` and the packaging. +`toyos/src` and `userland/libc/src`. Cargo fingerprints a compile on `rustc -vV`. A consumer that switches releases in a reused target directory sees no compiler change, keeps the old rlibs, and diff --git a/src/ci.rs b/src/ci.rs index 8d9e54294c2..652e1247815 100644 --- a/src/ci.rs +++ b/src/ci.rs @@ -28,10 +28,10 @@ const USAGE: &str = "cargo run -- --ci , where is one of: host every host test: the build system, the harness's own checks, the host workspace, the licences of what ships, clippy, the model controls, userland and the SDK (ci.yml, nightly) - toolchain whether a build of this tree's toolchain answers for it - bootstrap build this tree's toolchain, unless a build answers for it - guest the guest suite, on the build that answers for this tree - release put main's build up as its toolchain release: main's publisher alone + toolchain the cache entry of each store of this tree's toolchain + bootstrap build the stores of this tree's toolchain its job did not restore + guest the guest suite, on the sysroot its job restored + release put the sysroot main's nightly restored up as its toolchain release publish put main's SDK crates on crates.io (publish.yml)"; #[derive(Debug, PartialEq, Eq)] @@ -68,7 +68,7 @@ pub fn dispatch(root: &Path, args: &[String]) { }); let steps = match &job { Job::Host => host(root), - Job::Toolchain => vec![step("a build of this tree's toolchain", || release::toolchain(root))], + Job::Toolchain => vec![step("the stores of this tree's toolchain", || release::toolchain(root))], Job::Bootstrap => vec![step("this tree's toolchain", || release::bootstrap(root))], Job::Guest => guest(root, &suite_args(&["--jobs", "1"])), Job::Release => vec![step("main's toolchain release", || release::release(root))], @@ -868,7 +868,7 @@ fn publish(root: &Path) -> Result { /// Whether `HEAD` is `main`'s tip as `git ls-remote` printed it: a re-run of an /// older push would put older code up under a newer minor. -fn at_tip(ls_remote: &str, head: &str) -> Result<(), String> { +pub(crate) fn at_tip(ls_remote: &str, head: &str) -> Result<(), String> { match ls_remote.split_whitespace().next() { Some(tip) if tip == head => Ok(()), tip => Err(format!( @@ -1043,6 +1043,26 @@ mod tests { .unwrap_or_else(|e| panic!("{name}: {e}")) } + /// Every workflow, by file name. + fn workflows() -> Vec<(String, String)> { + let dir = repo_root().join(".github/workflows"); + let mut all: Vec<(String, String)> = std::fs::read_dir(&dir) + .expect(".github/workflows is readable") + .flatten() + .map(|e| (e.file_name().to_string_lossy().into_owned(), std::fs::read_to_string(e.path()).expect("a readable workflow"))) + .collect(); + all.sort(); + all + } + + /// `line` without the comment that ends it, and nothing of a comment line. + fn uncommented(line: &str) -> &str { + if line.trim_start().starts_with('#') { + return ""; + } + line.find(" #").map_or(line, |at| &line[..at]) + } + /// The lines of job `name` in `text`, empty if it has none. fn job<'a>(text: &'a str, name: &str) -> Vec<&'a str> { let head = format!(" {name}:"); @@ -1053,12 +1073,70 @@ mod tests { .collect() } + /// Each job of a workflow: its name and its lines. + fn jobs(text: &str) -> Vec<(&str, Vec<&str>)> { + let mut jobs: Vec<(&str, Vec<&str>)> = Vec::new(); + for line in text.split_once("\njobs:\n").map_or("", |(_, jobs)| jobs).lines() { + match line.strip_prefix(" ").and_then(|l| l.strip_suffix(':')) { + Some(name) if !name.starts_with([' ', '#']) => jobs.push((name, Vec::new())), + _ => jobs.last_mut().into_iter().for_each(|(_, lines)| lines.push(line)), + } + } + jobs + } + + /// The steps of a job's lines, each its lines from its `- ` on. + fn steps<'a>(lines: &[&'a str]) -> Vec> { + let mut steps: Vec> = Vec::new(); + let mut inside = false; + for &line in lines { + if line.starts_with(" ") && !line.starts_with(" ") && !line.trim_start().starts_with('#') { + inside = line.trim() == "steps:"; + } else if inside && line.starts_with(" - ") { + steps.push(vec![line]); + } else if let Some(step) = steps.last_mut().filter(|_| inside) { + step.push(line); + } + } + steps + } + /// The value of a job's own `:` line. fn field(job: &[&str], key: &str) -> Option { let line = format!(" {key}: "); job.iter().find_map(|l| l.strip_prefix(&line)).map(str::to_string) } + /// The value a step's own `:` line gives, its `- :` among them. + fn at<'a>(step: &[&'a str], key: &str) -> Option<&'a str> { + let (own, first) = (format!(" {key}: "), format!(" - {key}: ")); + step.iter().find_map(|l| l.strip_prefix(&own).or_else(|| l.strip_prefix(&first))) + } + + /// Each event a workflow runs on, with its lines. An `on:` that is not a + /// block of events is refused, so none hides from this reader. + fn events(name: &str, text: &str) -> Vec<(String, Vec)> { + let block = text.split_once("\non:\n").unwrap_or_else(|| panic!("{name}: no `on:` block")).1; + let mut events: Vec<(String, Vec)> = Vec::new(); + for line in block.lines().take_while(|l| l.is_empty() || l.starts_with([' ', '#'])) { + let line = uncommented(line); + if line.trim().is_empty() { + continue; + } + match line.strip_prefix(" ").filter(|l| !l.starts_with(' ')) { + Some(event) => { + let event = event.split(':').next().unwrap_or_default().to_string(); + events.push((event, vec![line.to_string()])); + } + None => match events.last_mut() { + Some((_, lines)) => lines.push(line.to_string()), + None => panic!("{name}: {line:?} under `on:` names no event"), + }, + } + } + events + } + /// A skipped job reads as green to a required check, so `guest` runs on /// every pull request and in the merge queue whatever `toolchain` /// concluded: its condition is `host`'s and `!cancelled()`, and nothing @@ -1080,42 +1158,235 @@ mod tests { assert_eq!(field(&tcg, "uses").as_deref(), Some("./.github/workflows/guest.yml")); } - /// No job a pull request, the merge queue or the nightly runs holds a token - /// that writes this repository: `ci.yml` and `nightly.yml` give their jobs - /// read alone, the workflows they call ask for nothing of their own, and - /// the one `contents: write` is `publish.yml`'s `release`, which main alone - /// triggers and which alone runs `--ci release`. + /// **`guest / suite` has no condition of its own**: it is a required + /// check, and a job its own `if:` skips reads as green to one. #[test] - fn only_mains_publisher_holds_a_write_token() { - let writes = |text: &str| -> Vec { - let grants = |l: &&str| l.trim_end().ends_with(": write") || l.contains("write-all"); - text.lines().filter(grants).map(|l| l.trim().to_string()).collect() - }; - for name in ["ci.yml", "nightly.yml"] { - let text = workflow(name); - assert!(text.contains("\npermissions:\n contents: read\n actions: read\n\n"), "{name}"); - assert!(writes(&text).is_empty() && !text.contains("--ci release"), "{name}: {:?}", writes(&text)); + fn the_required_guest_check_has_no_condition_of_its_own() { + let text = workflow("guest.yml"); + let all = jobs(&text); + let (_, suite) = all.iter().find(|(name, _)| *name == "suite").expect("guest.yml's `suite`"); + let own = suite.iter().find(|l| uncommented(l).starts_with(" if:")); + assert_eq!(own, None, "guest.yml's `suite` can skip itself"); + } + + /// **One job holds a token that writes this repository: the nightly's + /// `release`**, the one job that runs `--ci release`. Every `write` in a + /// workflow but a comment's is read, whatever YAML spells the grant; the + /// only other is crates.io's OIDC grant at the top of `publish.yml`. + #[test] + fn only_the_nightly_release_holds_a_write_token() { + let mut grants = Vec::new(); + let mut releases = Vec::new(); + for (name, text) in workflows() { + let mut job = None; + let mut in_jobs = false; + for line in text.lines() { + in_jobs |= line == "jobs:"; + if let Some(head) = line.strip_prefix(" ").and_then(|l| l.strip_suffix(':')).filter(|h| in_jobs && !h.starts_with([' ', '#'])) { + job = Some(head.to_string()); + } + let code = uncommented(line); + let words = code.split(|c: char| !(c.is_ascii_alphanumeric() || c == '-')); + if words.clone().any(|word| word.starts_with("write")) { + grants.push((name.clone(), job.clone(), code.trim().to_string())); + } + if code.contains("--ci release") { + releases.push((name.clone(), job.clone(), code.trim().to_string())); + } + } + } + let job = |name: &str| Some(name.to_string()); + let line = |name: &str, job: Option, text: &str| (name.to_string(), job, text.to_string()); + assert_eq!( + grants, + [line("nightly.yml", job("release"), "contents: write"), line("publish.yml", None, "id-token: write")] + ); + assert_eq!(releases, [line("nightly.yml", job("release"), "run: cargo run -- --ci release")]); + } + + /// **No workflow runs on a trigger from outside the repository**: + /// `pull_request_target`, `workflow_run`, `issue_comment` and every other + /// event that someone without write access starts runs on main, where a + /// save lands in what every ref restores. + #[test] + fn no_workflow_runs_on_a_low_trust_trigger() { + const TRUSTED: [&str; 6] = ["pull_request", "merge_group", "push", "schedule", "workflow_dispatch", "workflow_call"]; + for (name, text) in workflows() { + for (event, _) in events(&name, &text) { + assert!(TRUSTED.contains(&event.as_str()), "{name} runs on {event}"); + } + } + } + + /// **No job widens its cache access**: GitHub gives a run on a low-trust + /// trigger read access alone to main's entries, and a `cache-mode` of + /// `write` or `write-only` gives it back. Any `cache-mode` but `read` or + /// `none` is refused. + #[test] + fn no_job_widens_its_cache_mode() { + for (name, text) in workflows() { + for line in text.lines().map(uncommented) { + for (at, _) in line.match_indices("cache-mode") { + let value = line[at + "cache-mode".len()..].trim_start_matches([':', ' ', '"', '\'']); + let value: String = value.chars().take_while(|c| c.is_ascii_alphanumeric() || *c == '-').collect(); + assert!(value == "read" || value == "none", "{name}: {line:?}"); + } + } + } + } + + /// **Only main's runs save what other refs restore.** A run saves into + /// its own ref's scope, and main's alone is restored on other refs. So a + /// workflow that saves a cache entry, itself or through a workflow it calls, + /// runs on a push only to main, on a pull request only into main and on a + /// dispatch that takes no input; and a job that saves checks out its own + /// run's commit and no other. + #[test] + fn only_mains_runs_save_what_other_refs_restore() { + let all = workflows(); + let saves = |text: &str| text.contains("actions/cache/save@"); + let calls = |text: &str, callee: &str| text.lines().any(|l| l.trim() == format!("uses: ./.github/workflows/{callee}")); + let mut savers = Vec::new(); + for (name, text) in &all { + if !saves(text) && !all.iter().any(|(callee, called)| saves(called) && calls(text, callee)) { + continue; + } + savers.push(name.as_str()); + for (event, lines) in events(name, text) { + match event.as_str() { + "push" | "pull_request" => { + assert!(lines.contains(&" branches: [main]".to_string()), "{name}: a {event} on another ref: {lines:?}") + } + "workflow_dispatch" => assert!(lines.iter().all(|l| !l.contains("inputs")), "{name}: a dispatch that takes input"), + _ => {} + } + } + for (job, lines) in jobs(text) { + if !lines.iter().any(|l| l.contains("actions/cache/save@")) { + continue; + } + for step in steps(&lines) { + if at(&step, "uses").is_some_and(|uses| uses.starts_with("actions/checkout@")) { + let other = step.iter().find(|l| l.starts_with(" ref:") || l.starts_with(" repository:")); + assert_eq!(other, None, "{name} {job} saves a build of a commit its run is not"); + } + } + } + } + assert_eq!(savers, ["ci.yml", "nightly.yml", "publish.yml", "toolchain.yml"]); + } + + /// Holds one job's lines to [`every_job_that_saves_holds_to_the_allow_list`]. + fn holds_to_the_allow_list(lines: &[&str]) -> Result<(), String> { + let mut block: Option = None; + for line in lines { + let indent = line.len() - line.trim_start().len(); + let text = uncommented(line).trim(); + if text.is_empty() || block.is_some_and(|at| indent > at) { + continue; + } + block = None; + let entry = text.strip_prefix("- ").unwrap_or(text); + let column = indent + text.len() - entry.len(); + let (key, value) = entry + .split_once(": ") + .or(entry.strip_suffix(':').map(|key| (key, ""))) + .ok_or_else(|| format!("a line this reader cannot judge: {line:?}"))?; + let plain = !key.is_empty() && key.bytes().all(|b| b.is_ascii_alphanumeric() || b == b'-' || b == b'_'); + let value = value.trim(); + let path = key == "path"; + if !plain || value.starts_with(['{', '[']) || (!path && value.starts_with(['*', '&'])) { + return Err(format!("a line this reader cannot judge: {line:?}")); + } + if path && value.ends_with(['|', '>']) { + block = Some(column); + } + match key { + "run" => { + let job = value.strip_prefix("cargo run -- --ci ").unwrap_or_default(); + if job.is_empty() || !job.bytes().all(|b| b.is_ascii_lowercase()) { + return Err(format!("a step runs more than the driver: {line:?}")); + } + } + "shell" | "continue-on-error" => return Err(format!("{line:?}")), + _ => {} + } + } + for step in steps(lines) { + let guarded = step.iter().any(|l| l.starts_with(" - if:") || l.starts_with(" if:")); + let saves = at(&step, "uses").is_some_and(|uses| uses.starts_with("actions/cache/save@")); + if guarded && !saves { + return Err(format!("a step's own `if:` guards no save: {step:?}")); + } } - for name in ["guest.yml", "toolchain.yml"] { - let text = workflow(name); - assert!(!text.contains("permissions:") && !text.contains("--ci release"), "{name}"); + Ok(()) + } + + /// **A job that saves a cache entry runs the driver alone, and saves on its + /// save's own guard alone**: every `run:` is `cargo run -- --ci `, no + /// step sets its shell or outlives its own failure, and no step but a save + /// has a condition of its own, so an entry follows only the driver's green + /// steps. A line the reader cannot judge — an alias or anchor outside a + /// path, a merge key, a flow collection, a quoted key — is refused, so + /// nothing hides a step from it. + #[test] + fn every_job_that_saves_holds_to_the_allow_list() { + let mut held = Vec::new(); + for (name, text) in workflows() { + for (job, lines) in jobs(&text) { + if lines.iter().any(|l| l.contains("actions/cache/save@")) { + assert!(!text.contains("\ndefaults:"), "{name}: a shell for every step"); + holds_to_the_allow_list(&lines).unwrap_or_else(|why| panic!("{name} {job}: {why}")); + held.push(format!("{name} {job}")); + } + } } - let publish = workflow("publish.yml"); - assert!(publish.contains("\non:\n push:\n branches: [main]\n workflow_dispatch: {}\n"), "{publish}"); - assert_eq!(writes(&publish), ["id-token: write", "contents: write"]); - let release = job(&publish, "release"); - assert!(release.contains(&" contents: write"), "{release:?}"); - assert!(release.contains(&" run: cargo run -- --ci release"), "{release:?}"); - assert_eq!(publish.matches("--ci release").count(), 1); + assert_eq!(held, ["nightly.yml host", "toolchain.yml build"]); } - /// `toolchain.yml` uploads whole what `--ci bootstrap` leaves: a path that - /// missed it would upload nothing, and say nothing. + /// **Each toolchain store is restored and saved by the entry its job + /// wrote**: in `toolchain.yml`, each of `release::LAYERS` is restored by + /// exactly the `keys` step's `-key` and `-path` before the + /// `build` step, and saved by them after it only where that step built it; + /// and every job that installs the sysroot restores exactly the one the + /// toolchain job names, red on a miss. #[test] - fn the_toolchain_job_uploads_what_bootstrap_keeps() { + fn each_store_is_restored_and_saved_by_the_entry_its_job_wrote() { let text = workflow("toolchain.yml"); - let upload = format!(" path: {}/*.tar.zst\n archive: false\n", release::KEPT); - assert!(text.contains(&upload), "{text}"); + let action = |name: &str| { + let uses = |l: &str| l.trim().trim_start_matches("- ").strip_prefix("uses: ").map(str::to_string); + text.lines().find_map(|l| uses(l).filter(|a| a.starts_with(name))).unwrap_or_else(|| panic!("{name}")) + }; + let (restore, save) = (action("actions/cache/restore@"), action("actions/cache/save@")); + let place = |snippet: &str| { + assert_eq!(text.matches(snippet).count(), 1, "toolchain.yml holds {snippet:?} once"); + text.find(snippet).expect("found") + }; + let keys = place(" - id: keys\n run: cargo run -- --ci toolchain\n"); + let built = place(" - id: build\n run: cargo run -- --ci bootstrap\n"); + assert!(keys < built); + for (_, name) in release::LAYERS { + let entry = format!( + " with:\n path: ${{{{ steps.keys.outputs.{name}-path }}}}\n key: ${{{{ steps.keys.outputs.{name}-key }}}}\n" + ); + let restored = place(&format!(" - uses: {restore}\n{entry}\n")); + let saved = place(&format!(" - if: steps.build.outputs.{name} == 'built'\n uses: {save}\n{entry}")); + assert!(keys < restored && restored < built && built < saved, "{name}"); + } + assert!(!text.contains("restore-keys"), "a store restored by another entry than its key's"); + for output in ["key", "path"] { + place(&format!(" sysroot-{output}:\n value: ${{{{ jobs.build.outputs.sysroot-{output} }}}}\n")); + place(&format!(" sysroot-{output}: ${{{{ steps.keys.outputs.sysroot-{output} }}}}\n")); + } + let installs = " path: ${{ inputs.sysroot-path }}\n key: ${{ inputs.sysroot-key }}\n fail-on-cache-miss: true\n"; + assert!(workflow("guest.yml").contains(installs), "guest.yml installs another sysroot"); + let handed = " sysroot-key: ${{ needs.toolchain.outputs.sysroot-key }}\n sysroot-path: ${{ needs.toolchain.outputs.sysroot-path }}\n"; + for (name, callers) in [("ci.yml", 1), ("nightly.yml", 1)] { + assert_eq!(workflow(name).matches(handed).count(), callers, "{name} hands the guest lane another sysroot"); + } + let release = job(&workflow("nightly.yml"), "release").join("\n"); + let restored = " path: ${{ needs.toolchain.outputs.sysroot-path }}\n key: ${{ needs.toolchain.outputs.sysroot-key }}\n fail-on-cache-miss: true"; + assert!(release.contains(restored), "the nightly's release packs another sysroot: {release}"); } /// Every workflow's `pull_request:` trigger names `main` alone, and none @@ -1143,18 +1414,14 @@ mod tests { assert_eq!(seen, 5, "ci.yml, nightly.yml and publish.yml, and guest.yml and toolchain.yml"); } - /// Exactly one job writes each cache, on the nightly, so what a pull request - /// restores is one run's tree and never a race between two writers. - /// `guest.yml` saves only when its caller asks, and only the nightly asks. + /// Exactly one job writes each cache, so what a pull request restores is one + /// job's tree and never a race between two writers: the nightly's `host` + /// writes the host cache, and `toolchain.yml`'s `build` each toolchain store. #[test] fn each_cache_has_one_writer() { - let dir = repo_root().join(".github/workflows"); let mut writers = Vec::new(); - for entry in std::fs::read_dir(&dir).expect(".github/workflows is readable").flatten() { - let text = std::fs::read_to_string(entry.path()).expect("a readable workflow"); - let name = entry.file_name().to_string_lossy().into_owned(); + for (name, text) in workflows() { assert!(!text.contains("actions/cache@"), "{name}: the combined action saves too"); - assert!(name == "nightly.yml" || name == "guest.yml" || !text.contains("save-cache"), "{name}"); let lines: Vec<&str> = text.lines().collect(); for (at, line) in lines.iter().enumerate() { if line.contains("actions/cache/save@") { @@ -1162,20 +1429,17 @@ mod tests { .iter() .find_map(|l| l.trim_start().strip_prefix("key: ")) .expect("a save names its key"); - let asked = lines[at - 1].trim() == "- if: inputs.save-cache"; - writers.push((name.clone(), key.split('$').next().unwrap_or("").to_string(), asked)); + let cache = if key.starts_with("${{") { key } else { key.split('$').next().unwrap_or("") }; + writers.push((name.clone(), cache.to_string())); } } } assert!(!writers.is_empty(), "no job writes a cache, so every restore is cold"); writers.sort(); - let mut prefixes: Vec<&String> = writers.iter().map(|(_, p, _)| p).collect(); - prefixes.dedup(); - assert_eq!(prefixes.len(), writers.len(), "a cache with two writers: {writers:?}"); - assert!( - writers.iter().all(|(f, _, asked)| f == "nightly.yml" || (f == "guest.yml" && *asked)), - "{writers:?}" - ); + let mut caches: Vec<&String> = writers.iter().map(|(_, cache)| cache).collect(); + caches.dedup(); + assert_eq!(caches.len(), writers.len(), "a cache with two writers: {writers:?}"); + assert!(writers.iter().all(|(file, _)| file == "nightly.yml" || file == "toolchain.yml"), "{writers:?}"); } #[test] diff --git a/src/clang.rs b/src/clang.rs index ca7207a8e1e..35c9efdf886 100644 --- a/src/clang.rs +++ b/src/clang.rs @@ -44,7 +44,7 @@ const TOOLS: [&str; 3] = ["llvm-ar", "clang", "ld.lld"]; const APPLE_STRIP: &str = "rust-objcopy"; /// [`TOOLS`], and on an Apple host [`APPLE_STRIP`]. -fn tools() -> impl Iterator { +pub(crate) fn tools() -> impl Iterator { TOOLS.into_iter().chain(host_triple().ends_with("apple-darwin").then_some(APPLE_STRIP)) } diff --git a/src/compiler.rs b/src/compiler.rs index 8d786855b91..06de8fc04d1 100644 --- a/src/compiler.rs +++ b/src/compiler.rs @@ -2,17 +2,17 @@ //! primary's, or one of its own, content-addressed. //! //! **Every worktree builds with the compiler its own fork checkout names.** The -//! primary's `stage2` is built from what the primary's `rust/compiler/` holds, -//! and [`record`] writes which that is. A linked worktree whose fork checkout -//! holds the same `compiler/` ([`source`]) compiles with that one. One whose -//! `compiler/` differs — a new target spec, a codegen change — gets its own: -//! built by bootstrap in its own fork checkout, under that checkout's +//! primary's `stage2` is built from what the primary's fork checkout holds, and +//! [`record`] writes which that is ([`source`]). A linked worktree whose fork +//! checkout names the same compiler compiles with that one. One whose compiler +//! sources differ — a new target spec, a codegen change — gets its own: built +//! by bootstrap in its own fork checkout, under that checkout's //! `build/toyos-compiler/`, and placed at `rust/build/compilers//`, where //! the key ([`key`]) is the identity (`src/identity.rs`) of the checkout's -//! `compiler/`, `src/tools/`, `src/stage0` and `Cargo.lock`, the key of the -//! LLVM it links, which names `src/bootstrap`, and [`RECIPE`]. Nothing writes -//! that directory after its [`SOURCE`] file exists, and two worktrees naming -//! the same compiler share one copy. +//! [`KEYED`] sources, the key of the LLVM it links, which names +//! `src/bootstrap`, and the build itself ([`build_text`]). Nothing writes that +//! directory after its [`SOURCE`] file exists, and two worktrees naming the +//! same compiler share one copy. //! //! **LLVM is the host's, built from `src/llvm-project`** (`src/llvm.rs`), and //! linked through its `llvm-config`. [`source`] names that LLVM's key, so @@ -51,13 +51,23 @@ use crate::keystore::{self, Key}; use crate::sysroot::{clone_tree, git_bytes, git_out, short, tree_identity, Links}; use crate::toolchain::{self, host_triple}; -/// What changes how a key's sources become a compiler and is none of them: the -/// build below. Moving it moves every key. +/// What changes how a key's sources become a compiler and is neither them nor +/// [`config_text`]: the build below. Moving it moves every key. const RECIPE: &str = "bootstrap stage 2 of compiler/rustc and library, profile compiler, host only, with rust-lld, host linker pinned, LLVM, clang and LLD from the host's LLVM, no LLVM tool copied, rustc without debuginfo; 6"; /// What a compiler's key is the identity of, in its fork checkout. const KEYED: [&str; 4] = ["compiler", "src/tools", "src/stage0", "Cargo.lock"]; +/// What a compiler build is beyond its sources, as every key and record of one +/// reads it: [`RECIPE`], the configuration bootstrap is given ([`config_text`]) +/// with no path of this host in it, and the tools `clang::provision` puts beside +/// the compiler. +fn build_text() -> String { + let config = config_text(Path::new(""), &host_triple(), Path::new("")); + let provisioned: Vec<&str> = crate::clang::tools().collect(); + format!("{RECIPE}\n{config}provisioned {}", provisioned.join(" ")) +} + /// The submodule a compiler is built against by commit: its LLVM, which /// bootstrap builds from that commit, so its content is never read. pub(crate) const LLVM: &str = "src/llvm-project"; @@ -112,6 +122,20 @@ impl Compiler { .map_or(0, |d| d.as_nanos()); format!("{} {} {} {mtime}", source.trim(), driver.file_name().to_string_lossy(), meta.len()) } + + /// What a store built with this compiler is keyed by: its record, which + /// names what builds it and is known before it is built ([`primary_key`]). + pub fn key(&self) -> Key { + Key::of(&fs::read(&self.record).unwrap_or_else(|e| { + panic!("{} cannot be read ({e}), so nothing says which compiler a store is built with", self.record.display()) + })) + } +} + +/// [`Compiler::key`] of the primary's compiler `rust_dir`'s sources name, +/// whether or not it is built. +pub(crate) fn primary_key(rust_dir: &Path) -> Key { + Key::of(source(rust_dir).as_bytes()) } /// The primary's record of which `compiler/` its `stage2` was built from. @@ -124,23 +148,25 @@ pub fn compilers_dir(rust_dir: &Path) -> PathBuf { rust_dir.join("build/compilers") } -/// [`compiler_source`] and the key of the LLVM it links. +/// What `checkout`'s compiler is built from, as the primary records it: the +/// build ([`build_text`]), [`compiler_source`], and the key of the LLVM it links. pub fn source(checkout: &Path) -> String { source_with(checkout, &crate::llvm::key(checkout)) } /// [`source`], with the key of the LLVM `checkout` names. fn source_with(checkout: &Path, llvm: &Key) -> String { - format!("{} llvm {llvm}", compiler_source(checkout)) + format!("{}\n{} llvm {llvm}", build_text(), compiler_source(checkout)) } -/// What `checkout`'s `compiler/` is: its commit's tree, and whatever the working -/// tree changes in it — an edit, or a file git does not track yet, which is -/// what a new target spec is before its commit. +/// What `checkout`'s [`KEYED`] sources are: each one's entry at its commit, and +/// whatever the working tree changes in them — an edit, or a file git does not +/// track yet, which is what a new target spec is before its commit. fn compiler_source(checkout: &Path) -> String { - let tree = git_out(checkout, &["rev-parse", "HEAD:compiler"]); - let mut local = git_bytes(checkout, &["diff", "HEAD", "--", "compiler"]); - let untracked = git_bytes(checkout, &["ls-files", "-z", "--others", "--exclude-standard", "--", "compiler"]); + let in_keyed = |args: &[&'static str]| [args, &KEYED[..]].concat(); + let tree = git_out(checkout, &in_keyed(&["ls-tree", "HEAD", "--"])); + let mut local = git_bytes(checkout, &in_keyed(&["diff", "HEAD", "--"])); + let untracked = git_bytes(checkout, &in_keyed(&["ls-files", "-z", "--others", "--exclude-standard", "--"])); for name in untracked.split(|b| *b == 0).filter(|n| !n.is_empty()) { let path = checkout.join(String::from_utf8_lossy(name).as_ref()); local.extend_from_slice(name); @@ -175,8 +201,8 @@ pub fn forget(rust_dir: &Path) { } } -/// Whether the primary's `stage2` is the compiler `checkout`'s `compiler/` -/// names — `Err` when nothing records which compiler that is. +/// Whether the primary's `stage2` is the compiler `checkout` names — `Err` when +/// nothing records which compiler that is. /// /// **The source's content, never its files' times**: a checkout that rewrites a /// file with the bytes it had is no new compiler. @@ -185,8 +211,8 @@ fn primary_is(rust_dir: &Path, checkout: &Path, llvm: &Key) -> Result bool { match primary_is(rust_dir, rust_dir, &crate::llvm::key(rust_dir)) { Ok(current) => current, @@ -195,14 +221,19 @@ pub fn primary_is_current(rust_dir: &Path) -> bool { } } -/// The key of the compiler `fork`'s sources name: their content. +/// The key of the compiler `fork`'s sources name: their content, and the build. pub fn key(fork: &Path) -> Key { key_with(fork, &crate::llvm::key(fork)) } /// [`key`], with the key of the LLVM `fork` names. fn key_with(fork: &Path, llvm: &Key) -> Key { - let parts = [RECIPE, &tree_identity(fork, &KEYED, Links::Skipped), llvm.as_str()]; + key_of(fork, &build_text(), llvm) +} + +/// [`key`], with the build it reads. +fn key_of(fork: &Path, build: &str, llvm: &Key) -> Key { + let parts = [build, &tree_identity(fork, &KEYED, Links::Skipped), llvm.as_str()]; Key::of(parts.join("\n\0\n").as_bytes()) } @@ -341,10 +372,10 @@ fn build_in_fork(root: &Path, rust_dir: &Path, fork: &Path) -> PathBuf { stage2 } -/// Bootstrap's configuration for a compiler of a worktree's own: the primary's -/// `profile` and options, for the host alone, since every guest target's -/// libraries are the sysroot's to build, linking the LLVM at `llvm`. -fn config_text(build_dir: &Path, host: &str, llvm: &Path) -> String { +/// Bootstrap's configuration for every compiler, the primary's and a +/// worktree's own, built in `build_dir`: for the host alone, since every guest +/// target's libraries are the sysroot's to build, linking the LLVM at `llvm`. +pub(crate) fn config_text(build_dir: &Path, host: &str, llvm: &Path) -> String { format!( r#"change-id = "ignore" profile = "compiler" @@ -772,6 +803,46 @@ pub(crate) mod tests { assert_ne!(key(&fork), tools, "another LLVM commit did not move the key"); } + /// **The primary's record and every compiler's key read the whole build**: + /// the configuration bootstrap is given and the tools put beside the + /// compiler are in both, and another configuration is another key; the + /// record names every source of [`KEYED`], as the key does. What + /// [`record`] writes is the key a store is filed under before the + /// compiler is built ([`primary_key`]). + #[test] + fn a_compiler_is_keyed_and_recorded_by_its_whole_build() { + let scratch = TempDir::new("compiler-build"); + let (_primary, rust_dir, [same, _, _]) = estate(&scratch); + let fork = same.join("rust"); + let llvm = crate::llvm::key(&fork); + assert_eq!(key_of(&fork, &build_text(), &llvm), key(&fork)); + let config = config_text(Path::new(""), &host_triple(), Path::new("")); + let tools: Vec<&str> = crate::clang::tools().collect(); + for part in [config.as_str(), toolchain::LEAN, toolchain::HOST_LINKER_PIN, &tools.join(" ")] { + assert!(build_text().contains(part), "a compiler's key reads no {part:?}"); + assert!(source(&fork).contains(part), "the primary's record names no {part:?}"); + } + let more = build_text().replace("debuginfo-level-rustc = 0", "debuginfo-level-rustc = 1"); + assert_ne!(key_of(&fork, &more, &llvm), key(&fork), "another configuration kept the key"); + + let sources = [ + ("src/tools/lld-wrapper/src/main.rs", "fn main() {}\n"), + ("src/stage0", "compiler_version=nightly\n"), + ("Cargo.lock", "# relocked\n"), + ]; + for (file, text) in sources { + let before = source(&fork); + write(&fork.join(file), text); + assert_ne!(source(&fork), before, "{file} kept the primary's record"); + git(&fork, &["add", "-A"]); + git(&fork, &["commit", "-qm", file]); + assert_ne!(source(&fork), before, "{file}, committed, kept the primary's record"); + } + + record(&rust_dir); + assert_eq!(Compiler::primary(&rust_dir).key(), primary_key(&rust_dir)); + } + /// **A compiler of a worktree's own is built as the primary's is**: for /// the host alone, against the host's LLVM, copying none of its tools, with /// rustc without debuginfo and no codegen test. diff --git a/src/keystore.rs b/src/keystore.rs index c35e97f3dc1..93c7eec60f6 100644 --- a/src/keystore.rs +++ b/src/keystore.rs @@ -3,6 +3,13 @@ //! that removes a key no registered worktree records and nobody is making or //! using. //! +//! **A key hashes exactly what its product's build reads**: its sources, the +//! configuration its build is given, the tools that run that build and the keys +//! of the products it reads; and it is known before the product is. So a +//! product found under its key, made here or restored by a CI runner from +//! another run's cache, is the one this tree's build would make. An input a +//! build reads and its key does not is a defect of the key. +//! //! A product lives at `//`, whole once its maker renamed it there; //! any other name beginning `.` is one half-made or half-removed. A whole //! one is renamed out of the way before anything in it is removed ([`retire`]), diff --git a/src/lib.rs b/src/lib.rs index 06e45ea6a96..956d9f31d4b 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -115,6 +115,21 @@ pub fn ensure_submodules(repo_dir: &Path) { } } +/// `rust/` at the commit this tree pins and with no history, where a CI +/// runner's checkout has none: what reading the fork's sources and building +/// the toolchain from them need. +pub fn ensure_shallow_fork(root: &Path) -> Result<(), String> { + if root.join("rust/x.py").exists() { + return Ok(()); + } + let status = Command::new("git") + .args(["submodule", "update", "--init", "--depth", "1", "rust"]) + .current_dir(root) + .status() + .map_err(|e| format!("git submodule update --init --depth 1 rust: {e}"))?; + status.success().then_some(()).ok_or_else(|| format!("git submodule update --init --depth 1 rust exited {status}")) +} + /// Ensure a single git submodule is checked out. pub fn ensure_submodule(repo_dir: &Path, path: &str) { let dir = repo_dir.join(path); diff --git a/src/libc.rs b/src/libc.rs index 758721fd2d6..16713720040 100644 --- a/src/libc.rs +++ b/src/libc.rs @@ -10,6 +10,14 @@ pub const CRATE: &str = "userland/libc"; /// The features [`build`] gives [`CRATE`]. pub const FEATURES: &str = "std-runtime"; +/// What [`build`] asks cargo for, but the target, the manifest and the target +/// directory: what a sysroot's key reads of it. `--message-format=json` is how +/// it finds the exact rlib artifacts. +pub(crate) const BUILD: [&str; 5] = ["build", "--release", "--features", FEATURES, "--message-format=json"]; + +/// What [`build_c`] asks cargo for, as [`BUILD`] is [`build`]'s. +pub(crate) const BUILD_C: [&str; 4] = ["rustc", "--release", "--crate-type", "staticlib"]; + /// Build toyos-libc against the toolchain at `toolchain`, in `target_dir`, and /// install it there as `libtoyos_c.a`. Part of making a sysroot /// (`src/sysroot.rs`), whose key `userland/libc/src` is one of. @@ -20,24 +28,11 @@ pub fn build(root: &Path, toolchain: &Path, target_dir: &Path, arch: Arch) { // The one guest artifact `build::PROFILE` does not reach, so it is the one // place `overflow-checks` is off — and it is linked into std, so it is in - // every userland binary. Left deliberately, on two grounds: CLAUDE.md gives - // the POSIX compatibility layer explicitly relaxed rules, and a flag changed - // here does not move any sysroot's key unless `sysroot::RECIPE` moves with - // it, so the installed archive would not be rebuilt and the manifest would - // then claim something the artifact does not have. - // - // --message-format=json to discover the exact rlib artifacts. + // every userland binary. Left deliberately: CLAUDE.md gives the POSIX + // compatibility layer explicitly relaxed rules. let output = Command::new("cargo") - .args([ - "build", - "--release", - "--target", - arch.userland(), - "--features", - FEATURES, - "--message-format=json", - "--manifest-path", - ]) + .args(BUILD) + .args(["--target", arch.userland(), "--manifest-path"]) .arg(root.join(CRATE).join("Cargo.toml").to_str().unwrap()) .arg("--target-dir") .arg(target_dir) @@ -95,7 +90,8 @@ pub fn build(root: &Path, toolchain: &Path, target_dir: &Path, arch: Arch) { pub fn build_c(root: &Path, toolchain: &Path, target_dir: &Path, arch: Arch) { let target = arch.userland(); let output = Command::new("cargo") - .args(["rustc", "--release", "--target", target, "--crate-type", "staticlib", "--manifest-path"]) + .args(BUILD_C) + .args(["--target", target, "--manifest-path"]) .arg(root.join(CRATE).join("Cargo.toml")) .arg("--target-dir") .arg(target_dir) diff --git a/src/licence.rs b/src/licence.rs index b19a2d51b52..9fdd651de00 100644 --- a/src/licence.rs +++ b/src/licence.rs @@ -1161,12 +1161,7 @@ fn metadata( fn std_library(root: &Path) -> Result { let fork = crate::sysroot::fork_checkout(root); if !fork.join("library/Cargo.toml").exists() { - run( - Command::new("git") - .args(["submodule", "update", "--init", "--depth", "1", "rust"]) - .current_dir(root), - "git submodule update --init --depth 1 rust", - )?; + crate::ensure_shallow_fork(root)?; } Ok(fork.join("library")) } diff --git a/src/llvm.rs b/src/llvm.rs index 7829e197996..e22e2119888 100644 --- a/src/llvm.rs +++ b/src/llvm.rs @@ -158,7 +158,8 @@ struct HostTools { cc: PathBuf, cxx: PathBuf, /// The C and C++ compilers and CMake, each by its resolved path and all its - /// `--version` says, and on macOS the SDK path and version `xcrun` resolves. + /// `--version` says, n2 by the pin it is installed from, and on macOS the + /// SDK path and version `xcrun` resolves. identity: String, } @@ -176,6 +177,7 @@ fn tools_with(xcrun: impl Fn(&str) -> String) -> HostTools { let mut version = Command::new(tool); identity += &format!("{}\n{}", tool.display(), asked(version.arg("--version"))); } + identity += &format!("n2 {}\n", crate::n2::N2.join(" ")); if host_triple().ends_with("apple-darwin") { for question in ["--show-sdk-path", "--show-sdk-version"] { identity += &xcrun(question); @@ -240,7 +242,7 @@ fn tools() -> impl Iterator { } /// Why `dir` is not a finished LLVM, if it is not. -fn defect(dir: &Path) -> Option { +pub(crate) fn defect(dir: &Path) -> Option { if !dir.join(SOURCE).is_file() { return Some(format!("{} carries no {SOURCE}", dir.display())); } @@ -758,7 +760,7 @@ mod tests { /// **The tools the key names are the host's**: the C and C++ compilers the /// configuration names by path, and CMake, each by the file it resolves to - /// and what its `--version` says; on macOS, the SDK. + /// and what its `--version` says; n2 by its pin; on macOS, the SDK. #[test] fn the_key_names_the_host_s_tools() { let tools = host_tools(); @@ -769,6 +771,7 @@ mod tests { assert!(lines[at + 1].chars().any(|c| c.is_ascii_digit()), "{} said no version: {}", tool.display(), tools.identity); } assert!(lines.iter().any(|l| l.starts_with("cmake version")), "{}", tools.identity); + assert!(lines.contains(&format!("n2 {}", crate::n2::N2.join(" ")).as_str()), "no n2: {}", tools.identity); if host_triple().ends_with("apple-darwin") { assert!(lines.iter().any(|l| l.ends_with(".sdk") && Path::new(l).is_dir()), "no SDK: {}", tools.identity); } diff --git a/src/release.rs b/src/release.rs index d95839c53c4..bc40d8cb24f 100644 --- a/src/release.rs +++ b/src/release.rs @@ -1,107 +1,74 @@ -//! The toolchain a tree builds with: the tag that names it, the builds CI keeps -//! of it, and the release main publishes. +//! The toolchain a runner builds with, by the build system's own keys, and the +//! release main publishes of it. //! -//! **The tag is the content hash of [`trees`]**: the sources a toolchain is -//! built from and the build system's modules that build it ([`BUILDERS`]). A -//! tree that moved none of them names a toolchain somebody already built; a tree -//! that moved any names one nobody has. +//! **A toolchain is four stores, each a cache entry of the key the build system +//! files it under** ([`LAYERS`]). `toolchain.yml` restores each by the key +//! [`toolchain`] wrote, builds what none restored ([`bootstrap`]) and saves only +//! what it built. GitHub's ref scoping is the provenance: an entry a run on main +//! saved is restored on every ref, and any other run saves only into its own +//! ref's scope. //! -//! **CI installs a build only where it came from vouches for it, and only as the -//! bytes it was.** `toolchain.yml` bootstraps a tag no build answers for -//! ([`bootstrap`]) and keeps it as its run's artifact, uploaded whole; GitHub -//! records the SHA-256 of what was uploaded, and nothing rewrites an artifact. -//! [`install`] takes the newest build of its tree's tag that main's publisher -//! made and, failing that, the newest a run of a commit its tree vouches for made -//! ([`vouched`]); it refuses every other, and any download whose bytes hash to -//! anything but GitHub's digest. +//! **A guest job installs the sysroot its restore step put down** ([`install`]), +//! and main's nightly packs that store into the release a consumer outside CI +//! installs, then moves the SDK alias onto it ([`release`]); run anywhere else, +//! that job is refused before it reads anything. //! -//! **Only main publishes** ([`release`]): `publish.yml` on main puts main's own -//! build up as the release a consumer outside CI installs, and moves the SDK -//! alias onto it. Run anywhere else, that job is refused before it reads -//! anything. -//! -//! A build is `x86_64-unknown-linux-gnu`'s and is made on a GitHub-hosted -//! `ubuntu-24.04`; any other host is refused rather than building a tarball -//! nobody can install. A dev host never installs one: its build system -//! bootstraps from `rust/` as always. +//! A dev host installs none: its build system builds its own from `rust/`. -use std::collections::HashSet; use std::fs; use std::io::Write; use std::path::{Path, PathBuf}; -use std::process::{Command, Stdio}; +use std::process::Command; use serde_json::Value; use sha2::{Digest, Sha256}; use toyos_tmpdir::TempDir; -/// The build system's modules that build and pack the toolchain: every module -/// `src/toolchain.rs` and this file name through `crate::`, and every module -/// those name, so the tag moves with how the toolchain is built as well as with -/// what it is built from. -pub(crate) const BUILDERS: [&str; 18] = [ - "src/arch.rs", - "src/buildlock.rs", - "src/clang.rs", - "src/compiler.rs", - "src/flags.rs", - "src/gitfixture.rs", - "src/identity.rs", - "src/keystore.rs", - "src/libc.rs", - "src/libcxx.rs", - "src/llvm.rs", - "src/n2.rs", - "src/release.rs", - "src/sdkversion.rs", - "src/sync.rs", - "src/sysroot.rs", - "src/toolchain.rs", - "src/worktree.rs", -]; - -/// What the tag hashes, as `git rev-parse HEAD:` names them. -fn trees() -> Vec<&'static str> { - std::iter::once("rust") - .chain(crate::sysroot::SYSROOT_SOURCES) - .chain(crate::sysroot::SYSROOT_MANIFESTS) - .chain(BUILDERS) - .collect() -} +use crate::buildlock::Keyed; +use crate::keystore::Key; -/// The release's one asset, under the name a consumer's install fetches. const ASSET: &str = "toyos-toolchain.tar.zst"; -/// Where [`bootstrap`] leaves the build `toolchain.yml` uploads. -pub(crate) const KEPT: &str = "target/toolchain"; - -/// Main's publisher: the one workflow whose builds every tree takes, and the one -/// [`release`] runs under. -const PUBLISHER: &str = ".github/workflows/publish.yml"; +/// Main's publisher: the one workflow [`release`] runs under. +const PUBLISHER: &str = ".github/workflows/nightly.yml"; -/// The triple a build's host half runs on. const HOST: &str = "x86_64-unknown-linux-gnu"; /// The oldest glibc a consumer needs: `ubuntu-24.04`'s. A build naming a newer /// one is refused. const GLIBC_FLOOR: (u32, u32) = (2, 39); -/// `toolchain-linux-x86_64-<16 hex>`: the first 16 hex digits of the SHA-256 of -/// what `git rev-parse` prints for [`trees`], newline-terminated lines and all. -pub fn tag(root: &Path) -> Result { - let out = Command::new("git") - .arg("rev-parse") - .args(trees().iter().map(|t| format!("HEAD:{t}"))) - .current_dir(root) - .output() - .map_err(|e| format!("git rev-parse: {e}"))?; - if !out.status.success() { - return Err(format!( - "git rev-parse of the toolchain's trees: {}", - String::from_utf8_lossy(&out.stderr).trim() - )); +/// What every request this file makes of GitHub says it comes from. +const USER_AGENT: &str = "toyos-build (https://github.com/ToyOSOrg/ToyOS)"; + +/// The stores a toolchain is, in the order a build makes them, each under the +/// name its cache entry and its job's outputs carry. +pub(crate) const LAYERS: [(Keyed, &str); 4] = [ + (Keyed::Llvm, "llvm"), + (Keyed::Compiler, "compiler"), + (Keyed::Freestanding, "freestanding"), + (Keyed::Sysroot, "sysroot"), +]; + +/// One of [`LAYERS`] of this tree's toolchain: the key the build system files +/// it under, and the paths it is, relative to the checkout. +struct Layer { + kind: Keyed, + name: &'static str, + key: Key, + paths: Vec, +} + +impl Layer { + /// Its cache entry's key. + fn entry(&self) -> String { + format!("toolchain-{}-{}", self.name, self.key) } - Ok(format!("toolchain-linux-x86_64-{}", &sha256_hex(&out.stdout)[..16])) +} + +/// The release of the sysroot `key` names. +fn tag(key: &Key) -> String { + format!("toolchain-linux-x86_64-{key}") } pub(crate) fn sha256_hex(bytes: &[u8]) -> String { @@ -116,195 +83,164 @@ fn repo() -> String { std::env::var("GITHUB_REPOSITORY").unwrap_or_else(|_| "ToyOSOrg/ToyOS".into()) } -/// Whether this job is main's publisher — [`PUBLISHER`] on main, pushed or +/// Whether this job is main's publisher — [`PUBLISHER`] on main, scheduled or /// dispatched — as the runner names its workflow and event; refused by name if /// it is not. fn publisher(workflow: Option<&str>, event: Option<&str>, repo: &str) -> Result<(), String> { let mains = format!("{repo}/{PUBLISHER}@refs/heads/main"); match (workflow, event) { - (Some(workflow), Some("push" | "workflow_dispatch")) if workflow == mains => Ok(()), + (Some(workflow), Some("schedule" | "workflow_dispatch")) if workflow == mains => Ok(()), (workflow, event) => Err(format!( - "only {mains}, pushed or dispatched, publishes a toolchain, and this job is {} on {}", + "only {mains}, scheduled or dispatched, publishes a toolchain, and this job is {} on {}", workflow.unwrap_or("no workflow"), event.unwrap_or("no event") )), } } -fn this_job_publishes() -> bool { - let var = |name| std::env::var(name).ok(); - publisher(var("GITHUB_WORKFLOW_REF").as_deref(), var("GITHUB_EVENT_NAME").as_deref(), &repo()).is_ok() +/// This tree's toolchain as [`LAYERS`], keyed from its sources alone, before +/// any store is there. +fn layers(root: &Path) -> Vec { + let rust_dir = root.join("rust"); + let llvm = crate::llvm::key(&rust_dir); + let compiler = crate::compiler::primary_key(&rust_dir); + let freestanding = crate::sysroot::freestanding_key(root, &compiler, &rust_dir); + let sysroot = crate::sysroot::key(root, &freestanding); + LAYERS + .iter() + .map(|&(kind, name)| { + let (key, paths) = match kind { + Keyed::Llvm => (llvm.clone(), vec![crate::llvm::store(&rust_dir).join(&llvm)]), + Keyed::Compiler => ( + compiler.clone(), + vec![crate::toolchain::stage2(&rust_dir), crate::compiler::primary_record(&rust_dir)], + ), + Keyed::Freestanding => { + (freestanding.clone(), vec![crate::sysroot::freestanding_dir(&rust_dir).join(&freestanding)]) + } + Keyed::Sysroot => (sysroot.clone(), vec![crate::sysroot::sysroots_dir(&rust_dir).join(&sysroot)]), + }; + let paths = paths + .iter() + .map(|path| path.strip_prefix(root).unwrap_or_else(|_| panic!("{} is outside the checkout", path.display()))) + .map(Path::to_path_buf) + .collect(); + Layer { kind, name, key, paths } + }) + .collect() } -/// GitHub's answer to `GET https://api.github.com/`, or `None` when there -/// is no such thing. -fn api(path: &str) -> Result, String> { - let token = std::env::var("GH_TOKEN").map_err(|_| "GH_TOKEN is unset".to_string())?; - let url = format!("https://api.github.com/{path}"); - let out = Command::new("curl") - .args(["-sSL", "--retry", "3", "--retry-all-errors", "-w", "\n%{http_code}"]) - .args(["-H", &format!("Authorization: Bearer {token}")]) - .args(["-H", "Accept: application/vnd.github+json", &url]) - .output() - .map_err(|e| format!("curl: {e}"))?; - if !out.status.success() { - return Err(format!("curl {url} exited {}: {}", out.status, String::from_utf8_lossy(&out.stderr).trim())); - } - let text = String::from_utf8_lossy(&out.stdout).into_owned(); - let (body, status) = text.rsplit_once('\n').unwrap_or(("", text.as_str())); - match status { - "200" => serde_json::from_str(body).map(Some).map_err(|e| format!("{url} answered no JSON: {e}")), - "404" => Ok(None), - status => Err(format!("{url} answered {status}: {body}")), +/// Why `layer` is not whole in `root`, as the build that makes it decides, if +/// it is not. +fn defect(root: &Path, layer: &Layer) -> Option { + let dir = root.join(&layer.paths[0]); + match layer.kind { + Keyed::Llvm => crate::llvm::defect(&dir), + Keyed::Compiler => match fs::read(root.join(&layer.paths[1])) { + Ok(record) if Key::of(&record) == layer.key => crate::toolchain::toolchain_defect(&dir), + _ => Some(format!("{} records no compiler {}", layer.paths[1].display(), layer.key)), + }, + Keyed::Freestanding => crate::sysroot::unpublished(&dir), + Keyed::Sysroot => crate::sysroot::unfinished(&dir), } } -/// A toolchain build a run kept: its artifact, the digest GitHub recorded of its -/// bytes, and the run that uploaded it. -#[derive(Clone, Debug, PartialEq)] -struct Build { - artifact: u64, - /// `sha256:`. - digest: String, - run: u64, - /// The commit its run was for: the one pushed or queued, or a pull - /// request's head. - head: String, - /// Whether that run was main's publisher's. - mains: bool, -} - -impl Build { - fn provenance(&self) -> String { - let whose = if self.mains { "main's publisher" } else { "a commit this tree vouches for" }; - format!("artifact {} of run {}, {whose}, at {}", self.artifact, self.run, self.head) - } +/// The file a job's next steps read this step's outputs from: a runner's +/// `$GITHUB_OUTPUT`. +fn step_outputs() -> Result { + std::env::var_os("GITHUB_OUTPUT") + .map(PathBuf::from) + .ok_or_else(|| "not a runner: a dev host builds its own toolchain with `cargo run`".to_string()) } -/// The unexpired artifacts in GitHub's `listing`, newest first, each with the -/// branch its run was on. An artifact with no digest to hold its bytes to is -/// none of them. -fn listed(listing: &Value) -> Vec<(Build, String)> { - let artifacts = listing["artifacts"].as_array().map_or(&[][..], Vec::as_slice); - artifacts - .iter() - .filter(|a| a["expired"] == false) - .filter_map(|a| { - let run = &a["workflow_run"]; - let build = Build { - artifact: a["id"].as_u64()?, - digest: a["digest"].as_str()?.to_string(), - run: run["id"].as_u64()?, - head: run["head_sha"].as_str()?.to_string(), - mains: false, - }; - Some((build, run["head_branch"].as_str()?.to_string())) - }) - .collect() +/// Append `text` to the step outputs at `file`. +fn tell(file: &Path, text: &str) -> Result<(), String> { + fs::OpenOptions::new() + .append(true) + .open(file) + .and_then(|mut opened| opened.write_all(text.as_bytes())) + .map_err(|e| format!("{}: {e}", file.display())) } -/// Whether GitHub's `run` is main's publisher's: [`PUBLISHER`] on main, pushed -/// or dispatched, in `repo` and from it. -fn is_mains(run: &Value, repo: &str) -> bool { - run["path"] == PUBLISHER - && run["head_branch"] == "main" - && matches!(run["event"].as_str(), Some("push" | "workflow_dispatch")) - && run["repository"]["full_name"] == repo - && run["head_repository"]["full_name"] == repo -} - -/// Which of `builds`, newest first, a tree installs: the newest main's publisher -/// made, else — unless only main's will do — the newest a run of a commit in -/// `vouched` made. -fn choose<'a>(builds: &'a [Build], vouched: &HashSet, only_mains: bool) -> Option<&'a Build> { - builds.iter().find(|b| b.mains).or_else(|| { - if only_mains { - None - } else { - builds.iter().find(|b| vouched.contains(&b.head)) - } - }) +/// `cargo run -- --ci toolchain`: each of this tree's [`LAYERS`] as the cache +/// entry its job restores and saves, told to the job's next steps +/// ([`outputs`]). +pub fn toolchain(root: &Path) -> Result { + let file = step_outputs()?; + crate::ensure_shallow_fork(root)?; + let layers = layers(root); + tell(&file, &outputs(&layers))?; + Ok(layers.iter().map(|layer| format!("{} {}", layer.name, layer.key)).collect::>().join(", ")) } -/// The commits whose runs' builds this tree takes beside main's publisher's: -/// each commit on its first-parent chain, and the head each merge on that chain -/// took in — main's commits, each head main merged after its review, and a pull -/// request's own head — but no commit a branch passed on its way to the head -/// that was merged. -fn vouched(root: &Path) -> Result, String> { - Ok(merged_heads(&crate::sync::git(root, &["rev-list", "--first-parent", "--parents", "HEAD"])?)) +/// Each layer's entry as `-key` and its paths, one a line, as +/// `-path`. +fn outputs(layers: &[Layer]) -> String { + let mut text = String::new(); + for layer in layers { + let paths: Vec = layer.paths.iter().map(|path| path.display().to_string()).collect(); + text += &format!("{0}-key={1}\n{0}-path< HashSet { - rev_list - .lines() - .flat_map(|line| { - let mut words = line.split_whitespace(); - let commit = words.next(); - words.next(); - commit.into_iter().chain(words) +/// `cargo run -- --ci bootstrap`: this tree's toolchain made whole from what its +/// job restored, and each layer told to the job's save steps as built or kept +/// ([`built`]). A sysroot restored is all a guest job reads, so then nothing is +/// built; and a layer restored and not whole is refused, since its key reads +/// less than its build does. +pub fn bootstrap(root: &Path) -> Result { + let file = step_outputs()?; + let layers = layers(root); + let restored: Vec = layers.iter().map(|layer| root.join(&layer.paths[0]).exists()).collect(); + whole_as_restored(&layers, &restored, |layer| defect(root, layer))?; + if !restored[3] { + let mut lock = crate::buildlock::shared(root, "the toolchain"); + drop(crate::toolchain::ensure(root, &mut lock, false)); + for layer in &layers { + if let Some(why) = defect(root, layer) { + return Err(format!("the build left {} {} not whole: {why}", layer.name, layer.key)); + } + } + } + tell(&file, &built(&layers, &restored))?; + let said: Vec = layers + .iter() + .zip(&restored) + .map(|(layer, was)| match (*was, restored[3]) { + (true, _) => format!("{} {} restored", layer.name, layer.key), + (false, true) => format!("{} {} not needed", layer.name, layer.key), + (false, false) => format!("{} {} built", layer.name, layer.key), }) - .map(str::to_string) - .collect() + .collect(); + Ok(said.join(", ")) } -/// The build of `tag` this tree installs ([`choose`]), if a run kept one. -fn find(root: &Path, tag: &str, only_mains: bool) -> Result, String> { - let repo = repo(); - let listing = api(&format!("repos/{repo}/actions/artifacts?name={tag}.tar.zst&per_page=100"))? - .ok_or_else(|| format!("{repo} lists no artifacts"))?; - let mut builds = Vec::new(); - for (mut build, branch) in listed(&listing) { - if branch == "main" { - let run = api(&format!("repos/{repo}/actions/runs/{}", build.run))?; - build.mains = run.is_some_and(|run| is_mains(&run, &repo)); +/// Refused where a layer `restored` says its job restored is not whole, as +/// `defect` finds it: its key reads less than its build does, and a build +/// under that key could never be saved over the entry. +fn whole_as_restored(layers: &[Layer], restored: &[bool], defect: impl Fn(&Layer) -> Option) -> Result<(), String> { + for (layer, _) in layers.iter().zip(restored).filter(|(_, restored)| **restored) { + if let Some(why) = defect(layer) { + return Err(format!("{} {} was restored and is not whole: {why}", layer.name, layer.key)); } - builds.push(build); } - let vouched = if only_mains { HashSet::new() } else { vouched(root)? }; - Ok(choose(&builds, &vouched, only_mains).cloned()) -} - -/// `cargo run -- --ci toolchain`: whether a build answers for this tree's -/// toolchain, told to the job's next steps as `bootstrap` in `$GITHUB_OUTPUT`. -/// Main's publisher takes only its own. -pub fn toolchain(root: &Path) -> Result { - let output = std::env::var("GITHUB_OUTPUT") - .map_err(|_| "not a runner: a dev host builds its own toolchain with `cargo run`".to_string())?; - let tag = tag(root)?; - let found = find(root, &tag, this_job_publishes())?; - fs::OpenOptions::new() - .append(true) - .open(&output) - .and_then(|mut file| writeln!(file, "bootstrap={}", found.is_none())) - .map_err(|e| format!("{output}: {e}"))?; - Ok(match found { - Some(build) => format!("{tag}: {}", build.provenance()), - None => format!("{tag}: no build answers for it, so this job bootstraps one"), - }) + Ok(()) } -/// `cargo run -- --ci bootstrap`: this tree's toolchain built and left in -/// [`KEPT`] for its run to keep, unless a build already answers for it. -pub fn bootstrap(root: &Path) -> Result { - if !on_runner() { - return Err("not a runner: a dev host builds its own toolchain with `cargo run`".into()); - } - let tag = tag(root)?; - if let Some(build) = find(root, &tag, this_job_publishes())? { - return Ok(format!("{tag}: {}, so nothing is built", build.provenance())); - } - let kept = root.join(KEPT); - fs::create_dir_all(&kept).map_err(|e| format!("{}: {e}", kept.display()))?; - let tarball = kept.join(format!("{tag}.tar.zst")); - build(root, &tag, &tarball)?; - Ok(format!("{tag} built into {}", tarball.display())) +/// Each layer as `=built` where `restored` says its job restored neither +/// it nor the sysroot, and `=kept` otherwise: what the job saves. +fn built(layers: &[Layer], restored: &[bool]) -> String { + let sysroot = restored[3]; + layers + .iter() + .zip(restored) + .map(|(layer, restored)| format!("{}={}\n", layer.name, if *restored || sysroot { "kept" } else { "built" })) + .collect() } -/// Install the build of this tree's toolchain it takes ([`find`]) as rustup's -/// `toyos`, on a runner, once its bytes are the ones GitHub recorded. +/// Install the sysroot its job restored ([`lay_out`]) as rustup's `toyos`, on +/// a runner. /// /// Off a runner this says so and does nothing: the build system owns the dev /// host's toolchain. @@ -312,84 +248,52 @@ pub fn install(root: &Path) -> Result { if !on_runner() { return Ok("not a runner: the build system uses this checkout's own toolchain".into()); } - let tag = tag(root)?; - let build = find(root, &tag, false)?.ok_or_else(|| { - format!( - "no build of {tag} answers for this tree: main's publisher kept none, and no run of a \ - commit this tree vouches for kept one. `toolchain.yml` bootstraps it ahead of this job" - ) - })?; - let staging = TempDir::new("toolchain-install"); - let tarball = staging.join(ASSET); - fetch(&build, &tarball)?; - let into = root.join("rust/build"); - fs::create_dir_all(&into).map_err(|e| format!("{}: {e}", into.display()))?; - unpack(&tarball, &into)?; - let stage2 = into.join(format!("{HOST}/stage2")); + let key = lay_out(root)?; + let stage2 = crate::toolchain::stage2(&root.join("rust")); run(Command::new("rustup").args(["toolchain", "link", "toyos"]).arg(&stage2))?; run(Command::new(stage2.join("bin/rustc")).arg("-vV"))?; - Ok(format!("installed {tag} as `toyos`: {}", build.provenance())) -} - -/// `build`'s bytes at `to`, held to the digest GitHub recorded of them. Three -/// transfers at most: a body cut short is a wrong digest, not a curl failure. -fn fetch(build: &Build, to: &Path) -> Result<(), String> { - let mut last = String::new(); - for attempt in 1..=3 { - match download(build.artifact, to).and_then(|()| verify(to, &build.digest)) { - Ok(()) => return Ok(()), - Err(why) => last = why, - } - println!("toolchain download attempt {attempt}: {last}"); - } - Err(format!("artifact {} did not arrive as GitHub recorded it, in three attempts: {last}", build.artifact)) -} - -/// `artifact`'s bytes into `to`. GitHub answers with a redirect to storage that -/// carries its own credential, and curl sends the token to no other host. -fn download(artifact: u64, to: &Path) -> Result<(), String> { - let token = std::env::var("GH_TOKEN").map_err(|_| "GH_TOKEN is unset".to_string())?; - let url = format!("https://api.github.com/repos/{}/actions/artifacts/{artifact}/zip", repo()); - let status = Command::new("curl") - .args(["-sSfL", "--retry", "3", "--retry-all-errors", "-o"]) - .arg(to) - .args(["-H", &format!("Authorization: Bearer {token}"), &url]) - .status() - .map_err(|e| format!("curl: {e}"))?; - status.success().then_some(()).ok_or_else(|| format!("curl {url} exited {status}")) -} - -/// Whether `path` hashes to `digest`, GitHub's `sha256:`. -fn verify(path: &Path, digest: &str) -> Result<(), String> { - let want = digest.strip_prefix("sha256:").ok_or_else(|| format!("{digest:?} is not a SHA-256 digest"))?; - let mut file = fs::File::open(path).map_err(|e| format!("{}: {e}", path.display()))?; - let mut hasher = Sha256::new(); - std::io::copy(&mut file, &mut hasher).map_err(|e| format!("{}: {e}", path.display()))?; - let got: String = hasher.finalize().iter().map(|b| format!("{b:02x}")).collect(); - if got == want { - Ok(()) - } else { - Err(format!("{} hashes to {got}, and GitHub recorded {want}", path.display())) + Ok(format!("installed sysroot {key} as `toyos`")) +} + +/// Lay the one sysroot under `rust/build/sysroots`, which its job's restore +/// step put there, out as this checkout's installed toolchain +/// (`toolchain::Owner::Installed`): at `stage2`, with the witness it records and +/// its [`manifest`]. Refused unless that witness is this tree's. +fn lay_out(root: &Path) -> Result { + let rust_dir = root.join("rust"); + let store = crate::sysroot::sysroots_dir(&rust_dir); + let restored = fs::read_dir(&store) + .and_then(|entries| entries.map(|entry| entry.map(|entry| entry.path())).collect::>>()) + .map_err(|e| format!("{}: {e}", store.display()))?; + let [sysroot] = restored.as_slice() else { + return Err(format!( + "{} holds {} entries, and a job installs the one sysroot it restored: {restored:?}", + store.display(), + restored.len() + )); + }; + let key = sysroot + .file_name() + .and_then(|name| name.to_str()) + .and_then(Key::parse) + .ok_or_else(|| format!("{} is named by no key", sysroot.display()))?; + let witness = crate::sysroot::recorded_witness(sysroot)?; + if witness != crate::sysroot::witness(root) { + return Err(format!( + "sysroot {key} was built from other sources than this tree's, and this tree's key names it: \ + the key reads less than the sysroot is built from (`src/sysroot.rs`)" + )); } -} - -/// `zstd -dc | tar -C -x`. -fn unpack(tarball: &Path, into: &Path) -> Result<(), String> { - let mut zstd = Command::new("zstd") - .arg("-dc") - .arg(tarball) - .stdout(Stdio::piped()) - .spawn() - .map_err(|e| format!("zstd: {e}"))?; - let stream = zstd.stdout.take().expect("piped"); - let tar = Command::new("tar").arg("-C").arg(into).arg("-x").stdin(stream).status(); - let zstd = zstd.wait().map_err(|e| format!("zstd: {e}"))?; - let tar = tar.map_err(|e| format!("tar: {e}"))?; - if zstd.success() && tar.success() { - Ok(()) - } else { - Err(format!("unpacking: zstd exited {zstd}, tar {tar}")) + let stage2 = crate::toolchain::stage2(&rust_dir); + let host = stage2.parent().unwrap_or_else(|| panic!("{} has no parent", stage2.display())); + fs::create_dir_all(host).map_err(|e| format!("{}: {e}", host.display()))?; + fs::rename(sysroot, &stage2).map_err(|e| format!("{} -> {}: {e}", sysroot.display(), stage2.display()))?; + for (path, text) in + [(crate::toolchain::witness_path(&rust_dir), witness), (crate::toolchain::manifest_path(&rust_dir), manifest(&tag(&key)))] + { + fs::write(&path, text).map_err(|e| format!("{}: {e}", path.display()))?; } + Ok(key) } fn run(cmd: &mut Command) -> Result<(), String> { @@ -397,10 +301,10 @@ fn run(cmd: &mut Command) -> Result<(), String> { status.success().then_some(()).ok_or_else(|| format!("{cmd:?} exited {status}")) } -/// `cargo run -- --ci release`: main's own build of its toolchain put up as the -/// release a consumer outside CI installs, and the `sdk-` alias moved -/// onto it. Refused before anything is read unless this job is main's -/// publisher. +/// `cargo run -- --ci release`: the sysroot main's nightly restored, put up as +/// the release a consumer outside CI installs, and the `sdk-` alias +/// moved onto it. Refused before anything is read unless this job is main's +/// publisher, and before anything is put up unless it runs at main's tip. pub fn release(root: &Path) -> Result { let var = |name| std::env::var(name).ok(); release_as(root, var("GITHUB_WORKFLOW_REF").as_deref(), var("GITHUB_EVENT_NAME").as_deref()) @@ -409,119 +313,175 @@ pub fn release(root: &Path) -> Result { /// [`release`], run as the job the runner names by its workflow and event. fn release_as(root: &Path, workflow: Option<&str>, event: Option<&str>) -> Result { publisher(workflow, event, &repo())?; - let tag = tag(root)?; - let build = find(root, &tag, true)? - .ok_or_else(|| format!("main's publisher kept no build of {tag}: this run's `toolchain` job makes it"))?; + if !(cfg!(target_os = "linux") && crate::arch::Arch::HOST == Some(crate::arch::Arch::X86_64)) { + return Err(format!("a release is {HOST}'s and this host is not one; a tarball packed here would install nowhere")); + } + let tip = crate::sync::git(root, &["ls-remote", "origin", "refs/heads/main"])?; + crate::ci::at_tip(&tip, crate::sync::git(root, &["rev-parse", "HEAD"])?.trim())?; + let key = lay_out(root)?; + let rust_dir = root.join("rust"); + let need = shipped_glibc(&crate::toolchain::stage2(&rust_dir))?; + if need > GLIBC_FLOOR { + return Err(format!( + "the host half needs GLIBC_{}.{} and a release states {}.{}: build it on the oldest supported \ + glibc, or move GLIBC_FLOOR deliberately", + need.0, need.1, GLIBC_FLOOR.0, GLIBC_FLOOR.1 + )); + } let tmp = TempDir::new("toolchain-release"); let tarball = tmp.join(ASSET); - fetch(&build, &tarball)?; - let manifest = manifest(root, &tag, &build.head)?; - fs::write(tmp.join("notes.md"), notes(root, &tag, &manifest)?).map_err(|e| e.to_string())?; - let put = put_up(root, &tag, &build.digest, &tarball, &tmp.join("notes.md"))?; - Ok(format!("{put}; {}", alias(root, &manifest, &tmp)?)) -} - -/// `tag`'s release, made to carry exactly the bytes `digest` names: created if -/// there is none, its asset replaced if another writer's is there, and then held -/// to the digest GitHub records of what it carries. -fn put_up(root: &Path, tag: &str, digest: &str, tarball: &Path, notes: &Path) -> Result { - let path = format!("repos/{}/releases/tags/{tag}", repo()); - let carried = |release: Option| -> Option { - let assets = release?["assets"].as_array()?.clone(); - assets.iter().find(|a| a["name"] == ASSET)?["digest"].as_str().map(str::to_string) - }; - let said = match api(&path)? { - None => { - run(Command::new("gh") - .args(["release", "create", tag, "--title", tag, "--notes-file"]) - .arg(notes) - .arg(tarball) - .current_dir(root))?; - "published" + pack(&rust_dir.join("build"), &tarball)?; + let tag = tag(&key); + let notes = notes(root, &tag, &manifest(&tag))?; + let put = put_up(root, &tag, ¬es, &tarball, &tmp)?; + Ok(format!("{put}; {}", alias(root, &tag, ¬es, &tmp)?)) +} + +/// The tarball of the toolchain laid out under `build` ([`lay_out`]) at +/// `tarball`: `/stage2` but its `bin/cargo`, which names a path only this +/// runner has, then its witness and `TOOLCHAIN`, in sorted order with no owner +/// or time, so one sysroot packs to one digest. +fn pack(build: &Path, tarball: &Path) -> Result<(), String> { + let stage2 = Path::new(HOST).join("stage2"); + let mut entries = vec![stage2.clone()]; + walk(&build.join(&stage2), &stage2, &mut entries)?; + entries.retain(|entry| *entry != stage2.join("bin/cargo")); + entries.extend(["toyos-sysroot-witness", "TOOLCHAIN"].map(PathBuf::from)); + let mut tar = tar::Builder::new(Vec::new()); + tar.follow_symlinks(false); + tar.mode(tar::HeaderMode::Deterministic); + for entry in &entries { + tar.append_path_with_name(build.join(entry), entry).map_err(|e| format!("pack {}: {e}", entry.display()))?; + } + let bytes = tar.into_inner().map_err(|e| format!("pack {}: {e}", tarball.display()))?; + let file = fs::File::create(tarball).map_err(|e| format!("{}: {e}", tarball.display()))?; + let mut file = std::io::BufWriter::new(file); + ruzstd::encoding::compress(bytes.as_slice(), &mut file, ruzstd::encoding::CompressionLevel::Fastest); + file.flush().map_err(|e| format!("{}: {e}", tarball.display())) +} + +/// Every path under `dir`, named as it is under `prefix`, each directory's in +/// sorted order. +fn walk(dir: &Path, prefix: &Path, out: &mut Vec) -> Result<(), String> { + let mut names = fs::read_dir(dir) + .and_then(|entries| entries.map(|entry| entry.map(|entry| entry.file_name())).collect::>>()) + .map_err(|e| format!("{}: {e}", dir.display()))?; + names.sort(); + for name in names { + let path = dir.join(&name); + out.push(prefix.join(&name)); + let meta = fs::symlink_metadata(&path).map_err(|e| format!("{}: {e}", path.display()))?; + if meta.is_dir() { + walk(&path, &prefix.join(&name), out)?; } - Some(release) if carried(Some(release.clone())).as_deref() == Some(digest) => { - return Ok(format!("{tag} already carries main's build")); + } + Ok(()) +} + +/// What a release needs for its asset `name` to be the bytes `digest` names, +/// given GitHub's account of it, `None` where there is no release. +#[derive(Debug, PartialEq)] +enum Put { + Create, + Carried, + Upload, + /// Delete the asset another writer put there, then upload. + Replace { asset: u64 }, +} + +fn put(release: Option<&Value>, name: &str, digest: &str) -> Result { + let Some(release) = release else { return Ok(Put::Create) }; + let assets = release["assets"].as_array().ok_or("a release with no assets list")?; + match assets.iter().find(|asset| asset["name"] == name) { + None => Ok(Put::Upload), + Some(asset) if asset["digest"].as_str() == Some(digest) => Ok(Put::Carried), + Some(asset) => Ok(Put::Replace { asset: asset["id"].as_u64().ok_or("an asset with no id")? }), + } +} + +/// `tag`'s release, made to carry `file` as its asset by its name: created with +/// `notes` where there is none, given them where there is, and then held to the +/// digest GitHub records. +fn put_up(root: &Path, tag: &str, notes: &str, file: &Path, tmp: &Path) -> Result { + let name = file.file_name().and_then(|name| name.to_str()).ok_or_else(|| format!("{} has no name", file.display()))?; + let digest = format!("sha256:{}", file_sha256(file)?); + let at = api(&format!("repos/{}/releases/tags/{tag}", repo())); + let found = github("GET", &at, None)?; + let (release, said) = match put(found.as_ref(), name, &digest)? { + Put::Carried => return Ok(format!("{tag} already carries this {name}")), + Put::Create => { + let commit = crate::sync::git(root, &["rev-parse", "HEAD"])?; + let body = serde_json::json!({ "tag_name": tag, "name": tag, "body": notes, "target_commitish": commit.trim() }); + (send(tmp, "POST", &api(&format!("repos/{}/releases", repo())), &body)?, "put up") } - Some(_) => { - run(Command::new("gh").args(["release", "upload", tag, "--clobber"]).arg(tarball).current_dir(root))?; - "had another writer's asset, now main's build" + Put::Upload => (renote(tmp, found, notes)?, "given its asset"), + Put::Replace { asset } => { + github("DELETE", &api(&format!("repos/{}/releases/assets/{asset}", repo())), None)?; + (renote(tmp, found, notes)?, "had another writer's asset, now this one") } }; - let now = carried(api(&path)?); - if now.as_deref() != Some(digest) { - return Err(format!("{tag} carries {now:?} after the upload, and main's build is {digest}")); + let upload = release["upload_url"].as_str().ok_or("a release with no upload URL")?; + let upload = format!("{}?name={name}", upload.split('{').next().unwrap_or(upload)); + github("POST", &upload, Some((file, "application/octet-stream")))?; + let now = github("GET", &at, None)?; + if put(now.as_ref(), name, &digest)? != Put::Carried { + return Err(format!("{tag} does not carry {digest} as its {name} after the upload")); } Ok(format!("{tag} {said}")) } -/// Bootstrap this tree's toolchain, hold it to the glibc floor, and pack it into -/// `tarball`. -fn build(root: &Path, tag: &str, tarball: &Path) -> Result<(), String> { - if !(cfg!(target_os = "linux") && crate::arch::Arch::HOST == Some(crate::arch::Arch::X86_64)) { - return Err(format!( - "a build is {HOST}'s and this host is not one; a tarball built here would install \ - nowhere" - )); +/// The release `found` given `notes`. +fn renote(tmp: &Path, found: Option, notes: &str) -> Result { + let id = found.as_ref().and_then(|release| release["id"].as_u64()).ok_or("a release with no id")?; + send(tmp, "PATCH", &api(&format!("repos/{}/releases/{id}", repo())), &serde_json::json!({ "body": notes })) +} + +/// `body` sent to `url` by `method`, as JSON: what GitHub answered. +fn send(tmp: &Path, method: &str, url: &str, body: &Value) -> Result { + let file = tmp.join("request.json"); + fs::write(&file, body.to_string()).map_err(|e| format!("{}: {e}", file.display()))?; + github(method, url, Some((&file, "application/json")))?.ok_or_else(|| format!("{method} {url} found nothing")) +} + +fn api(path: &str) -> String { + format!("https://api.github.com/{path}") +} + +/// GitHub's answer to `method` on `url`, sent the file `body` names as its +/// content type where there is one: the JSON it answered, null for no content, +/// and `None` for a 404. +fn github(method: &str, url: &str, body: Option<(&Path, &str)>) -> Result, String> { + let token = std::env::var("GH_TOKEN").map_err(|_| "GH_TOKEN is unset".to_string())?; + let mut curl = Command::new("curl"); + curl.args(["-sSL", "--retry", "3", "-X", method, "-w", "\n%{http_code}", "-A", USER_AGENT]) + .args(["-H", &format!("Authorization: Bearer {token}"), "-H", "Accept: application/vnd.github+json"]); + if let Some((file, content_type)) = body { + curl.args(["-H", &format!("Content-Type: {content_type}"), "--data-binary"]).arg(format!("@{}", file.display())); } - let toyos = std::env::var("GITHUB_SHA").or_else(|_| crate::sync::git(root, &["rev-parse", "HEAD"]))?; - let manifest = manifest(root, tag, &toyos)?; - println!("{manifest}"); - run(Command::new("git").args(["submodule", "update", "--init", "rust"]).current_dir(root))?; - // Bootstrap takes `HEAD^1` as the upstream commit whose artifacts to fetch - // when it sees GitHub Actions; in this fork that is our own merge, which - // rust-lang's CI never built. - run(Command::new("cargo") - .args(["run", "--", "--build-only"]) - .env_remove("GITHUB_ACTIONS") - .env_remove("CI") - .current_dir(root))?; - - // What ships as `{HOST}/stage2` is the sysroot that build compiled against: - // the compiler with the guest libraries and `libtoyos_c.a` this tree's - // sources name, recorded beside the witness an installer checks it by. - let build = root.join("rust/build"); - let key = crate::keystore::recorded(root, crate::buildlock::Keyed::Sysroot).ok_or("the build recorded no sysroot key")?; - let sysroot = format!("sysroots/{key}"); - let stage2 = build.join(&sysroot); - fs::write(build.join("toyos-sysroot-witness"), crate::sysroot::witness(root)) - .map_err(|e| format!("recording the sysroot's witness: {e}"))?; - let need = shipped_glibc(&stage2)?; - if need > GLIBC_FLOOR { - return Err(format!( - "the host half needs GLIBC_{}.{} and a build states {}.{}: build it on the oldest \ - supported glibc, or move GLIBC_FLOOR deliberately", - need.0, need.1, GLIBC_FLOOR.0, GLIBC_FLOOR.1 - )); + let out = curl.arg(url).output().map_err(|e| format!("curl: {e}"))?; + if !out.status.success() { + return Err(format!("curl {method} {url} exited {}: {}", out.status, String::from_utf8_lossy(&out.stderr).trim())); } - fs::write(build.join("TOOLCHAIN"), &manifest).map_err(|e| e.to_string())?; - - // The sysroot's `bin/cargo` is a link into this runner's own toolchain; - // `Owner::Installed` recreates it. GNU tar's `--transform` renames the - // sysroot to the path an installer links. - let mut tar = Command::new("tar") - .arg("-C") - .arg(&build) - .arg(format!("--exclude={sysroot}/bin/cargo")) - .arg(format!("--transform=s,^{sysroot},{HOST}/stage2,")) - .args(["-c", &sysroot]) - .args(["toyos-sysroot-witness", "TOOLCHAIN"]) - .stdout(Stdio::piped()) - .spawn() - .map_err(|e| format!("tar: {e}"))?; - let stream = tar.stdout.take().expect("piped"); - let zstd = Command::new("zstd").args(["-T0", "-3", "-f", "-o"]).arg(tarball).stdin(stream).status(); - let tar = tar.wait().map_err(|e| format!("tar: {e}"))?; - let zstd = zstd.map_err(|e| format!("zstd: {e}"))?; - if !(tar.success() && zstd.success()) { - return Err(format!("packaging: tar exited {tar}, zstd {zstd}")); + let text = String::from_utf8_lossy(&out.stdout).into_owned(); + let (answer, status) = text.rsplit_once('\n').unwrap_or(("", text.as_str())); + match status { + "200" | "201" => serde_json::from_str(answer).map(Some).map_err(|e| format!("{method} {url} answered no JSON: {e}")), + "204" => Ok(Some(Value::Null)), + "404" => Ok(None), + status => Err(format!("{method} {url} answered {status}: {answer}")), } - Ok(()) +} + +fn file_sha256(path: &Path) -> Result { + let mut file = fs::File::open(path).map_err(|e| format!("{}: {e}", path.display()))?; + let mut hasher = Sha256::new(); + std::io::copy(&mut file, &mut hasher).map_err(|e| format!("{}: {e}", path.display()))?; + Ok(hasher.finalize().iter().map(|b| format!("{b:02x}")).collect()) } /// Every `GLIBC_x.y` the shipped host binaries and libraries name, as the /// newest: `rustc` and its libraries, and the `rust-lld`, clang and LLVM tools -/// beside them. A byte scan: it can only over-report, so its failure is a -/// refused build. +/// beside them. fn shipped_glibc(stage2: &Path) -> Result<(u32, u32), String> { let mut files: Vec = Vec::new(); let tools = stage2.join(format!("lib/rustlib/{HOST}/bin")); @@ -569,16 +529,11 @@ fn glibc_named(bytes: &[u8]) -> (u32, u32) { newest } -/// `TOOLCHAIN`: the pin a consumer writes down, and what it gets, `toyos` being -/// the commit that built it. Inside the tarball, and the alias release's own -/// asset. -fn manifest(root: &Path, tag: &str, toyos: &str) -> Result { - Ok(format!( - "toolchain {tag}\ntoyos {toyos}\nrust {}\nhost {HOST}\nglibc {}.{}\n", - crate::sync::git(root, &["rev-parse", "HEAD:rust"])?, - GLIBC_FLOOR.0, - GLIBC_FLOOR.1 - )) +/// `TOOLCHAIN`: the pin a consumer writes down, and what it gets. Inside the +/// tarball, and the start of the alias release's own asset; only what the +/// sysroot's key decides, so one sysroot packs to one digest. +fn manifest(tag: &str) -> String { + format!("toolchain {tag}\nhost {HOST}\nglibc {}.{}\n", GLIBC_FLOOR.0, GLIBC_FLOOR.1) } /// The release notes: how to install it, what glibc it needs. @@ -628,9 +583,10 @@ Until [rust-windowing/raw-window-handle#223](https://github.com/rust-windowing/r } /// `toolchain-linux-x86_64-sdk-`: -/// the name a consumer pins, moved onto this tree's toolchain. A second release -/// carrying only the manifest, because GitHub hangs an asset off one release id. -fn alias(root: &Path, manifest: &str, tmp: &Path) -> Result { +/// the name a consumer pins, moved onto `tag`. A second release carrying only a +/// `TOOLCHAIN` naming `tag`, the commit that put it up and the SDK crates, +/// because GitHub hangs an asset off one release id. +fn alias(root: &Path, tag: &str, notes: &str, tmp: &Path) -> Result { let plan = crate::sdkversion::plan(root)?; if let Some(owed) = plan.iter().find(|r| r.publish) { let name = owed.krate.name; @@ -639,168 +595,17 @@ fn alias(root: &Path, manifest: &str, tmp: &Path) -> Result { let abi = plan.iter().find(|r| r.krate.name == "toyos-abi").ok_or("toyos-abi is not published")?; let abi = abi.version.split('+').next().unwrap_or(&abi.version); let alias = format!("toolchain-linux-x86_64-sdk-{abi}"); - let notes = tmp.join("notes.md"); - let toolchain = tmp.join("TOOLCHAIN"); - // The tarball's copy names no crates.io version, so it never depends on crates.io. + let commit = |rev: &str| crate::sync::git(root, &["rev-parse", rev]).map(|sha| sha.trim().to_string()); let sdk: String = plan.iter().map(|r| format!("{} {}\n", r.krate.name, r.version)).collect(); - fs::write(&toolchain, format!("{manifest}{sdk}")).map_err(|e| e.to_string())?; - let gh = |args: &[&str], files: &[&Path]| { - Command::new("gh").args(args).args(files).current_dir(root).status().is_ok_and(|s| s.success()) - }; - let created = gh(&["release", "create", &alias, "--title", &alias, "--notes-file"], &[¬es, &toolchain]); - let moved = created - || (gh(&["release", "edit", &alias, "--notes-file"], &[¬es]) - && gh(&["release", "upload", &alias, "--clobber"], &[&toolchain])); - moved.then(|| format!("{alias} names it")).ok_or_else(|| format!("{alias} could not be moved")) + let toolchain = tmp.join("TOOLCHAIN"); + let text = format!("{}toyos {}\nrust {}\n{sdk}", manifest(tag), commit("HEAD")?, commit("HEAD:rust")?); + fs::write(&toolchain, text).map_err(|e| format!("{}: {e}", toolchain.display()))?; + put_up(root, &alias, notes, &toolchain, tmp).map(|said| format!("{said}, naming {tag}")) } #[cfg(test)] mod tests { use super::*; - use std::collections::BTreeSet; - - /// The packaging is one of the trees its own tag hashes. - #[test] - fn the_tag_hashes_this_file() { - assert!(trees().contains(&file!())); - for tree in trees() { - assert!( - Path::new(env!("CARGO_MANIFEST_DIR")).join(tree).exists(), - "{tree} is hashed into the tag and is not in the tree" - ); - } - } - - /// Every module `text` names as `crate::`, alone or in a - /// `crate::{…}` group. - fn crate_modules(text: &str) -> Vec { - let ident = |name: &str| -> String { - name.trim().chars().take_while(|c| c.is_ascii_alphanumeric() || *c == '_').collect() - }; - let mut found = Vec::new(); - for (at, _) in text.match_indices("crate::") { - let rest = &text[at + "crate::".len()..]; - match rest.strip_prefix('{') { - Some(group) => found.extend(group.split('}').next().unwrap_or("").split(',').map(ident)), - None => found.push(ident(rest)), - } - } - found.retain(|name| !name.is_empty()); - found - } - - /// [`BUILDERS`] is every module `src/toolchain.rs` and this file reach through - /// `crate::`: a module the toolchain's build starts calling is one more the - /// tag hashes, and this is what says so. - #[test] - fn the_tag_hashes_every_module_that_builds_the_toolchain() { - let here = Path::new(env!("CARGO_MANIFEST_DIR")); - let mut reached = BTreeSet::new(); - let mut todo = vec!["src/toolchain.rs".to_string(), file!().to_string()]; - while let Some(file) = todo.pop() { - if !reached.insert(file.clone()) { - continue; - } - let text = fs::read_to_string(here.join(&file)).unwrap_or_else(|e| panic!("{file}: {e}")); - for module in crate_modules(&text) { - let path = format!("src/{module}.rs"); - if here.join(&path).is_file() { - todo.push(path); - } - } - } - let declared: BTreeSet = BUILDERS.iter().map(|s| s.to_string()).collect(); - assert_eq!(reached, declared); - } - - #[test] - fn a_group_import_names_each_of_its_modules() { - let text = "use crate::{flags, release::tag, sync};\nlet x = crate::arch::Arch::HOST;"; - assert_eq!(crate_modules(text), ["flags", "release", "sync", "arch"]); - } - - fn git(dir: &Path, args: &[&str]) -> String { - let out = Command::new("git") - .args(["-c", "commit.gpgsign=false", "-c", "user.email=t@t", "-c", "user.name=t"]) - .args(["-c", "init.defaultBranch=main"]) - .args(crate::gitfixture::NO_AUTO_MAINTENANCE) - .args(args) - .current_dir(dir) - .output() - .expect("run git"); - assert!(out.status.success(), "git {args:?}: {}", String::from_utf8_lossy(&out.stderr)); - String::from_utf8_lossy(&out.stdout).trim().to_string() - } - - /// A commit to the C and C++ toolchain's declarations or headers, to the n2 - /// it is built under, or to any module that builds it moves the tag. - #[test] - fn the_tag_moves_with_what_the_toolchain_is_built_from_and_by() { - let repo = TempDir::new("release-tag"); - let here = Path::new(env!("CARGO_MANIFEST_DIR")); - let write = |path: &str, text: &str| { - let path = repo.join(path); - fs::create_dir_all(path.parent().unwrap()).unwrap(); - fs::write(path, text).unwrap(); - }; - git(&repo, &["init", "-q"]); - for tree in trees() { - match tree { - "rust" => { - git(&repo, &["update-index", "--add", "--cacheinfo", "160000,1111111111111111111111111111111111111111,rust"]); - } - file if here.join(file).is_file() => write(file, &fs::read_to_string(here.join(file)).unwrap()), - dir => write(&format!("{dir}/placeholder"), "x"), - } - } - fs::create_dir_all(repo.join("rust")).unwrap(); - git(&repo, &["add", "-A"]); - git(&repo, &["commit", "-qm", "the tree"]); - let mut before = tag(&repo).unwrap(); - - let clang = fs::read_to_string(here.join("src/clang.rs")).unwrap(); - let tools = r#"const TOOLS: [&str; 3] = ["llvm-ar", "clang", "ld.lld"];"#; - let objdump = r#"const TOOLS: [&str; 4] = ["llvm-ar", "clang", "ld.lld", "llvm-objdump"];"#; - let targets = r#"targets = \"AArch64;X86\""#; - let riscv = r#"targets = \"AArch64;RISCV;X86\""#; - assert!(clang.contains(tools) && clang.contains(targets), "src/clang.rs no longer declares what this mutates"); - let with_objdump = clang.replace(tools, objdump); - let cxx = fs::read_to_string(here.join("src/libcxx.rs")).unwrap(); - let (no_fs, fs_on) = (r#"("LIBCXX_ENABLE_FILESYSTEM", "OFF")"#, r#"("LIBCXX_ENABLE_FILESYSTEM", "ON")"#); - assert!(cxx.contains(no_fs), "src/libcxx.rs no longer declares what this mutates"); - let n2 = fs::read_to_string(here.join("src/n2.rs")).unwrap(); - let pin = crate::n2::N2[crate::n2::N2.len() - 1]; - assert!(n2.contains(pin), "src/n2.rs no longer declares what this mutates"); - let mut moves = |path: &str, text: String| { - write(path, &text); - git(&repo, &["add", "-A"]); - git(&repo, &["commit", "-qm", "a mutation"]); - let after = tag(&repo).unwrap(); - assert_ne!(after, before, "a commit to {path} kept the tag"); - before = after; - }; - moves("src/clang.rs", with_objdump.clone()); - moves("src/clang.rs", with_objdump.replace(targets, riscv)); - moves("src/libcxx.rs", cxx.replace(no_fs, fs_on)); - moves("src/n2.rs", n2.replace(pin, &"0".repeat(pin.len()))); - moves("userland/libc/include/placeholder", "y".to_string()); - for builder in BUILDERS { - let text = fs::read_to_string(here.join(builder)).unwrap(); - moves(builder, format!("{text}\nconst MOVED: () = ();\n")); - } - } - - /// `sha256sum`'s digest of the same bytes, cut to the same width. - #[test] - fn the_hash_is_sha256_of_what_git_printed() { - assert_eq!( - sha256_hex(b""), - "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" - ); - let tag = tag(Path::new(env!("CARGO_MANIFEST_DIR"))).expect("this checkout has a HEAD"); - let hex = tag.strip_prefix("toolchain-linux-x86_64-").expect("the prefix"); - assert!(hex.len() == 16 && hex.bytes().all(|b| b.is_ascii_hexdigit()), "{tag}"); - } #[test] fn the_glibc_scan_takes_the_newest_version_and_nothing_else() { @@ -814,24 +619,25 @@ mod tests { const REPO: &str = "ToyOSOrg/ToyOS"; /// The negative control on the publisher: the release job run as a pull - /// request's job, the merge queue's, the nightly's, or main's publisher - /// dispatched on a branch or run by any other event is refused by name - /// before it reads anything, and so is another repository's publisher. + /// request's job, the merge queue's, a push's, or main's nightly dispatched + /// on a branch or run by any other event is refused by name before it reads + /// anything, and so is another repository's nightly. #[test] fn only_mains_publisher_publishes() { - let mains = format!("{REPO}/.github/workflows/publish.yml@refs/heads/main"); - assert!(publisher(Some(&mains), Some("push"), REPO).is_ok()); + let mains = format!("{REPO}/.github/workflows/nightly.yml@refs/heads/main"); + assert!(publisher(Some(&mains), Some("schedule"), REPO).is_ok()); assert!(publisher(Some(&mains), Some("workflow_dispatch"), REPO).is_ok()); - let fork = "Fork/ToyOS/.github/workflows/publish.yml@refs/heads/main"; - assert!(publisher(Some(fork), Some("push"), REPO).unwrap_err().contains(fork)); + let fork = "Fork/ToyOS/.github/workflows/nightly.yml@refs/heads/main"; + assert!(publisher(Some(fork), Some("schedule"), REPO).unwrap_err().contains(fork)); let root = Path::new(env!("CARGO_MANIFEST_DIR")); let repo = repo(); let refused = [ (format!("{repo}/.github/workflows/ci.yml@refs/pull/671/merge"), "pull_request"), (format!("{repo}/.github/workflows/ci.yml@refs/heads/gh-readonly-queue/main/pr-671-59052827f"), "merge_group"), - (format!("{repo}/.github/workflows/nightly.yml@refs/heads/main"), "schedule"), - (format!("{repo}/.github/workflows/publish.yml@refs/heads/wt/toyos-guestci"), "workflow_dispatch"), - (format!("{repo}/.github/workflows/publish.yml@refs/heads/main"), "pull_request_target"), + (format!("{repo}/.github/workflows/publish.yml@refs/heads/main"), "push"), + (format!("{repo}/.github/workflows/nightly.yml@refs/heads/wt/toyos-guestci"), "workflow_dispatch"), + (format!("{repo}/.github/workflows/nightly.yml@refs/tags/main"), "push"), + (format!("{repo}/.github/workflows/nightly.yml@refs/heads/main"), "workflow_run"), ]; for (workflow, event) in refused { let why = release_as(root, Some(&workflow), Some(event)).expect_err(&workflow); @@ -840,105 +646,237 @@ mod tests { assert!(release_as(root, None, None).unwrap_err().starts_with("only ")); } - fn run_json(path: &str, branch: &str, event: &str, head_repo: &str) -> Value { - serde_json::json!({ - "path": path, "head_branch": branch, "event": event, - "repository": { "full_name": REPO }, "head_repository": { "full_name": head_repo }, - }) + fn release_json(assets: Value) -> Value { + serde_json::json!({ "id": 7, "upload_url": "https://uploads.github.com/repos/o/r/releases/7/assets{?name,label}", "assets": assets }) } + /// The negative control on what a release is made to carry: none is + /// created, one with no such asset is given it, one carrying these bytes is + /// left, and one carrying any other bytes, or bytes GitHub records no digest + /// for, has its asset replaced. #[test] - fn mains_builds_are_publish_yml_on_main_and_nothing_else() { - let publish = ".github/workflows/publish.yml"; - assert!(is_mains(&run_json(publish, "main", "push", REPO), REPO)); - assert!(is_mains(&run_json(publish, "main", "workflow_dispatch", REPO), REPO)); - assert!(!is_mains(&run_json(".github/workflows/nightly.yml", "main", "schedule", REPO), REPO)); - assert!(!is_mains(&run_json(".github/workflows/ci.yml", "main", "pull_request", REPO), REPO)); - assert!(!is_mains(&run_json(publish, "wt/toyos-guestci", "workflow_dispatch", REPO), REPO)); - assert!(!is_mains(&run_json(publish, "main", "pull_request", "Fork/ToyOS"), REPO)); - } - - fn artifact(id: u64, digest: Value, run: u64, head: &str, branch: &str, expired: bool) -> Value { - serde_json::json!({ - "id": id, "digest": digest, "expired": expired, - "workflow_run": { "id": run, "head_sha": head, "head_branch": branch }, - }) + fn a_release_is_made_to_carry_these_bytes_and_no_other() { + let digest = "sha256:aa"; + assert_eq!(put(None, ASSET, digest), Ok(Put::Create)); + let other = serde_json::json!({ "id": 3, "name": "notes.txt", "digest": digest }); + assert_eq!(put(Some(&release_json(serde_json::json!([other]))), ASSET, digest), Ok(Put::Upload)); + let ours = serde_json::json!([{ "id": 4, "name": ASSET, "digest": digest }]); + assert_eq!(put(Some(&release_json(ours)), ASSET, digest), Ok(Put::Carried)); + let theirs = serde_json::json!([{ "id": 5, "name": ASSET, "digest": "sha256:bb" }]); + assert_eq!(put(Some(&release_json(theirs)), ASSET, digest), Ok(Put::Replace { asset: 5 })); + let unrecorded = serde_json::json!([{ "id": 6, "name": ASSET, "digest": null }]); + assert_eq!(put(Some(&release_json(unrecorded)), ASSET, digest), Ok(Put::Replace { asset: 6 })); + assert!(put(Some(&serde_json::json!({ "message": "Not Found" })), ASSET, digest).is_err()); + } + + /// A toolchain laid out under `build` as [`lay_out`] leaves one, with a + /// file, an executable, a directory, a link and a `bin/cargo`. + fn laid_out(build: &Path, stamp: &str) { + let stage2 = build.join(HOST).join("stage2"); + for (file, text) in [("bin/rustc", "rustc"), ("lib/libstd.rlib", stamp), ("lib/rustlib/empty/.keep", "")] { + fs::create_dir_all(stage2.join(file).parent().unwrap()).unwrap(); + fs::write(stage2.join(file), text).unwrap(); + } + use std::os::unix::fs::PermissionsExt; + fs::set_permissions(stage2.join("bin/rustc"), fs::Permissions::from_mode(0o755)).unwrap(); + std::os::unix::fs::symlink("rustc", stage2.join("bin/ld.lld")).unwrap(); + std::os::unix::fs::symlink("/a/runner/s/cargo", stage2.join("bin/cargo")).unwrap(); + fs::write(build.join("toyos-sysroot-witness"), "toyos-abi/src/lib.rs:00").unwrap(); + fs::write(build.join("TOOLCHAIN"), manifest("toolchain-linux-x86_64-0123456789abcdef")).unwrap(); + } + + /// What the tarball at `path` holds, by name: a link by what it names, a + /// file by its bytes and mode. + fn unpacked(path: &Path) -> Vec<(String, String)> { + let bytes = fs::read(path).unwrap(); + let mut tar = Vec::new(); + ruzstd::decoding::StreamingDecoder::new(bytes.as_slice()).unwrap().read_to_end(&mut tar).unwrap(); + let mut archive = tar::Archive::new(tar.as_slice()); + let mut seen = Vec::new(); + for entry in archive.entries().unwrap() { + let mut entry = entry.unwrap(); + let name = entry.path().unwrap().display().to_string(); + let (kind, mode) = (entry.header().entry_type(), entry.header().mode().unwrap()); + let what = match kind { + tar::EntryType::Symlink => format!("-> {}", entry.link_name().unwrap().unwrap().display()), + tar::EntryType::Directory => "dir".to_string(), + _ => { + let mut text = String::new(); + entry.read_to_string(&mut text).unwrap(); + format!("{mode:o} {text}") + } + }; + seen.push((name, what)); + } + seen } - /// GitHub's list, read: an expired artifact and one with no digest are not - /// builds. + use std::io::Read; + + /// **One sysroot packs to one digest, and unpacks whole**: packed again + /// after its files were written again later, it is the same bytes; it holds + /// every file, mode and link of `stage2` but `bin/cargo`, then the witness + /// and `TOOLCHAIN`; and other bytes are another digest. #[test] - fn a_build_is_an_unexpired_artifact_with_a_digest() { - let listing = serde_json::json!({ "artifacts": [ - artifact(1, "sha256:aa".into(), 10, "h1", "main", false), - artifact(3, "sha256:cc".into(), 12, "h3", "wt/y", true), - artifact(4, Value::Null, 13, "h4", "wt/z", false), - artifact(5, "sha256:ee".into(), 14, "h5", "wt/q", false), - ]}); - let got: Vec<(u64, String)> = listed(&listing).into_iter().map(|(b, branch)| (b.artifact, branch)).collect(); - assert_eq!(got, [(1, "main".to_string()), (5, "wt/q".to_string())]); + fn one_sysroot_packs_to_one_digest_and_unpacks_whole() { + let tmp = TempDir::new("release-pack"); + let (first, again, other) = (tmp.join("first"), tmp.join("again"), tmp.join("other")); + laid_out(&first.join("build"), "std"); + laid_out(&again.join("build"), "std"); + laid_out(&other.join("build"), "another std"); + let later = std::time::SystemTime::now() + std::time::Duration::from_secs(3600); + fs::File::options().write(true).open(again.join("build").join(HOST).join("stage2/lib/libstd.rlib")).unwrap().set_modified(later).unwrap(); + for dir in [&first, &again, &other] { + pack(&dir.join("build"), &dir.join(ASSET)).unwrap(); + } + let digest = |dir: &Path| file_sha256(&dir.join(ASSET)).unwrap(); + assert_eq!(digest(&first), digest(&again), "one sysroot packed to two digests"); + assert_ne!(digest(&first), digest(&other)); + let stage2 = format!("{HOST}/stage2"); + let want: Vec<(String, String)> = [ + (stage2.clone(), "dir"), + (format!("{stage2}/bin"), "dir"), + (format!("{stage2}/bin/ld.lld"), "-> rustc"), + (format!("{stage2}/bin/rustc"), "755 rustc"), + (format!("{stage2}/lib"), "dir"), + (format!("{stage2}/lib/libstd.rlib"), "644 std"), + (format!("{stage2}/lib/rustlib"), "dir"), + (format!("{stage2}/lib/rustlib/empty"), "dir"), + (format!("{stage2}/lib/rustlib/empty/.keep"), "644 "), + ("toyos-sysroot-witness".to_string(), "644 toyos-abi/src/lib.rs:00"), + ("TOOLCHAIN".to_string(), "644 toolchain toolchain-linux-x86_64-0123456789abcdef\nhost x86_64-unknown-linux-gnu\nglibc 2.39\n"), + ] + .map(|(name, what)| (name, what.to_string())) + .to_vec(); + assert_eq!(unpacked(&first.join(ASSET)), want); + } + + /// A checkout whose witness reads `abi`, under `root`. + fn checkout(root: &Path, abi: &str) { + for tree in crate::sysroot::SYSROOT_SOURCES { + fs::create_dir_all(root.join(tree)).unwrap(); + } + fs::write(root.join("toyos-abi/src/lib.rs"), abi).unwrap(); + for manifest in crate::sysroot::SYSROOT_MANIFESTS { + fs::create_dir_all(root.join(manifest).parent().unwrap()).unwrap(); + fs::write(root.join(manifest), "[package]\n").unwrap(); + } + } + + /// A sysroot store under `root`'s `rust/build/sysroots`, as a job restores + /// one, recording `witness`. + fn restored(root: &Path, key: &str, witness: &str) -> PathBuf { + let dir = crate::sysroot::sysroots_dir(&root.join("rust")).join(key); + fs::create_dir_all(dir.join("bin")).unwrap(); + fs::write(dir.join("bin/rustc"), "rustc").unwrap(); + fs::write(dir.join("SOURCES"), format!("{key}\nfork /a/runner/s/rust\n{witness}\n")).unwrap(); + dir } - fn kept(artifact: u64, head: &str, mains: bool) -> Build { - Build { artifact, digest: format!("sha256:{artifact}"), run: artifact, head: head.into(), mains } + /// **A job installs the one sysroot it restored, and only one built from + /// its own tree's sources**: none, two, or one named by no key is refused; + /// one whose recorded witness is not this tree's is refused; the one that + /// is lands at `stage2` with that witness and its `TOOLCHAIN`. + #[test] + fn a_job_lays_out_the_one_sysroot_it_restored() { + let tmp = TempDir::new("release-lay-out"); + let root = tmp.join("checkout"); + checkout(&root, "pub struct A;\n"); + fs::create_dir_all(crate::sysroot::sysroots_dir(&root.join("rust"))).unwrap(); + assert!(lay_out(&root).unwrap_err().contains("holds 0 entries")); + let witness = crate::sysroot::witness(&root); + let one = restored(&root, "0123456789abcdef", &witness); + let two = restored(&root, "fedcba9876543210", &witness); + assert!(lay_out(&root).unwrap_err().contains("holds 2 entries")); + fs::remove_dir_all(&two).unwrap(); + fs::write(root.join("toyos-abi/src/lib.rs"), "pub struct A(u64);\n").unwrap(); + assert!(lay_out(&root).unwrap_err().contains("built from other sources"), "a sysroot of other sources was laid out"); + fs::write(root.join("toyos-abi/src/lib.rs"), "pub struct A;\n").unwrap(); + assert_eq!(lay_out(&root), Ok(Key::parse("0123456789abcdef").unwrap())); + let rust_dir = root.join("rust"); + assert!(!one.exists() && crate::toolchain::stage2(&rust_dir).join("bin/rustc").is_file()); + assert_eq!(fs::read_to_string(crate::toolchain::witness_path(&rust_dir)).unwrap(), witness); + let toolchain = fs::read_to_string(crate::toolchain::manifest_path(&rust_dir)).unwrap(); + assert_eq!(toolchain, manifest("toolchain-linux-x86_64-0123456789abcdef")); + fs::create_dir_all(rust_dir.join("build/sysroots/not-a-key")).unwrap(); + assert!(lay_out(&root).unwrap_err().contains("named by no key")); + } + + fn layer(name: &'static str, key: &str, paths: &[&str]) -> Layer { + let kind = LAYERS.iter().find(|(_, n)| *n == name).unwrap().0; + Layer { kind, name, key: Key::parse(key).unwrap(), paths: paths.iter().map(PathBuf::from).collect() } + } + + /// The job's outputs, as GitHub's multiline syntax reads them: each layer's + /// cache entry, and each path of it on a line of its own. + #[test] + fn a_job_is_told_each_layer_s_entry_and_paths() { + let layers = [ + layer("llvm", "1111111111111111", &["rust/build/llvm/1111111111111111"]), + layer("compiler", "2222222222222222", &["rust/build/h/stage2", "rust/build/toyos-compiler"]), + ]; + assert_eq!( + outputs(&layers), + "llvm-key=toolchain-llvm-1111111111111111\nllvm-path< = ["landed".to_string(), "own".to_string()].into(); - let builds = [kept(4, "passed-through", false), kept(3, "own", false), kept(2, "main", true)]; - assert_eq!(choose(&builds, &vouched, false), Some(&builds[2])); - assert_eq!(choose(&builds, &vouched, true), Some(&builds[2])); - let unpublished = [kept(4, "passed-through", false), kept(3, "own", false), kept(1, "landed", false)]; - assert_eq!(choose(&unpublished, &vouched, false), Some(&unpublished[1])); - assert_eq!(choose(&unpublished, &vouched, true), None); - assert_eq!(choose(&[kept(4, "passed-through", false)], &vouched, false), None); - } - - /// A tree vouches for its first-parent chain and the head each merge on it - /// took in, and for no commit a branch passed through before its head. + fn a_restored_layer_that_is_not_whole_is_refused() { + let layers: Vec = + LAYERS.iter().enumerate().map(|(at, (_, name))| layer(name, &at.to_string().repeat(16), &["p"])).collect(); + let broken = |layer: &Layer| (layer.name == "compiler").then(|| "stage2 carries no clang".to_string()); + let refused = whole_as_restored(&layers, &[true, true, false, false], broken).unwrap_err(); + assert!(refused.starts_with("compiler 1111111111111111 was restored") && refused.contains("no clang"), "{refused}"); + assert_eq!(whole_as_restored(&layers, &[true, false, false, false], broken), Ok(())); + assert_eq!(whole_as_restored(&layers, &[true; 4], |_| None), Ok(())); + } + + /// **Each layer is the store the build system makes, where it makes it**: + /// the LLVM, the freestanding libraries and the sysroot in their stores by + /// key, and the primary's compiler as its `stage2` and its record, each + /// relative to the checkout, keyed before any is built. #[test] - fn a_tree_vouches_for_its_first_parent_chain_and_the_heads_it_merged() { - let repo = TempDir::new("release-vouched"); - let commit = |message: &str| { - git(&repo, &["commit", "-q", "--allow-empty", "-m", message]); - git(&repo, &["rev-parse", "HEAD"]) - }; - git(&repo, &["init", "-q"]); - let m0 = commit("m0"); - git(&repo, &["switch", "-q", "-c", "landed"]); - let p0 = commit("p0"); - let p1 = commit("p1"); - git(&repo, &["switch", "-q", "main"]); - let m1 = commit("m1"); - git(&repo, &["merge", "-q", "--no-ff", "-m", "m2", "landed"]); - let m2 = git(&repo, &["rev-parse", "HEAD"]); - git(&repo, &["switch", "-q", "-c", "own", &m1]); - let q0 = commit("q0"); - let q1 = commit("q1"); - git(&repo, &["switch", "-q", "--detach", &m2]); - git(&repo, &["merge", "-q", "--no-ff", "-m", "the pull request's merge", "own"]); - let head = git(&repo, &["rev-parse", "HEAD"]); - let got = vouched(&repo).unwrap(); - let want: HashSet = [head, m2, m1, m0, p1, q1].into(); - assert_eq!(got, want, "p0 {p0} and q0 {q0} are what a branch passed through"); - } - - /// The negative control on the download: bytes that hash to anything but - /// GitHub's digest are refused, and so is a digest that is not SHA-256's. + fn the_layers_are_the_stores_the_build_makes() { + let scratch = TempDir::new("release-layers"); + let (primary, _rust_dir, _) = crate::compiler::tests::estate(&scratch); + checkout(&primary, "pub struct A;\n"); + let layers = layers(&primary); + let named: Vec<(&str, Vec)> = + layers.iter().map(|l| (l.name, l.paths.iter().map(|p| p.display().to_string()).collect())).collect(); + let key = |at: usize| layers[at].key.to_string(); + let host = crate::toolchain::host_triple(); + assert_eq!( + named, + [ + ("llvm", vec![format!("rust/build/llvm/{}", key(0))]), + ("compiler", vec![format!("rust/build/{host}/stage2"), "rust/build/toyos-compiler".to_string()]), + ("freestanding", vec![format!("rust/build/freestanding/{}", key(2))]), + ("sysroot", vec![format!("rust/build/sysroots/{}", key(3))]), + ] + ); + assert_eq!(layers[1].key, crate::compiler::primary_key(&primary.join("rust"))); + } + + /// **A job saves exactly the layers it built**: none where it restored the + /// sysroot, which is all a guest job reads, and otherwise each it did not + /// restore. #[test] - fn a_download_is_held_to_the_digest_github_recorded() { - let dir = TempDir::new("release-verify"); - let file = dir.join("toolchain.tar.zst"); - fs::write(&file, b"").unwrap(); - let empty = format!("sha256:{}", sha256_hex(b"")); - assert!(verify(&file, &empty).is_ok()); - fs::write(&file, b"another writer's").unwrap(); - assert!(verify(&file, &empty).unwrap_err().contains("hashes to")); - assert!(verify(&file, &sha256_hex(b"another writer's")).unwrap_err().contains("is not a SHA-256 digest")); + fn a_job_saves_only_what_it_built() { + let layers: Vec = LAYERS + .iter() + .enumerate() + .map(|(at, (_, name))| layer(name, &at.to_string().repeat(16), &["p"])) + .collect(); + let told = |restored: [bool; 4]| built(&layers, &restored); + assert_eq!(told([true, true, false, true]), "llvm=kept\ncompiler=kept\nfreestanding=kept\nsysroot=kept\n"); + assert_eq!(told([false, false, false, true]), "llvm=kept\ncompiler=kept\nfreestanding=kept\nsysroot=kept\n"); + assert_eq!(told([true, true, false, false]), "llvm=kept\ncompiler=kept\nfreestanding=built\nsysroot=built\n"); + assert_eq!(told([false; 4]), "llvm=built\ncompiler=built\nfreestanding=built\nsysroot=built\n"); } } diff --git a/src/sysroot.rs b/src/sysroot.rs index 4475eb4e9f0..03ecca52e78 100644 --- a/src/sysroot.rs +++ b/src/sysroot.rs @@ -4,8 +4,9 @@ //! **A sysroot is a function of its key.** The key ([`key`]) is the identity //! (`src/identity.rs`, so a comment is no change) of everything a sysroot is //! built from: the three trees std and `libtoyos_c.a` compile -//! ([`SYSROOT_SOURCES`]), the std fork's `library/` and `src/bootstrap/` in the -//! checkout that builds it, and the compiler that builds it. `rust/build/ +//! ([`SYSROOT_SOURCES`]) and how libc is built ([`SYSROOT_MANIFESTS`], +//! `libc::BUILD`), the std fork's `library/` and `src/bootstrap/` in the +//! checkout that builds it, and the compiler's key. `rust/build/ //! sysroots//` is a whole toolchain — the compiler's files cloned from its //! `stage2`, the guest targets' libraries built from this key's sources. A build //! compiles against the directory its own key names, so two worktrees with @@ -64,9 +65,15 @@ use whole_toolchain::{whole, Whole}; pub const SYSROOT_SOURCES: [&str; 4] = ["toyos-abi/src", "toyos/src", "userland/libc/src", "userland/libc/include"]; -/// Their manifests, whose features and versions decide the same build. -pub(crate) const SYSROOT_MANIFESTS: [&str; 3] = - ["toyos-abi/Cargo.toml", "toyos/Cargo.toml", "userland/libc/Cargo.toml"]; +/// Their manifests, and the lockfile and cargo configuration libc is built +/// under: the features, versions and flags of the same build. +pub(crate) const SYSROOT_MANIFESTS: [&str; 5] = [ + "toyos-abi/Cargo.toml", + "toyos/Cargo.toml", + "userland/libc/Cargo.toml", + "userland/libc/Cargo.lock", + "userland/.cargo/config.toml", +]; /// Of [`SYSROOT_MANIFESTS`], the ones std's lockfile resolves with the fork's /// own: what of a worktree can move a freestanding target's dependency versions. @@ -357,7 +364,7 @@ impl Keys { /// The keys of what `root` builds against with its std fork at `fork`, /// compiled by `compiler`. fn of(root: &Path, compiler: &Compiler, fork: &Path) -> Self { - let freestanding = freestanding_key(root, compiler, fork); + let freestanding = freestanding_key(root, &compiler.key(), fork); let sysroot = key(root, &freestanding); let identity = Identity::new(Key::of(compiler.identity().as_bytes()), &freestanding, &sysroot); Self { freestanding, sysroot, identity } @@ -365,9 +372,10 @@ impl Keys { } /// The key of the freestanding targets' libraries `root` builds against with -/// its std fork at `fork`, compiled by `compiler`: none of -/// [`SYSROOT_SOURCES`], and of `root` only [`STD_MANIFESTS`]. -fn freestanding_key(root: &Path, compiler: &Compiler, fork: &Path) -> Key { +/// its std fork at `fork`, compiled by the compiler whose key is `compiler` +/// (`Compiler::key`): none of [`SYSROOT_SOURCES`], and of `root` only +/// [`STD_MANIFESTS`]. +pub(crate) fn freestanding_key(root: &Path, compiler: &Key, fork: &Path) -> Key { freestanding_key_of(root, compiler, fork, RECIPE, &keyed_std_config()) } @@ -378,13 +386,13 @@ fn keyed_std_config() -> String { } /// [`freestanding_key`], with the recipe and std's configuration it reads. -fn freestanding_key_of(root: &Path, compiler: &Compiler, fork: &Path, recipe: &str, config: &str) -> Key { +fn freestanding_key_of(root: &Path, compiler: &Key, fork: &Path, recipe: &str, config: &str) -> Key { let parts = [ format!("{recipe}; cargo {STAGE0_CARGO}; targets {}", Libraries::Freestanding.targets().join(" ")), config.to_string(), STD_MANIFESTS.map(|manifest| manifest_line(root, manifest)).join("\n"), tree_identity(fork, &["library", "src/bootstrap"], Links::Refused), - compiler.identity(), + compiler.to_string(), ]; Key::of(parts.join("\n\0\n").as_bytes()) } @@ -392,12 +400,21 @@ fn freestanding_key_of(root: &Path, compiler: &Compiler, fork: &Path, recipe: &s /// The key of the sysroot `root` builds against, whose freestanding libraries /// are `freestanding`'s ([`freestanding_key`], which names the recipe, std's /// configuration, the fork and the compiler the rest is built with too). -fn key(root: &Path, freestanding: &Key) -> Key { - let parts = [ - format!("targets {}; C++ runtime {:?}", Libraries::Worktree.targets().join(" "), crate::libcxx::OPTIONS), - witness(root), - freestanding.to_string(), - ]; +pub(crate) fn key(root: &Path, freestanding: &Key) -> Key { + key_of(root, freestanding, &build_text()) +} + +/// What a sysroot's build is beyond its sources and its freestanding libraries: +/// its targets, the C++ runtime's options and libc's cargo invocations. +fn build_text() -> String { + let targets = Libraries::Worktree.targets().join(" "); + let (libc, staticlib) = (crate::libc::BUILD, crate::libc::BUILD_C); + format!("targets {targets}; C++ runtime {:?}; libc {libc:?} {staticlib:?}", crate::libcxx::OPTIONS) +} + +/// [`key`], with the build it reads. +fn key_of(root: &Path, freestanding: &Key, build: &str) -> Key { + let parts = [build.to_string(), witness(root), freestanding.to_string()]; Key::of(parts.join("\n\0\n").as_bytes()) } @@ -495,9 +512,25 @@ pub fn fork_checkout(root: &Path) -> PathBuf { fork } +/// What a sysroot's [`SOURCES`] says: its key, the fork checkout its std was +/// built in, and the witness of the sources it was built from. +fn sources_text(key: &Key, fork: &Path, witness: &str) -> String { + format!("{key}\nfork {}\n{witness}\n", fork.display()) +} + +/// The witness the sysroot at `dir` records it was built from ([`sources_text`]). +pub(crate) fn recorded_witness(dir: &Path) -> Result { + let path = dir.join(SOURCES); + let text = fs::read_to_string(&path).map_err(|e| format!("{}: {e}", path.display()))?; + match text.splitn(3, '\n').collect::>().as_slice() { + [_, fork, witness] if fork.starts_with("fork ") => Ok(witness.trim_end_matches('\n').to_string()), + _ => Err(format!("{} records no witness: {text:?}", path.display())), + } +} + /// Why `dir` is not a directory [`publish`] finished, if it is not: it carries /// no [`SOURCES`]. -fn unpublished(dir: &Path) -> Option { +pub(crate) fn unpublished(dir: &Path) -> Option { (!dir.join(SOURCES).is_file()).then(|| format!("{} carries no {SOURCES}", dir.display())) } @@ -506,7 +539,7 @@ fn unpublished(dir: &Path) -> Option { /// not the second is made again rather than trusted — all of it even when only /// its `bin/cargo` link dangles, because that is rare and a sysroot has no /// repair path. -fn unfinished(dir: &Path) -> Option { +pub(crate) fn unfinished(dir: &Path) -> Option { unpublished(dir).or_else(|| toolchain::toolchain_defect(dir)) } @@ -624,7 +657,7 @@ fn build(root: &Path, rust_dir: &Path, compiler: &Compiler, fork: &Path, keys: & "the sources moved while sysroot {key} was being built (they are now {again}); \ nothing was kept, and the next build makes the one they name" ); - format!("{key}\nfork {}\n{}\n", fork.display(), witness(root)) + sources_text(key, fork, &witness(root)) }); } @@ -645,7 +678,7 @@ fn build_freestanding(root: &Path, compiler: &Compiler, fork: &Path, key: &Key, for target in Libraries::Freestanding.targets() { place_std(&stamp(&built, target), &partial.join(target)); } - let again = freestanding_key(root, compiler, fork); + let again = freestanding_key(root, &compiler.key(), fork); assert!( again == *key, "the sources moved while the freestanding libraries {key} were being built (they are \ @@ -1050,10 +1083,20 @@ mod tests { fs::remove_file(&header).unwrap(); same("the C sysroot's headers as they were"); - write(&root.join("userland/libc/Cargo.toml"), "[package]\nversion = \"0.2.0\"\n"); - sysroot_only("libc's manifest, which std's lockfile does not resolve,"); - write(&root.join("userland/libc/Cargo.toml"), "[package]\nversion = \"0.1.0\"\n"); - same("libc's manifest as it was"); + for read in ["userland/libc/Cargo.toml", "userland/libc/Cargo.lock", "userland/.cargo/config.toml"] { + write(&root.join(read), "[package]\nversion = \"0.2.0\"\n"); + sysroot_only(&format!("{read}, which std's lockfile does not resolve,")); + write(&root.join(read), "[package]\nversion = \"0.1.0\"\n"); + same(&format!("{read} as it was")); + } + assert_eq!(key_of(&root, &was.freestanding, &build_text()), was.sysroot); + for flag in [crate::libc::BUILD[1], crate::libc::BUILD_C[1]] { + assert!(build_text().contains(flag), "the sysroot key reads none of libc's {flag}: {}", build_text()); + } + let options = format!("{:?}", crate::libcxx::OPTIONS); + assert!(build_text().contains(&options), "the sysroot key reads no C++ runtime option: {}", build_text()); + assert_ne!(key_of(&root, &was.freestanding, &build_text().replace("--release", "--profile=dev")), was.sysroot, + "libc's cargo invocation kept the sysroot"); let std = fork.join("library/std/src/lib.rs"); write(&std, "//! std, documented\npub fn exit() {}\n"); @@ -1086,7 +1129,7 @@ mod tests { same(manifest); } - let compiler = Compiler::primary(&rust_dir); + let compiler = Compiler::primary(&rust_dir).key(); let config = keyed_std_config(); assert_eq!(freestanding_key_of(&root, &compiler, &fork, RECIPE, &config), was.freestanding); assert_ne!(freestanding_key_of(&root, &compiler, &fork, RECIPE, ""), was.freestanding, @@ -1104,6 +1147,18 @@ mod tests { assert_eq!(now.identity.stale(Some(&stamp)), Some(Stale::All), "another compiler kept a crate's host half"); } + /// **A sysroot's recorded witness is the one its build wrote**, read back + /// whole; a `SOURCES` naming no fork records none. + #[test] + fn a_sysroot_records_the_witness_it_was_built_from() { + let dir = TempDir::new("recorded-witness"); + let witness = "toyos-abi/src/lib.rs:0011223344556677\ntoyos/Cargo.toml:8899aabbccddeeff"; + fs::write(dir.join(SOURCES), sources_text(&Key::of(b"a sysroot"), Path::new("/a/fork/rust"), witness)).unwrap(); + assert_eq!(recorded_witness(&dir), Ok(witness.to_string())); + fs::write(dir.join(SOURCES), "0123456789abcdef\n").unwrap(); + assert!(recorded_witness(&dir).is_err(), "a SOURCES with no fork line recorded a witness"); + } + /// **A crate's compiler is the one that built it, rebuilt in place or /// not**: the primary's is rebuilt where it stands, so a driver its rebuild /// left leaves all of a crate stale, and the same driver none of it. diff --git a/src/toolchain.rs b/src/toolchain.rs index 85a2d8aefaa..06913b03a66 100644 --- a/src/toolchain.rs +++ b/src/toolchain.rs @@ -10,7 +10,7 @@ use crate::buildlock::Scope; use crate::sysroot::{self, Sysroot, SYSROOT_SOURCES}; /// Whether the primary's compiler needs a bootstrap. `invalidate_hosted` -/// separates "the compiler changed" from "the rustup link is missing": only the +/// separates "the compiler changed" from "its `rustc` does not run": only the /// first makes the ToyOS-hosted rustc stale, and rebuilding that one costs /// minutes. #[derive(Clone, Copy, PartialEq, Debug)] @@ -266,12 +266,17 @@ pub(crate) fn assert_std_reads_no_worktree(root: &Path, fork: &Path, dep_info: & ); } -/// What an installed toolchain's sysroot was built from, as its publisher +/// What an installed toolchain's sysroot was built from, as its install /// recorded it (`src/release.rs`). -fn witness_path(rust_dir: &Path) -> PathBuf { +pub(crate) fn witness_path(rust_dir: &Path) -> PathBuf { rust_dir.join("build/toyos-sysroot-witness") } +/// The `TOOLCHAIN` an installed toolchain was installed with (`src/release.rs`). +pub(crate) fn manifest_path(rust_dir: &Path) -> PathBuf { + rust_dir.join("build/TOOLCHAIN") +} + /// The lines of a witness belonging to `trees`. fn witness_subset(text: &str, trees: &[&str]) -> String { @@ -379,11 +384,11 @@ fn cargo_link_stale(stage2: &Path) -> bool { /// Put a `cargo` beside the toolchain's `rustc`. /// /// **A symlink, and what survives the artifact round-trip is this step rather -/// than the link.** `src/release.rs` excludes it from the tarball for the reason -/// it excludes `lib/rustlib/`: it names a path only the publishing runner -/// has, and a copy would put a 32 MB host binary into a 401 MiB artifact to -/// stand in for a file the consumer can make in a microsecond. `Owner::Installed` -/// makes it, exactly as it makes the host target. +/// than the link.** `src/release.rs` excludes it from the tarball: it names a +/// path only the publishing runner has, and a copy would put a 32 MB host +/// binary into a 401 MiB artifact to stand in for a file the consumer can make +/// in a microsecond. `Owner::Installed` makes it, exactly as it makes the host +/// target. pub(crate) fn provision_toolchain_cargo(stage2: &Path) { let at = stage2.join("bin/cargo"); let _ = fs::remove_file(&at); @@ -482,16 +487,27 @@ fn rebuild_compiler(rust_dir: &Path, llvm: &Path, bootstrap: impl FnOnce()) { } /// What the primary bootstraps: a new compiler when `stage2` is not the one its -/// `compiler/` names, and the same one again when rustup has no `toyos` -/// toolchain to run. -fn bootstrap(current: bool, toolchain_exists: bool) -> Option { +/// fork checkout names, and the same one again when its `rustc` does not run. +/// A `stage2` that runs and has no rustup link, as one a runner restored, is +/// linked, not rebuilt. +fn bootstrap(current: bool, runs: bool) -> Option { if !current { Some(Bootstrap { invalidate_hosted: true }) } else { - (!toolchain_exists).then_some(Bootstrap { invalidate_hosted: false }) + (!runs).then_some(Bootstrap { invalidate_hosted: false }) } } +/// Whether the `rustc` in `stage2` runs. +fn runs(stage2: &Path) -> bool { + Command::new(stage2.join("bin/rustc")) + .arg("--version") + .stdout(std::process::Stdio::null()) + .stderr(std::process::Stdio::null()) + .status() + .is_ok_and(|s| s.success()) +} + /// Ensure the toolchain is up to date, and return the sysroot this checkout's /// sources name — made if nobody has made it (`src/sysroot.rs`). The primary /// builds the ToyOS-hosted rustc only for a build whose config ships it @@ -537,9 +553,9 @@ pub fn ensure(root: &Path, lock: &mut buildlock::Held, hosted_rustc: bool) -> Sy } Owner::Installed => { check_installed_toolchain(root, &rust_dir); - let release = rust_dir.join("build/TOOLCHAIN"); + let release = manifest_path(&rust_dir); let release = fs::read_to_string(&release).unwrap_or_else(|e| { - panic!("{}: {e}; an installed toolchain carries the TOOLCHAIN it was published with", release.display()) + panic!("{}: {e}; an installed toolchain carries the TOOLCHAIN it was installed with", release.display()) }); return Sysroot::installed(stage2(&rust_dir), &release); } @@ -550,17 +566,7 @@ pub fn ensure(root: &Path, lock: &mut buildlock::Held, hosted_rustc: bool) -> Sy lock.act_if( Scope::Global, "build the rust toolchain", - || { - let current = crate::compiler::primary_is_current(&rust_dir); - let toolchain_exists = Command::new("rustup") - .args(["run", "toyos", "rustc", "--version"]) - .stdout(std::process::Stdio::null()) - .stderr(std::process::Stdio::null()) - .status() - .map(|s| s.success()) - .unwrap_or(false); - bootstrap(current, toolchain_exists) - }, + || bootstrap(crate::compiler::primary_is_current(&rust_dir), runs(&stage2(&rust_dir))), |kind| { eprintln!("Building full toolchain (this takes a while on first run)..."); let llvm = crate::llvm::resolve(root, &rust_dir, &rust_dir); @@ -611,11 +617,6 @@ pub fn ensure(root: &Path, lock: &mut buildlock::Held, hosted_rustc: bool) -> Sy /// Everything a checkout may do with a toolchain it did not build: check that /// it is the one this tree needs, and say what to do when it is not. -/// -/// No amount of source here can rebuild a sysroot without `rust/`, so there is -/// nothing to decide and the answer is always the toolchain built from these -/// sources: the one the release tag names, which hashes every source and every -/// module it is built from (`src/release.rs`). fn check_installed_toolchain(root: &Path, rust_dir: &Path) { let stage2 = stage2(rust_dir); let linked = rustup_link(); @@ -648,8 +649,8 @@ fn check_installed_toolchain(root: &Path, rust_dir: &Path) { recorded.as_deref() == Some(want.as_str()), "this checkout and the installed toolchain at {} disagree about {}, so a build \ here would link its kernel against another tree's struct layouts.\n\ - Install the build this tree's release tag names; if that is the one installed, \ - the tag hashes less than the toolchain is built from (`src/release.rs`).", + A runner installs the sysroot its job restored by this tree's key; if that is the one \ + installed, the key reads less than the sysroot is built from (`src/sysroot.rs`).", stage2.display(), differing_trees(recorded.as_deref(), &want), ); @@ -718,7 +719,9 @@ pub(crate) fn x_build_compiler(rust_dir: &Path, args: &[&str], what: &str, llvm: } /// [`x_build`], with bootstrap's environment what `environment` makes of this -/// process's. +/// process's, less GitHub Actions' `GITHUB_ACTIONS` and `CI`: bootstrap takes +/// `HEAD^1` as the upstream commit whose artifacts to fetch when it sees them, +/// and in this fork that is our own merge, which rust-lang's CI never built. pub(crate) fn x_build_with( rust_dir: &Path, args: &[&str], @@ -735,6 +738,7 @@ pub(crate) fn x_build_with( let x = if rust_dir.join("x").exists() { "./x" } else { "./x.py" }; let mut command = Command::new(x); environment(&mut command); + command.env_remove("GITHUB_ACTIONS").env_remove("CI"); let mut child = command .args(args) .env("BOOTSTRAP_SKIP_TARGET_SANITY", "1") @@ -906,7 +910,8 @@ fn build_hosted_rustc(rust_dir: &Path, llvm: &Path) { } } -/// `bootstrap.toml` for the host-only toolchain, or with the ToyOS-hosted rustc. +/// `bootstrap.toml` for the host-only toolchain, every compiler's +/// (`compiler::config_text`), or with the ToyOS-hosted rustc. /// /// `lld = true` is what puts `rust-lld` in every stage's sysroot, where rustc /// finds the linker every guest target names. The hosted rustc's build cannot @@ -926,27 +931,18 @@ fn build_hosted_rustc(rust_dir: &Path, llvm: &Path) { /// The host-only toolchain builds no guest target's libraries: every sysroot /// builds its own (`src/sysroot.rs`). fn write_config(rust_dir: &Path, host: &str, with_hosted_rustc: bool, llvm: &Path) { - let host_line = if with_hosted_rustc { - format!("host = [\"{host}\", \"{}\"]", HOSTED_ARCH.userland()) - } else { - format!("host = [\"{host}\"]") - }; - let (guests, userland) = if with_hosted_rustc { - (GUEST_TARGETS.map(GuestTarget::triple).to_vec(), hosted_targets(llvm)) - } else { - (Vec::new(), String::new()) - }; - let targets = std::iter::once(host) - .chain(guests) - .map(|t| format!("\"{t}\"")) - .collect::>() - .join(", "); - let config = format!( - r#"change-id = "ignore" + let config = if with_hosted_rustc { + let targets = std::iter::once(host) + .chain(GUEST_TARGETS.map(GuestTarget::triple)) + .map(|t| format!("\"{t}\"")) + .collect::>() + .join(", "); + format!( + r#"change-id = "ignore" profile = "compiler" [build] -{host_line} +host = ["{host}", "{hosted}"] target = [{targets}] [llvm] @@ -954,7 +950,7 @@ target = [{targets}] [rust] incremental = true -lld = {lld} +lld = false {LEAN} [target.{host}] @@ -962,10 +958,14 @@ lld = {lld} {external} {userland}"#, - llvm = crate::clang::LLVM_CONFIG, - external = crate::llvm::host_lines(llvm), - lld = !with_hosted_rustc, - ); + hosted = HOSTED_ARCH.userland(), + llvm = crate::clang::LLVM_CONFIG, + external = crate::llvm::host_lines(llvm), + userland = hosted_targets(llvm), + ) + } else { + crate::compiler::config_text(&rust_dir.join("build"), host, llvm) + }; fs::write(rust_dir.join("bootstrap.toml"), config).unwrap(); } @@ -1242,9 +1242,41 @@ mod tests { llvm.display() ); assert!(config.contains(&host), "{config}"); + if !hosted { + let keyed = crate::compiler::config_text(&rust_dir.join("build"), "h", &llvm); + assert_eq!(config, keyed, "the primary's compiler is built under a configuration its key does not read"); + } } } + /// **Bootstrap never sees GitHub Actions' variables**, whatever its + /// caller's environment: it takes `HEAD^1`'s artifacts when it does. `./x` + /// here is this test binary, running [`a_fake_bootstrap_that_reads_ci`]. + #[test] + fn a_bootstrap_run_sees_no_ci_variables() { + let fork = TempDir::new("x-build-ci"); + fs::create_dir_all(fork.join("library")).unwrap(); + for lock in ["Cargo.lock", "library/Cargo.lock"] { + fs::write(fork.join(lock), "# as committed\n").unwrap(); + } + fs::write(fork.join(FAKE), "").unwrap(); + std::os::unix::fs::symlink(std::env::current_exe().unwrap(), fork.join("x")).unwrap(); + let args = ["--exact", "toolchain::tests::a_fake_bootstrap_that_reads_ci", "--include-ignored", "--nocapture"]; + let (ok, log) = x_build_with(&fork, &args, "a fake bootstrap", |command| { + command.env("GITHUB_ACTIONS", "true").env("CI", "true"); + }); + assert!(ok, "the fake bootstrap did not run: {log:?}"); + assert!(log.iter().any(|l| l == "GITHUB_ACTIONS none, CI none"), "{log:?}"); + } + + #[test] + #[ignore = "the bootstrap `a_bootstrap_run_sees_no_ci_variables` runs; never runs on its own"] + fn a_fake_bootstrap_that_reads_ci() { + assert!(Path::new(FAKE).is_file(), "a_fake_bootstrap_that_reads_ci ran outside a fake fork checkout; it is not a test"); + let read = |name| std::env::var(name).unwrap_or_else(|_| "none".to_string()); + println!("GITHUB_ACTIONS {}, CI {}", read("GITHUB_ACTIONS"), read("CI")); + } + /// **A bootstrap leaves the primary nothing that waits on another /// worktree's sysroot build**: the act that reassembles `stage2` completes it /// before its exclusive hold ends, so the step after it — run while a @@ -1398,17 +1430,30 @@ mod tests { fn the_primary_bootstraps_when_stale_or_missing() { let new = Some(Bootstrap { invalidate_hosted: true }); let again = Some(Bootstrap { invalidate_hosted: false }); - for (current, toolchain_exists, want) in [ + for (current, runs, want) in [ (true, true, None), (true, false, again), (false, true, new), (false, false, new), ] { - assert_eq!( - bootstrap(current, toolchain_exists), - want, - "current {current}, toolchain_exists {toolchain_exists}" - ); + assert_eq!(bootstrap(current, runs), want, "current {current}, runs {runs}"); + } + } + + /// **What decides a rebuild of a current `stage2` is whether its `rustc` + /// runs**, not whether rustup names it: one a runner restored has no + /// rustup link and is linked, not built again. + #[test] + fn a_compiler_s_rustc_runs_or_it_is_built_again() { + let stage2 = TempDir::new("runs"); + let rustc = stage2.join("bin/rustc"); + assert!(!runs(&stage2), "a stage2 with no rustc ran"); + fs::create_dir_all(rustc.parent().unwrap()).unwrap(); + // This test binary refuses `--version`; the host's rustc answers it. + for (what, ran) in [(std::env::current_exe().unwrap(), false), (host_sysroot().join("bin/rustc"), true)] { + let _ = fs::remove_file(&rustc); + std::os::unix::fs::symlink(&what, &rustc).unwrap(); + assert_eq!(runs(&stage2), ran, "{}", what.display()); } } From 8a219c13c292473da62ff25e8dc4ed46a479b517 Mon Sep 17 00:00:00 2001 From: japabu Date: Thu, 1 Oct 2026 20:53:31 +0200 Subject: [PATCH 11/30] The workflow rules are the reviewer's to read, not gates over the YAML By owner direction, rules a reviewer sees in the diff live in the review prompt, and code stays only where reading cannot see the defect: the layered stores, their keys and the build. `.claude/agents/reviewer.md` gains five sentences under "Workflows": - only main's runs save what other refs restore; - no trigger runs other code on main's ref; - no write-capable `cache-mode`; - `guest / suite` has no `if:` of its own, and its callers run it whatever `toolchain` concluded; - a job that saves runs only the driver. Deleted from src/ci.rs: the seven workflow-reading tests the previous commit added, the YAML readers they used, and this branch's write-token test. Main's `the_required_check_is_a_job_on_every_pull_request` is main's again. `workflows_run_against_main_on_hosted_runners` and `each_cache_has_one_writer` keep the only changes the new workflows force: two more files, and a cache named by a step's output. `release::LAYERS` is private again. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L --- .claude/agents/reviewer.md | 6 + src/ci.rs | 369 ++----------------------------------- src/release.rs | 2 +- 3 files changed, 21 insertions(+), 356 deletions(-) diff --git a/.claude/agents/reviewer.md b/.claude/agents/reviewer.md index 19330b044ab..3744d275aa2 100644 --- a/.claude/agents/reviewer.md +++ b/.claude/agents/reviewer.md @@ -80,6 +80,12 @@ above; otherwise it is a NOTE. or `[patch]`, which cargo ignores with only a warning; a new package without a `description` saying what it is. A new cargo feature or `cfg` arm of one, and every arm a changed `src/clippy.rs` shape stops building, is shown linted in the pull request body: a `mem::forget` planted in that arm turns `cargo run -- --clippy` red. +- **Workflows.** GitHub's cache scoping is the provenance of every cache entry, so a BLOCKER each: + - Only main's runs save a cache entry other refs restore. + - No workflow runs on `pull_request_target`, `workflow_run`, `issue_comment` or any other trigger that runs code other than main's on main's ref. + - No workflow or job declares `cache-mode: write` or `write-only`. + - `guest / suite` has no job-level `if:`, and a job that calls it runs whatever `toolchain` concluded: a skipped required check reads as green. + - A job that saves a cache entry runs only `cargo run -- --ci `. - **Growth.** Every line is a responsibility, not an asset. State the branch's net lines (`git diff --shortstat origin/main...HEAD`), production and tests apart. Production code that grows needs a reason you accept; a branch that could delete more than it adds and does not goes back diff --git a/src/ci.rs b/src/ci.rs index 652e1247815..cea1a2f7bf2 100644 --- a/src/ci.rs +++ b/src/ci.rs @@ -1038,355 +1038,12 @@ mod tests { assert!(at_tip("", tip).is_err()); } - fn workflow(name: &str) -> String { - std::fs::read_to_string(repo_root().join(".github/workflows").join(name)) - .unwrap_or_else(|e| panic!("{name}: {e}")) - } - - /// Every workflow, by file name. - fn workflows() -> Vec<(String, String)> { - let dir = repo_root().join(".github/workflows"); - let mut all: Vec<(String, String)> = std::fs::read_dir(&dir) - .expect(".github/workflows is readable") - .flatten() - .map(|e| (e.file_name().to_string_lossy().into_owned(), std::fs::read_to_string(e.path()).expect("a readable workflow"))) - .collect(); - all.sort(); - all - } - - /// `line` without the comment that ends it, and nothing of a comment line. - fn uncommented(line: &str) -> &str { - if line.trim_start().starts_with('#') { - return ""; - } - line.find(" #").map_or(line, |at| &line[..at]) - } - - /// The lines of job `name` in `text`, empty if it has none. - fn job<'a>(text: &'a str, name: &str) -> Vec<&'a str> { - let head = format!(" {name}:"); - text.lines() - .skip_while(|l| *l != head) - .skip(1) - .take_while(|l| l.is_empty() || l.starts_with(" ")) - .collect() - } - - /// Each job of a workflow: its name and its lines. - fn jobs(text: &str) -> Vec<(&str, Vec<&str>)> { - let mut jobs: Vec<(&str, Vec<&str>)> = Vec::new(); - for line in text.split_once("\njobs:\n").map_or("", |(_, jobs)| jobs).lines() { - match line.strip_prefix(" ").and_then(|l| l.strip_suffix(':')) { - Some(name) if !name.starts_with([' ', '#']) => jobs.push((name, Vec::new())), - _ => jobs.last_mut().into_iter().for_each(|(_, lines)| lines.push(line)), - } - } - jobs - } - - /// The steps of a job's lines, each its lines from its `- ` on. - fn steps<'a>(lines: &[&'a str]) -> Vec> { - let mut steps: Vec> = Vec::new(); - let mut inside = false; - for &line in lines { - if line.starts_with(" ") && !line.starts_with(" ") && !line.trim_start().starts_with('#') { - inside = line.trim() == "steps:"; - } else if inside && line.starts_with(" - ") { - steps.push(vec![line]); - } else if let Some(step) = steps.last_mut().filter(|_| inside) { - step.push(line); - } - } - steps - } - - /// The value of a job's own `:` line. - fn field(job: &[&str], key: &str) -> Option { - let line = format!(" {key}: "); - job.iter().find_map(|l| l.strip_prefix(&line)).map(str::to_string) - } - - /// The value a step's own `:` line gives, its `- :` among them. - fn at<'a>(step: &[&'a str], key: &str) -> Option<&'a str> { - let (own, first) = (format!(" {key}: "), format!(" - {key}: ")); - step.iter().find_map(|l| l.strip_prefix(&own).or_else(|| l.strip_prefix(&first))) - } - - /// Each event a workflow runs on, with its lines. An `on:` that is not a - /// block of events is refused, so none hides from this reader. - fn events(name: &str, text: &str) -> Vec<(String, Vec)> { - let block = text.split_once("\non:\n").unwrap_or_else(|| panic!("{name}: no `on:` block")).1; - let mut events: Vec<(String, Vec)> = Vec::new(); - for line in block.lines().take_while(|l| l.is_empty() || l.starts_with([' ', '#'])) { - let line = uncommented(line); - if line.trim().is_empty() { - continue; - } - match line.strip_prefix(" ").filter(|l| !l.starts_with(' ')) { - Some(event) => { - let event = event.split(':').next().unwrap_or_default().to_string(); - events.push((event, vec![line.to_string()])); - } - None => match events.last_mut() { - Some((_, lines)) => lines.push(line.to_string()), - None => panic!("{name}: {line:?} under `on:` names no event"), - }, - } - } - events - } - - /// A skipped job reads as green to a required check, so `guest` runs on - /// every pull request and in the merge queue whatever `toolchain` - /// concluded: its condition is `host`'s and `!cancelled()`, and nothing - /// more. The nightly's `tcg` runs the same way. - #[test] - fn the_guest_lanes_run_whatever_the_toolchain_concluded() { - let ci = workflow("ci.yml"); - assert!(ci.contains("\n pull_request:\n") && ci.contains("\n merge_group:")); - let host = field(&job(&ci, "host"), "if").expect("ci.yml's `host` has a condition"); - let guest = job(&ci, "guest"); - assert_eq!(field(&guest, "if"), Some(format!("${{{{ !cancelled() && ({host}) }}}}"))); - assert_eq!(field(&guest, "needs").as_deref(), Some("toolchain")); - assert_eq!(field(&guest, "uses").as_deref(), Some("./.github/workflows/guest.yml")); - assert_eq!(field(&job(&ci, "toolchain"), "uses").as_deref(), Some("./.github/workflows/toolchain.yml")); - let nightly = workflow("nightly.yml"); - let tcg = job(&nightly, "tcg"); - assert_eq!(field(&tcg, "if").as_deref(), Some("${{ !cancelled() }}")); - assert_eq!(field(&tcg, "needs").as_deref(), Some("toolchain")); - assert_eq!(field(&tcg, "uses").as_deref(), Some("./.github/workflows/guest.yml")); - } - - /// **`guest / suite` has no condition of its own**: it is a required - /// check, and a job its own `if:` skips reads as green to one. - #[test] - fn the_required_guest_check_has_no_condition_of_its_own() { - let text = workflow("guest.yml"); - let all = jobs(&text); - let (_, suite) = all.iter().find(|(name, _)| *name == "suite").expect("guest.yml's `suite`"); - let own = suite.iter().find(|l| uncommented(l).starts_with(" if:")); - assert_eq!(own, None, "guest.yml's `suite` can skip itself"); - } - - /// **One job holds a token that writes this repository: the nightly's - /// `release`**, the one job that runs `--ci release`. Every `write` in a - /// workflow but a comment's is read, whatever YAML spells the grant; the - /// only other is crates.io's OIDC grant at the top of `publish.yml`. #[test] - fn only_the_nightly_release_holds_a_write_token() { - let mut grants = Vec::new(); - let mut releases = Vec::new(); - for (name, text) in workflows() { - let mut job = None; - let mut in_jobs = false; - for line in text.lines() { - in_jobs |= line == "jobs:"; - if let Some(head) = line.strip_prefix(" ").and_then(|l| l.strip_suffix(':')).filter(|h| in_jobs && !h.starts_with([' ', '#'])) { - job = Some(head.to_string()); - } - let code = uncommented(line); - let words = code.split(|c: char| !(c.is_ascii_alphanumeric() || c == '-')); - if words.clone().any(|word| word.starts_with("write")) { - grants.push((name.clone(), job.clone(), code.trim().to_string())); - } - if code.contains("--ci release") { - releases.push((name.clone(), job.clone(), code.trim().to_string())); - } - } - } - let job = |name: &str| Some(name.to_string()); - let line = |name: &str, job: Option, text: &str| (name.to_string(), job, text.to_string()); - assert_eq!( - grants, - [line("nightly.yml", job("release"), "contents: write"), line("publish.yml", None, "id-token: write")] - ); - assert_eq!(releases, [line("nightly.yml", job("release"), "run: cargo run -- --ci release")]); - } - - /// **No workflow runs on a trigger from outside the repository**: - /// `pull_request_target`, `workflow_run`, `issue_comment` and every other - /// event that someone without write access starts runs on main, where a - /// save lands in what every ref restores. - #[test] - fn no_workflow_runs_on_a_low_trust_trigger() { - const TRUSTED: [&str; 6] = ["pull_request", "merge_group", "push", "schedule", "workflow_dispatch", "workflow_call"]; - for (name, text) in workflows() { - for (event, _) in events(&name, &text) { - assert!(TRUSTED.contains(&event.as_str()), "{name} runs on {event}"); - } - } - } - - /// **No job widens its cache access**: GitHub gives a run on a low-trust - /// trigger read access alone to main's entries, and a `cache-mode` of - /// `write` or `write-only` gives it back. Any `cache-mode` but `read` or - /// `none` is refused. - #[test] - fn no_job_widens_its_cache_mode() { - for (name, text) in workflows() { - for line in text.lines().map(uncommented) { - for (at, _) in line.match_indices("cache-mode") { - let value = line[at + "cache-mode".len()..].trim_start_matches([':', ' ', '"', '\'']); - let value: String = value.chars().take_while(|c| c.is_ascii_alphanumeric() || *c == '-').collect(); - assert!(value == "read" || value == "none", "{name}: {line:?}"); - } - } - } - } - - /// **Only main's runs save what other refs restore.** A run saves into - /// its own ref's scope, and main's alone is restored on other refs. So a - /// workflow that saves a cache entry, itself or through a workflow it calls, - /// runs on a push only to main, on a pull request only into main and on a - /// dispatch that takes no input; and a job that saves checks out its own - /// run's commit and no other. - #[test] - fn only_mains_runs_save_what_other_refs_restore() { - let all = workflows(); - let saves = |text: &str| text.contains("actions/cache/save@"); - let calls = |text: &str, callee: &str| text.lines().any(|l| l.trim() == format!("uses: ./.github/workflows/{callee}")); - let mut savers = Vec::new(); - for (name, text) in &all { - if !saves(text) && !all.iter().any(|(callee, called)| saves(called) && calls(text, callee)) { - continue; - } - savers.push(name.as_str()); - for (event, lines) in events(name, text) { - match event.as_str() { - "push" | "pull_request" => { - assert!(lines.contains(&" branches: [main]".to_string()), "{name}: a {event} on another ref: {lines:?}") - } - "workflow_dispatch" => assert!(lines.iter().all(|l| !l.contains("inputs")), "{name}: a dispatch that takes input"), - _ => {} - } - } - for (job, lines) in jobs(text) { - if !lines.iter().any(|l| l.contains("actions/cache/save@")) { - continue; - } - for step in steps(&lines) { - if at(&step, "uses").is_some_and(|uses| uses.starts_with("actions/checkout@")) { - let other = step.iter().find(|l| l.starts_with(" ref:") || l.starts_with(" repository:")); - assert_eq!(other, None, "{name} {job} saves a build of a commit its run is not"); - } - } - } - } - assert_eq!(savers, ["ci.yml", "nightly.yml", "publish.yml", "toolchain.yml"]); - } - - /// Holds one job's lines to [`every_job_that_saves_holds_to_the_allow_list`]. - fn holds_to_the_allow_list(lines: &[&str]) -> Result<(), String> { - let mut block: Option = None; - for line in lines { - let indent = line.len() - line.trim_start().len(); - let text = uncommented(line).trim(); - if text.is_empty() || block.is_some_and(|at| indent > at) { - continue; - } - block = None; - let entry = text.strip_prefix("- ").unwrap_or(text); - let column = indent + text.len() - entry.len(); - let (key, value) = entry - .split_once(": ") - .or(entry.strip_suffix(':').map(|key| (key, ""))) - .ok_or_else(|| format!("a line this reader cannot judge: {line:?}"))?; - let plain = !key.is_empty() && key.bytes().all(|b| b.is_ascii_alphanumeric() || b == b'-' || b == b'_'); - let value = value.trim(); - let path = key == "path"; - if !plain || value.starts_with(['{', '[']) || (!path && value.starts_with(['*', '&'])) { - return Err(format!("a line this reader cannot judge: {line:?}")); - } - if path && value.ends_with(['|', '>']) { - block = Some(column); - } - match key { - "run" => { - let job = value.strip_prefix("cargo run -- --ci ").unwrap_or_default(); - if job.is_empty() || !job.bytes().all(|b| b.is_ascii_lowercase()) { - return Err(format!("a step runs more than the driver: {line:?}")); - } - } - "shell" | "continue-on-error" => return Err(format!("{line:?}")), - _ => {} - } - } - for step in steps(lines) { - let guarded = step.iter().any(|l| l.starts_with(" - if:") || l.starts_with(" if:")); - let saves = at(&step, "uses").is_some_and(|uses| uses.starts_with("actions/cache/save@")); - if guarded && !saves { - return Err(format!("a step's own `if:` guards no save: {step:?}")); - } - } - Ok(()) - } - - /// **A job that saves a cache entry runs the driver alone, and saves on its - /// save's own guard alone**: every `run:` is `cargo run -- --ci `, no - /// step sets its shell or outlives its own failure, and no step but a save - /// has a condition of its own, so an entry follows only the driver's green - /// steps. A line the reader cannot judge — an alias or anchor outside a - /// path, a merge key, a flow collection, a quoted key — is refused, so - /// nothing hides a step from it. - #[test] - fn every_job_that_saves_holds_to_the_allow_list() { - let mut held = Vec::new(); - for (name, text) in workflows() { - for (job, lines) in jobs(&text) { - if lines.iter().any(|l| l.contains("actions/cache/save@")) { - assert!(!text.contains("\ndefaults:"), "{name}: a shell for every step"); - holds_to_the_allow_list(&lines).unwrap_or_else(|why| panic!("{name} {job}: {why}")); - held.push(format!("{name} {job}")); - } - } - } - assert_eq!(held, ["nightly.yml host", "toolchain.yml build"]); - } - - /// **Each toolchain store is restored and saved by the entry its job - /// wrote**: in `toolchain.yml`, each of `release::LAYERS` is restored by - /// exactly the `keys` step's `-key` and `-path` before the - /// `build` step, and saved by them after it only where that step built it; - /// and every job that installs the sysroot restores exactly the one the - /// toolchain job names, red on a miss. - #[test] - fn each_store_is_restored_and_saved_by_the_entry_its_job_wrote() { - let text = workflow("toolchain.yml"); - let action = |name: &str| { - let uses = |l: &str| l.trim().trim_start_matches("- ").strip_prefix("uses: ").map(str::to_string); - text.lines().find_map(|l| uses(l).filter(|a| a.starts_with(name))).unwrap_or_else(|| panic!("{name}")) - }; - let (restore, save) = (action("actions/cache/restore@"), action("actions/cache/save@")); - let place = |snippet: &str| { - assert_eq!(text.matches(snippet).count(), 1, "toolchain.yml holds {snippet:?} once"); - text.find(snippet).expect("found") - }; - let keys = place(" - id: keys\n run: cargo run -- --ci toolchain\n"); - let built = place(" - id: build\n run: cargo run -- --ci bootstrap\n"); - assert!(keys < built); - for (_, name) in release::LAYERS { - let entry = format!( - " with:\n path: ${{{{ steps.keys.outputs.{name}-path }}}}\n key: ${{{{ steps.keys.outputs.{name}-key }}}}\n" - ); - let restored = place(&format!(" - uses: {restore}\n{entry}\n")); - let saved = place(&format!(" - if: steps.build.outputs.{name} == 'built'\n uses: {save}\n{entry}")); - assert!(keys < restored && restored < built && built < saved, "{name}"); - } - assert!(!text.contains("restore-keys"), "a store restored by another entry than its key's"); - for output in ["key", "path"] { - place(&format!(" sysroot-{output}:\n value: ${{{{ jobs.build.outputs.sysroot-{output} }}}}\n")); - place(&format!(" sysroot-{output}: ${{{{ steps.keys.outputs.sysroot-{output} }}}}\n")); - } - let installs = " path: ${{ inputs.sysroot-path }}\n key: ${{ inputs.sysroot-key }}\n fail-on-cache-miss: true\n"; - assert!(workflow("guest.yml").contains(installs), "guest.yml installs another sysroot"); - let handed = " sysroot-key: ${{ needs.toolchain.outputs.sysroot-key }}\n sysroot-path: ${{ needs.toolchain.outputs.sysroot-path }}\n"; - for (name, callers) in [("ci.yml", 1), ("nightly.yml", 1)] { - assert_eq!(workflow(name).matches(handed).count(), callers, "{name} hands the guest lane another sysroot"); - } - let release = job(&workflow("nightly.yml"), "release").join("\n"); - let restored = " path: ${{ needs.toolchain.outputs.sysroot-path }}\n key: ${{ needs.toolchain.outputs.sysroot-key }}\n fail-on-cache-miss: true"; - assert!(release.contains(restored), "the nightly's release packs another sysroot: {release}"); + fn the_required_check_is_a_job_on_every_pull_request() { + let text = std::fs::read_to_string(repo_root().join(".github/workflows/ci.yml")) + .expect("ci.yml is readable"); + assert!(text.contains("\n pull_request:\n") && text.contains("\n merge_group:")); + assert!(text.contains("\n host:"), "ci.yml runs no job `host`"); } /// Every workflow's `pull_request:` trigger names `main` alone, and none @@ -1415,12 +1072,14 @@ mod tests { } /// Exactly one job writes each cache, so what a pull request restores is one - /// job's tree and never a race between two writers: the nightly's `host` - /// writes the host cache, and `toolchain.yml`'s `build` each toolchain store. + /// run's tree and never a race between two writers. #[test] fn each_cache_has_one_writer() { + let dir = repo_root().join(".github/workflows"); let mut writers = Vec::new(); - for (name, text) in workflows() { + for entry in std::fs::read_dir(&dir).expect(".github/workflows is readable").flatten() { + let text = std::fs::read_to_string(entry.path()).expect("a readable workflow"); + let name = entry.file_name().to_string_lossy().into_owned(); assert!(!text.contains("actions/cache@"), "{name}: the combined action saves too"); let lines: Vec<&str> = text.lines().collect(); for (at, line) in lines.iter().enumerate() { @@ -1436,10 +1095,10 @@ mod tests { } assert!(!writers.is_empty(), "no job writes a cache, so every restore is cold"); writers.sort(); - let mut caches: Vec<&String> = writers.iter().map(|(_, cache)| cache).collect(); - caches.dedup(); - assert_eq!(caches.len(), writers.len(), "a cache with two writers: {writers:?}"); - assert!(writers.iter().all(|(file, _)| file == "nightly.yml" || file == "toolchain.yml"), "{writers:?}"); + let mut prefixes: Vec<&String> = writers.iter().map(|(_, p)| p).collect(); + prefixes.dedup(); + assert_eq!(prefixes.len(), writers.len(), "a cache with two writers: {writers:?}"); + assert!(writers.iter().all(|(f, _)| f == "nightly.yml" || f == "toolchain.yml"), "{writers:?}"); } #[test] diff --git a/src/release.rs b/src/release.rs index bc40d8cb24f..4af08500805 100644 --- a/src/release.rs +++ b/src/release.rs @@ -43,7 +43,7 @@ const USER_AGENT: &str = "toyos-build (https://github.com/ToyOSOrg/ToyOS)"; /// The stores a toolchain is, in the order a build makes them, each under the /// name its cache entry and its job's outputs carry. -pub(crate) const LAYERS: [(Keyed, &str); 4] = [ +const LAYERS: [(Keyed, &str); 4] = [ (Keyed::Llvm, "llvm"), (Keyed::Compiler, "compiler"), (Keyed::Freestanding, "freestanding"), From 30c53c5e23c6dbe509265025340127cd6ae2e746 Mon Sep 17 00:00:00 2001 From: japabu Date: Thu, 1 Oct 2026 21:02:51 +0200 Subject: [PATCH 12/30] issues: CI no longer installs the published release, so the AArch64 red's toolchain line says what it runs CI's guest jobs restore the sysroot their toolchain job built, by its key. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L --- ...-aarch64-guest-dies-at-the-kernels-entry-on-cis-firmware.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/issues/kernel/every-aarch64-guest-dies-at-the-kernels-entry-on-cis-firmware.md b/issues/kernel/every-aarch64-guest-dies-at-the-kernels-entry-on-cis-firmware.md index f2dc2ea2efb..d4d8c222f90 100644 --- a/issues/kernel/every-aarch64-guest-dies-at-the-kernels-entry-on-cis-firmware.md +++ b/issues/kernel/every-aarch64-guest-dies-at-the-kernels-entry-on-cis-firmware.md @@ -30,7 +30,8 @@ branch (`test result: ok. 21 passed`). That host ran QEMU 11.1.1 from Homebrew, under the same TCG `-cpu max` for `VirtEl2`. Two parts of the instrument differ: - The firmware: the dev host runs QEMU's bundled `edk2-stable202408-prebuilt.qemu.org`. -- The toolchain: the dev host builds its own, and CI installs the published release. +- The toolchain: the dev host builds its own, and CI restores the sysroot its + toolchain job built. `.github/qemu-version` pins neither of the two. From 64d6036c3f9e94f7f6c8ea16e6eddb960f1e425e Mon Sep 17 00:00:00 2001 From: japabu Date: Thu, 1 Oct 2026 21:13:39 +0200 Subject: [PATCH 13/30] issues: the nested-NMI report is written under the console's registers since #675 The cause the issue named is past: `nested_nmi` wrote through `serial::panic_raw` until #675 (`bc68e5d78`). The exit stays open until CI's KVM `guest` check shows `nested_nmi_is_loud` green. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L --- ...rt-interleaves-with-another-cpus-console-line.md | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/issues/kernel/the-nested-nmi-report-interleaves-with-another-cpus-console-line.md b/issues/kernel/the-nested-nmi-report-interleaves-with-another-cpus-console-line.md index b5b68dcdbe4..00fa2b876d0 100644 --- a/issues/kernel/the-nested-nmi-report-interleaves-with-another-cpus-console-line.md +++ b/issues/kernel/the-nested-nmi-report-interleaves-with-another-cpus-console-line.md @@ -6,15 +6,18 @@ opened: 2026-10-01 # The nested-NMI report interleaves with another CPU's console line -`nested_nmi` (`kernel/src/arch/x86_64/idt/nmi.rs`) writes its report through -`serial::panic_raw`, which takes no lock. When cpu1 is writing its own record at -the same moment, the two lines interleave byte by byte on the 16550. The cpu1 +`nested_nmi` (`kernel/src/arch/x86_64/idt/nmi.rs`) wrote its report through +`serial::panic_raw`, which took no lock. With cpu1 writing its own record at +the same moment, the two lines interleaved byte by byte on the 16550. The cpu1 line was `[kernel 0.385 cpu1] CPU 1: joining scheduler`, and the 16550 carried: [[kenrnmel i0.38]5 cpNu1E] CSPUT 1E: Djo inNiMngI s choednule r -`NESTED NMI` is never whole on the console, so `nested_nmi_is_loud` times out -waiting for it, and the machine halts with its report unreadable. +`NESTED NMI` was never whole on the console, so `nested_nmi_is_loud` timed out +waiting for it, and the machine halted with its report unreadable. + +#675 (`bc68e5d78`) writes the report under the console's registers +(`serial::panic_registers`). **Evidence:** red under KVM in two runs, with byte-identical interleaving: - Main's nightly `guest` lane at `06788146b`, run 36843762360, job 110374194368. From 9e917f19b7ff2245095fce5fdc451d73993e6483 Mon Sep 17 00:00:00 2001 From: japabu Date: Thu, 1 Oct 2026 23:58:04 +0200 Subject: [PATCH 14/30] A key names a submodule by the commit its gitlink records, checked out or not Run 36913380100's `toolchain / build` failed after its cold build: `--ci toolchain` keyed the freestanding libraries e018c4aa81ffbf19 on the depth-1 fork clone, before bootstrap checked `library/backtrace` out ("Updating submodule library/backtrace" at 19:22:08Z), and the build then filed them under 736f8b4f48f3f08b. `tree_identity` read a submodule's files only where it was checked out, so the key a runner computes before its build was not the key its build reads. The post-build check refused it: "the build left freestanding e018c4aa81ffbf19 not whole: ... carries no SOURCES". `tree_identity` now names each gitlink by the commit `HEAD` records for it, the one a build checks out, through `sysroot::gitlink`, which is `compiler::llvm_commit` made general: a checkout holding what no commit does, or an index staging another commit, is refused. The LLVM key reads the same function, so its key does not move; the freestanding, sysroot and a worktree's own compiler keys do, once. `a_submodule_is_the_commit_its_gitlink_records_checked_out_or_not` keys a clone before and after `git submodule update --init library/backtrace`. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L --- .claude/agents/implementer.md | 3 +- CLAUDE.md | 2 +- ...se-is-compressed-at-zstds-fastest-level.md | 21 --- src/compiler.rs | 38 +----- src/llvm.rs | 10 +- src/sysroot.rs | 128 +++++++++++++++--- 6 files changed, 120 insertions(+), 82 deletions(-) delete mode 100644 issues/build/the-toolchain-release-is-compressed-at-zstds-fastest-level.md diff --git a/.claude/agents/implementer.md b/.claude/agents/implementer.md index 06184bff987..98db494669c 100644 --- a/.claude/agents/implementer.md +++ b/.claude/agents/implementer.md @@ -24,8 +24,7 @@ Where hardware or anything uncertain is involved, take the cheap measurement bef guess. Then build, then test before anyone reviews: - Host tests only: `cargo run -- --ci host`, and `cargo run -- --build-only` at most for the - image. Never a guest test or any other `cargo run`: the plain suite runs in CI's `guest` check, - and the orchestrator runs every guest mutation. + image. Never a guest test or any other `cargo run`: the orchestrator runs every guest test. - A result is the command's own exit code: ` > 2>&1; echo EXIT=$?`. A grepped `test result` line is not one, and a gate you did not run is a gate you do not claim. - Long commands run in the background with output to a file under the job scratchpad the brief diff --git a/CLAUDE.md b/CLAUDE.md index 6b6deca9629..3c5f2ec6c0e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -72,7 +72,7 @@ The bar is not yet the tree: `.claude/agents/reviewer.md`, "Arrivals", says wher The testing rules live where they are enforced: the PR gate and the nightly in `.github/workflows/`. Operationally: - `cargo run` builds everything (toolchain, kernel, bootloader, userland, image) and launches QEMU; `--build-only` skips the launch. `cargo test` runs the QEMU harness; `cargo run -- --ci host` runs every host suite, as the PR gate's required `host` check does. -- **Agents never run QEMU.** An agent verifies with host tests and builds the image at most; the plain suite runs in CI's `guest` check, and the orchestrator runs every patched guest run a negative control needs, one suite at a time. +- **Agents never run QEMU.** An agent verifies with host tests and builds the image at most; the orchestrator runs every guest test, one suite at a time. - **Both produce large output**: run them in the background and read the output file — `[N characters truncated]` means data was lost. A full boot is under a second; incremental builds finish in seconds. - **Leave the machine as you found it.** The development machine is shared: every agent stops what it started, killing only by PID and waiting out a build that holds the global lock, and removes the worktrees and scratch build output it no longer needs. diff --git a/issues/build/the-toolchain-release-is-compressed-at-zstds-fastest-level.md b/issues/build/the-toolchain-release-is-compressed-at-zstds-fastest-level.md deleted file mode 100644 index 1c16610758c..00000000000 --- a/issues/build/the-toolchain-release-is-compressed-at-zstds-fastest-level.md +++ /dev/null @@ -1,21 +0,0 @@ ---- -status: open -kind: tooling -opened: 2026-10-01 ---- - -# The toolchain release is compressed at zstd's fastest level - -`release::pack` (`src/release.rs`) compresses the release's tarball in-process -with ruzstd 0.9.0, whose encoder implements one level, `Fastest`, which its -documentation calls roughly zstd's level 1. The zstd binary that packed it -before compressed at level 3. Measured on the development host (macOS, arm64) -over one 1,369,952,256-byte tarball of sysroot `3b3ed0fb252fe96d`: ruzstd -wrote 566,444,661 bytes, the `zstd` CLI 379,391,222 at `-3` and 418,323,664 at -`-1`. Every consumer outside CI downloads the difference. - -Owner: the release module (`src/release.rs`). - -**Exit**: the release's tarball is compressed by a Rust encoder at least as -small as zstd's level 3 on the same tarball, ruzstd's `Default` level once it -is implemented. diff --git a/src/compiler.rs b/src/compiler.rs index 06de8fc04d1..04f14ec1758 100644 --- a/src/compiler.rs +++ b/src/compiler.rs @@ -237,42 +237,6 @@ fn key_of(fork: &Path, build: &str, llvm: &Key) -> Key { Key::of(parts.join("\n\0\n").as_bytes()) } -/// The LLVM commit `fork` builds against: the one its `HEAD` records, refused -/// when its index stages another, because bootstrap checks out the index's. An -/// LLVM change is a commit there and a gitlink here, so a checkout holding -/// anything no commit does is refused rather than named by its commit. -pub(crate) fn llvm_commit(fork: &Path) -> String { - let checkout = fork.join(LLVM); - // Exactly what bootstrap's LLVM stamp hashes beyond the commit; the untracked - // cache spares each call a walk of the whole tree. - let status = ["-c", "core.untrackedCache=true", "status", "--porcelain", "--untracked-files=normal"]; - let edited = checkout.join(".git").exists() && !git_bytes(&checkout, &status).is_empty(); - assert!( - !edited, - "{} holds changes no commit does, and a compiler is keyed on the commit its gitlink \ - names: commit them there and record that commit in {}", - checkout.display(), - fork.display(), - ); - let recorded = git_out(fork, &["ls-tree", "HEAD", LLVM]); - let committed = match recorded.split_whitespace().collect::>().as_slice() { - ["160000", "commit", sha, _] => sha.to_string(), - _ => panic!("{} records no {LLVM} gitlink: `git ls-tree HEAD {LLVM}` said {recorded:?}", fork.display()), - }; - let indexed = git_out(fork, &["ls-files", "--stage", LLVM]); - let staged = match indexed.split_whitespace().collect::>().as_slice() { - ["160000", sha, "0", _] => sha.to_string(), - _ => panic!("{} indexes no {LLVM} gitlink: `git ls-files --stage {LLVM}` said {indexed:?}", fork.display()), - }; - assert!( - staged == committed, - "{} stages {LLVM} at {staged}, and its HEAD records {committed}: bootstrap builds the one \ - staged, and nothing is keyed on what no commit holds; commit the gitlink, or unstage it", - fork.display(), - ); - committed -} - /// The compiler `root`'s fork checkout at `fork` names: the primary's where its /// `compiler/` is the one the primary's was built from, and otherwise its own, /// built if nobody has built it, held in use for as long as the returned value @@ -720,7 +684,7 @@ pub(crate) mod tests { /// Write the primary's record as it was written before its compiler linked /// the host's LLVM: its `compiler/` and its LLVM commit. pub(crate) fn record_before_the_store(rust_dir: &Path) { - fs::write(primary_record(rust_dir), format!("{} llvm {}", compiler_source(rust_dir), llvm_commit(rust_dir))).unwrap(); + fs::write(primary_record(rust_dir), format!("{} llvm {}", compiler_source(rust_dir), crate::sysroot::gitlink(rust_dir, LLVM))).unwrap(); } /// `fork`'s LLVM checked out at a commit of its own. diff --git a/src/llvm.rs b/src/llvm.rs index e22e2119888..5285737cdef 100644 --- a/src/llvm.rs +++ b/src/llvm.rs @@ -2,7 +2,7 @@ //! one per key on this host, shared by every compiler build that names it. //! //! **An LLVM is a function of its key** ([`key`]): the `src/llvm-project` commit -//! the fork checkout's gitlink names (`compiler::llvm_commit`, which refuses a +//! the fork checkout's gitlink names (`sysroot::gitlink`, which refuses a //! checkout holding what no commit does and a gitlink staged and not committed), //! the committed tree of its `src/bootstrap` (one holding what no commit does is //! refused), the bootstrap configuration below, [`RECIPE`], and the tools the @@ -41,9 +41,9 @@ use std::process::Command; use std::sync::OnceLock; use crate::buildlock::{Guard, Keyed}; -use crate::compiler::{llvm_commit, LLVM}; +use crate::compiler::LLVM; use crate::keystore::{self, Key}; -use crate::sysroot::{clone_tree, git_bytes, git_out}; +use crate::sysroot::{clone_tree, git_bytes, git_out, gitlink}; use crate::toolchain::{self, host_triple}; /// What changes how a key's sources become an LLVM and is none of the other @@ -139,7 +139,7 @@ pub fn key(fork: &Path) -> Key { fn key_of(fork: &Path, recipe: &str, config: &str, tools: &str) -> Key { refuse_uncommitted_bootstrap(fork); let bootstrap = git_out(fork, &["rev-parse", &format!("HEAD:{BOOTSTRAP}")]); - Key::of([recipe, config, &llvm_commit(fork), bootstrap.trim(), tools].join("\n\0\n").as_bytes()) + Key::of([recipe, config, &gitlink(fork, LLVM), bootstrap.trim(), tools].join("\n\0\n").as_bytes()) } /// Give `command` nothing of this process's environment but [`ENVIRONMENT`]. @@ -293,7 +293,7 @@ fn place(fork: &Path, key: &Key, dir: &Path, build: &impl Fn(&Path) -> PathBuf) ); let checkout = fork.join(LLVM); assert!(checkout.join(".git").exists(), "the LLVM build left no checkout at {}", checkout.display()); - let (built_from, commit) = (git_out(&checkout, &["rev-parse", "HEAD"]), llvm_commit(fork)); + let (built_from, commit) = (git_out(&checkout, &["rev-parse", "HEAD"]), gitlink(fork, LLVM)); // Bootstrap's `Llvm` step checks the gitlink's commit out before it builds, // so a checkout behind it, the key never reads, is moved first. assert!( diff --git a/src/sysroot.rs b/src/sysroot.rs index 03ecca52e78..c6e662cdec7 100644 --- a/src/sysroot.rs +++ b/src/sysroot.rs @@ -292,24 +292,65 @@ fn manifest_line(root: &Path, manifest: &str) -> String { /// as one hash. /// /// **Source as git sees it**: tracked files and untracked ones no ignore rule -/// covers, into every submodule checked out there — never what a build or the -/// desktop leaves beside them (bootstrap's `__pycache__`, Finder's -/// `.DS_Store`), which would make a key that moves while it is being built. +/// covers, and each submodule as the commit its gitlink records ([`gitlink`]), +/// checked out or not — never what a build or the desktop leaves beside them +/// (bootstrap's `__pycache__`, Finder's `.DS_Store`), which would make a key +/// that moves while it is being built. pub(crate) fn tree_identity(base: &Path, paths: &[&str], links: Links) -> String { - let mut files = Vec::new(); - source_files(base, paths, links, &mut files); - files.sort(); + let mut sources = Vec::new(); + source_files(base, paths, links, &mut sources); + sources.sort(); let mut hasher = Sha256::new(); - for path in files { - let data = fs::read(&path).unwrap_or_else(|e| panic!("read {}: {e}", path.display())); + for (path, commit) in sources { hasher.update(path.strip_prefix(base).unwrap_or(&path).to_string_lossy().as_bytes()); hasher.update([0]); - hasher.update(&*identity::of(&path, &data)); + match commit { + Some(commit) => hasher.update(commit.as_bytes()), + None => { + let data = fs::read(&path).unwrap_or_else(|e| panic!("read {}: {e}", path.display())); + hasher.update(&*identity::of(&path, &data)); + } + } hasher.update([0]); } hex(&hasher.finalize())[..16].to_string() } +/// The commit `checkout`'s `HEAD` records for its submodule at `path`, which is +/// the one a build checks out there; refused when the submodule's checkout +/// holds what no commit does, or the index stages another commit, because a +/// key names it by that commit. +pub(crate) fn gitlink(checkout: &Path, path: &str) -> String { + let submodule = checkout.join(path); + // The untracked cache spares each call a walk of a whole tree. + let status = ["-c", "core.untrackedCache=true", "status", "--porcelain", "--untracked-files=normal"]; + let edited = submodule.join(".git").exists() && !git_bytes(&submodule, &status).is_empty(); + assert!( + !edited, + "{} holds changes no commit does, and a key names it by the commit its gitlink records: \ + commit them there and record that commit in {}", + submodule.display(), + checkout.display(), + ); + let recorded = git_out(checkout, &["ls-tree", "HEAD", path]); + let committed = match recorded.split_whitespace().collect::>().as_slice() { + ["160000", "commit", sha, _] => sha.to_string(), + _ => panic!("{} records no {path} gitlink: `git ls-tree HEAD {path}` said {recorded:?}", checkout.display()), + }; + let indexed = git_out(checkout, &["ls-files", "--stage", path]); + let staged = match indexed.split_whitespace().collect::>().as_slice() { + ["160000", sha, "0", _] => sha.to_string(), + _ => panic!("{} indexes no {path} gitlink: `git ls-files --stage {path}` said {indexed:?}", checkout.display()), + }; + assert!( + staged == committed, + "{} stages {path} at {staged}, and its HEAD records {committed}: a build checks out the one \ + staged, and nothing is keyed on what no commit holds; commit the gitlink, or unstage it", + checkout.display(), + ); + committed +} + /// What [`tree_identity`] makes of a symbolic link, which git keeps as the path /// it names and a build reads through. #[derive(Clone, Copy)] @@ -320,16 +361,29 @@ pub(crate) enum Links { Skipped, } -fn source_files(checkout: &Path, paths: &[&str], links: Links, out: &mut Vec) { - let mut args = vec!["ls-files", "-z", "--cached", "--others", "--exclude-standard", "--"]; - args.extend(paths); - let listed = git_bytes(checkout, &args); +/// Each source under `paths` of `checkout`, with the commit of each that is a +/// submodule ([`gitlink`]). +fn source_files(checkout: &Path, paths: &[&str], links: Links, out: &mut Vec<(PathBuf, Option)>) { + let listed = |how: &[&str]| git_bytes(checkout, &[&["ls-files", "-z"][..], how, &["--"][..], paths].concat()); + let cached = listed(&["--stage"]); + let others = listed(&["--others", "--exclude-standard"]); + // ` \t`, and a gitlink's mode is 160000. + let cached = cached.split(|b| *b == 0).filter(|e| !e.is_empty()).map(|entry| { + let at = entry.iter().position(|b| *b == b'\t').unwrap_or_else(|| panic!("git ls-files --stage said {entry:?}")); + (&entry[at + 1..], entry.starts_with(b"160000 ")) + }); + let others = others.split(|b| *b == 0).filter(|e| !e.is_empty()).map(|entry| (entry, false)); let mut seen = BTreeSet::new(); - for entry in listed.split(|b| *b == 0).filter(|e| !e.is_empty()) { - let path = checkout.join(String::from_utf8_lossy(entry).as_ref()); + for (entry, submodule) in cached.chain(others) { + let name = String::from_utf8_lossy(entry); + let path = checkout.join(name.as_ref()); if !seen.insert(path.clone()) { continue; } + if submodule { + out.push((path, Some(gitlink(checkout, &name)))); + continue; + } let Ok(meta) = fs::symlink_metadata(&path) else { continue }; if meta.is_symlink() { match links { @@ -344,7 +398,7 @@ fn source_files(checkout: &Path, paths: &[&str], links: Links, out: &mut Vec Date: Thu, 1 Oct 2026 23:58:18 +0200 Subject: [PATCH 15/30] The release speaks to GitHub through ureq and packs gzip through flate2 Round 3's BLOCKERs on `src/release.rs`: - `Github` makes every REST call, the upload among them, through a ureq agent on rustls with RustCrypto's provider and webpki's roots, as `userland/doom/build.rs` already fetches; curl runs nowhere in the build system now. `sdkversion::index` reads the crates.io index through the same agent, so the ledger's curl row is met and goes, and with it the `env` row, whose exit bootstrap's own environment met. - `pack` streams the tar into flate2's pure-Rust gzip at its default level, and the asset is `toyos-toolchain.tar.gz`. ruzstd, twox-hash and the issue about zstd's fastest level go. Over one 1,372,971,008-byte tarball of sysroot 3c77313b0346745c on the development host, flate2 wrote 413,389,598 bytes in 36.4 s (35.9 s again, the same bytes), `gzip -6` 412,043,203 and `zstd -3` 379,443,021. - `release` refuses a job with no `GITHUB_REPOSITORY` instead of defaulting it, and the test names the repository it asks as. - A release that already carries its asset is still given the tree's notes. - The post-build check is `whole`, the function the restore's check is. `cargo tree -i cc -e build --target all` prints nothing: ring is in the lockfile as rustls-webpki's optional dependency and is built by no edge. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L --- Cargo.lock | 798 +++++++++++++++++- Cargo.toml | 8 +- ...d-runs-host-tools-outside-rust-and-qemu.md | 2 - issues/build/toyos-is-a-normal-target.md | 2 +- src/release.rs | 253 +++--- src/sdkversion.rs | 17 +- 6 files changed, 925 insertions(+), 155 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 8eadadd3255..6ee6917b550 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -8,6 +8,41 @@ version = "2.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa" +[[package]] +name = "aead" +version = "0.5.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d122413f284cf2d62fb1b7db97e02edb8cda96d769b16e443a4f6195e35662b0" +dependencies = [ + "crypto-common", + "generic-array", +] + +[[package]] +name = "aes" +version = "0.8.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b169f7a6d4742236a0a00c541b845991d0ac43e546831af1249753ab4c3aa3a0" +dependencies = [ + "cfg-if", + "cipher", + "cpufeatures", +] + +[[package]] +name = "aes-gcm" +version = "0.10.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "831010a0f742e1209b3bcea8fab6a8e149051ba6099432c8cb2cc117dec3ead1" +dependencies = [ + "aead", + "aes", + "cipher", + "ctr", + "ghash", + "subtle", +] + [[package]] name = "aho-corasick" version = "1.1.5" @@ -44,6 +79,24 @@ version = "1.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c08606f8c3cbf4ce6ec8e28fb0014a2c086708fe954eaa885384a6165172e7e8" +[[package]] +name = "base16ct" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4c7f02d4ea65f2c1853089ffd8d2787bdbc63de2f0d29dedbcf8ccdfa0ccd4cf" + +[[package]] +name = "base64" +version = "0.23.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ac07cdecf99051d9a5238b80f35af32cdeba5b336e55d957b318b50137e18da5" + +[[package]] +name = "base64ct" +version = "1.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2af50177e190e07a26ab74f8b1efbfe2ef87da2116221318cb1c2e82baf7de06" + [[package]] name = "bcachefs" version = "0.1.0" @@ -96,6 +149,12 @@ version = "0.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8f1fe948ff07f4bd06c30984e69f5b4899c516a3ef74f34df92a2df2ab535495" +[[package]] +name = "bytes" +version = "1.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fc652a48c352aef3ea3aed32080501cf3ef6ed5da78602a020c991775b0aff04" + [[package]] name = "cc" version = "1.2.56" @@ -112,6 +171,30 @@ version = "1.0.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" +[[package]] +name = "chacha20" +version = "0.9.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c3613f74bd2eac03dad61bd53dbe620703d4371614fe0bc3b9f04dd36fe4e818" +dependencies = [ + "cfg-if", + "cipher", + "cpufeatures", +] + +[[package]] +name = "chacha20poly1305" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "10cd79432192d1c0f4e1a0fef9527696cc039165d729fb41b3f4f4f354c2dc35" +dependencies = [ + "aead", + "chacha20", + "cipher", + "poly1305", + "zeroize", +] + [[package]] name = "chrono" version = "0.4.44" @@ -125,6 +208,23 @@ dependencies = [ "windows-link", ] +[[package]] +name = "cipher" +version = "0.4.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "773f3b9af64447d2ce9850330c473515014aa235e6a783b02db81ff39e4a3dad" +dependencies = [ + "crypto-common", + "inout", + "zeroize", +] + +[[package]] +name = "const-oid" +version = "0.9.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c2459377285ad874054d797f3ccebf984978aa39129f6eafde5cdc8315b612f8" + [[package]] name = "core-foundation-sys" version = "0.8.7" @@ -164,6 +264,18 @@ dependencies = [ "cfg-if", ] +[[package]] +name = "crypto-bigint" +version = "0.5.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0dc92fb57ca44df6db8059111ab3af99a63d5d0f8375d9972e319a379c6bab76" +dependencies = [ + "generic-array", + "rand_core 0.6.4", + "subtle", + "zeroize", +] + [[package]] name = "crypto-common" version = "0.1.7" @@ -174,6 +286,15 @@ dependencies = [ "typenum", ] +[[package]] +name = "ctr" +version = "0.9.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0369ee1ad671834580515889b80f2ea915f23b8be8d0daa4bbaf2ac5c7590835" +dependencies = [ + "cipher", +] + [[package]] name = "curve25519-dalek" version = "4.1.3" @@ -187,6 +308,7 @@ dependencies = [ "fiat-crypto", "rustc_version", "subtle", + "zeroize", ] [[package]] @@ -200,6 +322,17 @@ dependencies = [ "syn", ] +[[package]] +name = "der" +version = "0.7.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e7c1832837b905bbfb5101e07cc24c8deddf52f93225eee6ead5f4d63d53ddcb" +dependencies = [ + "const-oid", + "pem-rfc7468", + "zeroize", +] + [[package]] name = "digest" version = "0.10.7" @@ -207,7 +340,23 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292" dependencies = [ "block-buffer", + "const-oid", "crypto-common", + "subtle", +] + +[[package]] +name = "ecdsa" +version = "0.16.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ee27f32b5c5292967d2d4a9d7f1e0b0aed2c15daded5a60300e4abb9d8020bca" +dependencies = [ + "der", + "digest", + "elliptic-curve", + "rfc6979", + "signature", + "spki", ] [[package]] @@ -216,6 +365,7 @@ version = "2.2.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "115531babc129696a58c64a4fef0a8bf9e9698629fb97e9e40767d235cfbcd53" dependencies = [ + "pkcs8", "signature", ] @@ -227,8 +377,31 @@ checksum = "70e796c081cee67dc755e1a36a0a172b897fab85fc3f6bc48307991f64e4eca9" dependencies = [ "curve25519-dalek", "ed25519", + "serde", "sha2", "subtle", + "zeroize", +] + +[[package]] +name = "elliptic-curve" +version = "0.13.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b5e6043086bf7973472e0c7dff2142ea0b680d30e18d9cc40f267efbf222bd47" +dependencies = [ + "base16ct", + "crypto-bigint", + "digest", + "ff", + "generic-array", + "group", + "hkdf", + "pem-rfc7468", + "pkcs8", + "rand_core 0.6.4", + "sec1", + "subtle", + "zeroize", ] [[package]] @@ -249,6 +422,16 @@ dependencies = [ "log", ] +[[package]] +name = "ff" +version = "0.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c0b50bfb653653f9ca9095b427bed08ab8d75a137839d9ad64eb11810d5b6393" +dependencies = [ + "rand_core 0.6.4", + "subtle", +] + [[package]] name = "fiat-crypto" version = "0.2.9" @@ -326,6 +509,18 @@ checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a" dependencies = [ "typenum", "version_check", + "zeroize", +] + +[[package]] +name = "getrandom" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ff2abc00be7fca6ebc474524697ae276ad847ad0a6b3faa4bcb027e9a4614ad0" +dependencies = [ + "cfg-if", + "libc", + "wasi", ] [[package]] @@ -353,6 +548,16 @@ dependencies = [ "wasip3", ] +[[package]] +name = "ghash" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0d8a4362ccb29cb0b265253fb0a2728f592895ee6854fd9bc13f2ffda266ff1" +dependencies = [ + "opaque-debug", + "polyval", +] + [[package]] name = "gpt" version = "3.1.0" @@ -365,6 +570,17 @@ dependencies = [ "uuid", ] +[[package]] +name = "group" +version = "0.13.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0f9ef7462f7c099f518d754361858f86d8a07af53ba9af0fe635bbccb151a63" +dependencies = [ + "ff", + "rand_core 0.6.4", + "subtle", +] + [[package]] name = "hashbrown" version = "0.15.5" @@ -391,6 +607,40 @@ version = "0.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea" +[[package]] +name = "hkdf" +version = "0.12.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7b5f8eb2ad728638ea2c7d47a21db23b7b58a72ed6a38256b8a1849f15fbbdf7" +dependencies = [ + "hmac", +] + +[[package]] +name = "hmac" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6c49c37c09c17a53d937dfbb742eb3a961d65a994e6bcdcf37e7399d0cc8ab5e" +dependencies = [ + "digest", +] + +[[package]] +name = "http" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "918d3568bebf352712bc2ef3d46a8bcf1a75b373be6539de198e9105cbbf9ce0" +dependencies = [ + "bytes", + "itoa", +] + +[[package]] +name = "httparse" +version = "1.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6dbf3de79e51f3d586ab4cb9d5c3e2c14aa28ed23d180cf89b4df0454a69cc87" + [[package]] name = "iana-time-zone" version = "0.1.65" @@ -447,6 +697,15 @@ dependencies = [ "serde_core", ] +[[package]] +name = "inout" +version = "0.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "879f10e63c20629ecabbb64a8010319738c66a5cd0c29b02d63d272b03751d01" +dependencies = [ + "generic-array", +] + [[package]] name = "itoa" version = "1.0.17" @@ -476,6 +735,9 @@ name = "lazy_static" version = "1.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "bbd2bcb4c963f2ddae06a2efc7e9f3591312473c50c6685e1f298068316e66fe" +dependencies = [ + "spin", +] [[package]] name = "leb128fmt" @@ -489,6 +751,12 @@ version = "0.2.183" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b5b646652bf6661599e1da8901b3b9522896f01e736bad5f723fe7a3a27f899d" +[[package]] +name = "libm" +version = "0.2.16" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6d2cec3eae94f9f509c767b45932f1ada8350c4bdb85af2fcab4a3c14807981" + [[package]] name = "log" version = "0.4.29" @@ -548,7 +816,42 @@ version = "0.50.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7957b9740744892f114936ab4a57b3f487491bbeafaf8083688b16841a4240e5" dependencies = [ - "windows-sys", + "windows-sys 0.61.2", +] + +[[package]] +name = "num-bigint-dig" +version = "0.8.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e661dda6640fad38e827a6d4a310ff4763082116fe217f279885c97f511bb0b7" +dependencies = [ + "lazy_static", + "libm", + "num-integer", + "num-iter", + "num-traits", + "rand 0.8.8", + "smallvec", + "zeroize", +] + +[[package]] +name = "num-integer" +version = "0.1.47" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7ce2d95d4b3734dc35aa2f45e1aa22cd416814592a4f9d9205e11affd5b8e10b" +dependencies = [ + "num-traits", +] + +[[package]] +name = "num-iter" +version = "0.1.46" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c92800bd69a1eac91786bcfe9da64a897eb72911b8dc3095decbd07429e8048b" +dependencies = [ + "num-integer", + "num-traits", ] [[package]] @@ -558,6 +861,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841" dependencies = [ "autocfg", + "libm", ] [[package]] @@ -578,12 +882,118 @@ version = "1.21.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "42f5e15c9953c5e4ccceeb2e7382a716482c34515315f7b03532b8b4e8393d2d" +[[package]] +name = "opaque-debug" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c08d65885ee38876c4f86fa503fb49d7b507c2b62552df7c70b2fce627e06381" + +[[package]] +name = "p256" +version = "0.13.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c9863ad85fa8f4460f9c48cb909d38a0d689dba1f6f6988a5e3e0d31071bcd4b" +dependencies = [ + "ecdsa", + "elliptic-curve", + "primeorder", + "sha2", +] + +[[package]] +name = "p384" +version = "0.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fe42f1670a52a47d448f14b6a5c61dd78fce51856e68edaa38f7ae3a46b8d6b6" +dependencies = [ + "ecdsa", + "elliptic-curve", + "primeorder", + "sha2", +] + +[[package]] +name = "paste" +version = "1.0.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "57c0d7b74b563b49d38dae00a0c37d4d6de9b432382b2892f0574ddcae73fd0a" + +[[package]] +name = "pem-rfc7468" +version = "0.7.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "88b39c9bfcfc231068454382784bb460aae594343fb030d46e9f50a645418412" +dependencies = [ + "base64ct", +] + +[[package]] +name = "percent-encoding" +version = "2.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220" + [[package]] name = "pin-project-lite" version = "0.2.17" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" +[[package]] +name = "pkcs1" +version = "0.7.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c8ffb9f10fa047879315e6625af03c164b16962a5368d724ed16323b68ace47f" +dependencies = [ + "der", + "pkcs8", + "spki", +] + +[[package]] +name = "pkcs5" +version = "0.7.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e847e2c91a18bfa887dd028ec33f2fe6f25db77db3619024764914affe8b69a6" +dependencies = [ + "der", + "spki", +] + +[[package]] +name = "pkcs8" +version = "0.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f950b2377845cebe5cf8b5165cb3cc1a5e0fa5cfa3e1f7f55707d8fd82e0a7b7" +dependencies = [ + "der", + "pkcs5", + "spki", +] + +[[package]] +name = "poly1305" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8159bd90725d2df49889a078b54f4f79e87f1f8a8444194cdca81d38f5393abf" +dependencies = [ + "cpufeatures", + "opaque-debug", + "universal-hash", +] + +[[package]] +name = "polyval" +version = "0.6.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9d1fe60d06143b2430aa532c94cfe9e29783047f06c0d7fd359a9a51b729fa25" +dependencies = [ + "cfg-if", + "cpufeatures", + "opaque-debug", + "universal-hash", +] + [[package]] name = "ppv-lite86" version = "0.2.21" @@ -603,6 +1013,15 @@ dependencies = [ "syn", ] +[[package]] +name = "primeorder" +version = "0.13.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "353e1ca18966c16d9deb1c69278edbc5f194139612772bd9537af60ac231e1e6" +dependencies = [ + "elliptic-curve", +] + [[package]] name = "proc-macro2" version = "1.0.106" @@ -639,14 +1058,34 @@ version = "6.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" +[[package]] +name = "rand" +version = "0.8.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e058c7de0b26af77780c769414d6257830bb240f3c38477dbc2c16e5f54d6d4c" +dependencies = [ + "rand_chacha 0.3.1", + "rand_core 0.6.4", +] + [[package]] name = "rand" version = "0.9.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b9ef1d0d795eb7d84685bca4f72f3649f064e6641543d3a8c415898726a57b41" dependencies = [ - "rand_chacha", - "rand_core", + "rand_chacha 0.9.0", + "rand_core 0.9.5", +] + +[[package]] +name = "rand_chacha" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6c10a63a0fa32252be49d21e7709d4d4baf8d231c2dbce1eaa8141b9b127d88" +dependencies = [ + "ppv-lite86", + "rand_core 0.6.4", ] [[package]] @@ -656,7 +1095,16 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d3022b5f1df60f26e1ffddd6c66e8aa15de382ae63b3a0c1bfc0e4d3e3f325cb" dependencies = [ "ppv-lite86", - "rand_core", + "rand_core 0.9.5", +] + +[[package]] +name = "rand_core" +version = "0.6.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ec0be4795e2f6a28069bec0b5ff3e2ac9bafc99e6a9a7dc3547996c5c816922c" +dependencies = [ + "getrandom 0.2.17", ] [[package]] @@ -685,6 +1133,51 @@ version = "0.8.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4" +[[package]] +name = "rfc6979" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8dd2a808d456c4a54e300a23e9f5a67e122c3024119acbfd73e3bf664491cb2" +dependencies = [ + "hmac", + "subtle", +] + +[[package]] +name = "ring" +version = "0.17.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a4689e6c2294d81e88dc6261c768b63bc4fcdb852be6d1352498b114f61383b7" +dependencies = [ + "cc", + "cfg-if", + "getrandom 0.2.17", + "libc", + "untrusted", + "windows-sys 0.52.0", +] + +[[package]] +name = "rsa" +version = "0.9.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b8573f03f5883dcaebdfcf4725caa1ecb9c15b2ef50c43a07b816e06799bb12d" +dependencies = [ + "const-oid", + "digest", + "num-bigint-dig", + "num-integer", + "num-traits", + "pkcs1", + "pkcs8", + "rand_core 0.6.4", + "sha2", + "signature", + "spki", + "subtle", + "zeroize", +] + [[package]] name = "rustc-std-workspace-core" version = "1.0.1" @@ -701,26 +1194,106 @@ dependencies = [ ] [[package]] -name = "rustversion" -version = "1.0.22" +name = "rustls" +version = "0.23.45" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b39cdef0fa800fc44525c84ccb54a029961a8215f9619753635a9c0d2538d46d" +checksum = "0d41d731c7d2f962d1ccc364cec258de3c0e93b38c2fb3ba97ac74513048d634" +dependencies = [ + "log", + "once_cell", + "rustls-pki-types", + "rustls-webpki 0.103.15", + "subtle", + "zeroize", +] [[package]] -name = "ruzstd" -version = "0.9.0" +name = "rustls-pki-types" +version = "1.15.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2f4925028c7eb5d1fcdaf196971378ed9d2c1c4efc7dc5d011256f76c99c0a96" +dependencies = [ + "zeroize", +] + +[[package]] +name = "rustls-rustcrypto" +version = "0.0.2-alpha" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f12052947763ab8515f753315357599e9b0b4dab3b8ba15f30f725fe6d025557" +dependencies = [ + "aead", + "aes-gcm", + "chacha20poly1305", + "crypto-common", + "der", + "digest", + "ecdsa", + "ed25519-dalek", + "hmac", + "p256", + "p384", + "paste", + "pkcs8", + "rand_core 0.6.4", + "rsa", + "rustls", + "rustls-pki-types", + "rustls-webpki 0.102.8", + "sec1", + "sha2", + "signature", + "x25519-dalek", +] + +[[package]] +name = "rustls-webpki" +version = "0.102.8" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a252f5e20f038fe7b4ea53e073e65398d652c864cc162fc77c56c2f13717b888" +checksum = "64ca1bc8749bd4cf37b5ce386cc146580777b4e8572c7b97baf22c83f444bee9" dependencies = [ - "twox-hash", + "ring", + "rustls-pki-types", + "untrusted", ] +[[package]] +name = "rustls-webpki" +version = "0.103.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f3c3cf1d8b1e7d4927e2d154c3fcb02979afb9939629c62cd9048d4f07b60ac2" +dependencies = [ + "ring", + "rustls-pki-types", + "untrusted", +] + +[[package]] +name = "rustversion" +version = "1.0.22" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b39cdef0fa800fc44525c84ccb54a029961a8215f9619753635a9c0d2538d46d" + [[package]] name = "scoped-tls" version = "1.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e1cf6437eb19a8f4a6cc0f7dca544973b0b78843adbfeb3683d1a94a0024a294" +[[package]] +name = "sec1" +version = "0.7.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d3e97a565f76233a6003f9f5c54be1d9c5bdfa3eccfb189469f11ec4901c47dc" +dependencies = [ + "base16ct", + "der", + "generic-array", + "pkcs8", + "subtle", + "zeroize", +] + [[package]] name = "semver" version = "1.0.27" @@ -810,6 +1383,10 @@ name = "signature" version = "2.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "77549399552de45a898a580c1b41d445bf730df867cc44e6c0233bbc4b8329de" +dependencies = [ + "digest", + "rand_core 0.6.4", +] [[package]] name = "simd-adler32" @@ -823,6 +1400,22 @@ version = "1.15.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "67b1b7a3b5fe4f1376887184045fcf45c69e92af734b7aaddc05fb777b6fbd03" +[[package]] +name = "spin" +version = "0.9.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3763264f6b73151db08c50ff20d7d8a0b8796e021cdea7ceedad07b80155fa0e" + +[[package]] +name = "spki" +version = "0.7.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d91ed6c858b01f942cd56b37a94b3e0a1798290327d1236e4d9cf4eaca44d29d" +dependencies = [ + "base64ct", + "der", +] + [[package]] name = "subtle" version = "2.6.1" @@ -964,12 +1557,13 @@ version = "0.1.0" dependencies = [ "bcachefs", "fatfs", + "flate2", "fontdue", "getrandom 0.3.4", "gpt", "image", "libc", - "ruzstd", + "rustls-rustcrypto", "serde", "serde_json", "sha2", @@ -995,6 +1589,7 @@ dependencies = [ "toyos-userbound", "toyos-wallclock", "toyos-xhci", + "ureq", "uuid", ] @@ -1189,7 +1784,7 @@ dependencies = [ name = "toyos-sched-sim" version = "0.1.0" dependencies = [ - "rand", + "rand 0.9.5", "toyos-sched", ] @@ -1258,7 +1853,7 @@ version = "0.1.0" name = "toyos-xhci-sim" version = "0.1.0" dependencies = [ - "rand", + "rand 0.9.5", "toyos-xhci", ] @@ -1317,12 +1912,6 @@ version = "0.21.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2c591d83f69777866b9126b24c6dd9a18351f177e49d625920d19f989fd31cf8" -[[package]] -name = "twox-hash" -version = "2.1.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5283634e518fe9e82c7b20520bb4bc209009fd16c82077c802f8111ecbb0117a" - [[package]] name = "typenum" version = "1.19.0" @@ -1341,6 +1930,56 @@ version = "0.2.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ebc1c04c71510c7f702b52b7c350734c9ff1295c464a03335b00bb84fc54f853" +[[package]] +name = "universal-hash" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fc1de2c688dc15305988b563c3854064043356019f97a4b46276fe734c4f07ea" +dependencies = [ + "crypto-common", + "subtle", +] + +[[package]] +name = "untrusted" +version = "0.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8ecb6da28b8a351d773b68d5825ac39017e680750f980f3a1a85cd8dd28a47c1" + +[[package]] +name = "ureq" +version = "3.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9a7ac20be9b7726e0bbdbf974c059676d9acb1cd414961f570a4e8231cacd7fc" +dependencies = [ + "base64", + "log", + "percent-encoding", + "rustls", + "rustls-pki-types", + "ureq-proto", + "utf8-zero", + "webpki-roots", +] + +[[package]] +name = "ureq-proto" +version = "0.6.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f86fd172ccca569e458f61b6bdd6220965a9ef36e672a6852953b51a0e1583be" +dependencies = [ + "base64", + "http", + "httparse", + "log", +] + +[[package]] +name = "utf8-zero" +version = "0.8.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b8c0a043c9540bae7c578c88f91dda8bd82e59ae27c21baca69c8b191aaf5a6e" + [[package]] name = "uuid" version = "1.22.0" @@ -1364,6 +2003,12 @@ version = "0.9.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a" +[[package]] +name = "wasi" +version = "0.11.1+wasi-snapshot-preview1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b" + [[package]] name = "wasip2" version = "1.0.2+wasi-0.2.9" @@ -1461,6 +2106,15 @@ dependencies = [ "semver", ] +[[package]] +name = "webpki-roots" +version = "1.0.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7dcd9d09a39985f5344844e66b0c530a33843579125f23e21e9f0f220850f22a" +dependencies = [ + "rustls-pki-types", +] + [[package]] name = "windows-core" version = "0.62.2" @@ -1520,6 +2174,15 @@ dependencies = [ "windows-link", ] +[[package]] +name = "windows-sys" +version = "0.52.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "282be5f36a8ce781fad8c8ae18fa3f9beff57ec1b52cb3de0789201425d9a33d" +dependencies = [ + "windows-targets", +] + [[package]] name = "windows-sys" version = "0.61.2" @@ -1529,6 +2192,70 @@ dependencies = [ "windows-link", ] +[[package]] +name = "windows-targets" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9b724f72796e036ab90c1021d4780d4d3d648aca59e491e6b98e725b84e99973" +dependencies = [ + "windows_aarch64_gnullvm", + "windows_aarch64_msvc", + "windows_i686_gnu", + "windows_i686_gnullvm", + "windows_i686_msvc", + "windows_x86_64_gnu", + "windows_x86_64_gnullvm", + "windows_x86_64_msvc", +] + +[[package]] +name = "windows_aarch64_gnullvm" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32a4622180e7a0ec044bb555404c800bc9fd9ec262ec147edd5989ccd0c02cd3" + +[[package]] +name = "windows_aarch64_msvc" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "09ec2a7bb152e2252b53fa7803150007879548bc709c039df7627cabbd05d469" + +[[package]] +name = "windows_i686_gnu" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8e9b5ad5ab802e97eb8e295ac6720e509ee4c243f69d781394014ebfe8bbfa0b" + +[[package]] +name = "windows_i686_gnullvm" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0eee52d38c090b3caa76c563b86c3a4bd71ef1a819287c19d586d7334ae8ed66" + +[[package]] +name = "windows_i686_msvc" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "240948bc05c5e7c6dabba28bf89d89ffce3e303022809e73deaefe4f6ec56c66" + +[[package]] +name = "windows_x86_64_gnu" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "147a5c80aabfbf0c7d901cb5895d1de30ef2907eb21fbbab29ca94c5b08b1a78" + +[[package]] +name = "windows_x86_64_gnullvm" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "24d5b23dc417412679681396f2b49f3de8c1473deb516bd34410872eff51ed0d" + +[[package]] +name = "windows_x86_64_msvc" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec" + [[package]] name = "winnow" version = "0.7.15" @@ -1626,6 +2353,17 @@ dependencies = [ "wasmparser", ] +[[package]] +name = "x25519-dalek" +version = "2.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c7e468321c81fb07fa7f4c636c3972b9100f0346e5b6a9f2bd0603a52f7ed277" +dependencies = [ + "curve25519-dalek", + "rand_core 0.6.4", + "zeroize", +] + [[package]] name = "zerocopy" version = "0.8.47" @@ -1646,6 +2384,26 @@ dependencies = [ "syn", ] +[[package]] +name = "zeroize" +version = "1.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e13c156562582aa81c60cb29407084cdb54c4164760106ab78e6c5b0858cf64e" +dependencies = [ + "zeroize_derive", +] + +[[package]] +name = "zeroize_derive" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3c50655cbb0fe3fc43170059e702f1ce5e19b84cec58dc87b037a09935c2f328" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + [[package]] name = "zmij" version = "1.0.21" diff --git a/Cargo.toml b/Cargo.toml index 07f2acaf259..bf8092050e5 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -164,9 +164,13 @@ image = { version = "0.25", default-features = false, features = ["jpeg"] } # (`src/sourcegate.rs`). sha2 = "0.10" # The toolchain release's tarball, packed in-process (`src/release.rs`): Rust's -# own tar and zstd, where the binaries are hosts' tools. +# own tar and gzip, where the binaries are hosts' tools. tar = { version = "0.4.46", default-features = false } -ruzstd = "0.9.0" +flate2 = { version = "1", default-features = false, features = ["rust_backend"] } +# Every request the build system makes, GitHub's API and the crates.io index: +# rustls over RustCrypto, where curl is a host's tool and a TLS stack in C. +ureq = { version = "3", default-features = false, features = ["rustls-no-provider", "rustls-webpki-roots"] } +rustls-rustcrypto = "0.0.2-alpha" [dev-dependencies] # `USER_TOP`, so the harness judges a held `rsp` against the bound the kernel diff --git a/issues/build/the-build-runs-host-tools-outside-rust-and-qemu.md b/issues/build/the-build-runs-host-tools-outside-rust-and-qemu.md index ac1f9c74bd1..49e475200fc 100644 --- a/issues/build/the-build-runs-host-tools-outside-rust-and-qemu.md +++ b/issues/build/the-build-runs-host-tools-outside-rust-and-qemu.md @@ -23,11 +23,9 @@ arrives and is not one. M4 and M5 are stages of `issues/build/toyos-builds-itsel | the toolchain's own `clang`, `llvm-ar`, `rust-lld` and `llvm-config`, built from `ToyOSOrg/llvm-project` | rustc links every guest binary with `rust-lld`; `clang` compiles the C corpus (`tests/common/compile.rs`) and, with `llvm-ar`, doomgeneric through `cc::Build` (`src/clang.rs`); rustc's bootstrap asks `llvm-config` how to link LLVM | admitted: our fork's C++, which ToyOS can one day build and run; no Rust tool compiles C, `cc::Build` archives with an `ar`, bootstrap reads LLVM through `llvm-config`, and `CLAUDE.md` links everything with `rust-lld` | M5: no host in the loop | | `ovmf-generic`, `qemu-efi-aarch64` | the x86-64 and AArch64 UEFI firmware of CI's guest containers (`src/firmware.rs`), packaged by Debian apart from QEMU | admitted: QEMU's own firmware, and no Rust firmware does its job | the instrument's QEMU carries its own firmware | | `ca-certificates` | the trust store `git` and `curl` verify against in CI's containers | admitted: data both of them need | goes when neither runs there | -| `env` | the two portability jobs run `cargo run -- --build-only` under it, with `GITHUB_ACTIONS` and `CI` unset | refused: a Rust tool does it, the build system, which already runs bootstrap with both removed (`src/toolchain.rs`) | the build system removes both itself | | the T14's Ubuntu and every tool `src/metal.rs` runs on it over `ssh` | the metal loop, on the T14 and never on a development host | outside the rule: recovery equipment on a test machine, not the build's host | they leave with Ubuntu (`issues/boot-media/the-machine-updates-itself-without-ubuntu.md`) | | `sh` running `rust/x`, and Python running `x.py` and `bootstrap.py` | every toolchain build (`src/toolchain.rs`) | refused: a Rust tool does it, upstream's bootstrap binary, which builds with stable cargo, fetches its own stage0 (`rust/src/bootstrap/src/core/download.rs`) and needs no Python | `src/toolchain.rs` runs the bootstrap binary | | `curl` in rustc's bootstrap | fetches the stage0 `rust/src/stage0` pins, for a compiler or LLVM build whose build directory lacks it, whichever bootstrap runs | refused: a Rust tool does it, rustup installs the dated beta the pin names, and bootstrap takes a stage0 through `build.rustc` and `build.cargo`, as `src/sysroot.rs` hands it one | no toolchain build fetches with `curl` | -| `curl` in `src/release.rs` and `src/sdkversion.rs` | GitHub's API, which puts the toolchain release up and moves the `sdk-` alias on the nightly's `release` runner, on main alone; and the crates.io index; on CI runners only | refused: a Rust tool does it, `ureq`, which `userland/doom/build.rs` already fetches with | those fetches are Rust's | | `tar` and `zstd` | `actions/cache` packs and unpacks every cache entry with them, on CI runners; `guest.yml`'s `deps` installs `zstd`, without which the guest's restore names no entry the toolchain job saved | admitted: GitHub's cache action runs them, and CI keeps no store between runs without it | CI keeps no store between runs | | `ssh` | `src/metal.rs` reaches the T14's Ubuntu with it, only in the metal loop | refused: a Rust tool does it, the repository's own russh client `crate::build::ssh_client_host`, which `src/metaltalk.rs` already drives | `src/metal.rs` drives that client, or Ubuntu leaves the loop | | `cc`, `c++`, `ar` and `xcrun` on a macOS host, Apple's Command Line Tools | what the Linux row's tools do, and rustc asks `xcrun` for the SDK on every host link that names no `SDKROOT` (`rust/compiler/rustc_codegen_ssa/src/back/apple.rs`), as `src/llvm.rs` does for the LLVM's key | refused: one host OS alone | M5: no host in the loop | diff --git a/issues/build/toyos-is-a-normal-target.md b/issues/build/toyos-is-a-normal-target.md index 99f20333fd0..48a31bacb11 100644 --- a/issues/build/toyos-is-a-normal-target.md +++ b/issues/build/toyos-is-a-normal-target.md @@ -34,7 +34,7 @@ Stages, in order: the release notes of every toolchain release carry it: mkdir -p toyos-toolchain - curl -sSL "$asset" | tar --zstd -x -C toyos-toolchain + curl -sSL "$asset" | tar -xz -C toyos-toolchain stage2=toyos-toolchain/x86_64-unknown-linux-gnu/stage2 rustup toolchain link toyos "$stage2" ln -s "$(rustup which cargo)" "$stage2/bin/cargo" diff --git a/src/release.rs b/src/release.rs index 4af08500805..fbc42ebedde 100644 --- a/src/release.rs +++ b/src/release.rs @@ -19,6 +19,7 @@ use std::fs; use std::io::Write; use std::path::{Path, PathBuf}; use std::process::Command; +use std::sync::Arc; use serde_json::Value; use sha2::{Digest, Sha256}; @@ -27,7 +28,7 @@ use toyos_tmpdir::TempDir; use crate::buildlock::Keyed; use crate::keystore::Key; -const ASSET: &str = "toyos-toolchain.tar.zst"; +const ASSET: &str = "toyos-toolchain.tar.gz"; /// Main's publisher: the one workflow [`release`] runs under. const PUBLISHER: &str = ".github/workflows/nightly.yml"; @@ -38,7 +39,7 @@ const HOST: &str = "x86_64-unknown-linux-gnu"; /// one is refused. const GLIBC_FLOOR: (u32, u32) = (2, 39); -/// What every request this file makes of GitHub says it comes from. +/// What every request the build system makes says it comes from. const USER_AGENT: &str = "toyos-build (https://github.com/ToyOSOrg/ToyOS)"; /// The stores a toolchain is, in the order a build makes them, each under the @@ -79,8 +80,16 @@ fn on_runner() -> bool { std::env::var("GITHUB_ACTIONS").is_ok_and(|v| v == "true") } -fn repo() -> String { - std::env::var("GITHUB_REPOSITORY").unwrap_or_else(|_| "ToyOSOrg/ToyOS".into()) +/// The HTTP client of every request the build system makes: rustls with +/// RustCrypto's primitives and webpki's roots, so no C; a status is an answer, +/// not an error. +pub(crate) fn agent() -> ureq::Agent { + let tls = ureq::tls::TlsConfig::builder() + .provider(ureq::tls::TlsProvider::Rustls) + .root_certs(ureq::tls::RootCerts::WebPki) + .unversioned_rustls_crypto_provider(Arc::new(rustls_rustcrypto::provider())) + .build(); + ureq::Agent::config_builder().tls_config(tls).user_agent(USER_AGENT).http_status_as_error(false).build().new_agent() } /// Whether this job is main's publisher — [`PUBLISHER`] on main, scheduled or @@ -187,21 +196,17 @@ fn outputs(layers: &[Layer]) -> String { /// `cargo run -- --ci bootstrap`: this tree's toolchain made whole from what its /// job restored, and each layer told to the job's save steps as built or kept /// ([`built`]). A sysroot restored is all a guest job reads, so then nothing is -/// built; and a layer restored and not whole is refused, since its key reads -/// less than its build does. +/// built. A layer restored and not whole is refused, since its key reads less +/// than its build does, and so is one the build left not whole under its key. pub fn bootstrap(root: &Path) -> Result { let file = step_outputs()?; let layers = layers(root); let restored: Vec = layers.iter().map(|layer| root.join(&layer.paths[0]).exists()).collect(); - whole_as_restored(&layers, &restored, |layer| defect(root, layer))?; + whole(&layers, &restored, |layer| defect(root, layer)).map_err(|why| format!("restored, {why}"))?; if !restored[3] { let mut lock = crate::buildlock::shared(root, "the toolchain"); drop(crate::toolchain::ensure(root, &mut lock, false)); - for layer in &layers { - if let Some(why) = defect(root, layer) { - return Err(format!("the build left {} {} not whole: {why}", layer.name, layer.key)); - } - } + whole(&layers, &[true; 4], |layer| defect(root, layer)).map_err(|why| format!("built, {why}"))?; } tell(&file, &built(&layers, &restored))?; let said: Vec = layers @@ -216,13 +221,13 @@ pub fn bootstrap(root: &Path) -> Result { Ok(said.join(", ")) } -/// Refused where a layer `restored` says its job restored is not whole, as -/// `defect` finds it: its key reads less than its build does, and a build -/// under that key could never be saved over the entry. -fn whole_as_restored(layers: &[Layer], restored: &[bool], defect: impl Fn(&Layer) -> Option) -> Result<(), String> { - for (layer, _) in layers.iter().zip(restored).filter(|(_, restored)| **restored) { +/// Refused where a layer `which` names is not whole, as `defect` finds it: its +/// key reads less than its build does, and a build under that key could never +/// be saved over the entry. +fn whole(layers: &[Layer], which: &[bool], defect: impl Fn(&Layer) -> Option) -> Result<(), String> { + for (layer, _) in layers.iter().zip(which).filter(|(_, named)| **named) { if let Some(why) = defect(layer) { - return Err(format!("{} {} was restored and is not whole: {why}", layer.name, layer.key)); + return Err(format!("{} {} is not whole: {why}", layer.name, layer.key)); } } Ok(()) @@ -307,12 +312,14 @@ fn run(cmd: &mut Command) -> Result<(), String> { /// publisher, and before anything is put up unless it runs at main's tip. pub fn release(root: &Path) -> Result { let var = |name| std::env::var(name).ok(); - release_as(root, var("GITHUB_WORKFLOW_REF").as_deref(), var("GITHUB_EVENT_NAME").as_deref()) + let repo = var("GITHUB_REPOSITORY").ok_or("GITHUB_REPOSITORY is unset: only a runner publishes a toolchain")?; + release_as(root, &repo, var("GITHUB_WORKFLOW_REF").as_deref(), var("GITHUB_EVENT_NAME").as_deref()) } -/// [`release`], run as the job the runner names by its workflow and event. -fn release_as(root: &Path, workflow: Option<&str>, event: Option<&str>) -> Result { - publisher(workflow, event, &repo())?; +/// [`release`] of `repo`, run as the job the runner names by its workflow and +/// event. +fn release_as(root: &Path, repo: &str, workflow: Option<&str>, event: Option<&str>) -> Result { + publisher(workflow, event, repo)?; if !(cfg!(target_os = "linux") && crate::arch::Arch::HOST == Some(crate::arch::Arch::X86_64)) { return Err(format!("a release is {HOST}'s and this host is not one; a tarball packed here would install nowhere")); } @@ -332,32 +339,32 @@ fn release_as(root: &Path, workflow: Option<&str>, event: Option<&str>) -> Resul let tarball = tmp.join(ASSET); pack(&rust_dir.join("build"), &tarball)?; let tag = tag(&key); - let notes = notes(root, &tag, &manifest(&tag))?; - let put = put_up(root, &tag, ¬es, &tarball, &tmp)?; - Ok(format!("{put}; {}", alias(root, &tag, ¬es, &tmp)?)) + let notes = notes(root, repo, &tag, &manifest(&tag))?; + let github = Github::new(repo)?; + let put = put_up(&github, root, &tag, ¬es, &tarball)?; + Ok(format!("{put}; {}", alias(&github, root, &tag, ¬es, &tmp)?)) } /// The tarball of the toolchain laid out under `build` ([`lay_out`]) at -/// `tarball`: `/stage2` but its `bin/cargo`, which names a path only this -/// runner has, then its witness and `TOOLCHAIN`, in sorted order with no owner -/// or time, so one sysroot packs to one digest. +/// `tarball`, gzipped: `/stage2` but its `bin/cargo`, which names a path +/// only this runner has, then its witness and `TOOLCHAIN`, in sorted order with +/// no owner or time, so one sysroot packs to one digest. fn pack(build: &Path, tarball: &Path) -> Result<(), String> { let stage2 = Path::new(HOST).join("stage2"); let mut entries = vec![stage2.clone()]; walk(&build.join(&stage2), &stage2, &mut entries)?; entries.retain(|entry| *entry != stage2.join("bin/cargo")); entries.extend(["toyos-sysroot-witness", "TOOLCHAIN"].map(PathBuf::from)); - let mut tar = tar::Builder::new(Vec::new()); + let file = fs::File::create(tarball).map_err(|e| format!("{}: {e}", tarball.display()))?; + let gzip = flate2::write::GzEncoder::new(std::io::BufWriter::new(file), flate2::Compression::default()); + let mut tar = tar::Builder::new(gzip); tar.follow_symlinks(false); tar.mode(tar::HeaderMode::Deterministic); for entry in &entries { tar.append_path_with_name(build.join(entry), entry).map_err(|e| format!("pack {}: {e}", entry.display()))?; } - let bytes = tar.into_inner().map_err(|e| format!("pack {}: {e}", tarball.display()))?; - let file = fs::File::create(tarball).map_err(|e| format!("{}: {e}", tarball.display()))?; - let mut file = std::io::BufWriter::new(file); - ruzstd::encoding::compress(bytes.as_slice(), &mut file, ruzstd::encoding::CompressionLevel::Fastest); - file.flush().map_err(|e| format!("{}: {e}", tarball.display())) + let packed = tar.into_inner().and_then(|gzip| gzip.finish()).and_then(|mut file| file.flush()); + packed.map_err(|e| format!("{}: {e}", tarball.display())) } /// Every path under `dir`, named as it is under `prefix`, each directory's in @@ -400,83 +407,91 @@ fn put(release: Option<&Value>, name: &str, digest: &str) -> Result } /// `tag`'s release, made to carry `file` as its asset by its name: created with -/// `notes` where there is none, given them where there is, and then held to the -/// digest GitHub records. -fn put_up(root: &Path, tag: &str, notes: &str, file: &Path, tmp: &Path) -> Result { +/// `notes` where there is none and given them where there is, its asset put up +/// unless it already carries these bytes, and then held to the digest GitHub +/// records. +fn put_up(github: &Github, root: &Path, tag: &str, notes: &str, file: &Path) -> Result { let name = file.file_name().and_then(|name| name.to_str()).ok_or_else(|| format!("{} has no name", file.display()))?; - let digest = format!("sha256:{}", file_sha256(file)?); - let at = api(&format!("repos/{}/releases/tags/{tag}", repo())); - let found = github("GET", &at, None)?; - let (release, said) = match put(found.as_ref(), name, &digest)? { - Put::Carried => return Ok(format!("{tag} already carries this {name}")), - Put::Create => { + let bytes = fs::read(file).map_err(|e| format!("{}: {e}", file.display()))?; + let digest = format!("sha256:{}", sha256_hex(&bytes)); + let at = github.api(&format!("releases/tags/{tag}")); + let found = github.call("GET", &at, None)?; + let release = match &found { + None => { let commit = crate::sync::git(root, &["rev-parse", "HEAD"])?; let body = serde_json::json!({ "tag_name": tag, "name": tag, "body": notes, "target_commitish": commit.trim() }); - (send(tmp, "POST", &api(&format!("repos/{}/releases", repo())), &body)?, "put up") + github.send("POST", &github.api("releases"), &body)? } - Put::Upload => (renote(tmp, found, notes)?, "given its asset"), + Some(release) => { + let id = release["id"].as_u64().ok_or("a release with no id")?; + github.send("PATCH", &github.api(&format!("releases/{id}")), &serde_json::json!({ "body": notes }))? + } + }; + let said = match put(found.as_ref(), name, &digest)? { + Put::Carried => return Ok(format!("{tag} already carries this {name}")), + Put::Create => "put up", + Put::Upload => "given its asset", Put::Replace { asset } => { - github("DELETE", &api(&format!("repos/{}/releases/assets/{asset}", repo())), None)?; - (renote(tmp, found, notes)?, "had another writer's asset, now this one") + github.call("DELETE", &github.api(&format!("releases/assets/{asset}")), None)?; + "had another writer's asset, now this one" } }; let upload = release["upload_url"].as_str().ok_or("a release with no upload URL")?; let upload = format!("{}?name={name}", upload.split('{').next().unwrap_or(upload)); - github("POST", &upload, Some((file, "application/octet-stream")))?; - let now = github("GET", &at, None)?; + github.call("POST", &upload, Some((&bytes, "application/octet-stream")))?; + let now = github.call("GET", &at, None)?; if put(now.as_ref(), name, &digest)? != Put::Carried { return Err(format!("{tag} does not carry {digest} as its {name} after the upload")); } Ok(format!("{tag} {said}")) } -/// The release `found` given `notes`. -fn renote(tmp: &Path, found: Option, notes: &str) -> Result { - let id = found.as_ref().and_then(|release| release["id"].as_u64()).ok_or("a release with no id")?; - send(tmp, "PATCH", &api(&format!("repos/{}/releases/{id}", repo())), &serde_json::json!({ "body": notes })) -} - -/// `body` sent to `url` by `method`, as JSON: what GitHub answered. -fn send(tmp: &Path, method: &str, url: &str, body: &Value) -> Result { - let file = tmp.join("request.json"); - fs::write(&file, body.to_string()).map_err(|e| format!("{}: {e}", file.display()))?; - github(method, url, Some((&file, "application/json")))?.ok_or_else(|| format!("{method} {url} found nothing")) -} - -fn api(path: &str) -> String { - format!("https://api.github.com/{path}") -} - -/// GitHub's answer to `method` on `url`, sent the file `body` names as its -/// content type where there is one: the JSON it answered, null for no content, -/// and `None` for a 404. -fn github(method: &str, url: &str, body: Option<(&Path, &str)>) -> Result, String> { - let token = std::env::var("GH_TOKEN").map_err(|_| "GH_TOKEN is unset".to_string())?; - let mut curl = Command::new("curl"); - curl.args(["-sSL", "--retry", "3", "-X", method, "-w", "\n%{http_code}", "-A", USER_AGENT]) - .args(["-H", &format!("Authorization: Bearer {token}"), "-H", "Accept: application/vnd.github+json"]); - if let Some((file, content_type)) = body { - curl.args(["-H", &format!("Content-Type: {content_type}"), "--data-binary"]).arg(format!("@{}", file.display())); - } - let out = curl.arg(url).output().map_err(|e| format!("curl: {e}"))?; - if !out.status.success() { - return Err(format!("curl {method} {url} exited {}: {}", out.status, String::from_utf8_lossy(&out.stderr).trim())); - } - let text = String::from_utf8_lossy(&out.stdout).into_owned(); - let (answer, status) = text.rsplit_once('\n').unwrap_or(("", text.as_str())); - match status { - "200" | "201" => serde_json::from_str(answer).map(Some).map_err(|e| format!("{method} {url} answered no JSON: {e}")), - "204" => Ok(Some(Value::Null)), - "404" => Ok(None), - status => Err(format!("{method} {url} answered {status}: {answer}")), +/// GitHub's REST API for one repository, as the token its job was handed. +struct Github { + agent: ureq::Agent, + repo: String, + token: String, +} + +impl Github { + fn new(repo: &str) -> Result { + let token = std::env::var("GH_TOKEN").map_err(|_| "GH_TOKEN is unset".to_string())?; + Ok(Self { agent: agent(), repo: repo.to_string(), token }) + } + + /// The URL of `path` under the repository's API. + fn api(&self, path: &str) -> String { + format!("https://api.github.com/repos/{}/{path}", self.repo) + } + + /// GitHub's answer to `method` on `url`, sent `body` as its content type + /// where there is one: the JSON it answered, null for no content, and `None` + /// for a 404. + fn call(&self, method: &str, url: &str, body: Option<(&[u8], &str)>) -> Result, String> { + let request = ureq::http::Request::builder() + .method(method) + .uri(url) + .header("Authorization", format!("Bearer {}", self.token)) + .header("Accept", "application/vnd.github+json"); + let sent = match body { + Some((bytes, kind)) => request.header("Content-Type", kind).body(bytes).map(|request| self.agent.run(request)), + None => request.body(()).map(|request| self.agent.run(request)), + }; + let mut answer = sent.map_err(|e| format!("{method} {url}: {e}"))?.map_err(|e| format!("{method} {url}: {e}"))?; + let text = answer.body_mut().read_to_string().map_err(|e| format!("{method} {url}: {e}"))?; + match answer.status().as_u16() { + 200 | 201 => serde_json::from_str(&text).map(Some).map_err(|e| format!("{method} {url} answered no JSON: {e}")), + 204 => Ok(Some(Value::Null)), + 404 => Ok(None), + status => Err(format!("{method} {url} answered {status}: {text}")), + } } -} -fn file_sha256(path: &Path) -> Result { - let mut file = fs::File::open(path).map_err(|e| format!("{}: {e}", path.display()))?; - let mut hasher = Sha256::new(); - std::io::copy(&mut file, &mut hasher).map_err(|e| format!("{}: {e}", path.display()))?; - Ok(hasher.finalize().iter().map(|b| format!("{b:02x}")).collect()) + /// `body` sent to `url` by `method`, as JSON: what GitHub answered. + fn send(&self, method: &str, url: &str, body: &Value) -> Result { + let body = body.to_string(); + self.call(method, url, Some((body.as_bytes(), "application/json")))?.ok_or_else(|| format!("{method} {url} found nothing")) + } } /// Every `GLIBC_x.y` the shipped host binaries and libraries name, as the @@ -537,8 +552,8 @@ fn manifest(tag: &str) -> String { } /// The release notes: how to install it, what glibc it needs. -fn notes(root: &Path, tag: &str, manifest: &str) -> Result { - let url = format!("https://github.com/{}/releases/download/{tag}/{ASSET}", repo()); +fn notes(root: &Path, repo: &str, tag: &str, manifest: &str) -> Result { + let url = format!("https://github.com/{repo}/releases/download/{tag}/{ASSET}"); let (major, minor) = GLIBC_FLOOR; let userland = fs::read_to_string(root.join("userland/Cargo.toml")).map_err(|e| e.to_string())?; let rwh = userland @@ -552,7 +567,7 @@ fn notes(root: &Path, tag: &str, manifest: &str) -> Result { ## Install mkdir -p toyos-toolchain - curl -sSL {url} | tar --zstd -x -C toyos-toolchain + curl -sSL {url} | tar -xz -C toyos-toolchain rustup toolchain link toyos toyos-toolchain/{HOST}/stage2 ln -s \"$(rustup which cargo)\" toyos-toolchain/{HOST}/stage2/bin/cargo cargo +toyos build --target x86_64-unknown-toyos @@ -586,7 +601,7 @@ Until [rust-windowing/raw-window-handle#223](https://github.com/rust-windowing/r /// the name a consumer pins, moved onto `tag`. A second release carrying only a /// `TOOLCHAIN` naming `tag`, the commit that put it up and the SDK crates, /// because GitHub hangs an asset off one release id. -fn alias(root: &Path, tag: &str, notes: &str, tmp: &Path) -> Result { +fn alias(github: &Github, root: &Path, tag: &str, notes: &str, tmp: &Path) -> Result { let plan = crate::sdkversion::plan(root)?; if let Some(owed) = plan.iter().find(|r| r.publish) { let name = owed.krate.name; @@ -600,7 +615,7 @@ fn alias(root: &Path, tag: &str, notes: &str, tmp: &Path) -> Result Value { @@ -689,9 +703,7 @@ mod tests { /// file by its bytes and mode. fn unpacked(path: &Path) -> Vec<(String, String)> { let bytes = fs::read(path).unwrap(); - let mut tar = Vec::new(); - ruzstd::decoding::StreamingDecoder::new(bytes.as_slice()).unwrap().read_to_end(&mut tar).unwrap(); - let mut archive = tar::Archive::new(tar.as_slice()); + let mut archive = tar::Archive::new(flate2::read::GzDecoder::new(bytes.as_slice())); let mut seen = Vec::new(); for entry in archive.entries().unwrap() { let mut entry = entry.unwrap(); @@ -729,7 +741,7 @@ mod tests { for dir in [&first, &again, &other] { pack(&dir.join("build"), &dir.join(ASSET)).unwrap(); } - let digest = |dir: &Path| file_sha256(&dir.join(ASSET)).unwrap(); + let digest = |dir: &Path| sha256_hex(&fs::read(dir.join(ASSET)).unwrap()); assert_eq!(digest(&first), digest(&again), "one sysroot packed to two digests"); assert_ne!(digest(&first), digest(&other)); let stage2 = format!("{HOST}/stage2"); @@ -823,18 +835,19 @@ mod tests { ); } - /// **A layer restored and not whole is refused, never built again**: its - /// key reads less than its build does, and a build under that key could - /// never be saved. A layer not restored is the build's to make. + /// **A layer restored or built and not whole is refused, never built + /// again**: its key reads less than its build does, and a build under that + /// key could never be saved. A layer not restored is the build's to make. #[test] - fn a_restored_layer_that_is_not_whole_is_refused() { + fn a_layer_that_is_not_whole_is_refused() { let layers: Vec = LAYERS.iter().enumerate().map(|(at, (_, name))| layer(name, &at.to_string().repeat(16), &["p"])).collect(); let broken = |layer: &Layer| (layer.name == "compiler").then(|| "stage2 carries no clang".to_string()); - let refused = whole_as_restored(&layers, &[true, true, false, false], broken).unwrap_err(); - assert!(refused.starts_with("compiler 1111111111111111 was restored") && refused.contains("no clang"), "{refused}"); - assert_eq!(whole_as_restored(&layers, &[true, false, false, false], broken), Ok(())); - assert_eq!(whole_as_restored(&layers, &[true; 4], |_| None), Ok(())); + let refused = whole(&layers, &[true, true, false, false], broken).unwrap_err(); + assert!(refused.starts_with("compiler 1111111111111111 is not whole") && refused.contains("no clang"), "{refused}"); + assert_eq!(whole(&layers, &[true, false, false, false], broken), Ok(())); + assert!(whole(&layers, &[true; 4], broken).is_err(), "a build that left a layer not whole was taken"); + assert_eq!(whole(&layers, &[true; 4], |_| None), Ok(())); } /// **Each layer is the store the build system makes, where it makes it**: diff --git a/src/sdkversion.rs b/src/sdkversion.rs index 4fc47ffcb1d..76fcc4fd686 100644 --- a/src/sdkversion.rs +++ b/src/sdkversion.rs @@ -11,7 +11,6 @@ //! the index holds every version it names. use std::path::Path; -use std::process::Command; /// One published crate: the crates.io name, and its repository-relative /// directory. @@ -115,15 +114,12 @@ fn numbers(vers: &str) -> Option<(u64, u64, u64)> { /// for a crate never published. pub fn index(name: &str) -> Result { let url = format!("https://index.crates.io/{}/{}/{name}", &name[..2], &name[2..4]); - let out = Command::new("curl") - .args(["-sS", "-w", "\n%{http_code}", &url]) - .output() - .map_err(|e| format!("curl: {e}"))?; - let text = String::from_utf8_lossy(&out.stdout); - match text.rsplit_once('\n') { - Some((_, "404")) => Ok(String::new()), - Some((body, "200")) => Ok(body.to_string()), - _ => Err(format!("the crates.io index answered {text:?} for {name}")), + let mut answer = crate::release::agent().get(&url).call().map_err(|e| format!("{url}: {e}"))?; + let text = answer.body_mut().read_to_string().map_err(|e| format!("{url}: {e}"))?; + match answer.status().as_u16() { + 404 => Ok(String::new()), + 200 => Ok(text), + status => Err(format!("the crates.io index answered {status} for {name}: {text}")), } } @@ -147,6 +143,7 @@ mod tests { use super::*; use crate::gitfixture::{commit, repo}; use std::collections::BTreeMap; + use std::process::Command; use toyos_tmpdir::TempDir; const TWO: &[Crate] = From ccecfd5f7815a0e8f8c8af4e806a53ce1a68b308 Mon Sep 17 00:00:00 2001 From: japabu Date: Thu, 1 Oct 2026 23:58:38 +0200 Subject: [PATCH 16/30] File that a store's build code moves its key only through a RECIPE bumped by hand Round 3's BLOCKER on `src/keystore.rs`: the header said a key hashes exactly what its product's build reads, and the code that builds a store reaches its key only through that store's RECIPE. A key that read that code itself would move with every edit to `src/llvm.rs`, `src/sysroot.rs` or their tests, and an LLVM rebuild is the cold case of the merge queue's 180 minutes, so the weakness is filed with its exit instead, narrowed from the issue the release tag took with it, and "exactly" goes from the header. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L --- ...ey-only-through-a-recipe-bumped-by-hand.md | 26 +++++++++++++++++++ src/keystore.rs | 2 +- 2 files changed, 27 insertions(+), 1 deletion(-) create mode 100644 issues/build/a-stores-build-code-moves-its-key-only-through-a-recipe-bumped-by-hand.md diff --git a/issues/build/a-stores-build-code-moves-its-key-only-through-a-recipe-bumped-by-hand.md b/issues/build/a-stores-build-code-moves-its-key-only-through-a-recipe-bumped-by-hand.md new file mode 100644 index 00000000000..2f197828029 --- /dev/null +++ b/issues/build/a-stores-build-code-moves-its-key-only-through-a-recipe-bumped-by-hand.md @@ -0,0 +1,26 @@ +--- +status: open +kind: tooling +opened: 2026-10-01 +--- + +# A store's build code moves its key only through a RECIPE bumped by hand + +A keyed store's key (`src/keystore.rs`) reads its sources, the configuration +its build is given, the tools that run that build and the keys of the stores it +reads. The code that builds it — `src/llvm.rs`, `src/compiler.rs`, +`src/toolchain.rs`, `src/sysroot.rs`, `src/libc.rs`, `src/libcxx.rs` and +`src/clang.rs` — reaches the key only through the store's `RECIPE`, which moves +only when the change's author edits it. + +A change that does not keeps the old key. `toolchain.yml` then restores main's +entry under it on every pull request and on main, and no run can save over that +entry, so CI builds with a store the tree's code no longer makes. At +`src/libc.rs`, `.env_remove("RUSTFLAGS")` made +`.env("RUSTFLAGS", "-Coverflow-checks=on")` leaves `sysroot::key` where it was, +and no test reds. + +Owner: the keyed stores (`src/keystore.rs`). + +**Exit**: a change to the code that builds a keyed store moves that store's key +with no hand edit, and a test reds on the `src/libc.rs` patch above. diff --git a/src/keystore.rs b/src/keystore.rs index 93c7eec60f6..a4d144ace49 100644 --- a/src/keystore.rs +++ b/src/keystore.rs @@ -3,7 +3,7 @@ //! that removes a key no registered worktree records and nobody is making or //! using. //! -//! **A key hashes exactly what its product's build reads**: its sources, the +//! **A key hashes what its product's build reads**: its sources, the //! configuration its build is given, the tools that run that build and the keys //! of the products it reads; and it is known before the product is. So a //! product found under its key, made here or restored by a CI runner from From 74ee3fc8373305d9c2ebb3718f83ea1c31d049e7 Mon Sep 17 00:00:00 2001 From: japabu Date: Thu, 1 Oct 2026 23:58:48 +0200 Subject: [PATCH 17/30] Main builds one toolchain key at a time, and a merge group saves only the sysroot Round 3's NOTEs and its write-grant BLOCKER: - `toolchain.yml`'s job is one concurrency group per ref, never cancelled, so `publish.yml`'s and `nightly.yml`'s builds on main of one cold key run one after the other and the second restores what the first saved, where two groups built it twice and refused the later save. `publish.yml`'s own group goes. - A merge group saves no LLVM, compiler or freestanding libraries: its scope is read by nothing but its own guest job, which restores the sysroot alone. It still saves the sysroot it built, or that guest job restores nothing. - `.claude/agents/reviewer.md`'s Workflows names `nightly.yml`'s `release` as the one job granted `contents: write`. - `each_cache_has_one_writer` goes: its `${{` arm passed every toolchain save, and the Workflows rules are the reviewer's to read. - The portability jobs run `cargo run -- --build-only` without `env -u`: bootstrap's runs remove `GITHUB_ACTIONS` and `CI` themselves. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L --- .claude/agents/reviewer.md | 5 +++-- .github/workflows/nightly.yml | 4 ++-- .github/workflows/publish.yml | 3 --- .github/workflows/toolchain.yml | 12 +++++++++--- src/ci.rs | 30 ------------------------------ 5 files changed, 14 insertions(+), 40 deletions(-) diff --git a/.claude/agents/reviewer.md b/.claude/agents/reviewer.md index 3744d275aa2..b7ef288b7e7 100644 --- a/.claude/agents/reviewer.md +++ b/.claude/agents/reviewer.md @@ -80,8 +80,9 @@ above; otherwise it is a NOTE. or `[patch]`, which cargo ignores with only a warning; a new package without a `description` saying what it is. A new cargo feature or `cfg` arm of one, and every arm a changed `src/clippy.rs` shape stops building, is shown linted in the pull request body: a `mem::forget` planted in that arm turns `cargo run -- --clippy` red. -- **Workflows.** GitHub's cache scoping is the provenance of every cache entry, so a BLOCKER each: - - Only main's runs save a cache entry other refs restore. +- **Workflows.** A BLOCKER each: + - Only main's runs save a cache entry other refs restore: GitHub's cache scoping is every entry's provenance. + - `nightly.yml`'s `release` is the only job granted `contents: write`. - No workflow runs on `pull_request_target`, `workflow_run`, `issue_comment` or any other trigger that runs code other than main's on main's ref. - No workflow or job declares `cache-mode: write` or `write-only`. - `guest / suite` has no job-level `if:`, and a job that calls it runs whatever `toolchain` concluded: a skipped required check reads as green. diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index 0ee218ce736..9b46272c466 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -105,7 +105,7 @@ jobs: - &checkout uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 - - run: env -u GITHUB_ACTIONS -u CI cargo run -- --build-only + - run: cargo run -- --build-only portability-macos: runs-on: macos-latest @@ -119,6 +119,6 @@ jobs: curl --proto '=https' --tlsv1.2 -sSf -o "$RUNNER_TEMP/rustup-init.sh" https://sh.rustup.rs sh "$RUNNER_TEMP/rustup-init.sh" -y --profile minimal --default-toolchain stable echo "$HOME/.cargo/bin" >> "$GITHUB_PATH" - - run: env -u GITHUB_ACTIONS -u CI cargo run -- --build-only + - run: cargo run -- --build-only # The host suite's macOS arms run here alone: `ci.yml`'s `host` is Linux. - run: cargo run -- --ci host diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 43909793a6d..19ba55c7f17 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -37,9 +37,6 @@ jobs: run: cargo run -- --ci publish toolchain: - concurrency: - group: publish-toolchain - cancel-in-progress: false permissions: contents: read uses: ./.github/workflows/toolchain.yml diff --git a/.github/workflows/toolchain.yml b/.github/workflows/toolchain.yml index 4048cfb62a6..09face3d886 100644 --- a/.github/workflows/toolchain.yml +++ b/.github/workflows/toolchain.yml @@ -13,6 +13,10 @@ jobs: # Bare, not a container: its glibc is a build's floor. runs-on: ubuntu-24.04 timeout-minutes: 350 + # Main's two callers build one key once, one after the other. + concurrency: + group: toolchain-${{ github.ref }} + cancel-in-progress: false outputs: sysroot-key: ${{ steps.keys.outputs.sysroot-key }} sysroot-path: ${{ steps.keys.outputs.sysroot-path }} @@ -45,19 +49,21 @@ jobs: - id: build run: cargo run -- --ci bootstrap - - if: steps.build.outputs.llvm == 'built' + # A merge group's scope is read only by its own guest job, which restores + # the sysroot alone. + - if: steps.build.outputs.llvm == 'built' && github.event_name != 'merge_group' uses: actions/cache/save@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 with: path: ${{ steps.keys.outputs.llvm-path }} key: ${{ steps.keys.outputs.llvm-key }} - - if: steps.build.outputs.compiler == 'built' + - if: steps.build.outputs.compiler == 'built' && github.event_name != 'merge_group' uses: actions/cache/save@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 with: path: ${{ steps.keys.outputs.compiler-path }} key: ${{ steps.keys.outputs.compiler-key }} - - if: steps.build.outputs.freestanding == 'built' + - if: steps.build.outputs.freestanding == 'built' && github.event_name != 'merge_group' uses: actions/cache/save@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 with: path: ${{ steps.keys.outputs.freestanding-path }} diff --git a/src/ci.rs b/src/ci.rs index cea1a2f7bf2..b7b0d558552 100644 --- a/src/ci.rs +++ b/src/ci.rs @@ -1071,36 +1071,6 @@ mod tests { assert_eq!(seen, 5, "ci.yml, nightly.yml and publish.yml, and guest.yml and toolchain.yml"); } - /// Exactly one job writes each cache, so what a pull request restores is one - /// run's tree and never a race between two writers. - #[test] - fn each_cache_has_one_writer() { - let dir = repo_root().join(".github/workflows"); - let mut writers = Vec::new(); - for entry in std::fs::read_dir(&dir).expect(".github/workflows is readable").flatten() { - let text = std::fs::read_to_string(entry.path()).expect("a readable workflow"); - let name = entry.file_name().to_string_lossy().into_owned(); - assert!(!text.contains("actions/cache@"), "{name}: the combined action saves too"); - let lines: Vec<&str> = text.lines().collect(); - for (at, line) in lines.iter().enumerate() { - if line.contains("actions/cache/save@") { - let key = lines[at..] - .iter() - .find_map(|l| l.trim_start().strip_prefix("key: ")) - .expect("a save names its key"); - let cache = if key.starts_with("${{") { key } else { key.split('$').next().unwrap_or("") }; - writers.push((name.clone(), cache.to_string())); - } - } - } - assert!(!writers.is_empty(), "no job writes a cache, so every restore is cold"); - writers.sort(); - let mut prefixes: Vec<&String> = writers.iter().map(|(_, p)| p).collect(); - prefixes.dedup(); - assert_eq!(prefixes.len(), writers.len(), "a cache with two writers: {writers:?}"); - assert!(writers.iter().all(|(f, _)| f == "nightly.yml" || f == "toolchain.yml"), "{writers:?}"); - } - #[test] fn the_declared_version_is_a_version() { let declared = From 1a4f3abda78ae1d312e93c7d50a57f2370c22cee Mon Sep 17 00:00:00 2001 From: japabu Date: Thu, 1 Oct 2026 23:58:56 +0200 Subject: [PATCH 18/30] Review round 3's REMOVEs Deleted, not rewritten: the nested-NMI issue's "timed out waiting for it" and "with byte-identical interleaving"; the rustc -vV issue's clause on what the release tag hashes; `src/toolchain.rs`'s "a 401 MiB artifact" and its "fresh from the published artifact on every run". 9e917f19b carried two changes beside its key: `CLAUDE.md`'s and `.claude/agents/implementer.md`'s guest-test lines go back to main's, since the owner's direction that agents run their own guest tests supersedes what this branch wrote there, and the zstd-level issue goes with ruzstd (252c49e92). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L --- ...two-toolchain-releases-report-the-same-rustc-vv.md | 3 +-- ...port-interleaves-with-another-cpus-console-line.md | 6 +++--- src/toolchain.rs | 11 +++-------- 3 files changed, 7 insertions(+), 13 deletions(-) diff --git a/issues/build/two-toolchain-releases-report-the-same-rustc-vv.md b/issues/build/two-toolchain-releases-report-the-same-rustc-vv.md index 2c2514e6a20..105513d93e3 100644 --- a/issues/build/two-toolchain-releases-report-the-same-rustc-vv.md +++ b/issues/build/two-toolchain-releases-report-the-same-rustc-vv.md @@ -11,8 +11,7 @@ nor `description` under `[rust]`, so bootstrap's channel is `dev` and its default for that channel is `omit-git-hash = true` (`rust/src/bootstrap/src/core/config/config.rs`, the `omit_git_hash` line). Every release then reports `rustc 1.99.0-dev` with byte-identical `rustc -vV` -output — and the release tag hashes more than `rust` anyway: `toyos-abi/src`, -`toyos/src` and `userland/libc/src`. +output. Cargo fingerprints a compile on `rustc -vV`. A consumer that switches releases in a reused target directory sees no compiler change, keeps the old rlibs, and diff --git a/issues/kernel/the-nested-nmi-report-interleaves-with-another-cpus-console-line.md b/issues/kernel/the-nested-nmi-report-interleaves-with-another-cpus-console-line.md index 00fa2b876d0..c7aa009f12b 100644 --- a/issues/kernel/the-nested-nmi-report-interleaves-with-another-cpus-console-line.md +++ b/issues/kernel/the-nested-nmi-report-interleaves-with-another-cpus-console-line.md @@ -13,13 +13,13 @@ line was `[kernel 0.385 cpu1] CPU 1: joining scheduler`, and the 16550 carried: [[kenrnmel i0.38]5 cpNu1E] CSPUT 1E: Djo inNiMngI s choednule r -`NESTED NMI` was never whole on the console, so `nested_nmi_is_loud` timed out -waiting for it, and the machine halted with its report unreadable. +`NESTED NMI` was never whole on the console, and the machine halted with its +report unreadable. #675 (`bc68e5d78`) writes the report under the console's registers (`serial::panic_registers`). -**Evidence:** red under KVM in two runs, with byte-identical interleaving: +**Evidence:** red under KVM in two runs: - Main's nightly `guest` lane at `06788146b`, run 36843762360, job 110374194368. - PR #671's `guest` check, on its merge onto `59052827f`, run 36863809437, job 110375742604. diff --git a/src/toolchain.rs b/src/toolchain.rs index 06913b03a66..b9b513f79b1 100644 --- a/src/toolchain.rs +++ b/src/toolchain.rs @@ -385,10 +385,8 @@ fn cargo_link_stale(stage2: &Path) -> bool { /// /// **A symlink, and what survives the artifact round-trip is this step rather /// than the link.** `src/release.rs` excludes it from the tarball: it names a -/// path only the publishing runner has, and a copy would put a 32 MB host -/// binary into a 401 MiB artifact to stand in for a file the consumer can make -/// in a microsecond. `Owner::Installed` makes it, exactly as it makes the host -/// target. +/// path only the publishing runner has. `Owner::Installed` makes it, exactly as +/// it makes the host target. pub(crate) fn provision_toolchain_cargo(stage2: &Path) { let at = stage2.join("bin/cargo"); let _ = fs::remove_file(&at); @@ -631,10 +629,7 @@ fn check_installed_toolchain(root: &Path, rust_dir: &Path) { // Recreated rather than shipped: both of these point into whatever stable // toolchain this machine has, which is not a path any artifact can know. - // This is CI's whole share of the cargo provisioning — it links its - // toolchain fresh from the published artifact on every run, so nothing - // upstream of the download can have put one there. Its clang is the - // artifact's own, so this is not `complete`. + // Its clang is the artifact's own, so this is not `complete`. if host_target_missing(rust_dir) { link_host_target(rust_dir); } From 62999d902e4ba437bb8f2a582ab1e31494bef2d9 Mon Sep 17 00:00:00 2001 From: japabu Date: Fri, 2 Oct 2026 10:55:47 +0200 Subject: [PATCH 19/30] The nightly's release is skipped off main, and a toolchain job that waits is never cancelled nightly.yml's `release` ran on every dispatch and `--ci release` refuses it by name off main, so a nightly dispatched on a branch could only conclude failure. It now carries `if: github.ref == 'refs/heads/main'`, as `host`'s save step does; the refusal by name stays the boundary (`only_mains_publisher_publishes`). toolchain.yml's `build` ran in a concurrency group that holds one pending job: a third arriving cancelled it, and when that was the nightly's, the nightly ran no `release` and its `tcg` restored an empty key. The group now sets `queue: max`, under which up to 100 wait and none is cancelled for a later one (GitHub's `actions-group-concurrency` reusable, documented for github.com since github/docs 336b7f546d). `cancel-in-progress` keeps its default, false: the two may not be combined as true. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_013UDZQ6fSKw14e4w2TKTRfm --- .github/workflows/nightly.yml | 2 ++ .github/workflows/toolchain.yml | 5 +++-- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index 790d3d4d669..b26634d85f0 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -51,8 +51,10 @@ jobs: toolchain: uses: ./.github/workflows/toolchain.yml + # Skipped off main, where the driver refuses it by name. release: needs: toolchain + if: github.ref == 'refs/heads/main' runs-on: ubuntu-24.04 timeout-minutes: 30 permissions: diff --git a/.github/workflows/toolchain.yml b/.github/workflows/toolchain.yml index 09face3d886..22ba02bfbd9 100644 --- a/.github/workflows/toolchain.yml +++ b/.github/workflows/toolchain.yml @@ -13,10 +13,11 @@ jobs: # Bare, not a container: its glibc is a build's floor. runs-on: ubuntu-24.04 timeout-minutes: 350 - # Main's two callers build one key once, one after the other. + # Main's two callers build one key once, one after the other, and one that + # waits is never cancelled for a later one. concurrency: group: toolchain-${{ github.ref }} - cancel-in-progress: false + queue: max outputs: sysroot-key: ${{ steps.keys.outputs.sysroot-key }} sysroot-path: ${{ steps.keys.outputs.sysroot-path }} From 404df2c5586a244e1c09075603876eceb4657eae Mon Sep 17 00:00:00 2001 From: japabu Date: Fri, 2 Oct 2026 10:55:47 +0200 Subject: [PATCH 20/30] File the two compromises the toolchain job keeps, and close the two kernel issues CI met Filed, each with its owner, evidence and exit: - a runner image that moves cc, c++ or CMake moves every LLVM key, so every pull request and merge group builds the cold path until main saves the new keys; - a toolchain job that ends inside `--ci bootstrap` saves no layer it built: run 36913380100 built all four, reded in that step and saved none, and run 36934214557 built the LLVM again. Closed on job 110649069031 (run 36934214557, `guest / suite` under KVM on the merge of f1ccb0b3e into 76d0d9389: "test result: ok. 21 passed, 21 total"), which meets both exits as written: - every-aarch64-guest-dies-at-the-kernels-entry-on-cis-firmware: the sixteen `virt_*` tests are green in CI's `guest` check, since #677 enters the AArch64 kernel with the MMU off; - the-nested-nmi-report-interleaves-with-another-cpus-console-line: `nested_nmi_is_loud` is green there, since #675 writes the report under the console's registers, which `nested_nmi`'s own comment says. Neither was cited outside the other. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_013UDZQ6fSKw14e4w2TKTRfm --- ...or-cmake-makes-every-toolchain-job-cold.md | 26 +++++++++++++ ...does-not-finish-saves-no-layer-it-built.md | 26 +++++++++++++ ...es-at-the-kernels-entry-on-cis-firmware.md | 38 ------------------- ...erleaves-with-another-cpus-console-line.md | 31 --------------- 4 files changed, 52 insertions(+), 69 deletions(-) create mode 100644 issues/build/a-runner-image-that-moves-cc-or-cmake-makes-every-toolchain-job-cold.md create mode 100644 issues/build/a-toolchain-job-that-does-not-finish-saves-no-layer-it-built.md delete mode 100644 issues/kernel/every-aarch64-guest-dies-at-the-kernels-entry-on-cis-firmware.md delete mode 100644 issues/kernel/the-nested-nmi-report-interleaves-with-another-cpus-console-line.md diff --git a/issues/build/a-runner-image-that-moves-cc-or-cmake-makes-every-toolchain-job-cold.md b/issues/build/a-runner-image-that-moves-cc-or-cmake-makes-every-toolchain-job-cold.md new file mode 100644 index 00000000000..1edfc3196c3 --- /dev/null +++ b/issues/build/a-runner-image-that-moves-cc-or-cmake-makes-every-toolchain-job-cold.md @@ -0,0 +1,26 @@ +--- +status: open +kind: tooling +opened: 2026-10-02 +--- + +# A runner image that moves cc, c++ or CMake makes every toolchain job cold + +`llvm::key` reads the `cc`, `c++` and CMake that build an LLVM, each by its +resolved path and all its `--version` says (`src/llvm.rs`), and +`toolchain.yml`'s `build` takes all three from `ubuntu-24.04`'s image +(20260927.320.1 in job 110610545360). An image that moves one moves every LLVM +key, and with it every compiler, freestanding and sysroot key. Every pull +request and merge group then builds all four layers until a run on main has +saved the new keys: a pull request's saves reach no other ref, and a merge +group saves only its sysroot. + +That build is the cold path: 153:17 from the run's creation to `guest / suite` +green in run 36934214557, 2:09:01 of it `--ci bootstrap`, against the merge +queue's `check_response_timeout_minutes`. + +Owner: the toolchain job (`.github/workflows/toolchain.yml`). + +**Exit**: the tools an LLVM's key reads move only with a commit to this +repository, and a toolchain job on a runner image newer than that commit keys +the LLVM as the one before it did. diff --git a/issues/build/a-toolchain-job-that-does-not-finish-saves-no-layer-it-built.md b/issues/build/a-toolchain-job-that-does-not-finish-saves-no-layer-it-built.md new file mode 100644 index 00000000000..2121cf4038d --- /dev/null +++ b/issues/build/a-toolchain-job-that-does-not-finish-saves-no-layer-it-built.md @@ -0,0 +1,26 @@ +--- +status: open +kind: tooling +opened: 2026-10-02 +--- + +# A toolchain job that does not finish saves no layer it built + +`toolchain.yml`'s `build` makes every layer its restores missed in one step, +`cargo run -- --ci bootstrap`, and its four saves follow that step. A job that +ends inside it has saved nothing, whatever it had made: cancelled by a push to +its pull request, which cancels `ci.yml`'s run in progress, or red in the step +after its LLVM was placed. + +Run 36913380100's `toolchain / build` (job 110541308331) ran 2:08:24, built +all four layers and reded in that step's last check, so its saves were skipped. +The next run, 36934214557, +missed all four restores and built the LLVM again, 1:29:15 of its 2:09:01 +`--ci bootstrap` (job 110610545360). + +Owner: the toolchain job (`.github/workflows/toolchain.yml`, +`release::bootstrap`). + +**Exit**: each layer is saved by the step after the one that made it, and a +toolchain job stopped after its LLVM was placed leaves an entry the next run +restores. diff --git a/issues/kernel/every-aarch64-guest-dies-at-the-kernels-entry-on-cis-firmware.md b/issues/kernel/every-aarch64-guest-dies-at-the-kernels-entry-on-cis-firmware.md deleted file mode 100644 index d4d8c222f90..00000000000 --- a/issues/kernel/every-aarch64-guest-dies-at-the-kernels-entry-on-cis-firmware.md +++ /dev/null @@ -1,38 +0,0 @@ ---- -status: open -kind: defect -opened: 2026-10-01 ---- - -# Every AArch64 guest dies at the kernel's entry on CI's firmware - -All sixteen `virt_*` tests red in CI. Each boot's console ends the same way: -`Loader log: the kernel handoff begins`, then the firmware's -`Synchronous Exception at 0x00000000BC33EB20`. The PC differs per kernel build, -but always falls inside the kernel image the loader placed at `0xbc200000`. That -happens at EL1 entry (`Profile::Virt`) and at EL2 entry (`VirtEl2`), on one CPU -and on eight. The kernel prints nothing first, so the tests time out waiting for -their first marker. - -**Evidence**, identical in two runs on toolchain -`toolchain-linux-x86_64-48dd24f826263d6c`: -- Main's nightly `guest` lane at `06788146b`, run 36843762360, job 110374194368. -- PR #671's `guest` check, on its merge onto `59052827f`, run 36863809437, job - 110375742604. - -Both ran QEMU 11.1.1 under TCG `-cpu max` on an AMD EPYC 9V45 with 4 cores. The -firmware was Debian's `AAVMF_CODE.no-secboot.fd`, "version 2026.05-2". Both -logged `test result: FAILED. 4 passed, 17 failed`. The other red is -`issues/kernel/the-nested-nmi-report-interleaves-with-another-cpus-console-line.md`. - -The same sixteen tests pass on the dev host, in a whole-suite run of #670's -branch (`test result: ok. 21 passed`). That host ran QEMU 11.1.1 from -Homebrew, under the same TCG `-cpu max` for `VirtEl2`. Two parts of the -instrument differ: -- The firmware: the dev host runs QEMU's bundled `edk2-stable202408-prebuilt.qemu.org`. -- The toolchain: the dev host builds its own, and CI restores the sysroot its - toolchain job built. - -`.github/qemu-version` pins neither of the two. - -**Exit:** the sixteen `virt_*` tests are green in CI's `guest` check. diff --git a/issues/kernel/the-nested-nmi-report-interleaves-with-another-cpus-console-line.md b/issues/kernel/the-nested-nmi-report-interleaves-with-another-cpus-console-line.md deleted file mode 100644 index c7aa009f12b..00000000000 --- a/issues/kernel/the-nested-nmi-report-interleaves-with-another-cpus-console-line.md +++ /dev/null @@ -1,31 +0,0 @@ ---- -status: open -kind: defect -opened: 2026-10-01 ---- - -# The nested-NMI report interleaves with another CPU's console line - -`nested_nmi` (`kernel/src/arch/x86_64/idt/nmi.rs`) wrote its report through -`serial::panic_raw`, which took no lock. With cpu1 writing its own record at -the same moment, the two lines interleaved byte by byte on the 16550. The cpu1 -line was `[kernel 0.385 cpu1] CPU 1: joining scheduler`, and the 16550 carried: - - [[kenrnmel i0.38]5 cpNu1E] CSPUT 1E: Djo inNiMngI s choednule r - -`NESTED NMI` was never whole on the console, and the machine halted with its -report unreadable. - -#675 (`bc68e5d78`) writes the report under the console's registers -(`serial::panic_registers`). - -**Evidence:** red under KVM in two runs: -- Main's nightly `guest` lane at `06788146b`, run 36843762360, job 110374194368. -- PR #671's `guest` check, on its merge onto `59052827f`, run 36863809437, job - 110375742604. - -It is green under TCG: on the dev host in a whole-suite run of #670's branch, -and on a runner in main's nightly `tcg` lane at `06788146b` (job 110374194382). - -**Exit:** `nested_nmi_is_loud` is green in CI's KVM `guest` check, and a report -written while another CPU is writing a record reads whole. From 42a823b1f62f546ccb1cb6e78bc9734e7f292ad9 Mon Sep 17 00:00:00 2001 From: japabu Date: Fri, 2 Oct 2026 10:55:47 +0200 Subject: [PATCH 21/30] Review round 4's REMOVE: compile_error_at's doc is its first line The paragraph this branch had corrected, with run 31370078581's story, goes, and with it the sentence that named "that allowance" after it. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_013UDZQ6fSKw14e4w2TKTRfm --- src/toolchain.rs | 10 ---------- 1 file changed, 10 deletions(-) diff --git a/src/toolchain.rs b/src/toolchain.rs index b9b513f79b1..7e4c29e1ee6 100644 --- a/src/toolchain.rs +++ b/src/toolchain.rs @@ -794,16 +794,6 @@ impl Drop for Restore { } /// Where a compile error starts in an `x build` log, if there is one. -/// -/// The hosted rustc's build lets a non-zero `x build` through when the -/// artifacts it needs are on disk, because rustdoc for ToyOS does not link and -/// never has. That allowance used to be *anything at all*, as long as a `rustc` from -/// some earlier build was still there — so run `31370078581` compiled std with -/// `error[E0433]`, took the allowance, and died 83 seconds and 260 lines later -/// at a missing file. The reported failure was the consequence. -/// -/// A compile error cannot be a link failure, so it cannot be the thing that -/// allowance is for. fn compile_error_at(log: &[String]) -> Option { log.iter().position(|l| { let l = l.trim_start(); From cb19bca03fd18646b1ce7c73e9387a032b442e81 Mon Sep 17 00:00:00 2001 From: japabu Date: Fri, 2 Oct 2026 14:14:33 +0200 Subject: [PATCH 22/30] The build system's HTTP agent is rustls on ring The owner ruled it: one TLS provider everywhere. `release::agent` installed rustls-rustcrypto 0.0.2-alpha, which main's internet-clients track says does not come back and whose exit is that no manifest names it; the root manifest was a third one naming it. ureq's own `rustls` feature is rustls with ring and webpki's roots, so the agent configures no TLS at all and the root manifest names neither provider. The lockfile loses the RustCrypto stack: against origin/main it is +252 -1, where it was +801 -6. ring compiles C and assembly through the host's `cc` in the build system's own build, which needed none: `cargo tree -i cc` printed nothing and now prints cc under ring's build dependencies. The arrival is declared in the `cc` row of issues/build/the-build-runs-host-tools-outside-rust-and-qemu.md. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_013UDZQ6fSKw14e4w2TKTRfm --- Cargo.lock | 558 +----------------- Cargo.toml | 8 +- ...d-runs-host-tools-outside-rust-and-qemu.md | 2 +- src/release.rs | 14 +- 4 files changed, 16 insertions(+), 566 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index a0b45a495e5..f2824e05b5a 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -8,41 +8,6 @@ version = "2.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa" -[[package]] -name = "aead" -version = "0.5.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d122413f284cf2d62fb1b7db97e02edb8cda96d769b16e443a4f6195e35662b0" -dependencies = [ - "crypto-common", - "generic-array", -] - -[[package]] -name = "aes" -version = "0.8.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b169f7a6d4742236a0a00c541b845991d0ac43e546831af1249753ab4c3aa3a0" -dependencies = [ - "cfg-if", - "cipher", - "cpufeatures", -] - -[[package]] -name = "aes-gcm" -version = "0.10.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "831010a0f742e1209b3bcea8fab6a8e149051ba6099432c8cb2cc117dec3ead1" -dependencies = [ - "aead", - "aes", - "cipher", - "ctr", - "ghash", - "subtle", -] - [[package]] name = "aho-corasick" version = "1.1.5" @@ -85,24 +50,12 @@ version = "1.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c08606f8c3cbf4ce6ec8e28fb0014a2c086708fe954eaa885384a6165172e7e8" -[[package]] -name = "base16ct" -version = "0.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4c7f02d4ea65f2c1853089ffd8d2787bdbc63de2f0d29dedbcf8ccdfa0ccd4cf" - [[package]] name = "base64" version = "0.23.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ac07cdecf99051d9a5238b80f35af32cdeba5b336e55d957b318b50137e18da5" -[[package]] -name = "base64ct" -version = "1.8.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2af50177e190e07a26ab74f8b1efbfe2ef87da2116221318cb1c2e82baf7de06" - [[package]] name = "bcachefs" version = "0.1.0" @@ -186,30 +139,6 @@ version = "1.0.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" -[[package]] -name = "chacha20" -version = "0.9.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c3613f74bd2eac03dad61bd53dbe620703d4371614fe0bc3b9f04dd36fe4e818" -dependencies = [ - "cfg-if", - "cipher", - "cpufeatures", -] - -[[package]] -name = "chacha20poly1305" -version = "0.10.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "10cd79432192d1c0f4e1a0fef9527696cc039165d729fb41b3f4f4f354c2dc35" -dependencies = [ - "aead", - "chacha20", - "cipher", - "poly1305", - "zeroize", -] - [[package]] name = "chrono" version = "0.4.44" @@ -223,23 +152,6 @@ dependencies = [ "windows-link", ] -[[package]] -name = "cipher" -version = "0.4.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "773f3b9af64447d2ce9850330c473515014aa235e6a783b02db81ff39e4a3dad" -dependencies = [ - "crypto-common", - "inout", - "zeroize", -] - -[[package]] -name = "const-oid" -version = "0.9.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c2459377285ad874054d797f3ccebf984978aa39129f6eafde5cdc8315b612f8" - [[package]] name = "core-foundation-sys" version = "0.8.7" @@ -279,18 +191,6 @@ dependencies = [ "cfg-if", ] -[[package]] -name = "crypto-bigint" -version = "0.5.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0dc92fb57ca44df6db8059111ab3af99a63d5d0f8375d9972e319a379c6bab76" -dependencies = [ - "generic-array", - "rand_core 0.6.4", - "subtle", - "zeroize", -] - [[package]] name = "crypto-common" version = "0.1.7" @@ -301,15 +201,6 @@ dependencies = [ "typenum", ] -[[package]] -name = "ctr" -version = "0.9.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0369ee1ad671834580515889b80f2ea915f23b8be8d0daa4bbaf2ac5c7590835" -dependencies = [ - "cipher", -] - [[package]] name = "curve25519-dalek" version = "4.1.3" @@ -323,7 +214,6 @@ dependencies = [ "fiat-crypto", "rustc_version", "subtle", - "zeroize", ] [[package]] @@ -337,17 +227,6 @@ dependencies = [ "syn 2.0.117", ] -[[package]] -name = "der" -version = "0.7.10" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e7c1832837b905bbfb5101e07cc24c8deddf52f93225eee6ead5f4d63d53ddcb" -dependencies = [ - "const-oid", - "pem-rfc7468", - "zeroize", -] - [[package]] name = "derive-into-owned" version = "0.2.0" @@ -366,23 +245,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292" dependencies = [ "block-buffer", - "const-oid", "crypto-common", - "subtle", -] - -[[package]] -name = "ecdsa" -version = "0.16.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ee27f32b5c5292967d2d4a9d7f1e0b0aed2c15daded5a60300e4abb9d8020bca" -dependencies = [ - "der", - "digest", - "elliptic-curve", - "rfc6979", - "signature", - "spki", ] [[package]] @@ -391,7 +254,6 @@ version = "2.2.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "115531babc129696a58c64a4fef0a8bf9e9698629fb97e9e40767d235cfbcd53" dependencies = [ - "pkcs8", "signature", ] @@ -403,31 +265,8 @@ checksum = "70e796c081cee67dc755e1a36a0a172b897fab85fc3f6bc48307991f64e4eca9" dependencies = [ "curve25519-dalek", "ed25519", - "serde", "sha2", "subtle", - "zeroize", -] - -[[package]] -name = "elliptic-curve" -version = "0.13.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b5e6043086bf7973472e0c7dff2142ea0b680d30e18d9cc40f267efbf222bd47" -dependencies = [ - "base16ct", - "crypto-bigint", - "digest", - "ff", - "generic-array", - "group", - "hkdf", - "pem-rfc7468", - "pkcs8", - "rand_core 0.6.4", - "sec1", - "subtle", - "zeroize", ] [[package]] @@ -457,16 +296,6 @@ dependencies = [ "log", ] -[[package]] -name = "ff" -version = "0.13.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c0b50bfb653653f9ca9095b427bed08ab8d75a137839d9ad64eb11810d5b6393" -dependencies = [ - "rand_core 0.6.4", - "subtle", -] - [[package]] name = "fiat-crypto" version = "0.2.9" @@ -544,7 +373,6 @@ checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a" dependencies = [ "typenum", "version_check", - "zeroize", ] [[package]] @@ -583,16 +411,6 @@ dependencies = [ "wasip3", ] -[[package]] -name = "ghash" -version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f0d8a4362ccb29cb0b265253fb0a2728f592895ee6854fd9bc13f2ffda266ff1" -dependencies = [ - "opaque-debug", - "polyval", -] - [[package]] name = "gpt" version = "3.1.0" @@ -605,17 +423,6 @@ dependencies = [ "uuid", ] -[[package]] -name = "group" -version = "0.13.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f0f9ef7462f7c099f518d754361858f86d8a07af53ba9af0fe635bbccb151a63" -dependencies = [ - "ff", - "rand_core 0.6.4", - "subtle", -] - [[package]] name = "hashbrown" version = "0.15.5" @@ -642,24 +449,6 @@ version = "0.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea" -[[package]] -name = "hkdf" -version = "0.12.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7b5f8eb2ad728638ea2c7d47a21db23b7b58a72ed6a38256b8a1849f15fbbdf7" -dependencies = [ - "hmac", -] - -[[package]] -name = "hmac" -version = "0.12.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6c49c37c09c17a53d937dfbb742eb3a961d65a994e6bcdcf37e7399d0cc8ab5e" -dependencies = [ - "digest", -] - [[package]] name = "http" version = "1.5.0" @@ -732,15 +521,6 @@ dependencies = [ "serde_core", ] -[[package]] -name = "inout" -version = "0.1.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "879f10e63c20629ecabbb64a8010319738c66a5cd0c29b02d63d272b03751d01" -dependencies = [ - "generic-array", -] - [[package]] name = "itoa" version = "1.0.17" @@ -770,9 +550,6 @@ name = "lazy_static" version = "1.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "bbd2bcb4c963f2ddae06a2efc7e9f3591312473c50c6685e1f298068316e66fe" -dependencies = [ - "spin", -] [[package]] name = "leb128fmt" @@ -786,12 +563,6 @@ version = "0.2.183" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b5b646652bf6661599e1da8901b3b9522896f01e736bad5f723fe7a3a27f899d" -[[package]] -name = "libm" -version = "0.2.16" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b6d2cec3eae94f9f509c767b45932f1ada8350c4bdb85af2fcab4a3c14807981" - [[package]] name = "log" version = "0.4.29" @@ -854,41 +625,6 @@ dependencies = [ "windows-sys 0.61.2", ] -[[package]] -name = "num-bigint-dig" -version = "0.8.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e661dda6640fad38e827a6d4a310ff4763082116fe217f279885c97f511bb0b7" -dependencies = [ - "lazy_static", - "libm", - "num-integer", - "num-iter", - "num-traits", - "rand 0.8.8", - "smallvec", - "zeroize", -] - -[[package]] -name = "num-integer" -version = "0.1.47" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7ce2d95d4b3734dc35aa2f45e1aa22cd416814592a4f9d9205e11affd5b8e10b" -dependencies = [ - "num-traits", -] - -[[package]] -name = "num-iter" -version = "0.1.46" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c92800bd69a1eac91786bcfe9da64a897eb72911b8dc3095decbd07429e8048b" -dependencies = [ - "num-integer", - "num-traits", -] - [[package]] name = "num-traits" version = "0.2.19" @@ -896,7 +632,6 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841" dependencies = [ "autocfg", - "libm", ] [[package]] @@ -917,42 +652,6 @@ version = "1.21.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "42f5e15c9953c5e4ccceeb2e7382a716482c34515315f7b03532b8b4e8393d2d" -[[package]] -name = "opaque-debug" -version = "0.3.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c08d65885ee38876c4f86fa503fb49d7b507c2b62552df7c70b2fce627e06381" - -[[package]] -name = "p256" -version = "0.13.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c9863ad85fa8f4460f9c48cb909d38a0d689dba1f6f6988a5e3e0d31071bcd4b" -dependencies = [ - "ecdsa", - "elliptic-curve", - "primeorder", - "sha2", -] - -[[package]] -name = "p384" -version = "0.13.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fe42f1670a52a47d448f14b6a5c61dd78fce51856e68edaa38f7ae3a46b8d6b6" -dependencies = [ - "ecdsa", - "elliptic-curve", - "primeorder", - "sha2", -] - -[[package]] -name = "paste" -version = "1.0.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "57c0d7b74b563b49d38dae00a0c37d4d6de9b432382b2892f0574ddcae73fd0a" - [[package]] name = "pcap-file" version = "2.0.0" @@ -964,15 +663,6 @@ dependencies = [ "thiserror 1.0.69", ] -[[package]] -name = "pem-rfc7468" -version = "0.7.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "88b39c9bfcfc231068454382784bb460aae594343fb030d46e9f50a645418412" -dependencies = [ - "base64ct", -] - [[package]] name = "percent-encoding" version = "2.3.2" @@ -985,61 +675,6 @@ version = "0.2.17" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" -[[package]] -name = "pkcs1" -version = "0.7.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c8ffb9f10fa047879315e6625af03c164b16962a5368d724ed16323b68ace47f" -dependencies = [ - "der", - "pkcs8", - "spki", -] - -[[package]] -name = "pkcs5" -version = "0.7.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e847e2c91a18bfa887dd028ec33f2fe6f25db77db3619024764914affe8b69a6" -dependencies = [ - "der", - "spki", -] - -[[package]] -name = "pkcs8" -version = "0.10.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f950b2377845cebe5cf8b5165cb3cc1a5e0fa5cfa3e1f7f55707d8fd82e0a7b7" -dependencies = [ - "der", - "pkcs5", - "spki", -] - -[[package]] -name = "poly1305" -version = "0.8.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8159bd90725d2df49889a078b54f4f79e87f1f8a8444194cdca81d38f5393abf" -dependencies = [ - "cpufeatures", - "opaque-debug", - "universal-hash", -] - -[[package]] -name = "polyval" -version = "0.6.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9d1fe60d06143b2430aa532c94cfe9e29783047f06c0d7fd359a9a51b729fa25" -dependencies = [ - "cfg-if", - "cpufeatures", - "opaque-debug", - "universal-hash", -] - [[package]] name = "ppv-lite86" version = "0.2.21" @@ -1059,15 +694,6 @@ dependencies = [ "syn 2.0.117", ] -[[package]] -name = "primeorder" -version = "0.13.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "353e1ca18966c16d9deb1c69278edbc5f194139612772bd9537af60ac231e1e6" -dependencies = [ - "elliptic-curve", -] - [[package]] name = "proc-macro2" version = "1.0.106" @@ -1104,34 +730,14 @@ version = "6.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" -[[package]] -name = "rand" -version = "0.8.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e058c7de0b26af77780c769414d6257830bb240f3c38477dbc2c16e5f54d6d4c" -dependencies = [ - "rand_chacha 0.3.1", - "rand_core 0.6.4", -] - [[package]] name = "rand" version = "0.9.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b9ef1d0d795eb7d84685bca4f72f3649f064e6641543d3a8c415898726a57b41" dependencies = [ - "rand_chacha 0.9.0", - "rand_core 0.9.5", -] - -[[package]] -name = "rand_chacha" -version = "0.3.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e6c10a63a0fa32252be49d21e7709d4d4baf8d231c2dbce1eaa8141b9b127d88" -dependencies = [ - "ppv-lite86", - "rand_core 0.6.4", + "rand_chacha", + "rand_core", ] [[package]] @@ -1141,16 +747,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d3022b5f1df60f26e1ffddd6c66e8aa15de382ae63b3a0c1bfc0e4d3e3f325cb" dependencies = [ "ppv-lite86", - "rand_core 0.9.5", -] - -[[package]] -name = "rand_core" -version = "0.6.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ec0be4795e2f6a28069bec0b5ff3e2ac9bafc99e6a9a7dc3547996c5c816922c" -dependencies = [ - "getrandom 0.2.17", + "rand_core", ] [[package]] @@ -1179,16 +776,6 @@ version = "0.8.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4" -[[package]] -name = "rfc6979" -version = "0.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f8dd2a808d456c4a54e300a23e9f5a67e122c3024119acbfd73e3bf664491cb2" -dependencies = [ - "hmac", - "subtle", -] - [[package]] name = "ring" version = "0.17.14" @@ -1203,27 +790,6 @@ dependencies = [ "windows-sys 0.52.0", ] -[[package]] -name = "rsa" -version = "0.9.10" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b8573f03f5883dcaebdfcf4725caa1ecb9c15b2ef50c43a07b816e06799bb12d" -dependencies = [ - "const-oid", - "digest", - "num-bigint-dig", - "num-integer", - "num-traits", - "pkcs1", - "pkcs8", - "rand_core 0.6.4", - "sha2", - "signature", - "spki", - "subtle", - "zeroize", -] - [[package]] name = "rustc-std-workspace-core" version = "1.0.1" @@ -1247,8 +813,9 @@ checksum = "0d41d731c7d2f962d1ccc364cec258de3c0e93b38c2fb3ba97ac74513048d634" dependencies = [ "log", "once_cell", + "ring", "rustls-pki-types", - "rustls-webpki 0.103.15", + "rustls-webpki", "subtle", "zeroize", ] @@ -1262,47 +829,6 @@ dependencies = [ "zeroize", ] -[[package]] -name = "rustls-rustcrypto" -version = "0.0.2-alpha" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f12052947763ab8515f753315357599e9b0b4dab3b8ba15f30f725fe6d025557" -dependencies = [ - "aead", - "aes-gcm", - "chacha20poly1305", - "crypto-common", - "der", - "digest", - "ecdsa", - "ed25519-dalek", - "hmac", - "p256", - "p384", - "paste", - "pkcs8", - "rand_core 0.6.4", - "rsa", - "rustls", - "rustls-pki-types", - "rustls-webpki 0.102.8", - "sec1", - "sha2", - "signature", - "x25519-dalek", -] - -[[package]] -name = "rustls-webpki" -version = "0.102.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "64ca1bc8749bd4cf37b5ce386cc146580777b4e8572c7b97baf22c83f444bee9" -dependencies = [ - "ring", - "rustls-pki-types", - "untrusted", -] - [[package]] name = "rustls-webpki" version = "0.103.15" @@ -1326,20 +852,6 @@ version = "1.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e1cf6437eb19a8f4a6cc0f7dca544973b0b78843adbfeb3683d1a94a0024a294" -[[package]] -name = "sec1" -version = "0.7.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d3e97a565f76233a6003f9f5c54be1d9c5bdfa3eccfb189469f11ec4901c47dc" -dependencies = [ - "base16ct", - "der", - "generic-array", - "pkcs8", - "subtle", - "zeroize", -] - [[package]] name = "semver" version = "1.0.27" @@ -1429,10 +941,6 @@ name = "signature" version = "2.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "77549399552de45a898a580c1b41d445bf730df867cc44e6c0233bbc4b8329de" -dependencies = [ - "digest", - "rand_core 0.6.4", -] [[package]] name = "simd-adler32" @@ -1446,22 +954,6 @@ version = "1.15.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "67b1b7a3b5fe4f1376887184045fcf45c69e92af734b7aaddc05fb777b6fbd03" -[[package]] -name = "spin" -version = "0.9.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3763264f6b73151db08c50ff20d7d8a0b8796e021cdea7ceedad07b80155fa0e" - -[[package]] -name = "spki" -version = "0.7.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d91ed6c858b01f942cd56b37a94b3e0a1798290327d1236e4d9cf4eaca44d29d" -dependencies = [ - "base64ct", - "der", -] - [[package]] name = "subtle" version = "2.6.1" @@ -1640,7 +1132,6 @@ dependencies = [ "gpt", "image", "libc", - "rustls-rustcrypto", "serde", "serde_json", "sha2", @@ -1891,7 +1382,7 @@ dependencies = [ name = "toyos-sched-sim" version = "0.1.0" dependencies = [ - "rand 0.9.5", + "rand", "toyos-sched", ] @@ -1960,7 +1451,7 @@ version = "0.1.0" name = "toyos-xhci-sim" version = "0.1.0" dependencies = [ - "rand 0.9.5", + "rand", "toyos-xhci", ] @@ -2037,16 +1528,6 @@ version = "0.2.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ebc1c04c71510c7f702b52b7c350734c9ff1295c464a03335b00bb84fc54f853" -[[package]] -name = "universal-hash" -version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fc1de2c688dc15305988b563c3854064043356019f97a4b46276fe734c4f07ea" -dependencies = [ - "crypto-common", - "subtle", -] - [[package]] name = "untrusted" version = "0.9.0" @@ -2460,17 +1941,6 @@ dependencies = [ "wasmparser", ] -[[package]] -name = "x25519-dalek" -version = "2.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c7e468321c81fb07fa7f4c636c3972b9100f0346e5b6a9f2bd0603a52f7ed277" -dependencies = [ - "curve25519-dalek", - "rand_core 0.6.4", - "zeroize", -] - [[package]] name = "zerocopy" version = "0.8.47" @@ -2496,20 +1966,6 @@ name = "zeroize" version = "1.9.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e13c156562582aa81c60cb29407084cdb54c4164760106ab78e6c5b0858cf64e" -dependencies = [ - "zeroize_derive", -] - -[[package]] -name = "zeroize_derive" -version = "1.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3c50655cbb0fe3fc43170059e702f1ce5e19b84cec58dc87b037a09935c2f328" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.117", -] [[package]] name = "zmij" diff --git a/Cargo.toml b/Cargo.toml index 0f5f5f55210..47c28f7779a 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -169,10 +169,10 @@ sha2 = "0.10" # own tar and gzip, where the binaries are hosts' tools. tar = { version = "0.4.46", default-features = false } flate2 = { version = "1", default-features = false, features = ["rust_backend"] } -# Every request the build system makes, GitHub's API and the crates.io index: -# rustls over RustCrypto, where curl is a host's tool and a TLS stack in C. -ureq = { version = "3", default-features = false, features = ["rustls-no-provider", "rustls-webpki-roots"] } -rustls-rustcrypto = "0.0.2-alpha" +# Every request the build system makes, GitHub's API and the crates.io index, +# where curl is a host's tool: rustls on ring, the tree's one TLS provider, +# whose C and assembly the host's `cc` compiles. +ureq = { version = "3", default-features = false, features = ["rustls"] } [dev-dependencies] # `USER_TOP`, so the harness judges a held `rsp` against the bound the kernel diff --git a/issues/build/the-build-runs-host-tools-outside-rust-and-qemu.md b/issues/build/the-build-runs-host-tools-outside-rust-and-qemu.md index 5d2d8d9723f..62052a02f2e 100644 --- a/issues/build/the-build-runs-host-tools-outside-rust-and-qemu.md +++ b/issues/build/the-build-runs-host-tools-outside-rust-and-qemu.md @@ -19,7 +19,7 @@ arrives and is not one. M4 and M5 are stages of `issues/build/toyos-builds-itsel | `sh` running LLVM's `config.guess`, and the POSIX tools and `cc` it runs | LLVM's CMake, whenever this host builds an LLVM, and the C++ runtime's, in every sysroot build, ask it the host's triple, unconditionally (`get_host_triple` in `rust/src/llvm-project/llvm/cmake/modules/GetHostTriple.cmake`, which runs `sh` by name) | refused: a Rust tool does the shell's part, brush 0.4.0: on the development host (macOS, arm64) `config.guess` printed `/bin/sh`'s triple under it, `arm64-apple-darwin27.0.0`, exit 0 each. The script runs `sed`, `uname`, `mktemp`, `grep`, `rm`, `rmdir` and `cc` there under either shell, and that `cc` is the `cc` rows'. Five of the other six are refused, uutils' doing each: under brush with sed 0.2.0, grep 0.2.0 and coreutils 0.12.0's `mktemp`, `rm` and `rmdir`, and nothing else on `PATH` but the host's `uname` and `cc`, it printed that triple, exit 0. `uname` is admitted: coreutils 0.12.0's answers `-p` with `unknown` where macOS's answers `arm`, and `config.guess` reads that as PowerPC, `powerpc-apple-darwin27.0.0`, exit 0 | CMake finds brush as its `sh`, uutils' `sed`, `grep`, `mktemp`, `rm` and `rmdir`, and a Rust `uname` that answers `-p` as the host's does; or M5 runs it in the guest | | `git` for worktrees, submodules, checkouts, fixtures and rustc's bootstrap | adds worktrees (`src/sysroot.rs`); updates submodules (`src/lib.rs`, `src/sysroot.rs`); fetches the fork from the primary's and checks it out (`src/sysroot.rs`); makes the tests' fixture repositories; runs inside rustc's bootstrap | admitted: no Rust tool does the job, gitoxide 0.85 adds, removes and prunes no worktree, updates no submodule, stages, resets and pushes nothing, checks out only a fresh clone and fetches a local path by spawning `git`; a fixture must be what `git` makes, and bootstrap runs `git` itself | M4 runs it in the guest | | `git` for reads, a config write, a commit's paths written out, and clones and fetches over HTTPS | `rev-parse`, `merge-base`, `ls-tree`, `ls-files`, `cat-file`, `config --get-regexp`, `worktree list`, `status`, `diff`, `ls-remote` and `grep`, in the build system and its tests; `config --global --add safe.directory` in CI's containers; `checkout -- ` through an index of its own, which writes the C++ runtime's sources out of the LLVM commit into the stored LLVM (`src/llvm.rs`); every workflow's checkout | refused: a Rust tool does it, gitoxide 0.85, which reads refs, objects, the index, config, worktrees and status, adds a value to a config file and writes it (gix-config 0.58's `File::section_mut_or_create_new`, `SectionMut::push`, `File::write_to`), walks history, diffs, and lists, fetches and clones a remote over HTTPS; `grep` is a search of the files its index names; and gitoxide's CLI 0.59 (gix 0.88) wrote the runtimes' sources of LLVM `849da7d6` into an empty directory, each path's tree through `gix rev parse`, `gix index from-tree` and `gix free index checkout-exclusive`, exit 0 each: the 18759 files `git` writes there, byte for byte and mode for mode | those are gitoxide's | -| `cc`, `c++` and `ar` on a Linux host, `build-essential` in CI's containers | rustc links every host binary through `cc`; `cc` and `c++` compile LLVM, clang, LLD and `rustc_llvm` (`src/llvm.rs` names both to bootstrap); `ar` archives what `cc::Build` compiles | admitted: no Rust tool compiles C or C++, or takes rustc's host link | M5: no host in the loop | +| `cc`, `c++` and `ar` on a Linux host, `build-essential` in CI's containers | rustc links every host binary through `cc`; `cc` and `c++` compile LLVM, clang, LLD and `rustc_llvm` (`src/llvm.rs` names both to bootstrap); `cc` compiles ring's C and assembly in every build of the build system itself, whose HTTP agent is rustls on ring (`src/release.rs`); `ar` archives what `cc::Build` compiles, ring's included | admitted: no Rust tool compiles C or C++, or takes rustc's host link; ring is the tree's one TLS provider by the owner's ruling, over a provider in Rust alone | M5: no host in the loop | | the toolchain's own `clang`, `llvm-ar`, `rust-lld` and `llvm-config`, built from `ToyOSOrg/llvm-project` | rustc links every guest binary with `rust-lld`; `clang` compiles the C corpus (`tests/common/compile.rs`) and, with `llvm-ar`, doomgeneric through `cc::Build` (`src/clang.rs`); rustc's bootstrap asks `llvm-config` how to link LLVM | admitted: our fork's C++, which ToyOS can one day build and run; no Rust tool compiles C, `cc::Build` archives with an `ar`, bootstrap reads LLVM through `llvm-config`, and `CLAUDE.md` links everything with `rust-lld` | M5: no host in the loop | | `ovmf-generic`, `qemu-efi-aarch64` | the x86-64 and AArch64 UEFI firmware of CI's guest containers (`src/firmware.rs`), packaged by Debian apart from QEMU | admitted: QEMU's own firmware, and no Rust firmware does its job | the instrument's QEMU carries its own firmware | | `ca-certificates` | the trust store `git` and `curl` verify against in CI's containers | admitted: data both of them need | goes when neither runs there | diff --git a/src/release.rs b/src/release.rs index d87ba4d3d0c..257465f60bf 100644 --- a/src/release.rs +++ b/src/release.rs @@ -19,7 +19,6 @@ use std::fs; use std::io::Write; use std::path::{Path, PathBuf}; use std::process::Command; -use std::sync::Arc; use serde_json::Value; use sha2::{Digest, Sha256}; @@ -80,16 +79,11 @@ fn on_runner() -> bool { std::env::var("GITHUB_ACTIONS").is_ok_and(|v| v == "true") } -/// The HTTP client of every request the build system makes: rustls with -/// RustCrypto's primitives and webpki's roots, so no C; a status is an answer, -/// not an error. +/// The HTTP client of every request the build system makes: rustls on ring +/// with webpki's roots, as ureq configures it; a status is an answer, not an +/// error. pub(crate) fn agent() -> ureq::Agent { - let tls = ureq::tls::TlsConfig::builder() - .provider(ureq::tls::TlsProvider::Rustls) - .root_certs(ureq::tls::RootCerts::WebPki) - .unversioned_rustls_crypto_provider(Arc::new(rustls_rustcrypto::provider())) - .build(); - ureq::Agent::config_builder().tls_config(tls).user_agent(USER_AGENT).http_status_as_error(false).build().new_agent() + ureq::Agent::config_builder().user_agent(USER_AGENT).http_status_as_error(false).build().new_agent() } /// Whether this job is main's publisher — [`PUBLISHER`] on main, scheduled or From 585e6ea92986226c85d03e4976b85e7da46f6d6c Mon Sep 17 00:00:00 2001 From: japabu Date: Fri, 2 Oct 2026 14:14:56 +0200 Subject: [PATCH 23/30] A landing on main during a nightly is not that nightly's failure Main's nightly 36985427800, at 74a2e703b, ended its three-hour toolchain step red: "crates.io holds no toyos-abi of this tree, so no sdk alias can name it". #650 and #659 had landed meanwhile and their pushes had put newer toyos-abi versions up, so the tree the nightly checked out was no longer the one crates.io's newest named. This branch deleted that step and kept the red: `release_as` ran behind `ci::at_tip`, which refuses with "HEAD ... is not main's tip" whenever a landing precedes the `release` job, and `alias` kept the crates.io refusal for a landing whose crates went up after that check. `sdk_at_tip` is now the release's one decision, taken before anything is laid out, packed or put up: it reads crates.io, then main's tip. A tree main has moved past puts nothing up and the job is green, saying so; the tip's nightly publishes. At the tip the plan it read is the one the alias is written from, so nothing read later can disagree with it. crates.io before the tip is the order that matters: a landing whose crates the first read shows has moved the tip the second read sees, and the other order leaves a landing between the two reads refused. What stays refused is the tip's own crates not being up, which publish.yml owes, and a remote that names no main. A run of an older tree still moves no alias back, which is what `at_tip` was put there for; `at_tip` is `publish`'s alone again, and private as on main. `a_landing_during_the_nightly_puts_nothing_up_and_is_no_failure` lands once before the release's first read, between its two reads and not at all, with and without newer SDK crates. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_013UDZQ6fSKw14e4w2TKTRfm --- src/ci.rs | 2 +- src/release.rs | 87 ++++++++++++++++++++++++++++++++++++++++++-------- 2 files changed, 74 insertions(+), 15 deletions(-) diff --git a/src/ci.rs b/src/ci.rs index 66960eaf279..c1f90245836 100644 --- a/src/ci.rs +++ b/src/ci.rs @@ -933,7 +933,7 @@ fn publish(root: &Path) -> Result { /// Whether `HEAD` is `main`'s tip as `git ls-remote` printed it: a re-run of an /// older push would put older code up under a newer minor. -pub(crate) fn at_tip(ls_remote: &str, head: &str) -> Result<(), String> { +fn at_tip(ls_remote: &str, head: &str) -> Result<(), String> { match ls_remote.split_whitespace().next() { Some(tip) if tip == head => Ok(()), tip => Err(format!( diff --git a/src/release.rs b/src/release.rs index 257465f60bf..81977b1c9a0 100644 --- a/src/release.rs +++ b/src/release.rs @@ -10,8 +10,9 @@ //! //! **A guest job installs the sysroot its restore step put down** ([`install`]), //! and main's nightly packs that store into the release a consumer outside CI -//! installs, then moves the SDK alias onto it ([`release`]); run anywhere else, -//! that job is refused before it reads anything. +//! installs, then moves the SDK alias onto it ([`release`]). Run as any other +//! job, that is refused before it reads anything; run on a tree main has moved +//! past, it puts nothing up. //! //! A dev host installs none: its build system builds its own from `rust/`. @@ -26,6 +27,7 @@ use toyos_tmpdir::TempDir; use crate::buildlock::Keyed; use crate::keystore::Key; +use crate::sdkversion::Release; const ASSET: &str = "toyos-toolchain.tar.gz"; @@ -303,7 +305,7 @@ fn run(cmd: &mut Command) -> Result<(), String> { /// `cargo run -- --ci release`: the sysroot main's nightly restored, put up as /// the release a consumer outside CI installs, and the `sdk-` alias /// moved onto it. Refused before anything is read unless this job is main's -/// publisher, and before anything is put up unless it runs at main's tip. +/// publisher; a tree main has moved past puts nothing up ([`sdk_at_tip`]). pub fn release(root: &Path) -> Result { let var = |name| std::env::var(name).ok(); let repo = var("GITHUB_REPOSITORY").ok_or("GITHUB_REPOSITORY is unset: only a runner publishes a toolchain")?; @@ -317,8 +319,11 @@ fn release_as(root: &Path, repo: &str, workflow: Option<&str>, event: Option<&st if !(cfg!(target_os = "linux") && crate::arch::Arch::HOST == Some(crate::arch::Arch::X86_64)) { return Err(format!("a release is {HOST}'s and this host is not one; a tarball packed here would install nowhere")); } - let tip = crate::sysroot::git_out(root, &["ls-remote", "origin", "refs/heads/main"]); - crate::ci::at_tip(&tip, crate::sysroot::git_out(root, &["rev-parse", "HEAD"]).trim())?; + let head = crate::sysroot::git_out(root, &["rev-parse", "HEAD"]); + let tip = || crate::sysroot::git_out(root, &["ls-remote", "origin", "refs/heads/main"]); + let Some(sdk) = sdk_at_tip(|| crate::sdkversion::plan(root), tip, head.trim())? else { + return Ok(format!("main has moved past {}, and its tip's nightly is the one that publishes", head.trim())); + }; let key = lay_out(root)?; let rust_dir = root.join("rust"); let need = shipped_glibc(&crate::toolchain::stage2(&rust_dir))?; @@ -336,7 +341,30 @@ fn release_as(root: &Path, repo: &str, workflow: Option<&str>, event: Option<&st let notes = notes(root, repo, &tag, &manifest(&tag))?; let github = Github::new(repo)?; let put = put_up(&github, root, &tag, ¬es, &tarball)?; - Ok(format!("{put}; {}", alias(&github, root, &tag, ¬es, &tmp)?)) + Ok(format!("{put}; {}", alias(&github, root, &sdk, &tag, ¬es, &tmp)?)) +} + +/// The SDK crates as crates.io holds them, where `head` is main's tip as +/// `ls_remote` prints it, and `None` where main has moved past `head`: a +/// landing during a nightly is not its failure, and a run of an older tree +/// moves no alias back. crates.io is read before the tip, so a landing whose +/// crates that read shows is one the tip shows too. Refused where crates.io's +/// newest is not the tip's own, which `publish.yml` owes. +fn sdk_at_tip( + plan: impl FnOnce() -> Result, String>, + ls_remote: impl FnOnce() -> String, + head: &str, +) -> Result>, String> { + let sdk = plan()?; + let said = ls_remote(); + let tip = said.split_whitespace().next().ok_or("origin names no main")?; + if tip != head { + return Ok(None); + } + match sdk.iter().find(|r| r.publish) { + Some(owed) => Err(format!("crates.io holds no {} of this tree, main's tip, so no sdk alias can name it", owed.krate.name)), + None => Ok(Some(sdk)), + } } /// The tarball of the toolchain laid out under `build` ([`lay_out`]) at @@ -595,17 +623,12 @@ Until [rust-windowing/raw-window-handle#223](https://github.com/rust-windowing/r /// the name a consumer pins, moved onto `tag`. A second release carrying only a /// `TOOLCHAIN` naming `tag`, the commit that put it up and the SDK crates, /// because GitHub hangs an asset off one release id. -fn alias(github: &Github, root: &Path, tag: &str, notes: &str, tmp: &Path) -> Result { - let plan = crate::sdkversion::plan(root)?; - if let Some(owed) = plan.iter().find(|r| r.publish) { - let name = owed.krate.name; - return Err(format!("crates.io holds no {name} of this tree, so no sdk alias can name it")); - } - let abi = plan.iter().find(|r| r.krate.name == "toyos-abi").ok_or("toyos-abi is not published")?; +fn alias(github: &Github, root: &Path, sdk: &[Release], tag: &str, notes: &str, tmp: &Path) -> Result { + let abi = sdk.iter().find(|r| r.krate.name == "toyos-abi").ok_or("toyos-abi is not published")?; let abi = abi.version.split('+').next().unwrap_or(&abi.version); let alias = format!("toolchain-linux-x86_64-sdk-{abi}"); let commit = |rev: &str| crate::sysroot::git_out(root, &["rev-parse", rev]).trim().to_string(); - let sdk: String = plan.iter().map(|r| format!("{} {}\n", r.krate.name, r.version)).collect(); + let sdk: String = sdk.iter().map(|r| format!("{} {}\n", r.krate.name, r.version)).collect(); let toolchain = tmp.join("TOOLCHAIN"); let text = format!("{}toyos {}\nrust {}\n{sdk}", manifest(tag), commit("HEAD"), commit("HEAD:rust")); fs::write(&toolchain, text).map_err(|e| format!("{}: {e}", toolchain.display()))?; @@ -654,6 +677,42 @@ mod tests { assert!(release_as(root, REPO, None, None).unwrap_err().starts_with("only ")); } + /// crates.io's newest `toyos-abi` as a plan reads it: this tree's, or owed. + fn sdk(owed: bool) -> Vec { + let krate = &crate::sdkversion::PUBLISHED[0]; + vec![Release { krate, key: String::new(), version: "0.28.0+k".into(), publish: owed, manifest: String::new() }] + } + + /// **A landing on main during a nightly is not that nightly's failure**: + /// whether it comes before the release reads anything, between its read of + /// crates.io and its read of main's tip, or not at all, and whether or not + /// it put newer SDK crates up, the release is the tip's or nothing is put + /// up, and neither is refused. What is refused is the tip's own crates not + /// being up, and a remote that names no main. + #[test] + fn a_landing_during_the_nightly_puts_nothing_up_and_is_no_failure() { + const HEAD: &str = "0123456789abcdef0123456789abcdef01234567"; + const LANDED: &str = "fedcba9876543210fedcba9876543210fedcba98"; + let main = |tip: &str| format!("{tip}\trefs/heads/main\n"); + // The release of `HEAD` on a main that lands once, after `after` of + // the release's reads: whether it publishes. + let publishes = |after: usize, moves_sdk: bool| { + let reads = std::cell::Cell::new(0); + let landed = || reads.replace(reads.get() + 1) >= after; + let read = sdk_at_tip(|| Ok(sdk(landed() && moves_sdk)), || main(if landed() { LANDED } else { HEAD }), HEAD); + read.map(|sdk| sdk.is_some()) + }; + for moves_sdk in [false, true] { + assert_eq!(publishes(0, moves_sdk), Ok(false), "a landing before the release read anything"); + assert_eq!(publishes(1, moves_sdk), Ok(false), "a landing between the release's two reads"); + assert_eq!(publishes(2, moves_sdk), Ok(true), "no landing"); + } + let owed = sdk_at_tip(|| Ok(sdk(true)), || main(HEAD), HEAD).err().expect("the tip's crates are not up"); + assert!(owed.contains("toyos-abi") && owed.contains("main's tip"), "{owed}"); + let unnamed = sdk_at_tip(|| Ok(sdk(false)), String::new, HEAD).err().expect("no main"); + assert!(unnamed.contains("names no main"), "{unnamed}"); + } + fn release_json(assets: Value) -> Value { serde_json::json!({ "id": 7, "upload_url": "https://uploads.github.com/repos/o/r/releases/7/assets{?name,label}", "assets": assets }) } From 3f59d1c2f78ec42ef87cfb58229e50df4e83c7d8 Mon Sep 17 00:00:00 2001 From: japabu Date: Fri, 2 Oct 2026 14:14:56 +0200 Subject: [PATCH 24/30] Two kernel issues no longer call the KVM guests the nightly's The nightly's one guest lane is TCG; the EPYC KVM guests are every pull request's and the merge queue's. The branch had dropped the same words from two other issues. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_013UDZQ6fSKw14e4w2TKTRfm --- issues/kernel/the-kernel-loads-no-cpu-microcode.md | 2 +- .../kernel/toyos-uses-what-modern-hardware-offers-for-speed.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/issues/kernel/the-kernel-loads-no-cpu-microcode.md b/issues/kernel/the-kernel-loads-no-cpu-microcode.md index 3003800d615..5e7fdb143d4 100644 --- a/issues/kernel/the-kernel-loads-no-cpu-microcode.md +++ b/issues/kernel/the-kernel-loads-no-cpu-microcode.md @@ -50,7 +50,7 @@ pages (Linux `amd.c`, `__apply_microcode_amd`). On families 0x17, 0x19 and part of 0x1a below a per-CPU cutoff revision the CPU's own signature check is broken (EntrySign; `cpu_has_entrysign`, `need_sha_check`), so the hash pin is the only check, as `amd_shas.c` is Linux's. linux-firmware's `LICENSE.amd-ucode` is -unread. ToyOS has no AMD metal: the nightly's EPYCs are KVM guests, which load +unread. ToyOS has no AMD metal: CI's EPYCs are KVM guests, which load nothing. **Licence.** `LicenseRef-Intel-Microcode` is in no `ALLOWED` row of diff --git a/issues/kernel/toyos-uses-what-modern-hardware-offers-for-speed.md b/issues/kernel/toyos-uses-what-modern-hardware-offers-for-speed.md index f6b97784fe0..e0994691323 100644 --- a/issues/kernel/toyos-uses-what-modern-hardware-offers-for-speed.md +++ b/issues/kernel/toyos-uses-what-modern-hardware-offers-for-speed.md @@ -8,7 +8,7 @@ opened: 2026-09-29 ToyOS takes each hardware feature that makes it faster on every CPU that enumerates it; a CPU without one runs the plain path, or is refused by name -where the stage says so. The proving machines are the T14 and the nightly's +where the stage says so. The proving machines are the T14 and AMD EPYC KVM guests, and a stage's correctness test runs on each that has its feature. A figure is the T14's, since the EPYC guests are shared CI runners: the median of 11 runs with its spread, taken by the program of From 482d4873f444b274bc5709a037259af9a508f84f Mon Sep 17 00:00:00 2001 From: japabu Date: Fri, 2 Oct 2026 14:15:47 +0200 Subject: [PATCH 25/30] The landing test takes the landing that moves the SDK first With the tip read before crates.io, the test's first red was the landing that moves no SDK, which that order publishes over rather than refuses. The landing that puts newer crates up is the race, so it is asserted first and the mutation's red is the refusal itself. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_013UDZQ6fSKw14e4w2TKTRfm --- src/release.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/release.rs b/src/release.rs index 81977b1c9a0..2ba36b343ec 100644 --- a/src/release.rs +++ b/src/release.rs @@ -702,7 +702,7 @@ mod tests { let read = sdk_at_tip(|| Ok(sdk(landed() && moves_sdk)), || main(if landed() { LANDED } else { HEAD }), HEAD); read.map(|sdk| sdk.is_some()) }; - for moves_sdk in [false, true] { + for moves_sdk in [true, false] { assert_eq!(publishes(0, moves_sdk), Ok(false), "a landing before the release read anything"); assert_eq!(publishes(1, moves_sdk), Ok(false), "a landing between the release's two reads"); assert_eq!(publishes(2, moves_sdk), Ok(true), "no landing"); From 68dc21ce255766e5e0739bfbd1603b6aeebf7d7e Mon Sep 17 00:00:00 2001 From: japabu Date: Fri, 2 Oct 2026 14:27:59 +0200 Subject: [PATCH 26/30] ring is the provider the tree takes, not yet its only one doom's build script and tests/toyos-rust-tests still name rustls-rustcrypto, as main's internet-clients track records, so "the tree's one TLS provider" was false of the tree. The manifest's comment and the `cc` row say what the owner ruled, and the comment names the track that holds it. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_013UDZQ6fSKw14e4w2TKTRfm --- Cargo.toml | 5 +++-- .../build/the-build-runs-host-tools-outside-rust-and-qemu.md | 2 +- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/Cargo.toml b/Cargo.toml index 47c28f7779a..0b1b3ce1363 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -170,8 +170,9 @@ sha2 = "0.10" tar = { version = "0.4.46", default-features = false } flate2 = { version = "1", default-features = false, features = ["rust_backend"] } # Every request the build system makes, GitHub's API and the crates.io index, -# where curl is a host's tool: rustls on ring, the tree's one TLS provider, -# whose C and assembly the host's `cc` compiles. +# where curl is a host's tool: rustls on ring, the one TLS provider the tree +# takes (`issues/design-debt/the-internet-clients-work-unchanged.md`), whose C +# and assembly the host's `cc` compiles. ureq = { version = "3", default-features = false, features = ["rustls"] } [dev-dependencies] diff --git a/issues/build/the-build-runs-host-tools-outside-rust-and-qemu.md b/issues/build/the-build-runs-host-tools-outside-rust-and-qemu.md index 62052a02f2e..70e7f15513b 100644 --- a/issues/build/the-build-runs-host-tools-outside-rust-and-qemu.md +++ b/issues/build/the-build-runs-host-tools-outside-rust-and-qemu.md @@ -19,7 +19,7 @@ arrives and is not one. M4 and M5 are stages of `issues/build/toyos-builds-itsel | `sh` running LLVM's `config.guess`, and the POSIX tools and `cc` it runs | LLVM's CMake, whenever this host builds an LLVM, and the C++ runtime's, in every sysroot build, ask it the host's triple, unconditionally (`get_host_triple` in `rust/src/llvm-project/llvm/cmake/modules/GetHostTriple.cmake`, which runs `sh` by name) | refused: a Rust tool does the shell's part, brush 0.4.0: on the development host (macOS, arm64) `config.guess` printed `/bin/sh`'s triple under it, `arm64-apple-darwin27.0.0`, exit 0 each. The script runs `sed`, `uname`, `mktemp`, `grep`, `rm`, `rmdir` and `cc` there under either shell, and that `cc` is the `cc` rows'. Five of the other six are refused, uutils' doing each: under brush with sed 0.2.0, grep 0.2.0 and coreutils 0.12.0's `mktemp`, `rm` and `rmdir`, and nothing else on `PATH` but the host's `uname` and `cc`, it printed that triple, exit 0. `uname` is admitted: coreutils 0.12.0's answers `-p` with `unknown` where macOS's answers `arm`, and `config.guess` reads that as PowerPC, `powerpc-apple-darwin27.0.0`, exit 0 | CMake finds brush as its `sh`, uutils' `sed`, `grep`, `mktemp`, `rm` and `rmdir`, and a Rust `uname` that answers `-p` as the host's does; or M5 runs it in the guest | | `git` for worktrees, submodules, checkouts, fixtures and rustc's bootstrap | adds worktrees (`src/sysroot.rs`); updates submodules (`src/lib.rs`, `src/sysroot.rs`); fetches the fork from the primary's and checks it out (`src/sysroot.rs`); makes the tests' fixture repositories; runs inside rustc's bootstrap | admitted: no Rust tool does the job, gitoxide 0.85 adds, removes and prunes no worktree, updates no submodule, stages, resets and pushes nothing, checks out only a fresh clone and fetches a local path by spawning `git`; a fixture must be what `git` makes, and bootstrap runs `git` itself | M4 runs it in the guest | | `git` for reads, a config write, a commit's paths written out, and clones and fetches over HTTPS | `rev-parse`, `merge-base`, `ls-tree`, `ls-files`, `cat-file`, `config --get-regexp`, `worktree list`, `status`, `diff`, `ls-remote` and `grep`, in the build system and its tests; `config --global --add safe.directory` in CI's containers; `checkout -- ` through an index of its own, which writes the C++ runtime's sources out of the LLVM commit into the stored LLVM (`src/llvm.rs`); every workflow's checkout | refused: a Rust tool does it, gitoxide 0.85, which reads refs, objects, the index, config, worktrees and status, adds a value to a config file and writes it (gix-config 0.58's `File::section_mut_or_create_new`, `SectionMut::push`, `File::write_to`), walks history, diffs, and lists, fetches and clones a remote over HTTPS; `grep` is a search of the files its index names; and gitoxide's CLI 0.59 (gix 0.88) wrote the runtimes' sources of LLVM `849da7d6` into an empty directory, each path's tree through `gix rev parse`, `gix index from-tree` and `gix free index checkout-exclusive`, exit 0 each: the 18759 files `git` writes there, byte for byte and mode for mode | those are gitoxide's | -| `cc`, `c++` and `ar` on a Linux host, `build-essential` in CI's containers | rustc links every host binary through `cc`; `cc` and `c++` compile LLVM, clang, LLD and `rustc_llvm` (`src/llvm.rs` names both to bootstrap); `cc` compiles ring's C and assembly in every build of the build system itself, whose HTTP agent is rustls on ring (`src/release.rs`); `ar` archives what `cc::Build` compiles, ring's included | admitted: no Rust tool compiles C or C++, or takes rustc's host link; ring is the tree's one TLS provider by the owner's ruling, over a provider in Rust alone | M5: no host in the loop | +| `cc`, `c++` and `ar` on a Linux host, `build-essential` in CI's containers | rustc links every host binary through `cc`; `cc` and `c++` compile LLVM, clang, LLD and `rustc_llvm` (`src/llvm.rs` names both to bootstrap); `cc` compiles ring's C and assembly in every build of the build system itself, whose HTTP agent is rustls on ring (`src/release.rs`); `ar` archives what `cc::Build` compiles, ring's included | admitted: no Rust tool compiles C or C++, or takes rustc's host link; ring is the one TLS provider the tree takes (owner, 2026-10-02), over a provider in Rust alone | M5: no host in the loop | | the toolchain's own `clang`, `llvm-ar`, `rust-lld` and `llvm-config`, built from `ToyOSOrg/llvm-project` | rustc links every guest binary with `rust-lld`; `clang` compiles the C corpus (`tests/common/compile.rs`) and, with `llvm-ar`, doomgeneric through `cc::Build` (`src/clang.rs`); rustc's bootstrap asks `llvm-config` how to link LLVM | admitted: our fork's C++, which ToyOS can one day build and run; no Rust tool compiles C, `cc::Build` archives with an `ar`, bootstrap reads LLVM through `llvm-config`, and `CLAUDE.md` links everything with `rust-lld` | M5: no host in the loop | | `ovmf-generic`, `qemu-efi-aarch64` | the x86-64 and AArch64 UEFI firmware of CI's guest containers (`src/firmware.rs`), packaged by Debian apart from QEMU | admitted: QEMU's own firmware, and no Rust firmware does its job | the instrument's QEMU carries its own firmware | | `ca-certificates` | the trust store `git` and `curl` verify against in CI's containers | admitted: data both of them need | goes when neither runs there | From bf9983ad10944d51da901c8250a760cdd085fdaa Mon Sep 17 00:00:00 2001 From: japabu Date: Fri, 2 Oct 2026 15:38:22 +0200 Subject: [PATCH 27/30] The release decision's two weaknesses are filed Round 6's review found them recorded only under the pull request body's "Unsure": a nightly whose `release` decides before `publish.yml` has put its own tip's crates up is red for no defect of the tree, and a nightly a landing overtakes puts no release up and moves no alias, green. Each is an issue with its owner, the runs that measure its window and an exit a test or a request can fail. Read for them on 2026-10-02: runs 36843762360, 36988155706 and 36985427800 (`gh run view`), main's publish runs (`gh run list --workflow publish.yml`), main's first-parent log, and the `toolchain / build` jobs of run 36988764929 attempt 2 and run 36934214557. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_013UDZQ6fSKw14e4w2TKTRfm --- ...akes-leaves-no-release-and-no-sdk-alias.md | 33 +++++++++++++++++++ ...its-tips-crates-are-up-reds-the-nightly.md | 32 ++++++++++++++++++ 2 files changed, 65 insertions(+) create mode 100644 issues/build/a-nightly-a-landing-overtakes-leaves-no-release-and-no-sdk-alias.md create mode 100644 issues/build/a-release-that-decides-before-its-tips-crates-are-up-reds-the-nightly.md diff --git a/issues/build/a-nightly-a-landing-overtakes-leaves-no-release-and-no-sdk-alias.md b/issues/build/a-nightly-a-landing-overtakes-leaves-no-release-and-no-sdk-alias.md new file mode 100644 index 00000000000..4aadb23c802 --- /dev/null +++ b/issues/build/a-nightly-a-landing-overtakes-leaves-no-release-and-no-sdk-alias.md @@ -0,0 +1,33 @@ +--- +status: open +kind: tooling +opened: 2026-10-02 +--- + +# A nightly a landing overtakes leaves no release and no SDK alias + +The nightly's `release` puts nothing up where main's tip is no longer its +HEAD (`release::sdk_at_tip`), and is green: "main has moved past ``, and +its tip's nightly is the one that publishes". The toolchain release and the +`sdk-` alias then wait for a nightly whose `release` runs at the tip, +or a dispatch there. No landing runs one, the job's conclusion is success +either way, and a nightly overtaken every night publishes nothing until one is +not. + +The window runs from the nightly's creation to its `release`'s decision: the +wait in `nightly-`'s concurrency group, then the `toolchain` job, 2:38 at +its fastest measured (run 36988764929, attempt 2) and 2:11:46 cold (run +36934214557). `cron: '0 3 * * *'` created its scheduled runs at 09:35:54Z on +2026-10-01 (36843762360) and 09:09:27Z on 2026-10-02 (36988155706), and main +took seven landings on 2026-10-02, pushed between 08:39:27Z and 11:20:48Z. +Run 36988155706, at `46af79d5d`, waited behind a dispatched nightly +(36985427800) and started its jobs at 11:45:51Z, four landings after its HEAD +(#643, #664, #679, #659). The release this decision replaced read no tip: a +landing that moved no SDK crate did not stop it. + +Owner: the release module (`src/release.rs`) and `nightly.yml`'s `release`. + +**Exit**: a landing during a nightly does not leave main's tip without its +toolchain release and SDK alias: after a day on which every nightly was +overtaken, `releases/tags/toolchain-linux-x86_64-` +answers 200 and the tip's `sdk-` alias names it. diff --git a/issues/build/a-release-that-decides-before-its-tips-crates-are-up-reds-the-nightly.md b/issues/build/a-release-that-decides-before-its-tips-crates-are-up-reds-the-nightly.md new file mode 100644 index 00000000000..0a74b5c0094 --- /dev/null +++ b/issues/build/a-release-that-decides-before-its-tips-crates-are-up-reds-the-nightly.md @@ -0,0 +1,32 @@ +--- +status: open +kind: tooling +opened: 2026-10-02 +--- + +# A release that decides before its tip's crates are up reds the nightly + +The nightly's `release` puts a toolchain up only where crates.io's newest SDK +crates are the tree's own (`release::sdk_at_tip`). Where HEAD is main's tip +and they are not, it refuses: "crates.io holds no `` of this tree, +main's tip, so no sdk alias can name it". That refusal does not tell a tip +`publish.yml` failed to publish from one whose `publish` run has not finished, +and on the second the nightly is red for no defect of the tree. Nothing is put +up, and a re-run of that one job after the publish recovers it. + +The decision follows the nightly's `toolchain` job, 2:38 at its fastest +measured (run 36988764929, attempt 2), and the `release` job's checkout, +restore and driver build. Main's six green `publish` runs of 2026-10-02 took +between 1:04 and 1:54 from creation (36985281365, 36986108744, 36986326239, +36991892914, 36995618637, 37000495781). `publish` runs one at a time +(`publish.yml`'s `concurrency`), and after each crate it reads the index up to +60 times, 5 s apart (`ci::publish`). A publish that queues or waits longer +than that margin, behind a landing made just before the nightly, is read as a +missing one. + +Owner: the release module (`src/release.rs`). + +**Exit**: a `release` at main's tip is red for crates crates.io does not hold +only once that tip's `publish` run has concluded: a test stages a tip whose +crates come up after the release's first read of crates.io, and sees the +release put up. From e99547ae63d2cf5827405beb4f756efcb69ad80a Mon Sep 17 00:00:00 2001 From: japabu Date: Fri, 2 Oct 2026 15:38:23 +0200 Subject: [PATCH 28/30] Two more issues no longer call the EPYC guests the nightly's The nightly's one guest lane is TCG; the EPYC KVM guests are every pull request's and the merge queue's. Four lines in two issues still said "nightly EPYC guest" after the branch dropped the words from four others. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_013UDZQ6fSKw14e4w2TKTRfm --- .../features-no-proving-machine-is-known-to-offer.md | 6 +++--- ...-decides-a-cpus-speculation-mitigations-as-linux-does.md | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/issues/hardware/features-no-proving-machine-is-known-to-offer.md b/issues/hardware/features-no-proving-machine-is-known-to-offer.md index c771b54bc67..1709affbdb9 100644 --- a/issues/hardware/features-no-proving-machine-is-known-to-offer.md +++ b/issues/hardware/features-no-proving-machine-is-known-to-offer.md @@ -9,7 +9,7 @@ opened: 2026-09-29 No stage of `issues/kernel/the-kernel-is-at-least-as-secure-as-linux-on-every-machine-toyos-supports.md` or `issues/kernel/toyos-uses-what-modern-hardware-offers-for-speed.md` takes -these, since neither the T14 nor a nightly EPYC KVM guest is known to +these, since neither the T14 nor an EPYC KVM guest is known to enumerate one. It is blocked on a proving machine that does; its exit is every row a stage issue under the track it serves, or a `rejected` issue. Intel's parts are named by the SDM 325462-093US, Vol. 1 Table 5-2 ("SDM"), or by the @@ -35,9 +35,9 @@ cores were not verified against Arm's. The Arm rows belong to | TDX | Emerald Rapids (ISE) | | Total Storage Encryption | Panther Lake (ISE) | | TME | 11th-generation Core lines that set CPUID.(7,0):ECX bit 13, which varies by line (datasheet 631121-012 §1.3); the T14 reads ECX 0x18c05fde, bit 13 clear | -| INVLPGB, TLBSYNC | AMD, CPUID 0x80000008:EBX bit 3 (`arch/x86/include/asm/cpufeatures.h:331,335`); whether a nightly EPYC guest sees it waits on its runner's CPUID capture | +| INVLPGB, TLBSYNC | AMD, CPUID 0x80000008:EBX bit 3 (`arch/x86/include/asm/cpufeatures.h:331,335`); whether an EPYC guest sees it waits on its runner's CPUID capture | | SEV-SNP | AMD, CPUID 0x8000001F:EAX bit 4 (`cpufeatures.h:448,453`) | -| Shadow stack on AMD | AMD; unverified: that AMD enumerates it through the bit Linux reads for Intel, CPUID.(7,0):ECX bit 7 (`cpufeatures.h:390,397`), and which parts set it; whether a nightly EPYC guest sees it waits on its runner's CPUID capture | +| Shadow stack on AMD | AMD; unverified: that AMD enumerates it through the bit Linux reads for Intel, CPUID.(7,0):ECX bit 7 (`cpufeatures.h:390,397`), and which parts set it; whether an EPYC guest sees it waits on its runner's CPUID capture | | MTE | Armv8.5 (`arch/arm64/Kconfig:2150-2170`) | | Pointer authentication | Armv8.3 (`arch/arm64/Kconfig:1967-1974`) | | BTI | Armv8.5 (`arch/arm64/Kconfig:2087-2093`) | diff --git a/issues/kernel/a-pure-function-decides-a-cpus-speculation-mitigations-as-linux-does.md b/issues/kernel/a-pure-function-decides-a-cpus-speculation-mitigations-as-linux-does.md index e09ddc3efde..35182bbddb9 100644 --- a/issues/kernel/a-pure-function-decides-a-cpus-speculation-mitigations-as-linux-does.md +++ b/issues/kernel/a-pure-function-decides-a-cpus-speculation-mitigations-as-linux-does.md @@ -13,7 +13,7 @@ selects, built by the kernel and by a host test. Pull request #602 holds it. **Exit**: each committed fixture's facts give its lines: the T14's and the TCG model's from `issues/hardware/linuxs-readings-of-the-t14-and-the-tcg-model-lack-reads-owed-before-the-t14s-wipe.md`, -and each nightly EPYC guest's once its runner is captured. **Mutation**: `GDS` +and each EPYC guest's once its runner is captured. **Mutation**: `GDS` deleted from `cpu_vuln_blacklist`'s TIGERLAKE_L row reds the T14's fixture, and `SRSO` deleted from its family 0x19 row reds a family-0x19 EPYC guest's. **Oracle**: those lines, and Linux's `cpu_vuln_whitelist` and From 281602b72bc4b07df616fb00c28caea895956848 Mon Sep 17 00:00:00 2001 From: japabu Date: Fri, 2 Oct 2026 15:38:23 +0200 Subject: [PATCH 29/30] The TLS stage says which builds compile no ring "No build compiles it" stopped being true when the build system's HTTP agent moved to rustls on ring: every build of the build system compiles ring for the host. What is untried is the ToyOS target, and the sentence says so: `cargo tree -i ring --target all --workspace` prints nothing for `userland` and for `tests/toyos-rust-tests` (exit 0 each). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_013UDZQ6fSKw14e4w2TKTRfm --- issues/design-debt/the-internet-clients-work-unchanged.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/issues/design-debt/the-internet-clients-work-unchanged.md b/issues/design-debt/the-internet-clients-work-unchanged.md index 053f8a76a96..6aa2b22830b 100644 --- a/issues/design-debt/the-internet-clients-work-unchanged.md +++ b/issues/design-debt/the-internet-clients-work-unchanged.md @@ -34,9 +34,9 @@ netd, `toyos::net` and std's ToyOS networking in the `rust/` fork. `rustls-rustcrypto` 0.0.2-alpha, and #660 cut both. The row comes back on `ring` and waits for it; `rustls-rustcrypto` does not come back (owner, 2026-10-02). Whether `ring` builds for the ToyOS target is open and - untried: no build compiles it. `rustls-rustcrypto` is still named by - doom's build script, which installs it on the host, and by a line the cut - left in `tests/toyos-rust-tests` + untried: no build for that target compiles it. `rustls-rustcrypto` is + still named by doom's build script, which installs it on the host, and by + a line the cut left in `tests/toyos-rust-tests` (`issues/build/the-guest-test-crate-depends-on-three-crates-no-test-uses.md`). Open: what doom's build script installs instead. **Exit**: `https_tls13` is a `METAL` row: on the T14's I219 an unmodified From a305253e908177adcd924d82bad753c4c75d591e Mon Sep 17 00:00:00 2001 From: japabu Date: Fri, 2 Oct 2026 16:01:07 +0200 Subject: [PATCH 30/30] The capture issue names the KVM runners, and the unrequired guest check is filed Review round 7's second NOTE: `issues/build/no-nightly-runner-has-had-its- cpuid-and-vulnerability-lines-captured.md` named the nightly for the KVM runners in its slug and in its exit. At this branch no nightly job is given `/dev/kvm` (`nightly.yml`'s `tcg` calls `guest.yml` with `kvm: false`); the KVM guests are `ci.yml`'s `guest`. The slug is renamed `no-kvm-runner-...`, the heading and the exit corrected with it, and its one citation, `toyos-cpuvuln/src/tests.rs:605`, moved. The body's "nightly run 36496779560" stays: that run is `nightly.yml`'s, dispatched on 2026-09-28. `issues/build/the-guest-check-gates-nothing.md` is filed on the orchestrator's instruction. Rounds 5 to 7 hold that it is filed if `guest / suite` is not named a required check in the sitting that lands #671, and it will not be: the naming waits on main's cold `toolchain` job. Its evidence is ruleset 20589156 as read at 13:59:20Z on 2026-10-02 (`check_response_timeout_minutes` 240, one required check, `host`) and main's cache scope as read at 14:00:46Z (two `host-sealed-` entries, no toolchain layer); its exit is the ruleset listing `guest / suite`. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_013UDZQ6fSKw14e4w2TKTRfm --- ...cpuid-and-vulnerability-lines-captured.md} | 6 ++--- issues/build/the-guest-check-gates-nothing.md | 27 +++++++++++++++++++ toyos-cpuvuln/src/tests.rs | 2 +- 3 files changed, 31 insertions(+), 4 deletions(-) rename issues/build/{no-nightly-runner-has-had-its-cpuid-and-vulnerability-lines-captured.md => no-kvm-runner-has-had-its-cpuid-and-vulnerability-lines-captured.md} (77%) create mode 100644 issues/build/the-guest-check-gates-nothing.md diff --git a/issues/build/no-nightly-runner-has-had-its-cpuid-and-vulnerability-lines-captured.md b/issues/build/no-kvm-runner-has-had-its-cpuid-and-vulnerability-lines-captured.md similarity index 77% rename from issues/build/no-nightly-runner-has-had-its-cpuid-and-vulnerability-lines-captured.md rename to issues/build/no-kvm-runner-has-had-its-cpuid-and-vulnerability-lines-captured.md index 426e1f94360..79a5bfb9181 100644 --- a/issues/build/no-nightly-runner-has-had-its-cpuid-and-vulnerability-lines-captured.md +++ b/issues/build/no-kvm-runner-has-had-its-cpuid-and-vulnerability-lines-captured.md @@ -4,7 +4,7 @@ kind: tooling opened: 2026-09-29 --- -# No nightly runner has had its CPUID and vulnerability lines captured +# No KVM runner has had its CPUID and vulnerability lines captured `toyos-cpuvuln`'s `fixtures/awaiting-capture/` holds the lines this crate reads out of the pinned Linux for an EPYC Milan and an EPYC Turin as KVM @@ -14,8 +14,8 @@ host's microcode gives `IBPB_BRTYPE` are guesses there. In nightly run 36496779560 the 14 KVM shards ran on EPYC 7763 (11), 9V74 (2) and 9V45 (1), each as `-cpu host,+rdrand,+smap,+fsgsbase,+x2apic,+smep` with `-smp cores=N`. -**Exit**: a nightly step boots the pinned Linux under that command line on -the runner and records CPUID leaves 0, 1, 7.0, 7.2, 0x80000000, 0x80000008 and +**Exit**: a CI step given `/dev/kvm` boots the pinned Linux under that command +line on its runner and records CPUID leaves 0, 1, 7.0, 7.2, 0x80000000, 0x80000008 and 0x80000021, MSR 0x10A where enumerated, the microcode revision, and `grep .` over `/sys/devices/system/cpu/vulnerabilities/`; the two fixtures are replaced by what it records, and one is added per CPU model it meets. diff --git a/issues/build/the-guest-check-gates-nothing.md b/issues/build/the-guest-check-gates-nothing.md new file mode 100644 index 00000000000..b88174c5277 --- /dev/null +++ b/issues/build/the-guest-check-gates-nothing.md @@ -0,0 +1,27 @@ +--- +status: assigned +kind: tooling +opened: 2026-10-02 +--- + +# The guest check gates nothing + +`ci.yml`'s `guest` runs the guest suite as `guest / suite` on every non-draft +pull request and every merge group, and main's ruleset does not name it: a +pull request or a merge group whose `guest / suite` is red still lands. +`gh api repos/ToyOSOrg/ToyOS/rulesets/20589156` at 13:59:20Z on 2026-10-02 +reads `check_response_timeout_minutes` 240 and one required check, `host`. +Pull request #671, which made the check, says so of itself: "Until step 4 the +guest check gates nothing." + +The naming is a ruleset edit, with no commit, and it waits on main. Main's +cache scope holds no toolchain layer (`actions/caches` for `refs/heads/main` +at 14:00:46Z on 2026-10-02: two `host-sealed-` entries and nothing else), so +until `publish.yml`'s `toolchain` has saved the four there, every merge group +builds all four, as pull request run 36934214557 did in 2:11:46, and a check +named before that makes every landing wait on it. + +Owner: the orchestrator. + +**Exit**: `gh api repos/ToyOSOrg/ToyOS/rulesets/20589156` lists `guest / suite` +among the required checks. diff --git a/toyos-cpuvuln/src/tests.rs b/toyos-cpuvuln/src/tests.rs index f6b624a50ef..35d4808a1a6 100644 --- a/toyos-cpuvuln/src/tests.rs +++ b/toyos-cpuvuln/src/tests.rs @@ -602,7 +602,7 @@ fn rtm_always_abort_decides_tsx_first() { // The two fixtures below are this crate's reading of the pinned Linux, awaiting // a capture on a nightly runner: -// `issues/build/no-nightly-runner-has-had-its-cpuid-and-vulnerability-lines-captured.md`. +// `issues/build/no-kvm-runner-has-had-its-cpuid-and-vulnerability-lines-captured.md`. /// A guest's lines do not read its microcode: `tsa_init` returns under a /// hypervisor (`amd.c:519-520`) before `amd_check_tsa_microcode`.