From d7094d04767bb481aef89bec82b8f57e6e0a678e Mon Sep 17 00:00:00 2001 From: japabu Date: Thu, 1 Oct 2026 10:32:39 +0200 Subject: [PATCH 1/8] toyos-fat32: the suite reads committed volumes macOS wrote, and the host job runs on Linux The FAT32 host suite formatted every volume with `/sbin/newfs_msdos` through an `hdiutil` device node and populated it through a macOS mount, so the PR gate's required `host` check could only run on `macos-latest`. A macOS binary is a hard no, "only for tests" included. The volumes are the oracle: bytes a FAT implementation that is not ours wrote. So they are generated once and committed, and the runtime dependency goes. - Six fixtures under toyos-fat32/tests/fixtures/: two blank volumes (512-byte and 4 KiB clusters) host_write.rs writes into, host_read.rs's two populated volumes, and hostile.rs's corpus and empty-file volume. Made on macOS 27.0.1 (26A434), msdosfs SourceVersion 845000003000001, by the invocations common/mod.rs made at fa134154c with a fixed label; NOTICE records the commands, the files, and each raw digest. - Compact form: runs of nonzero 512-byte sectors (u64 image length, then offset, length, bytes), expanded by `Image::fixture` into a sparse file, a dozen lines of our own Rust. 441,328 bytes for all six; gzip -9 of the runs would be 384,430 and of the raw images 1,263,669, measured. 85% of the bytes are the tests' own pseudorandom file contents, which do not compress, so a compression crate would buy 13% for a new dependency edge. Each expansion was checked to reproduce the raw image's sha256. - The read-back of what *our* writer wrote went through the macOS mount; committed bytes cannot serve that. It is now the `fatfs` crate, a third party's FAT implementation already in the lockfile, which the guest harness already reads volumes with. It is a dev-dependency only. The timestamp test now asserts the exact fields (2024-06-01 12:34:56) instead of a 24-hour window, because fatfs applies no zone. - Boot code: every boot sector carries newfs_msdos's 84-byte `bootcode[]` at 0x5A, byte for byte Robert Nordier's array in FreeBSD's mkfs_msdos.c (BSD-2-Clause) and in Apple's msdosfs-788.100.31. The rows say `(MIT OR Apache-2.0) AND BSD-2-Clause` and licenses/ carries the notice verbatim. The freed clusters keep the AppleDouble sidecars macOS wrote and the generator deleted; that is machine-written data, recorded in NOTICE. - ci.yml's and nightly.yml's `host` jobs move to ubuntu-24.04, on a cache key of their own: restoring the 4.15 GiB macOS `host-` cache onto Linux would carry dead macOS artifacts forward in every later save. - issues/filesystem/fat32-suite-needs-macos-binaries.md and its row in the host-tools issue are deleted; nothing else cited them. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L --- .github/workflows/ci.yml | 8 +- .github/workflows/nightly.yml | 8 +- Cargo.lock | 1 + NOTICE | 88 ++++++ ...d-runs-host-tools-outside-rust-and-qemu.md | 1 - .../fat32-suite-needs-macos-binaries.md | 50 ---- licenses/BSD-2-Clause-newfs_msdos.txt | 24 ++ src/licence.rs | 40 +++ toyos-fat32-check/tests/common/mod.rs | 10 +- toyos-fat32/Cargo.toml | 4 + toyos-fat32/tests/common/mod.rs | 149 ++-------- toyos-fat32/tests/fixtures/blank-4k.runs | Bin 0 -> 3672 bytes toyos-fat32/tests/fixtures/blank-512.runs | Bin 0 -> 3672 bytes toyos-fat32/tests/fixtures/empty-file.runs | Bin 0 -> 6264 bytes toyos-fat32/tests/fixtures/hostile.runs | Bin 0 -> 36536 bytes toyos-fat32/tests/fixtures/read-4k.runs | Bin 0 -> 55032 bytes toyos-fat32/tests/fixtures/read-512.runs | Bin 0 -> 336152 bytes toyos-fat32/tests/host_read.rs | 61 ++-- toyos-fat32/tests/host_write.rs | 276 ++++++++---------- toyos-fat32/tests/hostile.rs | 15 +- 20 files changed, 342 insertions(+), 393 deletions(-) delete mode 100644 issues/filesystem/fat32-suite-needs-macos-binaries.md create mode 100644 licenses/BSD-2-Clause-newfs_msdos.txt create mode 100644 toyos-fat32/tests/fixtures/blank-4k.runs create mode 100644 toyos-fat32/tests/fixtures/blank-512.runs create mode 100644 toyos-fat32/tests/fixtures/empty-file.runs create mode 100644 toyos-fat32/tests/fixtures/hostile.runs create mode 100644 toyos-fat32/tests/fixtures/read-4k.runs create mode 100644 toyos-fat32/tests/fixtures/read-512.runs diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 4f348c1b995..6e4e1cfe272 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -15,11 +15,9 @@ concurrency: cancel-in-progress: true jobs: - # macOS, because `toyos-fat32`'s fixtures are formatted with `newfs_msdos` - # and `hdiutil`. host: if: github.event_name == 'merge_group' || github.event.pull_request.draft == false - runs-on: macos-latest + runs-on: ubuntu-24.04 timeout-minutes: 45 steps: - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 @@ -37,7 +35,7 @@ jobs: toyos/target kernel/target bootloader/target - key: host-${{ github.run_id }} - restore-keys: host- + key: host-linux-${{ github.run_id }} + restore-keys: host-linux- - run: cargo run -- --ci host diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index 36d2aa93359..df5bdf718cc 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -17,7 +17,7 @@ concurrency: jobs: host: - runs-on: macos-latest + runs-on: ubuntu-24.04 timeout-minutes: 90 steps: - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 @@ -35,8 +35,8 @@ jobs: toyos/target kernel/target bootloader/target - key: host-${{ github.run_id }} - restore-keys: host- + key: host-linux-${{ github.run_id }} + restore-keys: host-linux- - run: cargo run -- --ci host @@ -46,7 +46,7 @@ jobs: uses: actions/cache/save@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 with: path: *host-paths - key: host-${{ github.run_id }} + key: host-linux-${{ github.run_id }} # Publishes this tree's toolchain if nobody has, and on main moves the SDK # alias onto it. Bare `ubuntu-24.04`, not a container: its glibc is the diff --git a/Cargo.lock b/Cargo.lock index f3c87c3c550..ff0cf17ca84 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1059,6 +1059,7 @@ version = "0.1.0" name = "toyos-fat32" version = "0.1.0" dependencies = [ + "fatfs", "toyos-fat32-check", "toyos-tmpdir", "toyos-wallclock", diff --git a/NOTICE b/NOTICE index 1b5c29be846..3e8ccba5736 100644 --- a/NOTICE +++ b/NOTICE @@ -364,3 +364,91 @@ the tool, not the bytes a user's own data becomes when the tool writes it: the layout is a published on-disk format, not a copyrightable work of the program, and the file contents are ours. The tool is not distributed with this repository in any form. + + +toyos-fat32/tests/fixtures/ — volumes macOS's FAT tools wrote +-------------------------------------------------------------- + + blank-512.runs 3,672 bytes, of a 67,108,864-byte raw volume + sha256 4f1686152769fb4b064a78b0a2287c8b54bff3ec1e279b35c8f337d538a45f78 + raw 51dfc22ac6ab939a28f8cf113da48cbb75eb0b4f5a4d2b79c9f09522a58fe052 + blank-4k.runs 3,672 bytes, of a 314,572,800-byte raw volume + sha256 14e98fc505450882f4d284fb712d32b4f9e4d50c1bfc86702ace8f0adee745e0 + raw 788974e418ac37cb43f65057f609bd4234b4f5fffb10e8be7563a608b423e137 + read-512.runs 336,152 bytes, of a 67,108,864-byte raw volume + sha256 6bef14efca36070830f5a4a2b1dc9b69a0e2627679cc5c9ed542acaa4ece416c + raw 911903c9338144f65df86a6f98d87c375725806a709db3bb48d772fe3b719fbb + read-4k.runs 55,032 bytes, of a 314,572,800-byte raw volume + sha256 cab78e1ea6c46b0a04a0de32a24f1a6ac90a2017dd7188745de2d603c5ab121b + raw cb4cefe9a9b2b17371ec20fa8a49b16475f2434ff5ed5d9277c99bef3cdd592c + hostile.runs 36,536 bytes, of a 67,108,864-byte raw volume + sha256 c570b3c6944f47d2cd1498da14f12c4a98dc85683f782624181917eba68f9066 + raw 1750198122459dee8f1b4d54ac460d9689279189d6ff59f6072d57d00dbbd68e + empty-file.runs 6,264 bytes, of a 67,108,864-byte raw volume + sha256 68a6abf1626e4bdf6b3586d3f52b2d7051390e4e18beb31258dd0021bb3e96b9 + raw ff0e162a36ccf76d1aa5cb36ce47c0d728442bf926d59af498d0d55820bcfd85 + Boot stub: Copyright (c) 1998 Robert Nordier + Licence text: licenses/BSD-2-Clause-newfs_msdos.txt + SPDX-License-Identifier: (MIT OR Apache-2.0) AND BSD-2-Clause + +`toyos-fat32`'s host tests read these volumes, and write into the two blank +ones. The only honest judge of a FAT reader is bytes a FAT implementation that +is not ours wrote, and these are those bytes. Each is committed as its runs of +nonzero 512-byte sectors, which `Image::fixture` in +`toyos-fat32/tests/common/mod.rs` expands into a sparse file; the raw digest is +of that expansion, and equals the digest of what the tools wrote. + +They were made once, on macOS 27.0.1 (build 26A434), with Apple's `msdosfs` +project at SourceVersion 845000003000001 (`/sbin/newfs_msdos`, sha256 +29b3d7daa299583015c823ea6ebce643c030a79c8bc26b9bf9c23deb4c6d1a73) and +`/usr/bin/hdiutil`, by a throwaway program making the invocations +`toyos-fat32/tests/common/mod.rs` made at fa134154c with a fixed label. Per +volume, on a sparse file of the raw size: + + hdiutil attach -imagekey diskimage-class=CRawDiskImage -nobrowse -nomount + newfs_msdos -F 32 -S 512 -c -v