Repository navigation
ToyOS's own SHA-256 and SHA-512, as fast as sha2's soft backend, replace the sha2 crate in the loader's and update's image verification, swap, pkg and the build - #812
Conversation
…n the loader's and update's image verification, swap, pkg and the build
The owner's ruling on the dependency audit ("Own SHA-2"): the hash the
signed-image trust boundary rests on is ours. `toyos-sha2` is SHA-256 and
SHA-512 written from FIPS 180-4 (§4.2, §5.1, §5.3, §6.2, §6.4), no_std and
forbid(unsafe_code), scalar on every target. SHA-512 is in because
`toyos-update::sig` builds the SSHSIG message over SHA-512(header).
Its API is the subset of sha2's `Digest` calls the tree makes, as inherent
methods: `new`, `update(impl AsRef<[u8]>)`, `finalize() -> [u8; N]` and
`digest(impl AsRef<[u8]>) -> [u8; N]`. Every caller drops its
`use sha2::Digest` and its `.into()`; cicache's `{:x}` of a GenericArray
becomes `release::sha256_hex`.
Callers moved: toyos-update (`sha256`, and `sig`'s SHA-512), toyos-swap,
userland/update's streamed ROOT hash, userland/pkg, and the build's
cicache, sourcegate, sysroot, image and release.
`ed25519-dalek` 2.2 still hashes with `sha2` 0.10.9 inside a verification
(its `verifying.rs` takes `sha2::Sha512` for H(R || A || M)), so `sha2`
stays in the lock and the loader keeps its `force-soft` line, recommented
as being for that alone.
Oracles: NIST's CAVP SHAVS response files, committed under
toyos-sha2/cavp/ and recorded in NOTICE (byte-oriented ShortMsg, LongMsg
and Monte for both hashes, bit-oriented ShortMsg and Monte; the
bit-oriented LongMsg files, 16.8 MB, are not), and RustCrypto's sha2 as a
dev-dependency over every length 0..=4096 in random splits.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017cSFvbD35xJ2kGANVdm23C
…IST-PD Clippy's chunks_exact_to_as_chunks and manual_is_multiple_of, under `--ci host`'s -D warnings: a block's words and the whole blocks of an update are `as_chunks`, which leaves no `expect` on a length the type already fixes. The licence gate judges every file under a shipped path package's directory as shipped, so the CAVP vectors under toyos-sha2/cavp/ are judged by their NOTICE line, NIST-PD: NIST's notice that a work of the US government is public domain (17 U.S.C. §105), allowed beside CC0-1.0, the other public-domain dedication the list carries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017cSFvbD35xJ2kGANVdm23C
… a pass The rounds moved all eight working variables every round; the scalar hash measured 1.47x the CPU time of sha2's force-soft backend over a 56 MiB message on the host. Each round now writes only the two it changes (d and h, §6.2.2 step 3), and the loop takes eight rounds a pass with the names rotated, which is the same computation: 1.19x to 1.25x. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017cSFvbD35xJ2kGANVdm23C
|
Negative controls, applied with
--- a/toyos-sha2/src/lib.rs
+++ b/toyos-sha2/src/lib.rs
@@ -24,7 +24,7 @@
/// SHA-256's round constants, §4.2.2.
#[rustfmt::skip]
const K256: [u32; 64] = [
- 0x428a2f98, 0x71374491, 0xb5c0fbcf, 0xe9b5dba5, 0x3956c25b, 0x59f111f1, 0x923f82a4, 0xab1c5ed5,
+ 0x428a2f99, 0x71374491, 0xb5c0fbcf, 0xe9b5dba5, 0x3956c25b, 0x59f111f1, 0x923f82a4, 0xab1c5ed5,
0xd807aa98, 0x12835b01, 0x243185be, 0x550c7dc3, 0x72be5d74, 0x80deb1fe, 0x9bdc06a7, 0xc19bf174,
0xe49b69c1, 0xefbe4786, 0x0fc19dc6, 0x240ca1cc, 0x2de92c6f, 0x4a7484aa, 0x5cb0a9dc, 0x76f988da,
0x983e5152, 0xa831c66d, 0xb00327c8, 0xbf597fc7, 0xc6e00bf3, 0xd5a79147, 0x06ca6351, 0x14292967,
--- a/toyos-sha2/src/lib.rs
+++ b/toyos-sha2/src/lib.rs
@@ -62,7 +62,7 @@
0xca273eceea26619c, 0xd186b8c721c0c207, 0xeada7dd6cde0eb1e, 0xf57d4f7fee6ed178,
0x06f067aa72176fba, 0x0a637dc5a2c898a6, 0x113f9804bef90dae, 0x1b710b35131c471b,
0x28db77f523047d84, 0x32caab7b40c72493, 0x3c9ebe0a15c9bebc, 0x431d67c49c100d4c,
- 0x4cc5d4becb3e42b6, 0x597f299cfc657e2a, 0x5fcb6fab3ad6faec, 0x6c44198c4a475817,
+ 0x4cc5d4becb3e42b6, 0x597f299cfc657e2a, 0x5fcb6fab3ad6faec, 0x6c44198c4a475816,
];
/// SHA-512's initial hash value, §5.3.5.The script: #!/bin/sh
# Apply one checked mutation patch, run toyos-sha2's tests, report, restore.
set -u
tree=<path>
logs=<path>
for m in mutation-k256-first mutation-k512-last; do
cd "$tree" || exit 9
git apply --check "$logs/$m.patch" || { echo "$m: does not apply"; exit 9; }
git apply "$logs/$m.patch"
cargo build -p toyos-sha2 > "$logs/$m.build.log" 2>&1; echo "$m build EXIT=$?"
cargo test -p toyos-sha2 > "$logs/$m.test.log" 2>&1; echo "$m test EXIT=$?"
git apply -R "$logs/$m.patch"
git status --porcelain
doneHost benchmark source (scratch crate, [package]
name = "sha2bench"
version = "0.0.0"
edition = "2024"
publish = false
[features]
soft = ["sha2/force-soft"]
[dependencies]
toyos-sha2 = { path = "<worktree>/toyos-sha2" }
sha2 = "0.10"
libc = "0.2"
[profile.release]
opt-level = 2
[workspace]use sha2::Digest as _;
fn cpu() -> f64 {
let mut t = libc::timespec { tv_sec: 0, tv_nsec: 0 };
assert_eq!(unsafe { libc::clock_gettime(libc::CLOCK_THREAD_CPUTIME_ID, &mut t) }, 0);
t.tv_sec as f64 + t.tv_nsec as f64 * 1e-9
}
fn best<const N: usize>(f: impl Fn() -> [u8; N]) -> (f64, [u8; N]) {
let mut best = f64::MAX;
let mut out = [0; N];
for _ in 0..21 {
let t = cpu();
out = std::hint::black_box(f());
best = best.min(cpu() - t);
}
(best, out)
}
fn main() {
let size: usize = std::env::args().nth(1).expect("bytes").parse().expect("a size");
let data: Vec<u8> = (0..size).map(|i| (i as u32).wrapping_mul(2654435761) as u8).collect();
let mib = size as f64 / (1 << 20) as f64;
let (a, da) = best(|| toyos_sha2::Sha256::digest(&data));
let (b, db) = best(|| sha2::Sha256::digest(&data).into());
assert_eq!(da, db);
let (c, dc) = best(|| toyos_sha2::Sha512::digest(&data));
let (d, dd) = best(|| sha2::Sha512::digest(&data).into());
assert_eq!(dc, dd);
let soft = if cfg!(feature = "soft") { "sha2 force-soft" } else { "sha2 default" };
println!("{size} bytes ({mib:.1} MiB), best of 21, thread CPU time");
println!("SHA-256 toyos-sha2 {:8.2} ms {:7.1} MiB/s | {soft} {:8.2} ms {:7.1} MiB/s | ratio {:.2}", a * 1e3, mib / a, b * 1e3, mib / b, a / b);
println!("SHA-512 toyos-sha2 {:8.2} ms {:7.1} MiB/s | {soft} {:8.2} ms {:7.1} MiB/s | ratio {:.2}", c * 1e3, mib / c, d * 1e3, mib / d, c / d);
}Output at db2e1f3 (Apple M4 Pro, load average 40-48 during it): |
|
Review of #812 at head Net lines (
Correctness against FIPS 180-4, read by hand at this head:
The oracles (CAVP byte ShortMsg, LongMsg and Monte, plus the BLOCKER
NOTE
SEND BACK |
|
T14 at |
…and the bit-oriented ones go The licence gate counts every file under a shipped path package's directory as shipped, so vectors under toyos-sha2/cavp/ needed NIST-PD in ALLOWED. They are test input only: under tests/, the build's own directory, no shipped package holds them, NOTICE's section says NIST-PD as before, and ALLOWED is main's again. The bit-oriented ShortMsg and Monte files go with the test code that read them. The crate hashes whole bytes; the bit test padded in its own helper and drove the compression function directly, which the byte ShortMsg, LongMsg and Monte files already drive over thousands of blocks, and the bit Monte files differ from the byte ones only by their seed. The host-tools issue named sha2 as what src/release.rs hashes with; it is toyos-sha2. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017cSFvbD35xJ2kGANVdm23C
…schedule computed a pass ahead, every round's indices constant, and the hash value held across a run of blocks The loader's ROOT hash took 2.653 s on the T14 against main's 2.209 s with sha2's soft backend. Three changes close it, measured on the host at opt-level 2 with [profile.toyos]'s debug assertions and overflow checks, 56 MiB, best of 21 runs' thread CPU time: - The schedule is a sixteen-word ring, W(t) written over W(t-16) a pass at a time, in place of a 64- or 80-word array. - Rounds and schedule words are written out sixteen to a pass through const generics, so every index into the working variables and the schedule is a constant; LLVM had kept the rounds a loop over memory. - One call compresses every whole block of an update, the hash value in locals between them; Maj is b ^ ((a ^ b) & (b ^ c)), whose b ^ c is the previous round's a ^ b. SHA-256 went from 124.95 ms to 99.10 ms and SHA-512 from 80.03 ms to 61.63 ms, against sha2 force-soft's 103.33-105.29 ms and 66.98-68.22 ms. The two compression functions are now one macro over their word, block, constants and rotations. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017cSFvbD35xJ2kGANVdm23C
update's ROOT hash, pkg and swap on the T14 and the build's hashing on x86-64 CI runners took sha2's x86 backend; toyos-sha2 is scalar and forbids unsafe. The loss is at most the scalar time, which the T14's loader measured on main: 2.21 s for the testcases image's 306 MiB ROOT. Whether that is worth unsafe intrinsics in the crate every signed image is verified with is the owner's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017cSFvbD35xJ2kGANVdm23C
|
Round 2 evidence for #812 at b9132eb (the code at c8aad54; b9132eb adds only the issue file). Paths are shown relative to the worktree or as Host benchmark:
|
|
Review of #812 at head Round-1 BLOCKERs
I re-read the c8aad54 compression against FIPS 180-4 §6.2.2 and §6.4.2.
Net lines (
Round 1's NOTEs are closed: the bit vectors and BLOCKER
NOTE
SEND BACK |
|
T14 at |
|
Review of #812 at head Earlier BLOCKERs
Main has moved to Net lines ( NOTE
LAND |
The owner's ruling on the dependency audit, "Own SHA-2": every SHA-256 and SHA-512 that ToyOS's own code takes is now
toyos-sha2, written here from FIPS 180-4. Headb9132ebb7.What changed, per decision
toyos-sha2is a new member:no_std,forbid(unsafe_code), 255 lines. It implements SHA-256 and SHA-512 per FIPS 180-4 §4.2, §5.1, §5.3, §6.2 and §6.4.toyos-update::sigbuilds the SSHSIG message over SHA-512(header).sha2's soft backend. Round 1 was 1.2x slower, and the T14 showed it: the loader's ROOT hash took 2.653 s against main's 2.209 s. Three changes close the gap; they are measured under Speed.update, with the hash value held in locals between blocks.b ^ ((a ^ b) & (b ^ c)), whoseb ^ cis the previous round'sa ^ b.unsafeandcore::arch, and this crate forbidsunsafe. The loader is also a soft-float UEFI application that may not assume the SIMD/SHA registers are its own, which is why it already forcedsha2soft. The x86-64 callers that used SHA-NI/AVX2 are bounded under Speed and recorded inissues/x86-64-hashing-runs-scalar-where-the-cpu-has-sha-instructions.md(status: owner).Digestcalls the tree makes, as inherent methods:new(),update(impl AsRef<[u8]>),finalize() -> [u8; N],digest(impl AsRef<[u8]>) -> [u8; N]andDefault. Each caller drops itsuse sha2::Digestand its.into().toyos-update:sha256(), which covers the loader's andupdate's section hashes,floor, andsigning.rsthrough it. Alsosig's SHA-512.toyos-swap.userland/update's streamed ROOT hash.userland/pkg.src/cicache.rs,sourcegate.rs,sysroot.rs,image.rsandrelease.rs.cicache's{:x}of aGenericArraybecamerelease::sha256_hex.ed25519-dalekkeepssha2.sha2 0.10.9without making it optional, and itssrc/verifying.rstakessha2::Sha512for H(R‖A‖M) insideverify_strict. Sosha2 0.10.9stays inCargo.lock.sha2 … features = ["force-soft"]line, with a comment saying it exists only for that.sha2 0.11.0-rc.5belongs to russh and is untouched.sha2any more, excepttoyos-sha2's dev-dependency.tests/cavp/, outside every shipped package's directory. They are ShortMsg, LongMsg and Monte for both hashes, byte for byte as upstream, CRLF included.NOTICE's sectiontests/cavp/gives each file's hash and size, the upstream archive's hash, andNIST-PD.src/licence.rsis identical to main (git diff origin/main -- src/licence.rsis empty). This follows the orchestrator's ruling that this branch does not widen the licence list.padded,bit_fileand their tests. They drove the compression function under the test's own padding, which the byte files already drive. The bit Monte files differ from the byte ones only by their seed.issues/the-build-runs-host-tools-outside-rust-and-qemu.md'sshasumrow now namestoyos-sha2as whatsrc/release.rshashes with.Checks of high-risk code (the signed-image trust boundary)
toyos-sha2/src/tests.rs).[L = n]header is held against its digests' length.sha20.10.9. It is a dev-dependency only and shares no code with this crate.b ^ cchanged toa ^ c: all five red.I % 4: all five red.toyos-update's existing RFC 8032 andsshsig_oracletests (OpenSSH'sssh-keygen -Youtput) pass through the new SHA-512.Speed
[profile.toyos]'s debug assertions and overflow checks; each figure is the best of 21 runs' thread CPU time. The source and the log are in the PR comment.toyos-sha2took 99.8–102.1 ms againstsha2force-soft's 103.3–105.3 ms, a ratio of 0.97–0.98.sha2default gives the same numbers on this host, because its aarch64 hardware backend needs theasmfeature.testcasesROOT (320,864,256 bytes) in 4,757,794,201 ticks at 2,419,200,000 Hz (1.967 s, 14.83 ticks/byte) against main's 5,343,577,520 (2.209 s, 16.65 ticks/byte): 0.89x. That loader was built by the compiler before The fork's LLVM gives a loop counter's recurrence only the wrap flags proven for it, a sysroot whose compilers lose a loop's last exit is refused, and the ScalarEvolution issue is closed #790 (merge based6298c83e); the merge queue builds with main's compiler.x86_64-unknown-uefiwith the same profile, this head's SHA-256 block executes about 3,300 instructions: 65 to load the block, 3 schedule passes of 341, and 4 round passes of 536 plus the loop's own. That is counted from the emitted listing.sha2soft's fully unrolled block loop is 3,427 instructions. The T14 reading below is the timed verdict.update,pkgandswapon the T14, and the build's hashing on CI. The loss is at most their scalar hash time. The T14's loader on main measured that rate: 320,864,256 bytes in 5,343,577,520 ticks at 2,419,200,000 Hz, which is 2.21 s or 138.5 MiB/s.updateloses at most 2.2 s of a 306 MiBtestcasesROOT. The same loader read that ROOT off the stick in 9.31 s.updateloses at most 0.4 s.cicache's 47,906,184 tracked bytes, at most 0.33 s at that rate.unsafeintrinsics into the crate every signed image is verified with, and the loader could not take that path anyway. Whether that trade is wanted is the owner's call, recorded in the issue.Gates (head b9132eb)
cargo run -- --ci host: EXIT=0, "Host: 78 step(s), all green".cargo test -p toyos-sha2: EXIT=0, 5 tests.cargo run -- --build-only: EXIT=0.cargo test --test toyos-build: EXIT=0, "41 passed, 41 total" over 44 guests.uptimeload averages were 8.82 / 9.75 / 13.08 before and 15.01 / 11.18 / 13.45 after.boot:testcasesatb9132ebb7, run by the orchestrator: both image hashes matchedrequest.txt, bothtoyos-metal --fat32-checkexit 0; judgecargo test --test toyos-build -- --metal --metal-readback <dir> boot:testcasesEXIT=0,[metal] 249 passed, 0 failed, 2 boot(s).No new guest test: the hash is a pure function, and the host tiers above cover it completely.
Net lines: 2785 insertions and 36 deletions.
toyos-sha2/src/lib.rs.Coordination with #808 (
wt/toyos-pkgrepo)#808 had not landed at this head. Whichever branch lands second resolves these, and nothing else:
toyos-update/Cargo.toml: keeptoyos-sha2, with nosha2line.Cargo.toml: keeptoyos-sha2in place ofsha2 = "0.10", never both.sig.rs:use sha2::{Digest as _, Sha512};becomesuse toyos_sha2::Sha512;.repo.rs: deleteuse sha2::Digest as _;and writesha2::Sha256astoyos_sha2::Sha256in both places.Digest::from(self.hash.finalize())compiles unchanged.Cargo.lock.Unsure
update,pkg,swap) is not measured; the loader's reading above bounds the scalar path.🤖 Generated with Claude Code
https://claude.ai/code/session_017cSFvbD35xJ2kGANVdm23C