Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@ members = [
"toyos-net-wire",
"toyos-osrelease",
"toyos-quiesce",
"toyos-random",
"toyos-rootimage",
"toyos-swap",
"toyos-symbols",
Expand Down
2 changes: 2 additions & 0 deletions bootloader/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,8 @@ toyos-gpt = { path = "../toyos-gpt" }
toyos-rootimage = { path = "../toyos-rootimage" }
bcachefs = { path = "../bcachefs", default-features = false }
toyos-tco = { path = "../toyos-tco" }
# What bytes are a seed: the judgment the kernel makes of what this hands it.
toyos-random = { path = "../toyos-random" }
# The counter's rate each line's stamp is converted at, decoded as the kernel
# decodes it.
toyos-tsc = { path = "../toyos-tsc" }
Expand Down
5 changes: 5 additions & 0 deletions bootloader/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ mod gcd;
mod loaderlog;
mod rootbridge;
mod rootimage;
mod seed;
mod slot;
mod stamp;
mod watchdog;
Expand Down Expand Up @@ -627,7 +628,11 @@ fn start_kernel(kernel: LoadedKernel, kernel_elf_bytes: vec::Vec<u8>, cmdline: v
loader_entry_counter: entry_counter,
loader_handoff_counter: 0,
root_read_ticks,
// Read into this struct below, and nowhere else: the kernel zeroes it here.
loader_seed: [0; toyos_abi::boot::SEED_LEN],
loader_seed_len: 0,
};
kernel_args.loader_seed_len = seed::read(&system_table, &mut kernel_args.loader_seed);
report_reach(
"Kernel arguments",
&kernel_args as *const KernelArgs as u64,
Expand Down
42 changes: 42 additions & 0 deletions bootloader/src/seed.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
//! The seed firmware gives the kernel's random generator: 32 bytes from
//! `EFI_RNG_PROTOCOL` (UEFI 2.11 §37.5), asked for once, before
//! `ExitBootServices`. Firmware without the protocol is a machine and not an
//! error: the kernel is handed none and keys its generator from what its CPU
//! has, or refuses. One line says which, and no line carries a byte.

use toyos_abi::boot::SEED_LEN;
use toyos_random::{wipe, Seed};
use uefi::prelude::*;
use uefi::proto::rng::Rng;

use crate::protocol;

/// Fill `into` with firmware's seed and answer its length: [`SEED_LEN`], or 0
/// with `into` zero. Judged with the kernel's own [`Seed::judge`], so the line
/// here says what the kernel will find.
pub fn read(system_table: &SystemTable<Boot>, into: &mut [u8; SEED_LEN]) -> u64 {
let bs = system_table.boot_services();
let none = |why: core::fmt::Arguments| {
println!("Seed: {why}, so the kernel's generator is handed none");
0
};
let Ok(handle) = bs.get_handle_for_protocol::<Rng>() else {
return none(format_args!("firmware has no EFI_RNG_PROTOCOL"));
};
// GET_PROTOCOL: an exclusive open would stop the driver behind it.
let mut rng = match protocol::get::<Rng>(bs, handle) {
Ok(rng) => rng,
Err(e) => return none(format_args!("EFI_RNG_PROTOCOL would not open ({e})")),
};
// No algorithm named: firmware's default (§37.5.2).
if let Err(e) = rng.get_rng(None, into) {
wipe(into);
return none(format_args!("EFI_RNG_PROTOCOL's GetRNG failed ({e})"));
}
if let Err(why) = Seed::judge(into) {
wipe(into);
return none(format_args!("EFI_RNG_PROTOCOL's {SEED_LEN} bytes are refused, {why}"));
}
println!("Seed: {SEED_LEN} bytes from EFI_RNG_PROTOCOL for the kernel's generator");
SEED_LEN as u64
}
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,11 @@ When a host's QEMU installation ships an edk2 from stable202502 to
stable202608, `virt` guests under HVF never return from `ExitBootServices`.
Owner: the orchestrator.

Every `virt_` test whose profile is `Profile::Virt` or `Profile::VirtNoRng`
(`tests/toyos.rs`), which is most of them, boots under HVF on an Apple host,
so a QEMU upgrade there that bundles an edk2 from this range reds all of
those, where it once would have red two.

Under QEMU 11.1.1's HVF a `virt` guest reads `ID_AA64PFR0_EL1.GIC` as 0,
though its GICv3's system registers answer. `hvf_arch_init_vcpu`
(`target/arm/hvf/hvf.c:1481`) writes that register once, setting `GIC` only if
Expand Down
22 changes: 0 additions & 22 deletions issues/every-random-byte-is-one-rdrand.md

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
---
status: open
kind: defect
opened: 2026-10-09
---

# The boot's last word can miss the console: the stop's drain declines while `klogd` holds the wire

A boot that was asked to stop can power off without `Shutting down.` on its
console. Seen on 8-CPU `virt` guests under HVF, six times, each a test red as
`STALLED: waiting for the boot's last word — it went quiet`: the guest was not
quiet but gone.

**What the code does, by reading.** `quiesce` (`kernel/src/syscall/machine.rs`)
logs the last word and calls `log::console::drain_inline`, which takes the
wire with `serial::try_wire` and returns at once where it is held: "whoever
holds the wire drains it too". The holder is `klogd`, on another CPU, part-way
through its backlog. Nothing waits for it. The caller goes on to the power-off
or the reset, which turns `klogd`'s CPU off before it reaches the record. The
stop's record and the census, logged above the last word, go the same way.

**What was measured.** The entropy stage's branch at `7522ec61e`, an
Apple-silicon host of 14 cores at 1-minute load 28 to 46, QEMU 11.1.1,
`tests/virtsmpcase` on eight CPUs under `Profile::Virt` (HVF), whose first job
`unmap_touch` has the kernel report eight faults, each at length:

- `virt_el1_smp`, `virt_mask_windows` and `virt_off_names_the_cpus_left_on`
each wait for the last word. Side by side with `virt_smp` and
`virt_failed_ap_leaves_no_hole`, 131 runs: five reds, two of `virt_el1_smp`,
two of `virt_mask_windows`, one of `virt_off_names_the_cpus_left_on`. A
sixth, `virt_el1_smp`, in the whole suite at load 41 to 47.
- The same five tests with `Profile::Virt` emulated at EL1 (`-cpu max`), the
same session: 50 runs, no red. Five in 393 guests against none in 150 does
not separate the two by itself (Fisher's exact test, p = 0.33): the place
the reds stop in does.
- One red `virt_el1_smp`, captured: QEMU's `arm_psci_call` trace holds seven
`CPU_OFF` and one `SYSTEM_OFF`, so the kernel powered the machine off. Its
console's last line is the supervisor's `power: the machine stops, and
logkeeper makes the log whole first (Shutdown)`, stamped 2.361, in among
the kernel's fault-report records stamped 2.264: `klogd` was about 100 ms
of guest clock behind when the stop began. No stop record, no census, no
`Shutting down.`.
- The red `virt_off_names_the_cpus_left_on` said, after the wait began,
`power: cpu6 is not off by PSCI's answer inside the budget; SYSTEM_OFF
regardless` and the same of cpu7, which `arch::power::off` writes straight
to the UART after `quiesce` has returned, and no `Shutting down.`.

The probe that captured it is `debug-shutdown-regs.patch` in
https://github.com/ToyOSOrg/ToyOS/pull/802#issuecomment-6080912601.

**Not the counters-read silences.** The third and fourth silent guests of
`issues/a-counters-read-under-host-load-can-go-silent-for-15-s.md` stop at
another step: each console ends at the kernel's `spawn:` record of
`test_rs_counters_read`, with no `===TEST_END test_rs_counters_read` and no
supervisor stop line. In this defect's capture every userland line through
`power: the machine stops …` reached the wire, as `drain_for_the_stop` waits
for the wire and drains the queue; only the kernel's own records after it
went missing. The two `virt_mask_windows` stalls that issue records with no
console, and no PSCI trace, are not decidable either way. What tells the
next capture's silence apart: whether `===TEST_END test_rs_counters_read`
and the supervisor's stop line are on its console, and QEMU's PSCI trace of
it, a powered-off guest's `CPU_OFF` and `SYSTEM_OFF` being this defect's.

**Not known.** Whether the wire's holder was `klogd`: no capture names it.
Whether an x86-64 guest with a 16550 loses its last word the same way.
Whether `virt_reboot`, which waits for `Rebooting.` through the same
`quiesce`, loses it under HVF: it was moved there and back unmeasured.

**Until it is fixed** these stay emulated (`tests/toyos.rs`): the three
tests above, and `virt_reboot`; `Profile::VirtTcg` stays for
`virt_el1_smp`.

**Exit**: the stop puts its last word on the wire before it takes a CPU down,
waiting for the wire's holder as `drain_for_the_stop` does, or the cause is
shown to be another from a capture. It is shown by a test that reds without
the fix on every boot, the wire held across the stop by an actuator, and by
the four tests under `Profile::Virt` on an Apple host: 300 side-by-side
runs at load 30 or more with no red, the harness's PSCI trace allowed under
HVF as the probe allowed it. They then move to `Profile::Virt`,
`virt_reboot` with an EL2 row of its own kept for `SYSTEM_RESET` through
the SMC conduit, and `Profile::VirtTcg` is deleted.

Owner: the kernel's AArch64 bring-up, `issues/toyos-runs-on-arm64.md`, held
by the orchestrator's next kernel worker.
33 changes: 33 additions & 0 deletions issues/the-kernels-generator-is-keyed-once-and-never-reseeded.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
---
status: open
kind: defect
opened: 2026-09-24
---

# The kernel's generator is keyed once at boot and never reseeded, and a source is checked only for a constant

`kernel/src/random.rs` keys a ChaCha20 generator (`toyos-random`) once, before
the first hash container, from the seed the loader read from firmware's
`EFI_RNG_PROTOCOL` and from the CPU's own sources: `RDSEED` and `RDRAND`, or
`RNDR`. The hash seed and every `SYS_RANDOM` byte descend from that key. What
is still owed:

- **Nothing reseeds.** Each draw replaces the key, so the generator's memory
read later gives no earlier draw; but a key read once gives every later draw
until the machine restarts. Nothing mixes a fresh draw in on a schedule or
after a resume.
- **The sources are firmware and the CPU, and nothing else.** There is no
jitter source. A guest under HVF has no CPU source, so its key is whatever
the host's generator gave QEMU's virtio-rng when edk2 read it, once.
- **The health test is one comparison.** A source whose 32 bytes are four
equal words is refused by name: that is the all-ones `RDRAND` some AMD parts
returned after a resume, and a source stuck on one value. A source that
repeats with a longer period, or is biased, is mixed.
- **A key's copies outside the named buffers are not wiped.** The generator,
a draw's stream and every block buffer are zeroed with volatile writes; what
the compiler spilled of them to a stack frame or left in a register is not.

**Exit**: the generator is reseeded on a schedule from every source the
machine has, a jitter source among them. At boot and at each reseed a health
test refuses a source that repeats or is stuck, by name. A host test feeds a
reseed a stuck source and shows it refused and the key it had kept.
64 changes: 53 additions & 11 deletions issues/toyos-runs-on-arm64.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,17 @@ Every x86 guest on this host runs under TCG emulation instead — there is no
HVF on this Mac?", he answered: "Yes, once ARM userland boots
(Recommended)".

## The default architecture

`cargo run` and the build pick x86-64 when no architecture is named, whatever
the host (`src/build.rs`'s `arch_for`, the `None => Arch::X86_64` arm), and
the owner's word of 2026-10-09 makes the default the host's own, which
`src/arch.rs`'s `Arch::HOST` already knows. It cannot be flipped without
taking the desktop away from an Apple-silicon host: the AArch64 image has no
virtio devices and no userland servers yet (stages 6 and 7). The flip is an
exit item of stage 7, and until then the default staying x86-64 on an ARM
host is a weakness of this track, not a choice.

## Measured, on `main` at `03b1b4db`

**Size.**
Expand Down Expand Up @@ -277,17 +288,39 @@ Each stage names its exit; "measured" means a number from a run.
the kernel's own tables (`TTBR1_EL1` holding memory and nothing else, each
user space on `TTBR0_EL1` under a 16-bit ASID from `kernel/pure/pcid`), the
GICv3's SGIs and the virtual timer's PPI, the EL0 entry, and the context
switch carrying FP/SIMD; the `virt_` tests other than `virt_early_panic`,
`virt_early_fault` and `virt_el2_drop` judge it under the EL2 profile, emulated, because HVF
exposes no RNDR and the kernel's hash seed refuses there until stage 6's
virtio-rng. Each judges an event, never a rate: no QEMU test measures time.
switch carrying FP/SIMD. The `virt_` tests judge it under HVF on an Apple
host and emulated at EL1 elsewhere, the kernel's generator keyed from the
seed the loader reads from firmware's `EFI_RNG_PROTOCOL`, which edk2
answers from a virtio-rng (`kernel/src/random.rs`); `virt_el2_drop`,
`virt_smp` and `virt_jobs_at_el2`, one boot of the job case with the timer
and FP jobs in it, stay emulated at EL2, which HVF gives no guest, and
`virt_reboot` and three more stay emulated at EL2 until
`issues/the-boots-last-word-can-miss-the-console-when-klogd-holds-the-wire.md`
is fixed. Each judges an event, never a rate: no QEMU test measures time.
**Accepted with the move to HVF:** `virt_user_mode`, `virt_irq_storm`,
`virt_timer_floor`, `virt_failed_ap_leaves_no_hole` and
`virt_fatal_halts_the_others_first` no longer boot entered at EL2. The
entry's drop from EL2 stays judged by `virt_el2_drop`, `virt_smp` and
`virt_jobs_at_el2`, and PSCI through the SMC conduit by `virt_smp`
(`CPU_ON`, `CPU_OFF`, `SYSTEM_OFF`) and `virt_reboot` (`SYSTEM_RESET`);
what those five judge after the entry is the same kernel at EL1 either way.
Owed before the exit holds: the interrupts-off window against x86's, a
measurement only metal can make, with no instrument on either arch yet; the
instruction-cache maintenance before a mapping is executable
(`cache::make_executable`), the break-before-make ordering of a live
entry's replacement, and the TLB flush before a reclaimed ASID is issued
again, which QEMU's TCG, the only oracle this stage has, cannot fail on:
the first HVF run, once stage 6 gives HVF its RNDR, is their exit; and the
again, which QEMU's TCG cannot fail on. Under HVF since the entropy stage
every program the `virt_` tests run at EL0 is mapped executable through
`cache::make_executable`, with no test red, which is no proof: a stale
instruction is not certain to show in one boot. Each of the three closes
on a guest test under HVF that is red with its step deleted, and stays
owed with that test until one is: for `make_executable`, a program that
runs code it wrote over code it ran at the same address, on a host whose
`CTR_EL0.DIC` the test reads and says clear; for break-before-make, two
CPUs, one reading a page whose live entry the other replaces with
another frame's, every read the old frame's value or the new one's; for
the ASID flush, a test kernel whose ASIDs an actuator bounds to two, and
three processes each reading back its own frame at one address. And the
three deletions shown red. They are shown red on a machine whose
firmware leaves the registers otherwise, or by a loader that writes the
opposite values before the handoff. The ITS moves to stage 6: a claimed
Expand Down Expand Up @@ -317,13 +350,21 @@ Each stage names its exit; "measured" means a number from a run.
(`sched/dump.rs`'s `probe_silent`), which reaches `irqchip::send_nmi`'s
`owed!` on a machine of more than one CPU, and which nothing but the
`dump-deaf-cpu` actuator asks for until AArch64 has a keyboard; and, for
the first HVF run, the clean of an AP's start block to the point of
coherency, which TCG cannot fail on.
the clean of an AP's start block to the point of coherency, which TCG
cannot fail on: `virt_el1_smp` under HVF started eight CPUs through PSCI
in each of 131 boots of the entropy stage's measurement, all eight online
and scheduling every time, and `virt_failed_ap_leaves_no_hole` and
`virt_fatal_halts_the_others_first` start theirs under HVF in the suite;
it stays owed until `virt_el1_smp` under HVF is shown red with the clean
deleted.

6. **Virtio on `virt`.** virtio-pci (ECAM from MCFG) for blk, net, gpu,
sound, input and rng. virtio-input replaces the i8042 as the
key-transition source. virtio-rng, or SMCCC TRNG, feeds `sys_random`
alongside RNDR. SMMUv3 is on `virt` (`-M virt,iommu=smmuv3`), decoded from
key-transition source. The rng is done, and is no ToyOS driver: edk2
drives the virtio-rng, the loader reads the seed once, and the kernel's
generator is keyed from it and RNDR (`kernel/src/random.rs`); SMCCC TRNG
is absent from QEMU 11.1.1's `virt` under HVF and TCG alike
(`TRNG_VERSION` answers -1 through HVC). SMMUv3 is on `virt` (`-M virt,iommu=smmuv3`), decoded from
IORT. **Exit**: netd claims its NIC through an SMMUv3 domain; a
foreign-DMA test faults into a `DMA FAULT` record, not a crash.
Stage 0's three DMA-ordering fixes (NVMe's phase before its body, xHCI
Expand All @@ -350,7 +391,8 @@ Each stage names its exit; "measured" means a number from a run.
**Exit**: the desktop comes up on virtio-gpu; `calc`, `snake` and `doom`,
each started on it, each map a window and present a frame, read by a test
that is red when one of them does not; `ssh` works from the host; `/log` survives a reboot; the same `system.toml`
drives both arches.
drives both arches; `cargo run` with no architecture named boots the
host's.

8. **The harness boots aarch64.** `tests/common/qemu.rs` takes an
`Arch`: `virt`, edk2-aarch64, HVF on Apple hosts (TCG otherwise).
Expand Down
1 change: 1 addition & 0 deletions kernel/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -498,6 +498,7 @@ toyos-tsc = { path = "../toyos-tsc" }
toyos-ps2 = { path = "ps2" }
toyos-rootimage = { path = "../toyos-rootimage" }
toyos-quiesce = { path = "../toyos-quiesce" }
toyos-random = { path = "../toyos-random" }
toyos-symbols = { path = "../toyos-symbols" }
toyos-untrusted = { path = "../toyos-untrusted" }
toyos-userbound = { path = "../toyos-userbound" }
Expand Down
Loading
Loading