Fatal paths write the console UART only under its registers, whose lock knows which CPU's fatal path holds it

[Japabu]

nested_nmi_is_loud failed under KVM in CI. The cause is in the kernel: nested_nmi wrote its report through serial::panic_raw, which took no lock, while klogd on another CPU was writing records to the same 16550. On this branch:

• Every fatal path writes the console UART through a serial::PanicUart, which it gets only by asking for the console's registers.
• Every arch::console_uart function that touches the UART takes a serial::Registers, which only serial.rs builds.
• The registers' lock knows which CPU's fatal path holds it:
  • a fatal path entered on top of its own CPU's hold has the registers at once;
  • a burst asked for beneath that hold is refused loudly instead of spinning for ever.
• The CPU id the lock reads is held, at each AP's bring-up, against the id its roster slot and every IPI use; on x86-64 the boot CPU's is too.

Root cause

• Two KVM runs went red on this branch's base code. Each base is an ancestor of 0edf266ab: 59052827f is the merge base of The guest suite is every pull request's guest check, on toolchain stores CI caches by the build system's own keys; a landing during a nightly does not red its release #671's run head 09bda2f3c, and the other is 06788146b. git log <base>..0edf266ab touches none of serial.rs, serial_lock.rs, console_uart.rs, nmi.rs, log/, panic.rs, sched/driver.rs, or the harness's faults.rs, qemu.rs and serial.rs. The guest suite is every pull request's guest check, on toolchain stores CI caches by the build system's own keys; a landing during a nightly does not red its release #671's own diff touches none of them either.
  • The guest suite is every pull request's guest check, on toolchain stores CI caches by the build system's own keys; a landing during a nightly does not red its release #671's guest check, run 36863809437, job 110375742604, timed out after 39 s. NESTED NMI never arrived whole.
  • main's nightly guest lane at 06788146b, run 36843762360, job 110374194368, ended with QEMU died before NESTED NMI (status 0) after 63 s.
• In the first run, the first spliced line is exactly a byte interleaving of cpu1's record [kernel 0.385 cpu1] CPU 1: joining scheduler and the report's [nmi] NESTED NMI on (64 = 44 + 20 bytes). This was checked by an interleaving test against the line as that job's log carries it.
• drivers/serial.rs holds the registers' lock for one burst, and klogd writes every UART byte under it. The panic path "takes the registers alone, and bypasses them once they stay held". panic_flush did that. The report did not, and neither did panic::last_words, percpu::ist1_report, panic_reboot::arm's raw lines or panic_reboot::reboot_now.

Change

• The lock (kernel/src/drivers/serial_lock.rs, which kernel-loom compiles under loom). Its word is free, LIVE (a burst holder), or FATAL plus the CPU whose fatal path holds it. The lock reads that CPU itself, through crate::arch::cpu::hardware_id. kernel-loom supplies that function as a per-model-thread shim (arch::cpu::become_cpu). No caller passes a CPU id.

  • seize(tries) is the fatal paths' acquire. It answers one of three ways:
    • Taken: the lock was free, or let go inside the bound.
    • Reentered: this CPU's own fatal path already holds it.
    • Expired: another holder kept it through every try.
  • The bound is within, the one bounded wait for a console lock (seize, and flush_final's wait for the wire). It asks an attempt that never answers exactly tries times. Under feature = "loom" its pause is loom::hint::spin_loop, imported beside the loom atomics.
  • lock(), a burst's acquire, asserts that the word is not this CPU's FATAL. A burst beneath its own CPU's fatal hold would wait for a frame that cannot let go while it waits. The review's last_words patch produced exactly that hang: the hold was kept across the record, and before klogd runs the record's inline drain is a burst on the same CPU. Now the assertion's panic ends in the reentry guard's raw report, which names it.

• kernel-loom builds the lock, the roster and the CPU shim both read under loom alone. Neither not-loom test (log_zeroed_init, log_body_words) names them. So the shim's not-loom arm, which no src/clippy.rs shape built, is deleted rather than given a shape, and without its gate the roster does not build there (g1). The not-loom arms of the lock and the roster are the kernel's, which the kernel's ten shapes lint.

• The registers' token.

  • arch::console_uart's init, rx_ready, tx_ready, read_byte and write_byte take &mut serial::Registers, on both architectures. AArch64's init touches no register; it takes them for the call it shares with the 16550's. AArch64's frame() answers the frame's address and touches no register.
  • serial::init asks for them with BackendGuard::lock. What the architecture's init logs under it is not drained there: drain_inline returns while has_console is false, which it is until UART_PRESENT is stored from that call's answer.
  • Registers(()) has a private field, so only serial.rs builds one: at BackendGuard::lock, and in panic_registers.
  • virtio-console's write_bytes_locked, try_read_byte_locked and has_data_locked take the BackendGuard itself.

• Each CPU's id read, at its bring-up (kernel/src/smp_roster.rs). The lock and the panic path's per-CPU slots (panic.rs) name a CPU by arch::cpu::hardware_id. On x86-64 that is CPUID's answer, while the roster and every IPI use LAPIC ids: the boot CPU's own (apic::id), and the MADT's for the APs.

  • Roster::echo(token) runs on the AP and reads that CPU's arch::cpu::hardware_id itself, as the lock does. The read and the token are one AtomicU64 store, so the read cannot land after the token, and no caller picks which read the echo carries: the review's swapped arguments do not build (w1), and apic::id() in place of the read builds for x86-64 alone, while the AArch64 kernel and kernel-loom, both compiled by the host gate, refuse it (w2).
  • Roster::commit refuses an AP whose echoed read is not the id it commits the AP under: the one its roster slot holds and every IPI names it by, the MADT's LAPIC id, or on AArch64 the affinity of the MADT's MPIDR. Both architectures commit only through it.
  • x86-64's boot_aps holds the boot CPU's own read against its LAPIC id before the roster takes it.
  • On AArch64 the boot CPU's roster slot is its read (ROSTER.set_bsp(me)), which also picks its GIC redistributor (irqchip.rs:184), so nothing independent holds it. That is filed.
  • Each refusal is the boot CPU's panic, before smp::set_ready, so a wrong read stops the boot. It is not the AP's own: before the release an AP's panic stops no other CPU (stop_other_cpus sends nothing then), and the boot CPU boots on without it once await_echo runs out.

• PanicUart (write, hex, dec): panic_registers() is its only constructor. On Expired it writes one raw line before anything else is written over the holder.

• panic_flush drains through virtio-console only under registers it took clean.

  • Under Reentered, the outer path may have stopped inside a virtqueue publish, so the flush says so in one raw line and drains raw, as it does after Expired.
  • For that reason nested_nmi lets its report's registers go before halt_all_cpus, and last_words lets them go before its record.

• nested_nmi_is_loud reads through the halt's flush to the reboot's arm line (panic: rebooting in), its ready marker.

  • It asserts the report's three lines whole and back to back.
  • It refuses both lines the registers write when they were not clean (faults::UNCLEAN).
  • Its verdict is faults::report.

• tests/checks.rs nested_nmi_verdict runs that verdict on the host:

  • on a whole report;
  • on the spliced capture CI's KVM lane recorded in run 36863809437, which must be refused;
  • on a burst of another CPU's line, with no line end, cut into the middle of each of the report's three lines in turn;
  • on each refused line.

  It also holds the refused words to kernel/src/drivers/serial.rs.

• Issues.

  • Filed:
    • issues/panic-path/a-fatal-path-inside-its-own-cpus-console-burst-waits-out-the-bound.md
    • issues/panic-path/a-panic-write-between-two-console-bursts-can-split-a-character.md
    • issues/kernel/a-safe-mmio-window-reaches-any-physical-address.md
    • issues/kernel/a-safe-port-read-reaches-any-io-port.md
    • issues/panic-path/a-panic-reentry-inside-a-fatal-paths-console-hold-halts-with-it-held.md
    • issues/kernel/the-aarch64-boot-cpus-id-read-is-held-against-nothing.md
  • Main's nightly run 36843762360 failed nested_nmi_is_loud in its KVM lane (guest, job 110374194368). Its capture stops in the line before the — of cpu1's i8042: record, and that — lies inside one 16-byte burst: the unlocked report could split it there, and the report on this branch cannot, so the branch closes that instance.
  • issues/panic-path/a-panic-write-between-two-console-bursts-can-split-a-character.md, filed here and left open, is another way for nested_nmi_is_loud to fail under KVM: a report taken between two bursts can still split a character that straddles them.
  • No run of this branch measures the KVM arm; the nightly dispatched on main after this lands does.

Where each property is held

• Unrepresentable. Each of these fails to build (mutations 3a to 3g):

  • a call into arch::console_uart that touches the UART without the registers: init, rx_ready, tx_ready, read_byte or write_byte, on either architecture;
  • a forged Registers;
  • a virtio-console *_locked call made without a BackendGuard.

  The type does not reach a module that touches the registers by itself. Such a module could use unsafe { outb(0x3f8, …) }, which breaks outb's contract that the caller owns the port. It could also use a safe inb(0x3f8), which takes a received byte, or on AArch64 a safe Mmio::new over console_uart::frame(). Both are filed.

• Loom (kernel-loom/tests/serial_lock.rs, nine models) holds:

  • within's count;
  • seize's bound, and the wait inside it;
  • seize's re-entry, and its exclusion against a burst;
  • the refusal of a burst beneath its own CPU's fatal hold;
  • a burst on another CPU waiting that hold out.

  The CPU id read is the one inside the lock, and the models drive it from two CPUs.

• Loom, bring-up (kernel-loom/tests/smp_bringup.rs, an_ap_that_reads_another_id_is_refused): an AP whose CPU reads 5 (become_cpu(5)) echoes on its own thread, the boot CPU awaits that echo with Roster::await_echo and commits the AP as 7, and the commit must panic with the refusal. m1 reds it. The read and the token are one store, so there is no second store left to swap.

• The bounded wait at three preemption bounds. The model of the wait inside the bound runs at preemption bounds 0, 1 and 2, with k + 2 tries at bound k.

  • The review asked for seize(_, 2) with Taken in every execution. Under loom's default, unbounded exploration, no try count makes Taken hold in every execution. Measured with two tries: LOOM_MAX_PREEMPTIONS=0 exit 0, =1 exit 101, =2 exit 101.
  • Loom's trace of the red execution shows the cause. The seize looks, yields to the holder, and runs again before the holder's store executes. That is the holder preempted by the waiter, and each such preemption costs the seize one try.
  • The size of PANIC_LOCK_SPIN_LIMIT, a count and not a time, is issues/panic-path/the-panic-lock-spin-limit-is-a-count-its-comment-calls-a-second.md.

• Host check: nested_nmi_verdict holds:

  • the verdict;
  • each of whole's clauses alone (4a to 4d);
  • the kernel's refused words (r4).

• Every boot: each AP's id read against its roster id (7a to 7c), and x86-64's boot CPU's against its LAPIC id (7d, 7e).

• Guest:

  • A second NMI on IST2. It enters through the naked nmi_entry.
  • The registers let go before the halt (row 5). A guest run reds it. Refusing it at compile time is not clean: a token halt_all_cpus takes by value must reach all eleven of its callers, and a mint they can all reach, nested_nmi can reach a second time, so keeping the registers into the halt would still compile.
  • The report meeting another CPU's klogd on one 16550. Only KVM shows this: TCG was green on the base.

• Metal: none. The T14 has no 16550.

Mutations

Each mutation went through one script:

1. The patch was checked with git apply --check.
2. It was applied, then built, then run.
3. It was reversed, and the script showed the tree's diff and untracked set unchanged.

Where each row was measured:

• At the head, 3b961d5bf: m1, w1, w2 and g1, and the builds of 7a, 7b, 7d and 7e. Every other build and host run at d31faab90; git diff d31faab90 3b961d5bf touches only Roster::echo and its two callers, kernel-loom's gate on the roster, the two models that echo, and their doc comments.
• The guest runs, the orchestrator's under TCG: 7a to 7e at d31faab90, and row 5 at 1a9380fed. Row 5's red stands for the head: git diff 1a9380fed 3b961d5bf touches neither kernel/src/arch/x86_64/idt/nmi.rs nor kernel/src/drivers/serial.rs.

The commands were:

• loom runs: cargo test -p kernel-loom --test serial_lock, and --test smp_bringup for m1;
• kernel-loom builds: cargo build -p kernel-loom --lib, with --no-default-features where a row says so, and cargo test -p kernel-loom --test smp_bringup --no-run;
• check runs: cargo test --test toyos-checks nested_nmi_verdict;
• kernel builds: cargo build --target x86_64-unknown-none --features boot-actuators in kernel/, and aarch64-unknown-none-softfloat where a row says so; w1 and w2 with default features.

"(review)" marks the patch a review named.

#	mutation	build	run, and what fails
1a	compile_error! planted in kernel-loom's arch::cpu shim	cargo build -p kernel-loom --no-default-features --lib: 0. Without --no-default-features: 101, "planted in the cpu shim"	—
1b	assert!(true); planted in the shim's become_cpu	cargo clippy -p kernel-loom --all-targets with the workspace shape's lint arguments: 101, clippy::assertions_on_constants	—
2a	seize's Err(_) => None → Err(_) => Some(Seized::Expired) (review)	0	101: a_seize_takes_a_burst_let_go_inside_the_bound
2b	within's 0..tries → 0..tries.min(1) (review)	0	101: the same, and within_asks_a_silent_attempt_exactly_its_tries ("a wait bounded at 100 tries asked 1 times")
2c	within's pause is core::hint::spin_loop, not loom's	0	101: a_seize_takes_a_burst_let_go_inside_the_bound
2d	within's 0..tries → 0..tries.min(4) (review)	0	101: within_asks_a_silent_attempt_exactly_its_tries ("a wait bounded at 100 tries asked 4 times"). With that model skipped, the other eight pass: 0
3a	crate::arch::console_uart::write_byte(b'!'); at the head of nested_nmi	101: error[E0061], "argument #1 of type &mut serial::Registers is missing"	—
3b	the same with &mut crate::drivers::serial::Registers(())	101: error[E0603], "tuple struct constructor Registers is private"	—
3c	crate::drivers::virtio_console::write_bytes_locked(b"!"); there	101: error[E0061], "argument #1 of type &mut BackendGuard is missing"	—
3d	crate::arch::console_uart::rx_ready(); there	101: error[E0061], "argument #1 of type &mut serial::Registers is missing"	—
3e	crate::arch::console_uart::tx_ready(); there	101: the same	—
3f	crate::arch::console_uart::init(0); there	101: the same	—
3g	crate::arch::console_uart::tx_ready(); at the head of AArch64's ap_entry, built for aarch64-unknown-none-softfloat	101: the same	—
4a	whole's body → report.len() == 3 (review)	0	101: "a burst inside the report's line 1 was read as a whole report"
4b	whole's first-line clause dropped	0	101: line 1
4c	whole's registers clause dropped	0	101: line 2
4d	whole's last-line clause dropped	0	101: line 3
5a	lock()'s refusal deleted	0	101: a_burst_beneath_its_own_cpus_fatal_path_is_refused, with loom's "Model exceeded maximum number of branches" in place of the refusal
5b	last_words' block braces dropped, so its registers live across the record (review)	0	No host test compiles panic.rs, and no test boots a DOUBLE PANIC. By reading, the hang it made is now 5a's refusal: alert! → log::emit → drain_inline → write_wire → BackendGuard::lock → BackendLock::lock on this CPU's FATAL word.
6a	the lock's own read crate::arch::cpu::hardware_id() → 0 (review)	0	101: a_seize_reenters_its_own_cpus_hold ("a fatal path took another CPU's hold") and a_burst_waits_out_another_cpus_fatal_path (the burst is refused on CPU 2)
7a	x86-64 cpu::hardware_id's return edx; → return edx & 0; (review; its literal return 0; fails -Dwarnings on the unused edx, build 101)	0, also with default features	guest, nested_nmi_is_loud: 0/1, the boot CPU's panic, "assertion left == right failed: smp: cpu1 reads its own hardware id as 0x0, and its roster slot and every IPI name it 0x1"
7b	AArch64 cpu::hardware_id's packed_affinity(mpidr) → packed_affinity(mpidr) & 0, built for aarch64-unknown-none-softfloat	0, also with default features	guest, virt_smp: 0/1, the same panic
7c	7a with the AP check deleted: the refusal in Roster::commit at the head	0, also with default features	guest, nested_nmi_is_loud: 1/1 green
7d	x86-64 cpu::hardware_id's return edx; → return if edx == 0 { 1 } else { edx };, so QEMU's boot CPU alone reads 1 (review)	0, also with default features	guest, nested_nmi_is_loud: 0/1, the boot CPU's panic in boot_aps, "assertion left == right failed: smp: the boot CPU reads its own hardware id as 0x1, and its roster slot and every IPI name it 0x0"
7e	7d with the boot CPU's check deleted from boot_aps (review)	0, also with default features	guest, nested_nmi_is_loud: 1/1 green
m1	the refusal deleted from Roster::commit (review)	0	101: an_ap_that_reads_another_id_is_refused, "test did not panic as expected"; the other two smp_bringup models pass
w1	x86-64's ap_entry echoes with the review's swapped arguments, ROSTER.echo(cpu::hardware_id(), percpu::ap_token())	101: error[E0061], "this method takes 1 argument but 2 arguments were supplied"	—
w2	Roster::echo reads crate::arch::apic::id(), the review's other mistake	x86-64: 0. AArch64: 101, and kernel-loom's smp_bringup: 101, each error[E0433], "cannot find apic in arch"	—
g1	kernel-loom's smp_roster without its #[cfg(feature = "loom")]	--no-default-features: 101, error[E0433], "cannot find cpu in arch"; default features: 0	—
r1	seize → return Seized::Taken(self.lock()); (round 2's row)	0	101: a_burst_beneath_its_own_cpus_fatal_path_is_refused, a_seize_gives_up_on_a_holder_that_never_lets_go, a_seize_reenters_its_own_cpus_hold
r2	within's for _ in 0..tries → loop	0	101: a_seize_gives_up_on_a_holder_that_never_lets_go, a_seize_reenters_its_own_cpus_hold, within_asks_a_silent_attempt_exactly_its_tries
r3	seize loses its Reentered arm	0	101: a_seize_reenters_its_own_cpus_hold ("a fatal path waited out its own CPU's hold")
r4	DRAINED_RAW drifts (own dropped)	0	101: serial.rs "writes no" the refused line
row 5	nested_nmi's block braces dropped, so its registers live into halt_all_cpus (sha256 f718ce25…)	0, also with default features	guest, nested_nmi_is_loud: 0/1, "the report's registers were not clean: [serial] this cpu's own fatal path held the console registers; drained raw"

Gates

At 3b961d5bf:

• cargo run -- --ci host: EXIT=0, [ci] Host: 56 step(s), all green.
  • Its clippy step ran 17 shapes, clean, ten of them the kernel's, four of those for aarch64-unknown-none-softfloat.
  • Its workspace step ran kernel-loom's serial_lock (9 passed) and smp_bringup (3 passed).
  • It ran toyos-checks: nested_nmi_verdict ok.
  • It ran kernel-loom without loom (log_zeroed_init, log_body_words): 3 passed.
  • It ran the serial-try-lock-then-some control (2 verdicts reached), and smp_bringup's roster-commit-relaxed and smp-ready-split (1 verdict each).
• cargo run -- --clippy: EXIT=0, clippy: 17 invocations clean.
• cargo run -- --build-only: EXIT=0; with --arch aarch64: EXIT=0.
• cargo test -p kernel-loom, every model: EXIT=0.

Guest, the orchestrator's runs under TCG: at d31faab90, the whole suite 21/21; 7a 0/1, 7b 0/1 and 7d 0/1, each on its predicted line; 7c 1/1 and 7e 1/1. At 1a9380fed, row 5 0/1 on its predicted line.

Device driver and a lock on the panic path: the two checks

• Negative control:
  • The whole change reverted is main's kernel. KVM recorded it red twice (above).
  • Every mechanism has a mutation above, and every host mutation reds except 5b. 5b is the review's patch, whose effect is now the refusal 5a holds.
  • Row 5 reds under TCG.
  • The bring-up check: 7a and 7b red where 7c, the same read with the check removed, stays green; m1 reds the loom model when the refusal is deleted from Roster::commit.
  • x86-64's boot CPU's check: 7d reds where 7e, the same read with the check removed, stays green.
• Independent oracle:
  • loom, a third-party model checker, explores every interleaving of seize, lock and a burst within the bounds stated above.
  • rustc's type and privacy checks refuse 3a to 3g.
  • The architecture manuals, now held at bring-up. Re-entry rests on arch::cpu::hardware_id differing on every CPU. On x86-64 that is the x2APIC id CPUID leaf 0BH or 1FH reports in EDX for the current logical processor, which the SDM gives as the one the local APIC's ID register holds (leaf 1's initial APIC id on a CPU without those leaves); on AArch64, MPIDR's affinity. Each AP's read is now held against the id it was started by, and on x86-64 the boot CPU's against its LAPIC id. No test puts two CPUs on fatal paths at once.
  • The recorded KVM failures are an oracle of the defect only. TCG cannot tell the base from the fix: the base was green under it on the dev host and in the nightly tcg lane. The fix's KVM arm is a nightly the orchestrator dispatches on main after this lands.

Unsure

• The KVM green arm is not measured: it is a nightly the orchestrator dispatches on main after this lands.
• The bring-up check on metal. It runs on every boot, the T14's included, and no metal run has measured it. Its claim that CPUID's x2APIC id is the LAPIC's own rests on the SDM, and on Linux's reading of the T14, toyos-cpuvuln/fixtures/t14/cpuinfo.txt: processors 0 to 7 give apicid and initial apicid alike as 0, 2, 4, 6, 1, 3, 5, 7, the T14's MADT order (MADT cpus=[0, 2, 4, 6, 1, 3, 5, 7], issues/hardware/a-metal-session-runs-a-pre-flash-gate-first.md:62).
• 5b. What the review's last_words patch now does is established by reading, not by a run: no test boots a DOUBLE PANIC, and I added none. A DOUBLE PANIC needs a guest. The refusal such a panic would end in is held by loom (5a).
• When a burst holder on this CPU never lets go, every panic_registers on that path waits out the bound again. This is filed.
• A panic inside a fatal path's hold, before halt_all_cpus has stopped the other CPUs, halts its CPU with the hold kept. This is filed. The refusal is one more way in: a log! under a hold before klogd runs.

🤖 Generated with Claude Code

https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L

[Japabu]

Readiness: CI host is green at 21e3fc8. Run 36871322929's job and its cargo run -- --ci host step both concluded success; gh pr checks still lists the draft-time skipped run 36871310254. nested_nmi_is_loud, the test this branch changes, has no run at this head under any tier. By reviewer.md that is NOT READY FOR REVIEW. The brief asks for these judgments now, so the review follows, and the missing run is the first BLOCKER.

Net: +106/−13. Production +20/−7, tests +28/−6, issues +58.

BLOCKER

• tests/common/faults.rs:10 — nested_nmi_is_loud has no run at 21e3fc8, so "another CPU's burst never lands inside it" stands on no measurement of the fixed kernel. TCG cannot tell: the base passed it there (nightly 36843762360, tcg job 110374194382), and the KVM arm is owed. Close with a TCG run green at this head, and, before landing, the KVM guest run green at this branch's head (The guest suite is every pull request's guest check, on toolchain stores CI caches by the build system's own keys; a landing during a nightly does not red its release #671's PR job once this branch carries it).
• kernel/src/arch/x86_64/idt/nmi.rs:153 — patch: let registers → let _registers at :143, and delete drop(registers);. It stays green, because the capture ends at STOPS. After it, halt_all_cpus's panic_flush spins out the bound on this CPU's own hold and bypasses without a word, so reading on would not tell either. panic_registers' expiry must say itself in one raw line; it is silent today in both callers, against the brief's "fail loud". nested_nmi_is_loud must then read on to the flush's last record (panic: rebooting) and refuse that line, and the patch must turn it red.
• kernel/src/drivers/serial.rs:133 — patch: the body becomes Some(BackendGuard::lock()). It stays green in every suite: nothing puts the registers on the NMI'd CPU's interrupted frame (the first NMI comes from the idle loop, kernel/src/sched/driver.rs:694, which holds nothing), and no actuator wedges a holder for panic_flush. So the no-deadlock claim at nmi.rs:141-142, "bounded, so a hold the interrupted context left only delays it", cannot fail. Cheapest red: move the bounded acquire into serial_lock.rs's BackendLock, where kernel-loom goes red when the same patch removes the bound. Otherwise, a guest arm that sends the first NMI under a held BackendGuard.
• PR body, "Why this stays a guest test" — the type reason is a cost ("means changing every caller of panic_raw"), not a reason a type cannot reach the behaviour, and reviewer.md refuses a cost reason. The type: a token that panic_raw, panic_raw_hex and panic_raw_dec require, handed out by panic_registers (held or expired), with bypassed() named at every other raw writer. It makes a report written without asking unrepresentable, and it is the enforcement the rule proposal wants. Those writers sit outside the implementer's fence, which the orchestrator moves or rules on.

NOTE

• kernel/src/arch/x86_64/idt/nmi.rs:143 — no deadlock by reading. try_lock never blocks, and stop_other_cpus' fixed (maskable) 0xFD IPI cannot stop a holder, since BackendGuard keeps IF clear. A holder that halted itself, or this CPU's own interrupted frame, costs the whole bound here and again in panic_flush. The bound is 100M try_lock+pause with no unit of time, and this is its third consumer (issues/panic-path/the-panic-lock-spin-limit-is-a-count-its-comment-calls-a-second.md). Nothing at that point runs under PANIC_BOUND_MS, which halt_all_cpus arms later; kernel/src/main.rs:160's flush before the halt is in the same position.
• kernel/src/drivers/serial.rs:180 — the registers have one wait left: panic_flush and nested_nmi share panic_registers. But flush_final keeps the same bounded loop over PANIC_LOCK_SPIN_LIMIT for the wire. If panic_registers took the try-function, it would serve both and delete the copy.
• issues/panic-path/a-fatal-report-written-raw-can-splice-into-another-cpus-console-burst.md:14 — stays filed: the same mechanism is wrong for it. On a reentry inside panic_flush's drain, last_words would wait out its own CPU's hold, and the exit needs a holder-aware wait in the loom-checked lock. The list omits two raw writers: panic_reboot::arm(false)'s three raw lines (kernel/src/main.rs:123, the reentry arm, which never stops the others) and panic_reboot::reboot_now.
• issues/panic-path/a-panic-write-between-two-console-bursts-can-split-a-character.md:9 — a different mechanism, and cheap here: end each uart_write_fifo burst on a character boundary, a pure function with a host test. It is not live for this boot: kernel/src/arch/x86_64/i8042/mod.rs:303's — falls at bytes 71-73 of its record, inside burst 64-79, and write_line hands write_wire a whole record at once.
• kernel/src/drivers/serial.rs:8 — the rule proposal belongs in neither CLAUDE.md nor a comment at panic_registers. This module header already says "The panic path takes the registers alone, and bypasses them once they stay held", and four raw writers break it today. The token above is its enforcement.
• PR body, the two checks — the body says TCG cannot tell. The recorded captures are an independent oracle of the defect, and they show the new assertion red on the base: run 36863809437's three lines before STOPS are the spliced report, and run 36843762360 never reaches STOPS. Both red bases carry this base's kernel and harness files: git log from either base to 0edf266ab over them is empty. They measure nothing about the fix.

REMOVE

• kernel/src/arch/x86_64/idt/nmi.rs:14 — "and it obeys this file's discipline exactly — no lock, no allocation, nothing that logs" — false now that nested_nmi in this file spins on BackendGuard::try_lock.
• PR body, Gates and the two checks — the scratchpad paths (…/orch/nmiloud/negctl-report-unlocked.patch, nmiloud/jobs.txt) — main's record would cite files that go with the job.
• PR body, Unsure — "either run gh workflow run nightly.yml --ref wt/toyos-nmiloud and read the guest job, or" — a branch nightly the owner forbids.
• PR body, Gates — "It ran on this head's code; the two issue files were written or reworded afterwards, and no gate reads issues/." — false (src/build.rs:2821 resolves cited issue files), and run 36871322929 ran at this head anyway.

SEND BACK

[Japabu]

Orchestrator guest run at f9835df2f (TCG on the dev host, Homebrew firmware; it covers all seven requested fatal-path tests): the whole suite, exit 0.

test result: ok. 21 passed, 21 total (125.9s)

KVM evidence comes from the batch with #671 and #676, whose own CI run is the proof.

[Japabu]

Round 2 at f9835df2f. Readiness: CI host is green at this head. Run 36879963042, job 110428923696: the cargo run -- --ci host step concluded success ([ci] Host: 56 step(s), all green, checks::nested_nmi_verdict ... ok, kernel-loom serial_lock 5 ok, control serial-try-lock-then-some: 2 verdicts reached). Guest: the orchestrator's TCG whole suite at this head, 21 passed (comment 5934051852). KVM arm: the batch's, per the brief.

Net origin/main...HEAD: +416/−155. Production (kernel/src) +226/−146; tests (tests/, kernel-loom/tests) +142/−9; issues +48. Round 2 alone: production +221/−154, tests +127/−16, issues +20/−30. The production growth is the type and the CPU-naming lock that round 1 asked for. Accepted.

Round 1

• BLOCKER 1, tests/common/faults.rs:10 (no run at head) — CLOSED for this PR. The orchestrator's TCG whole-suite run at f9835df2f passed 21, with nested_nmi_is_loud and the six other fatal-path tests among them. The KVM green arm, the only arm that tells the fix from the base, is the batch's precondition.
• BLOCKER 2, kernel/src/arch/x86_64/idt/nmi.rs:142 (registers kept into the halt) — OPEN. The expiry now speaks, and the test reads to panic: rebooting in and refuses both lines. But the patch (the block's braces removed, row 5 of the table) has no red run, and the body records that honestly: "a guest run, requested". The batch's CI cannot cover it, because it boots the unmutated head. The red needs no race: the halt's flush finds this CPU's own FATAL word and writes DRAINED_RAW ahead of the arm line. One TCG run of nested_nmi_is_loud on the patched kernel closes it.
• BLOCKER 3, kernel/src/drivers/serial.rs:133 (unbounded acquire) — CLOSED. The bound is now BackendLock::seize's. Seized::Taken(self.lock()) and within's for → loop each turn kernel-loom serial_lock red (exit 101, three models), per the body's table.
• BLOCKER 4, the guest test's type reason — CLOSED. panic_raw* are deleted, PanicUart is the token only panic_registers builds, and the reasons given are capabilities, not cost. One layer down, the claim fails (second BLOCKER below).

BLOCKER

• kernel/src/drivers/serial_lock.rs:88 — the wait inside the bound is untested. Patch Err(_) => None, → Err(_) => Some(Seized::Expired),, or :96 0..tries → 0..tries.min(1). Either stays green in all five serial_lock models and every host check: no model has a holder let go while a seize is still looking. So Taken's "let go inside the bound" rests only on a KVM race, and that wait is the fix: it keeps klogd's burst out of the report. Must turn red: a model in which the lock is held as a burst when seize(0, 2) begins and the holder lets go on another thread, asserting Taken in every execution. Under feature = "loom", within's pause must be loom::hint::spin_loop (imported beside the loom atomics) so the holder runs between two looks, as two_writers_never_overlap yields.
• kernel/src/arch/x86_64/console_uart.rs:72 — the type does not close the raw write. pub fn write_byte here and at kernel/src/arch/aarch64/console_uart.rs:85 is safe, and every kernel module reaches it as crate::arch::console_uart::write_byte (arch/mod.rs:13, arch/*/mod.rs:17). A fatal path that writes the UART without asking still compiles. So three claims are false one layer down: "A raw write that does not ask no longer compiles", "the one way a fatal path writes it" (serial.rs:124), and "PanicUart makes it unrepresentable". This writer is the sibling of the deleted panic_raw. It must need something only serial.rs hands to BackendGuard and PanicUart. Red: crate::arch::console_uart::write_byte(b'!') added to nested_nmi must fail the kernel build.
• tests/common/faults.rs:59 — the host check cannot fail on whole. Patch whole's body to report.len() == 3, and nested_nmi_verdict stays green: position at :47 refuses its one spliced capture before whole runs (no line starts [nmi] NESTED NMI on cpu ). Nothing on the host holds the registers line or the STOPS line, so a burst landing after the report's first line passes. Must turn red: nested_nmi_verdict on a capture whose first line is whole and whose second is spliced.
• kernel/src/panic.rs:271 — a brace is all that keeps out a hang this branch made possible (last_words held nothing before). Patch: drop the block's braces (:272, :317) so the registers live across the alert!s. Every tier stays green, because no test boots a DOUBLE PANIC. Before klogd runs, that alert! drains inline into BackendLock::lock(), which spins for ever on this CPU's own FATAL word. PanicUart's "Never held across a log!" (serial.rs:125) is the same rule, unenforced. Must turn red: an arm with a DOUBLE PANIC before klogd starts, or a structure under which the patch cannot be written.
• kernel/src/drivers/serial.rs:150 — CPU A cannot enter B's hold by reading: CPUID 0x1F/0xB x2APIC ids and MPIDR affinities are unique per CPU, and the compare is the whole word. Only the call site is unheld. Patch crate::arch::cpu::hardware_id() → 0 and every tier stays green: loom passes its own ids, and no test puts two CPUs on fatal paths at once. With the patch, a second fatal CPU writes into the first's report, and its flush disables virtio-console and drains raw under the first's live drain. Must turn red: an arm in which a second CPU's fatal path asks while the first's holds, refusing DRAINED_RAW.

NOTE

• kernel/src/drivers/serial_lock.rs:87 — re-entry is sound by reading. Reentered needs the whole word FATAL + this CPU, which only this CPU's seize stores and only its Held's drop clears. It builds no Held, so the inner path cannot release the outer hold. No path that gets it returns into the frame beneath: each ends in halt_all_cpus, the reentry guard's halt, or a reset.
• kernel/src/drivers/serial_lock.rs:95 — the bound has one owner: within with PANIC_LOCK_SPIN_LIMIT, called by seize and flush_final. flush_final's copy is gone, and no other console-lock wait counts tries.
• kernel/src/drivers/virtio_console.rs:95 — on "no pub raw writer anywhere": write_bytes_locked, try_read_byte_locked (:169) and has_data_locked (:194) are safe pub fns whose exclusion is only a doc comment. Each reaches the &mut VConsole in a static UnsafeCell, so if two CPUs call them without a BackendGuard, both hold it. This is pre-existing and outside the UART claim; the same proof argument closes it.
• PR body, "Closes issues/kernel/the-nested-nmi-report-interleaves-with-another-cpus-console-line.md" — on The guest suite is every pull request's guest check, on toolchain stores CI caches by the build system's own keys; a landing during a nightly does not red its release #671's head 2a0e045c4 that file is cited from issues/kernel/every-aarch64-guest-dies-at-the-kernels-entry-on-cis-firmware.md:26. The batch deletes both in one merge, once its KVM guest run meets that file's exit.

REMOVE

• kernel-loom/tests/serial_lock.rs:5 — "The console drain takes the backend with try_lock from every CPU that logs, so a losing attempt is the common case beside another CPU's write." False: klogd is the one console writer, and BackendLock::lock is try_lock's one kernel caller.

SEND BACK

[Japabu]

Guest runs at f71a5fd19, orchestrator's, this Mac (x86-64 under TCG), one at a time:

job	result
r3-row5: nested_nmi_is_loud with target/r3-row5.patch (sha256 f718ce25cfda03243cc1c2da216480faf1a01745feaace416e899035e489888d, row 5: the block's braces removed)	0/1, red: nested_nmi_is_loud: the report's registers were not clean: [serial] this cpu's own fatal path held the console registers; drained raw
r3-suite: whole suite, unmutated	21/21 (29.0 s)

Round 1's BLOCKER 2 now has its red run, for the reason the review predicted: the halt's flush finds this CPU's own FATAL word and drains raw ahead of the arm line.

[Japabu]

Round 3 at f71a5fd19. Readiness: CI host is green at this head. Run 36889923469, job 110462658634: the job and its cargo run -- --ci host step both concluded success. Guest at this head: the orchestrator's TCG runs (comment 5935483958) give r3-row5 red and r3-suite 21/21. The KVM arm is the batch's.

Net origin/main...HEAD: +627/−179.

• Production (kernel/src): +287/−168.
• Tests and the loom shim: +252/−11.
• Issues: +88.

Round 3 alone: production +88/−49, tests +123/−15, issues +40. The production growth is the Registers token and the lock reading its own CPU, which round 2's BLOCKERs asked for. Accepted.

Round 2's BLOCKERs

• 1, row 5 (nested_nmi's braces dropped) — CLOSED. r3-row5 at f71a5fd19 (patch sha256 f718ce25…) is 0/1, refused on [serial] this cpu's own fatal path held the console registers; drained raw. That line is the defect itself: the halt's flush finds this CPU's FATAL word.

• 2, the wait inside the bound — CLOSED. 2a, 2b and 2c each red a_seize_takes_a_burst_let_go_inside_the_bound (exit 101):

  • 2a: Err(_) => Some(Seized::Expired).
  • 2b: 0..tries.min(1).
  • 2c: core::hint::spin_loop as the pause.

  The bound is sound, and it is stated at the model (kernel-loom/tests/serial_lock.rs:60). Each look the seize takes before the holder's store costs loom one preemption of the holder. So k + 2 tries hold Taken at bound k, and no count holds it unbounded. The body's two-try measurement agrees: exit 0 at bound 0, exit 101 at bounds 1 and 2. The model holds that a release between looks is seen. It does not hold the 100M count (NOTE).

• 3, a raw write without asking — CLOSED. 3a (E0061), 3b (E0603) and 3c (E0061) each fail the kernel build at this head. Registers(()) is built only in serial.rs: at BackendGuard::lock and in panic_registers' three arms.

• 4, whole unheld on the host — CLOSED. 4a to 4d each red nested_nmi_verdict. A cut adds no line and keeps the NESTED prefix, so only the cut line's own clause can refuse it.

• 5, the last_words brace — CLOSED. 5a (the refusal deleted) reds a_burst_beneath_its_own_cpus_fatal_path_is_refused. 5b (the braces dropped) is still exit 0. By reading, it no longer hangs. Traced with 5b applied, a DOUBLE PANIC before klogd starts prints, and is bounded:

  • The three raw !!! DOUBLE PANIC lines go out under Taken.
  • The alert!'s inline drain reaches the assertion at serial_lock.rs:68 on this CPU's FATAL word. The path is log/mod.rs:220 → drain_inline → write_wire → uart_write_fifo or write_burst → BackendGuard::lock.
  • That panic is depth 1, so the reentry guard (main.rs:121) gets Reentered at once. It writes !!! PANIC REENTRY: CPU halted, whose second: line is that assertion and its message. Then arm(false)'s raw line (main.rs:123). Then it holds the panel to the bound, or halts.
  • All of that goes to the 16550, and none of it through the raw drain: no panic_flush runs. So a virtio-console console gets nothing after its last drained record, as for every reentry. The DOUBLE PANIC record's own drain is the burst that was refused.
  • The filed issue says the same ("writes under it and halts this CPU alone"). It does not describe a silent halt.
  • Before klogd starts, the APs wait in ap_idle, because log::console::start (main.rs:557) runs before smp::set_ready (:565). So the kept hold blocks no other CPU's burst there.

• 6, the CPU id read — CLOSED. 6a reds a_seize_reenters_its_own_cpus_hold and a_burst_waits_out_another_cpus_fatal_path. 6a is fatal_here's crate::arch::cpu::hardware_id() → 0 (serial_lock.rs:28). That read now sits in serial_lock.rs, which both the kernel and kernel-loom compile, so it is the call site round 2 found unheld. The read's own body is a NOTE.

BLOCKER

• kernel-loom/src/lib.rs:112 — the cpu shim adds two new arms of kernel-loom's loom feature, and the body shows neither linted.
  • The not(feature = "loom") arm is built by no src/clippy.rs shape. Kernel-loom is linted only with its default loom, in the two workspace shapes. So a core::mem::forget planted in that arm leaves cargo run -- --clippy green.
  • Close it one of two ways. Show both arms linted, which for this one takes a shape that builds kernel-loom --no-default-features. Or delete the arm: compile serial_lock and the shim only under loom. Neither not-loom test (log_zeroed_init, log_body_words) names them.

NOTE

• kernel/src/arch/x86_64/cpu.rs:472 — on x86-64, a wrong id at the real read reaches no red, and this read is not the SMP primitive there.
  • The roster and every IPI take the LAPIC's own id (apic::id(); x86_64/smp.rs:167, apic.rs:150, :168). This CPUID read serves only panic.rs and the lock.
  • I expect return edx; → return 0; to stay green on every TCG suite. KVM can red it only when klogd bursts during the report.
  • On AArch64 the same function picks the GIC redistributor (aarch64/irqchip.rs:184) and the self-SGI's target (:307), so there it is the primitive.
  • The body's oracle is the manuals. The cheapest red asserts cpu::hardware_id() == apic::id() once on each CPU at bring-up.
• kernel/src/drivers/serial_lock.rs:111 — the wait is held to four looks.
  • Patch 0..tries → 0..tries.min(4). It passes all eight models (the largest is seize(4), at bound 2) and every host check.
  • With it, every fatal path gives up on a live burst after four looks and writes over it.
  • The red is a host test that within(n, …) asks an attempt that never answers exactly n times. The size of the 100M count stays the tracked timing weakness.
• kernel/src/arch/x86_64/cpu.rs:327 — inb is a safe pub fn that every module reaches as crate::arch::cpu::inb. So inb(0x3f8) takes a received console byte with no Registers and no unsafe. The body's list of escapes names outb and the AArch64 Mmio window, not this, and no issue records it.
• kernel/src/arch/x86_64/console_uart.rs:27 — two more paths touch the 16550 with no Registers. init reprograms it, clearing the FIFOs and writing the loopback byte. rx_ready and tx_ready read LSR. So the body's "a write or read through arch::console_uart that did not ask for the registers" fails to build only for write_byte and read_byte.
• PR body: row 5, the Gates guest line and "Guest runs requested" still say the runs are requested. The red (r3-row5, 0/1 on DRAINED_RAW) and r3-suite 21/21 at f71a5fd19 are only in comment 5935483958, so main's record carries neither.

REMOVE

• PR body, Guest: "Rust cannot require a value to be dropped before a -> ! call, so this red needs a guest run" — false. A token that halt_all_cpus takes by value, and that PanicUart hands back only when its hold ends, refuses row 5 at compile time.

SEND BACK

[Japabu]

Guest runs at 1a9380fed, orchestrator's, this Mac (x86-64 and virt_smp under TCG), one at a time:

job	patch	result	predicted
nmiloud-r4-suite	none	21/21 (44.7 s)	21/21
nmiloud-r4-id-zero (nested_nmi_is_loud)	x86-64 hardware_id → edx & 0	0/1: assertion `left == right` failed: smp: cpu1 reads its own hardware id as 0x0, and its roster slot and every IPI name it 0x1	red, that line
nmiloud-r4-row5 (nested_nmi_is_loud)	row 5, braces dropped	0/1: the report's registers were not clean: [serial] this cpu's own fatal path held the console registers; drained raw	red, that line
nmiloud-r4-id-zero-unchecked (nested_nmi_is_loud)	id → 0 with smp::commit's assertion removed	1/1 green	green
nmiloud-r4-a64-id-zero (virt_smp)	AArch64 hardware_id → … & 0	0/1: the same smp: cpu1 reads its own hardware id as 0x0 … assertion	red

The id → 0 pair shows the new bring-up assertion is what separates a wrong id read from a green boot.

[Japabu]

Round 4 at 1a9380fed. Readiness: CI host is green at this head. In run 36898082761, job 110490041087, both the job and its cargo run -- --ci host step concluded success. Guest runs at this head are the orchestrator's TCG runs (comment 5936673848):

• the suite, 21/21;
• x86-64 id → 0, red on the new assertion;
• row 5, red;
• id → 0 with the assertion removed, green;
• AArch64 id → 0, red on virt_smp.

The KVM arm is the batch's.

Net origin/main...HEAD: +753/−203.

• Production (kernel/src): +351/−191.
• Tests and the loom shim (tests/, kernel-loom/): +269/−12.
• Issues: +133.

Round 4 alone: production +67/−26, tests and shim +26/−10, issues +45. The production growth is the Registers parameters (ruling 5) and the bring-up check (ruling 2). The check's ECHOED_ID, smp::echo and smp::commit go under the BLOCKER below.

Round 3's BLOCKER

• kernel-loom/src/lib.rs:112, the shim's unlinted not-loom arm — CLOSED. The shim and the lock compile under loom alone (kernel-loom/src/lib.rs:107, :376), so no not-loom arm is left.
  • 1a: a compile_error! in the shim builds with --no-default-features (exit 0) and reds without it (101).
  • 1b: a planted assert!(true) reds the workspace shape's clippy (101, assertions_on_constants).

Rulings

1. Holds (above).
2. Holds for every AP on both architectures: 7a red on the cpu1 message, 7b red on virt_smp, 7c green. Where the check lives is the BLOCKER. The boot CPU's check has no red (NOTE).
3. Holds. within_asks_a_silent_attempt_exactly_its_tries runs at 0, 1 and 100, and 2d reds it (101, "asked 4 times").
4. Holds. The issue is accurate: inb and inw are safe pub fns (kernel/src/arch/x86_64/cpu.rs:327, :344), with the callers it names.
5. Holds on both architectures (3a to 3g). What serial::init logs under the hold never drains there, for two reasons:
   • drain_inline returns on !has_console() (kernel/src/log/console.rs:92).
   • serial::init runs once per boot path (kernel/src/main.rs:241, :261), before UART_PRESENT is stored and before virtio-console exists.
6. Holds, and the reason that replaced the sentence is true.
   • halt_all_cpus has eleven call sites, and ten of them have no registers to hand back.
   • So a by-value token needs a mint that every caller reaches, nested_nmi included. Keeping its PanicUart and minting a second token compiles: the token refuses the literal brace-drop, not the bug.
   • The runtime refusal, DRAINED_RAW, is what row 5 reds on at this head.
7. Not met yet (NOTE).

The handshake. Sound by reading:

• The AP's relaxed store comes before Roster::echo's release, and the boot CPU loads after await_echo's acquire of that token.
• Only the AP being started writes the word. Neither architecture starts another AP after a timeout (x86_64/smp.rs:234, aarch64/smp.rs:68) or after a refused CPU_ON (aarch64/smp.rs:61), so no stale writer exists.
• The refusal belongs on the boot CPU: before is_ready, stop_other_cpus sends nothing on either architecture (x86_64/apic.rs:192, aarch64/irqchip.rs:320).
• Both architectures commit only through smp::commit.
• A failure is loud, since both reds carry the message, and bounded by halt_all_cpus' reboot bound.

What holds that order is the BLOCKER.

Issue files. Both are accurate and have exits. The AArch64 file's two consequences check out:

• Gic::redistributor panics on an affinity no redistributor answers for (irqchip.rs:135).
• start breaks on a refused CPU_ON (aarch64/smp.rs:59-61).

BLOCKER

• kernel/src/smp.rs:28 — ECHOED_ID is a second echo word beside Roster::echoed.
  • It sits outside smp_roster.rs and outside the models that kernel-loom keeps for bring-up's ordering edges (kernel-loom/tests/smp_bringup.rs, "invisible to a guest test on x86 TSO").
  • Its order rests on a comment and on the echo's release/acquire, which no model drives.
  • ROSTER.commit stays a public way around the check.
  • Patch: swap :33 and :34, so the echo goes before the store. The boot CPU can then read 0 or the previous AP's id and refuse a good AP.
  • I expect every tier to stay green with that patch. On x86-64, tsc_inside logs a line between await_echo and commit (x86_64/smp.rs:226-229). On AArch64 the window is one store.
  • Fix: carry the read in the roster's own echo. One AtomicU64 of token and read makes the order unwritable. Roster::commit refuses a mismatch, and ECHOED_ID, smp::echo and smp::commit go.
  • Must turn red: a smp_bringup.rs model when the refusal is deleted from Roster::commit. If the read stays a second word, the same model must also red under the swap.

NOTE

• kernel/src/arch/x86_64/smp.rs:168 — the boot CPU's check has no red.
  • 7a to 7c leave QEMU's boot CPU at id 0, which is its LAPIC id, so the body's "(7a to 7c)" for it measures nothing.
  • Guest run: in kernel/src/arch/x86_64/cpu.rs:472, return edx; becomes return if edx == 0 { 1 } else { edx };. Args: cargo test -- nested_nmi_is_loud. It must be 0/1 on smp: the boot CPU reads its own hardware id as 0x1, and its roster slot and every IPI name it 0x0.
  • Control: the same patch with :168-173 deleted, which I expect green under TCG. The boot CPU and cpu1 then share id 1, and nothing else holds that.
• PR body — ruling 7 is not met. Rows 7a, 7b, 7c and row 5, the Gates guest line and the bring-up control still say "owed". Their results are only in comment 5936673848, so main's record carries none of them.
• PR body, Unsure, "rests on the SDM" — toyos-cpuvuln/fixtures/t14/cpuinfo.txt is Linux's reading of the T14.
  • It gives each CPU's id as 0, 2, 4, 6, 1, 3, 5, 7, which is the T14's MADT order (issues/hardware/a-metal-session-runs-a-pre-flash-gate-first.md:62).
  • That is the AP check's relation on the T14, an oracle the body can name beside the SDM.

REMOVE

• kernel-loom/tests/serial_lock.rs:2 — "within asks an attempt exactly the tries it was given;" restates the model's own doc (:56-57).
• kernel/src/arch/x86_64/smp.rs:297 — "with the hardware id this AP reads as its own" restates smp::echo's doc (kernel/src/smp.rs:30-31).

SEND BACK

[Japabu]

Guest runs at d31faab90, orchestrator's, this Mac (x86-64 and virt_smp under TCG), one at a time:

job	patch	result	predicted
nmiloud-r5-suite	none	21/21 (33.4 s)	21/21
nmiloud-r5-boot-id-one (nested_nmi_is_loud)	x86-64 cpu.rs:472 → return if edx == 0 { 1 } else { edx };	0/1: assertion `left == right` failed: smp: the boot CPU reads its own hardware id as 0x1, and its roster slot and every IPI name it 0x0	red, that line
nmiloud-r5-boot-id-one-unchecked (nested_nmi_is_loud)	the same with the boot CPU's check deleted	1/1 green	green
nmiloud-r5-id-zero (nested_nmi_is_loud)	x86-64 AP id → 0	0/1: smp: cpu1 reads its own hardware id as 0x0, and its roster slot and every IPI name it 0x1	red in Roster::commit
nmiloud-r5-a64-id-zero (virt_smp)	AArch64 AP id → 0	0/1: the same cpu1 assertion	red
nmiloud-r5-id-zero-unchecked (nested_nmi_is_loud)	x86-64 AP id → 0 with Roster::commit's refusal deleted	1/1 green	green

Each pair shows the check is what separates a wrong id read from a green boot, for the boot CPU and for an AP.

[Japabu]

Round 5 at d31faab90. Readiness: CI host is green at this head. In run 36903521752, job 110508251323, the job and its cargo run -- --ci host step both concluded success:

• [ci] Host: 59 step(s), all green;
• smp_bringup 3 passed, among them an_ap_that_reads_another_id_is_refused - should panic ... ok;
• the controls roster-commit-relaxed and smp-ready-split reached 1 verdict each.

The run was in progress when this round began and concluded before this comment. Guest runs at this head are the orchestrator's TCG runs (comment 5937395775). The KVM arm is a nightly on main after landing, by the owner's decision.

Net origin/main...HEAD: +783/−211.

• Production (kernel/src): +346/−198.
• Tests and the loom shim (tests/, kernel-loom/): +304/−13.
• Issues: +133.

Round 5 alone: production +33/−45, tests +39/−5, issues +1/−1. Production shrinks by 12. Accepted.

Round 4's BLOCKER

• kernel/src/smp.rs:28, ECHOED_ID beside the roster's echo, with ROSTER.commit a way around the check — CLOSED.
  • ECHOED_ID, smp::echo and smp::commit are gone, and git grep finds no use of them.
  • Both architectures commit through ROSTER.commit (x86_64/smp.rs:229, aarch64/smp.rs:70), and it does the refusing itself (smp_roster.rs:142-148).
  • m1, the refusal deleted, reds an_ap_that_reads_another_id_is_refused: "test did not panic as expected", exit 101.
  • At this head (comment 5937395775):
    • x86-64 AP id → 0: 0/1 in Roster::commit, and 1/1 green with the refusal deleted;
    • AArch64 AP id → 0: 0/1 on virt_smp.

Round 4's NOTE on the boot CPU's check is closed: nmiloud-r5-boot-id-one is 0/1 on the predicted line, and its control 1/1 green. Both of round 4's REMOVEs are applied.

The echo is sound on both architectures, by reading:

• Packing. The token is bits 63..32 and the read bits 31..0 (smp_roster.rs:109). echoed compares the high half to the token (:115), and commit takes the low half (:142). No token is 0, since next_token starts at 1 and there are at most seven attempts, so the initial word is no echo.
• Ordering. The AP makes one 64-bit store-release and the boot CPU one load-acquire. commit's relaxed reload of the same word is sequenced after that acquire, so coherence gives it the acquired value or a later one.
• Stale writer. A later value needs a second echo. Each AP echoes once, and neither architecture starts another AP after a timeout or a refused CPU_ON (x86_64/smp.rs:231-235, aarch64/smp.rs:59-61, :66-68). If a stale echo existed it would carry its own token, so echoed would not take it. Landing between the acquire and the reload, it could only make commit refuse a good AP, loudly.

The loom model reaches the refusal.

• Its expected string is the refusal's own message with the model's 5 and 7, so nothing else satisfies it, and m1 turns it into "did not panic".
• ap.join() comes before commit, so the model holds no ordering edge. With one word, none is left to hold.
• Its id 7 against token 1 catches what no QEMU boot can. A commit that read the token half would pass every guest boot, where cpu N's token and id are both N, and would red both smp_bringup models that commit.

BLOCKER

• none.

NOTE

• kernel/src/smp_roster.rs:108 — echo(token: u32, read: u32) takes the read from its caller, so nothing holds which read it carries.

  • Swapped arguments at x86_64/smp.rs:297 or aarch64/smp.rs:85 pass every QEMU boot, where cpu N's id and token are both N. Only a T14 row reds them, through metal::cpus()'s "failed to start" (tests/common/metal.rs:479).
  • Echoing apic::id() instead of cpu::hardware_id() passes every tier and leaves the check holding nothing.
  • Fix: echo(token) reads crate::arch::cpu::hardware_id() itself, as serial_lock::fatal_here (:27) does through the shim kernel-loom supplies under loom. smp_roster is then gated under loom there, as serial_lock is (kernel-loom/src/lib.rs:376), and the models call become_cpu before they echo. That makes both mistakes unwritable.

• PR body — main's record lacks the guest runs at this head. They are only in comment 5937395775:

  • the suite, 21/21;
  • the boot CPU's red and its control;
  • the AP reds and their control.

  The Gates guest line and "Where each row was measured" name 1a9380fed, and the two checks' negative control has no boot-CPU pair. Row 5's red at 1a9380fed stands for this head, because round 5 touches neither nmi.rs nor serial.rs.

• PR body, Issues, "Fixes the red of main's nightly run 36843762360" — no run of this branch measures it: the body's Unsure puts the KVM arm after landing.

  • The branch's own issues/panic-path/a-panic-write-between-two-console-bursts-can-split-a-character.md records that run's capture as lost at a split —.
  • That capture's — falls inside one 16-byte burst, so this branch closes that instance only.
  • A report taken between two bursts can still split another record's character. So the post-landing nightly can still red nested_nmi_is_loud, through a defect the branch files and leaves open.

• Landing alone — The guest suite is every pull request's guest check, on toolchain stores CI caches by the build system's own keys; a landing during a nightly does not red its release #671's head c1564508b still carries issues/kernel/the-nested-nmi-report-interleaves-with-another-cpus-console-line.md, cited at issues/kernel/every-aarch64-guest-dies-at-the-kernels-entry-on-cis-firmware.md:26.

  • That file's exit is the KVM guest green that the post-landing nightly measures.
  • The guest suite is every pull request's guest check, on toolchain stores CI caches by the build system's own keys; a landing during a nightly does not red its release #671's next merge of main deletes the file and its citation on a green nightly, or keeps the file on a red one.

REMOVE

• PR body, Unsure — "No guest has run the head. The guest results above are 1a9380fed's." — false since comment 5937395775.
• PR body, Unsure — "The boot CPU's check has no red. …" — false: nmiloud-r5-boot-id-one is 0/1 on the boot CPU's assertion at this head.
• PR body, Mutations — "The guest runs, the orchestrator's under TCG, at 1a9380fed. There the AP check read the id from a second word, in smp::commit; …" — names a second word and a function main never has, and the head's runs measure Roster::commit itself.
• PR body, row 7c — "smp::commit's assertion at 1a9380fed," — the same.

LAND AFTER NAMED CHANGES

[Japabu]

Mutation patches behind the body's rows m1, w1, w2 and g1 (host, at 3b961d5bf) and 7a, 7b, 7d and 7e (guest jobs; each one's build at 3b961d5bf), each applied with git apply from the repository root.

m1 (sha256 77fa4904eec0a189…)

diff --git a/kernel/src/smp_roster.rs b/kernel/src/smp_roster.rs
index 7436176..097464f 100644
--- a/kernel/src/smp_roster.rs
+++ b/kernel/src/smp_roster.rs
@@ -140,14 +140,6 @@ impl Roster {
     /// the release an AP's own panic stops no other CPU.
     pub fn commit(&self, at: Attempt, hardware_id: u32) {
         debug_assert!(at.id == self.count.load(Ordering::Relaxed));
-        // Relaxed: the read is the echo's own word, which `await_echo` acquired.
-        let read = self.echoed.load(Ordering::Relaxed) as u32;
-        assert_eq!(
-            read,
-            hardware_id,
-            "smp: cpu{} reads its own hardware id as {read:#x}, and its roster slot and every IPI name it {hardware_id:#x}",
-            at.id
-        );
         self.hardware_ids[at.id as usize].store(hardware_id, Ordering::Relaxed);
         // Release: the slot store above lands before the count exposes it; the
         // control drops it to relaxed and the model finds the unfilled slot.

w1 (sha256 f2541eae13c85f76…)

diff --git a/kernel/src/arch/x86_64/smp.rs b/kernel/src/arch/x86_64/smp.rs
index 721f93d..48a9ce9 100644
--- a/kernel/src/arch/x86_64/smp.rs
+++ b/kernel/src/arch/x86_64/smp.rs
@@ -294,7 +294,7 @@ extern "C" fn ap_entry() -> ! {
     apic::init_ap();
 
     // Echo this attempt's token, so the BSP counts this AP for its own attempt.
-    ROSTER.echo(percpu::ap_token());
+    ROSTER.echo(cpu::hardware_id(), percpu::ap_token());
 
     process::ap_idle();
 }

w2 (sha256 e486dea5672d4e1d…)

diff --git a/kernel/src/smp_roster.rs b/kernel/src/smp_roster.rs
index 7436176..db63043 100644
--- a/kernel/src/smp_roster.rs
+++ b/kernel/src/smp_roster.rs
@@ -107,7 +107,7 @@ impl Roster {
     /// first interrupt: the token of the attempt that started it, and the
     /// hardware id this CPU reads as its own.
     pub fn echo(&self, token: u32) {
-        let read = crate::arch::cpu::hardware_id();
+        let read = crate::arch::apic::id();
         self.echoed.store((u64::from(token) << 32) | u64::from(read), Ordering::Release);
     }
 

g1 (sha256 19a977103920edf2…)

diff --git a/kernel-loom/src/lib.rs b/kernel-loom/src/lib.rs
index afcade9..fe3cd2e 100644
--- a/kernel-loom/src/lib.rs
+++ b/kernel-loom/src/lib.rs
@@ -170,7 +170,6 @@ pub mod shootdown;
 
 /// The CPU roster and the release/answer word, driven by `tests/smp_bringup.rs`;
 /// under `loom` alone, as is the [`arch::cpu`] shim it names.
-#[cfg(feature = "loom")]
 #[path = "../../kernel/src/smp_roster.rs"]
 pub mod smp_roster;
 

7a (sha256 010f506482c2523c…)

diff --git a/kernel/src/arch/x86_64/cpu.rs b/kernel/src/arch/x86_64/cpu.rs
--- a/kernel/src/arch/x86_64/cpu.rs
+++ b/kernel/src/arch/x86_64/cpu.rs
@@ -469,7 +469,7 @@
             // SDM Vol. 2A, CPUID leaf 0BH: EBX[15:0] == 0 means unimplemented,
             // not id 0, so the leaf-1 fallback below must still run.
             if ebx & 0xFFFF != 0 {
-                return edx;
+                return edx & 0;
             }
         }
     }

7b (sha256 475cda465f4e5e6b…)

diff --git a/kernel/src/arch/aarch64/cpu.rs b/kernel/src/arch/aarch64/cpu.rs
--- a/kernel/src/arch/aarch64/cpu.rs
+++ b/kernel/src/arch/aarch64/cpu.rs
@@ -122,5 +122,5 @@
     let mpidr: u64;
     // SAFETY: reads an ID register.
     unsafe { asm!("mrs {}, mpidr_el1", out(reg) mpidr, options(nomem, nostack, preserves_flags)) };
-    toyos_gicv3::packed_affinity(mpidr)
+    toyos_gicv3::packed_affinity(mpidr) & 0
 }

7d (sha256 a2b7723e385ad2fc…)

diff --git a/kernel/src/arch/x86_64/cpu.rs b/kernel/src/arch/x86_64/cpu.rs
--- a/kernel/src/arch/x86_64/cpu.rs
+++ b/kernel/src/arch/x86_64/cpu.rs
@@ -469,7 +469,7 @@
             // SDM Vol. 2A, CPUID leaf 0BH: EBX[15:0] == 0 means unimplemented,
             // not id 0, so the leaf-1 fallback below must still run.
             if ebx & 0xFFFF != 0 {
-                return edx;
+                return if edx == 0 { 1 } else { edx };
             }
         }
     }

7e (sha256 97d947478684b1b8…)

diff --git a/kernel/src/arch/x86_64/cpu.rs b/kernel/src/arch/x86_64/cpu.rs
--- a/kernel/src/arch/x86_64/cpu.rs
+++ b/kernel/src/arch/x86_64/cpu.rs
@@ -469,7 +469,7 @@
             // SDM Vol. 2A, CPUID leaf 0BH: EBX[15:0] == 0 means unimplemented,
             // not id 0, so the leaf-1 fallback below must still run.
             if ebx & 0xFFFF != 0 {
-                return edx;
+                return if edx == 0 { 1 } else { edx };
             }
         }
     }
diff --git a/kernel/src/arch/x86_64/smp.rs b/kernel/src/arch/x86_64/smp.rs
--- a/kernel/src/arch/x86_64/smp.rs
+++ b/kernel/src/arch/x86_64/smp.rs
@@ -165,12 +165,6 @@
 /// Boot all Application Processors found in the MADT, using `boot_cr3` — the bootloader's identity+high-half PML4 — until each AP switches to the kernel PML4 in `ap_entry`.
 pub fn boot_aps(madt: &MadtInfo, boot_cr3: u64) {
     let bsp_id = apic::id();
-    let read = cpu::hardware_id();
-    assert_eq!(
-        read,
-        bsp_id,
-        "smp: the boot CPU reads its own hardware id as {read:#x}, and its roster slot and every IPI name it {bsp_id:#x}"
-    );
     ROSTER.set_bsp(bsp_id);
     copy_trampoline();
 

The four guest patches are byte-identical to round 5's.

[Japabu]

Guest runs at 3b961d5bf (this head), orchestrator's, this Mac (x86-64 and virt_smp under TCG), one at a time:

job	patch	result
nmiloud-r6-suite	none	21/21 (33.4 s)
nmiloud-r6-id-zero (nested_nmi_is_loud)	x86-64 AP id → 0	0/1: smp: cpu1 reads its own hardware id as 0x0, and its roster slot and every IPI name it 0x1
nmiloud-r6-a64-id-zero (virt_smp)	AArch64 AP id → 0	0/1: the same cpu1 assertion
nmiloud-r6-boot-id-one (nested_nmi_is_loud)	x86-64 boot CPU id → 1	0/1: smp: the boot CPU reads its own hardware id as 0x1, and its roster slot and every IPI name it 0x0
nmiloud-r6-boot-id-one-unchecked (nested_nmi_is_loud)	the same with the boot CPU's check deleted	1/1 green

All as predicted. The named changes of review round 5 are applied: git diff d31faab90..3b961d5bf is item 1 alone in code (5 files, +21 −11) plus the body.