A syscall's body runs with interrupts open, and a kernel-mode timer expiry re-arms one quantum

issues/a-muted-screen-test-pays-its-ceiling-before-any-boot-has-measured-the-host.md

@@ -0,0 +1,28 @@
+---
+status: open
+kind: tooling
+opened: 2026-10-04
+---
+
+# A muted screen test pays its ceiling before any boot has measured the host
+
+`QemuInstance::screendump_while` (`tests/common/qemu.rs`) fixes its deadline
+once, at its start, through `budget_smp`, whose host factor is the run's
+fastest boot so far (`host_scale`). A muted guest reaches no ready marker, so
+its own boot never feeds that factor, and a muted test that starts before any
+other guest of the run has booted pays its ceiling at 1×, however loaded the
+host is.
+
+Seen once, on the dev host at load averages 80.26, 74.24 and 66.53 (14
+cores), in a whole `cargo test --test toyos-build` run at `7c550fa1f` on
+`wt/toyos-irqon`: `screen_panic_muted` was one of the twelve tests the run
+started at 07:17:52, its kernel built at 07:18:20, and it was red at 07:18:51,
+before any test of the run had passed, with `"PANIC:" not on
+screen of a guest with no serial port at all`. The decoded screen held only
+the kernel's first three lines, all stamped 0.000. The same run ended with
+`fastest boot 2162 ms against the reference 1424 ms — liveness ceilings paid
+at 1.52x`. The whole run at `2b9697284`, at 1.01x, had it green.
+
+**Exit**: a muted screen test's wait is bounded by the guest's progress or by a
+host factor measured before its deadline is fixed, and a muted test run first
+on a loaded host is green.

issues/a-process-lengthens-an-interrupts-off-walk-by-the-threads-it-parks-on-one-ring.md

@@ -32,20 +32,19 @@ holder, decides how long a CPU runs with interrupts masked:
 
 Nothing caps N: a thread costs its process a 128 KiB kernel stack
 (`kernel/src/process.rs`) and no count. The first walk starts in
-`SYS_INBOX_SUBMIT`, and a syscall runs with interrupts masked from entry to
-exit (`issues/syscall-preemption-is-incidental.md`), with #634 and
-without it: #634 did not mask it, and reverting #634 does not shorten it.
-The second runs in the device's handler since #634.
+`SYS_INBOX_SUBMIT`, masked by the ring's `IrqLock`. The second runs in the
+device's handler since #634.
 
-How long the second is has not been read. The first has, at one size: the
+How long the second is has not been read. The first has, at one size, while
+syscalls still ran with interrupts masked from entry to exit: the
 three T14 boots of #649 at `8b73eba69` (comment 5959415453, readbacks
 `649-r5/1-head`, `649-r5/2-report-halved`, whose kernel prints half of every
 span, and `649-r5/3-idle-halt-counted`) ran it with N = 256
 (`test_rs_ring_park_herd`) and read it from the herd's own report, which
 spans the runner's spawn of the herd as well. The longest `irqs_off_ns` on
 any CPU there is 1032292 (`windowscase/kernel.log:428`), 2 × 519358 (`:426`)
 and 1175943 (`:430`). The third is cpu7's, the CPU that spawned the herd
-(`issues/the-cpu-that-spawns-a-toybox-applet-reads-1-4-ms-of-interrupts-and-preemption-off-on-the-t14.md`),
+(`issues/the-cpu-that-spawns-a-toybox-applet-reads-1-4-ms-of-preemption-off-on-the-t14.md`),
 and the other seven read 257646 to 364953 in that boot.
 
 The three at `0aa8d4c88` (comment 5960575031, readbacks `649-r6/1-head`,
@@ -68,6 +67,14 @@ machine's own events reached, and which boot carried one decides a single
 reading. None of the six took an interrupt a handler #634 changed serves
 (`userdev=0 sound=0 dmafault=0 hda=0` on every CPU).
 
+Those readings were the whole masked syscall, not the walk. With a syscall's
+body open to interrupts, #716's six head boots of the same load at the same N
+(comment 5979107466; readbacks `irqon/metal/head/mask_windows/runN` and
+`irqon/metal/head-full`) read the longest `irqs_off_ns` in the herd's report
+at 60479, 62947, 56462, 57842, 58396 and 59488, and every CPU between 41469
+and 62947 (`windowscase/kernel.log:424` to `:438`). That bounds the walk at
+N = 256 from above at 63 µs; it says nothing of how the walk grows with N.
+
 **Exit**: the interrupts-off window step 2's instrument reads on the T14
 under N threads parked in `submit` on one ring, a sibling thread completing
 into it, does not grow with N: read at two sizes on one kernel, each size the

issues/a-shutdown-on-a-held-usb-disk-left-a-cpu-deaf-to-a-tlb-shootdown.md

@@ -32,7 +32,11 @@ trigger and not the deaf CPU.
 `issues/xhci-waits-are-spins.md` carries the arithmetic for one disk
 operation outrunning `time::DEAF_CPU`; this is a boot
 that did outrun it, in `quiesce`, across several operations each inside its
-own budget. Whether `quiesce` holds `IF` clear between them is not measured.
+own budget. Whether `quiesce` holds `IF` clear between them is not measured;
+both sightings ran while a syscall's body ran with interrupts masked, and the
+shutdown syscall's now runs with them open
+(`issues/syscall-preemption-is-incidental.md`), which no boot has
+read since.
 
 **Second sighting, with the roles swapped**: `usb_transport_break --nightly` at
 `e889d03e` (#554), the `AnotherStick` boot (`554r6-usb_transport_break.log` in

issues/a-tls-block-is-zeroed-twice-with-interrupts-masked.md → issues/a-tls-block-is-zeroed-twice-with-preemption-off.md

@@ -4,15 +4,15 @@ kind: defect
 opened: 2026-10-03
 ---
 
-# A TLS block is zeroed twice with interrupts masked
+# A TLS block is zeroed twice with preemption off
 
 `kernel/src/loader/tls.rs`'s `build_combined` takes its frames from
 `PageAlloc::new`, which is `pmm::alloc_contiguous`, which zeroes every 2 MiB
 frame it hands out; then it zeroes the whole block again with `write_bytes`.
 Every `SYS_THREAD_SPAWN` (`process::spawn_thread`) and every `SYS_SPAWN`
-(`loader`) builds one, and a syscall runs with interrupts masked from entry to
+(`loader`) builds one, and a syscall runs with preemption off from entry to
 exit (`issues/syscall-preemption-is-incidental.md`), so each spawn pays
-two writes of the block, at least 2 MiB each, in one interrupts-off window.
+two writes of the block, at least 2 MiB each, in one preemption-off window.
 
 By reading, not measured.
 

issues/no-program-measures-toyos-against-linux-on-one-machine.md

@@ -21,3 +21,11 @@ two sides' IDs match or either side's changed across the window.
 **Mutation**, each red: a pair on one CPU passes; only the readings before the
 window compared, so a sibling that moved onto the initiator's CPU inside it
 passes. **Oracle**: Linux.
+
+What stands nearest is `counters_metal`'s `loaded` phase
+(`tests/toyos-rust-tests/src/bin/counters_metal.rs`), the one lateness reading
+the shipped kernel gives under load, and it is not this program's timer
+lateness: it reads how late each CPU's kick handler ran after a round's
+kicks, measured against the round's earliest stamp and so short by however
+late that one was, once a round rather than at 1 kHz, on ToyOS alone, and it
+holds no figure.

issues/nothing-in-the-machine-can-read-the-trace-ring.md

@@ -38,9 +38,9 @@ anything more is built on it.
    trace. A system call held past the threshold in a shipped-configuration
    kernel reads back from the trace with its number and its program. A
    window's record names its opener by address, which
-   `issues/the-cpu-that-spawns-a-toybox-applet-reads-1-4-ms-of-interrupts-and-preemption-off-on-the-t14.md`
+   `issues/the-cpu-that-spawns-a-toybox-applet-reads-1-4-ms-of-preemption-off-on-the-t14.md`
    and
-   `issues/the-supervisors-claim-of-a-pci-function-the-t14-lacks-holds-interrupts-off-for-3-8-ms.md`
+   `issues/the-supervisors-claim-of-a-pci-function-the-t14-lacks-holds-preemption-off-for-3-8-ms.md`
    need.
 
 **Ruled** (owner, 2026-10-04), on when these steps start, **"Both in

issues/syscall-preemption-is-incidental.md

@@ -4,7 +4,7 @@ kind: defect
 opened: 2026-07-31
 ---
 
-# A syscall runs with interrupts masked, and only incidentally with preemption disabled
+# A syscall runs with preemption disabled from entry to exit, only incidentally
 
 `syscall_entry` raises the preempt count before `call {handler}` and lowers it
 after, so `preempt::enable`'s `count() == 0` slow path can never fire inside a
@@ -14,23 +14,14 @@ latency by "the longest preempt-disabled section"; in syscall context that
 section is *the entire syscall*, and the real bound is the next
 `kernel_exit_to_user_check`.
 
-The preempt count is the weaker of two independent blockers, and it is not the
-one that decides the bound. `MSR_FMASK = 0x40200` (`arch/syscall.rs:57`) clears
-IF on every SYSCALL entry, and nothing on the straight-line syscall path sets it
-again — the only `sti`s in the kernel are `cpu::enable_interrupts`
-(`arch/cpu.rs:113`, reached from `trap_dispatch`'s #PF arm and from init),
-`kernel_exit_to_user_check`'s own yield window (`arch/idt/mod.rs:232`), and the
-idle loop (`sched/driver.rs:494`). With IF=0 the CPU cannot even be *told* to
-reschedule: `KernelHw::need_resched` (`hw.rs:96-108`) documents that a remote
-CPU's `need_resched` byte is unreachable from here, so a remote request is only
-deliverable as a kick IPI — an interrupt the masked target will not take until
-it leaves the syscall.
-
-That makes the entry level's fix ineffective on its own: dropping it around
-blocking-capable handler regions cannot move remote RT wake latency at all
-while IF stays 0. A fix has to unmask interrupts over those regions — which
-means auditing what each one is safe to be interrupted in — or the bound is
-accepted and that model corrected.
+A syscall's body runs with interrupts open on both architectures: the gate
+masks them only from the entry through the user-state save and from the
+handler's return to the user return (`kernel/src/arch/x86_64/syscall.rs`,
+`kernel/src/arch/aarch64/trap.rs`). A timer's expiry or a kick inside one sets
+`need_resched`, which the syscall's exit serves, and the Ring 0 expiry re-arms
+one quantum on. What still runs masked is the Ring 3 tick's and kick's pass,
+which the handler runs, and the walk under an `IrqWatch`'s lock
+(`issues/a-process-lengthens-an-interrupts-off-walk-by-the-threads-it-parks-on-one-ring.md`).
 
 This was masked until the preempt count was made conserved across a context
 switch (the scheduler's own baselines needed it): before that the count drifted,
@@ -41,11 +32,27 @@ assumes.
 Owner: `issues/toyos-beats-linuxs-latency-on-the-t14.md`, whose second
 step is syscalls running with interrupts on (owner, 2026-10-03).
 
-**Exit**, on the T14: the longest interrupts-off window the `mask_windows` row
-reads under its load (`herd irqs_off_ns=`, printed by `windows_on_metal` in
-`tests/toyos.rs`) is no longer than the longest lateness of the timer's
-interrupt in Linux's reading of this machine, which that track keeps: a masked
-window makes a timer's interrupt late by at most its own length. No figure is
-set here. Which of Linux's figures is the bar is that track's open question,
-and what the row reads today, and which syscalls those windows are, is in
-`issues/xhci-waits-are-spins.md`, "On the T14".
+The T14 read the step at #716 (comment 5979107466; readbacks
+`irqon/metal/{base,head}/mask_windows/runN` and `irqon/metal/head-full`),
+five `mask_windows` boots an arm, interleaved, and the head's full profile
+once, `windows_on_metal`'s `herd` line:
+
+| arm | `irqs_off_ns` | `preempt_off_ns` |
+|---|---|---|
+| base, main at `d47b383cf` | 18863127, 1185364, 1084719, 1265962, 4703201 | 18862957, 1185277, 1065082, 1265899, 4703058 |
+| head, images at `f89e73128` | 60479, 62947, 56462, 57842, 58396; 59488 | 1175425, 4711060, 4702163, 4714590, 9806929; 10784336 |
+
+The interrupts-off window is under 131 µs on every head boot; the
+preemption-off window is not on any. The three head readings of 4.70 to
+4.71 ms are an SMI's window by reading, no SMI count being read in that boot;
+nothing names the 9.8 and 10.8 ms ones.
+
+**Exit**, on the T14: the longest preemption-off window the `mask_windows` row
+reads under its load (`herd preempt_off_ns=`, printed by `windows_on_metal` in
+`tests/toyos.rs`) is no longer than 131 µs, the track's ruled bar. The row
+cannot read it before step 1: a `mask-windows` kernel charges each of the
+firmware's 4.5 ms SMIs
+(`issues/the-t14s-firmware-interrupts-every-cpu-every-2-2-s-under-toyos.md`)
+to whatever window it lands in. This exit is the orchestrator's amendment at
+#716's first review: it read the interrupts-off window, which step 2 shortened
+while the defect this file names, preemption off for a whole syscall, stood.

issues/the-cpu-that-spawns-a-toybox-applet-reads-1-4-ms-of-interrupts-and-preemption-off-on-the-t14.md → issues/the-cpu-that-spawns-a-toybox-applet-reads-1-4-ms-of-preemption-off-on-the-t14.md

@@ -4,7 +4,7 @@ kind: defect
 opened: 2026-10-02
 ---
 
-# The CPU that spawns a toybox applet reads 1.4 ms of interrupts and preemption off on the T14
+# The CPU that spawns a toybox applet reads 1.4 ms of preemption off on the T14
 
 Read by the `mask-windows` kernel (`kernel/src/windows.rs`) on LENOVO
 20W0003AMZ, BIOS N34ET71W (1.71), in the three `mask_windows` boots of #649
@@ -42,12 +42,24 @@ else, `pwd`'s and `echo`'s. cpu7's line in each
   `1-head` (`:430`): 468787 past that boot's applet spawn, which the spawn
   does not account for and nothing names.
 
-By reading, not measured: it is `SYS_SPAWN`, which runs with interrupts
-masked from entry to exit like every syscall
-(`issues/syscall-preemption-is-incidental.md`), and whose own record
+By reading, not measured: it is `SYS_SPAWN`, which ran with interrupts
+masked from entry to exit like every syscall then, and whose own record
 reads `total=1ms` for each of these applets. A report carries a span and no
 address, so nothing names it.
 
+A syscall's body now runs with interrupts open and preemption off
+(`issues/syscall-preemption-is-incidental.md`), and the section left
+`irqs_off_ns` and stayed in `preempt_off_ns`. #716's interleaved
+`mask_windows` boots (comment 5979107466; readbacks
+`irqon/metal/{base,head}/mask_windows/runN` and `irqon/metal/head-full`) ran
+`pwd` alone of the applets, spawned from cpu7 on every boot; cpu7's line in
+its report (`windowscase/kernel.log:396`, `:397` in `head-full`):
+
+| arm | `irqs_off_ns` | `preempt_off_ns` |
+|---|---|---|
+| base, main at `d47b383cf` | 1429697, 1334122, 1460438, 1385896, 1373747 | 1429402, 1333862, 1460269, 1385648, 1373471 |
+| head, images at `f89e73128` | 1146, 1178, 1136, 1781, 1058; 1155 | 1317184, 1508859, 1463782, 1494243, 1511276; 1481928 |
+
 **Exit**: the section is named on the T14 by the address its opening hook was
 called from, and the spawning CPU's longest window no longer includes it, or
 this file is replaced by the bound it is held to and the derivation of it.

issues/the-supervisors-claim-of-a-pci-function-the-t14-lacks-holds-interrupts-off-for-3-8-ms.md → issues/the-supervisors-claim-of-a-pci-function-the-t14-lacks-holds-preemption-off-for-3-8-ms.md

@@ -4,7 +4,7 @@ kind: defect
 opened: 2026-10-02
 ---
 
-# init's claim of a PCI function the T14 lacks holds interrupts off for 3.8 ms
+# init's claim of a PCI function the T14 lacks holds preemption off for 3.8 ms
 
 On LENOVO 20W0003AMZ, BIOS N34ET71W (1.71), one boot of `main` at `c59e09ed6`
 with a throwaway instrument that names a window's opener and samples its CPU
@@ -16,6 +16,23 @@ is in `PciDevice::is_id` under `pcidev::claim`. By reading, the claim reads the
 vendor ID of each of the 24 functions the kernel enumerated
 (`kernel/src/pcidev/mod.rs`); nothing has said what it spends 3.8 ms on.
 
+A syscall's body now runs with interrupts open and preemption off
+(`issues/syscall-preemption-is-incidental.md`). #716's interleaved
+`mask_windows` boots (comment 5979107466; readbacks
+`irqon/metal/{base,head}/mask_windows/runN` and `irqon/metal/head-full`) each
+made this claim (`supervisor: diskserver: no pci:1b36:0010 on this machine`)
+before the first job's report, which spans it and the partition claims below.
+cpu0's line there:
+
+| arm | `irqs_off_ns` | `preempt_off_ns` |
+|---|---|---|
+| base, main at `d47b383cf` | 6592629, 6565352, 6491279, 6614740, 6509061 | 6592463, 6565169, 6491091, 6614595, 6508897 |
+| head, images at `f89e73128` | 3061, 10315, 14214, 15492, 3510; 15409 | 6550562, 6517919, 6632892, 6549746, 6619444; 5706681 |
+
+On the head no CPU but the one `hold_once` held reads more than 44537 ns of
+interrupts off in that report, which bounds every window inside the claim
+from above; no reading separates the claim's own windows from the rest.
+
 Before the first job's exit cpu0 also carries init's partition claims, 6.0
 and 6.6 ms in that boot (`issues/xhci-waits-are-spins.md`), and the
 `mask-windows` kernel (`kernel/src/windows.rs`) prints each CPU's longest

issues/xhci-waits-are-spins.md

@@ -4,7 +4,7 @@ kind: defect
 opened: 2026-08-03
 ---
 
-# The xHCI driver's waits are spins, and a USB disk call spins with interrupts masked
+# The xHCI driver's waits are spins, under a lock and with preemption off
 
 Every wait in the kernel's xHCI driver spins against a wall-clock deadline
 while holding `XHCI` (`kernel/src/drivers/xhci/mod.rs`), a ticket spinlock and
@@ -19,24 +19,21 @@ and a disk call.
 
 A call on a USB disk holds `XHCI` from its first command (`with_disk`,
 `kernel/src/drivers/xhci/wait/msc.rs`) inside a syscall, which runs with
-interrupts masked from entry to exit
-(`issues/syscall-preemption-is-incidental.md`): a partition claim's
+preemption off from entry to exit and, since the gate opens them, interrupts
+on (`issues/syscall-preemption-is-incidental.md`): a partition claim's
 `SYS_PARTITION_READ`, `SYS_PARTITION_WRITE` and `SYS_FSYNC`, the last a cache
 flush through `xhci::storage_flush` (`partition_fsync`,
 `kernel/src/object/ops.rs`), and the table `SYS_DEVICE_CLAIM` reads for one
 (`gpt::claimable`, `kernel/src/gpt.rs`). logd's `fsync` is one of these: the
 LOG fsd answers each with `SYS_FSYNC` on its claim. So its CPU holds
-interrupts and preemption off for as long as the device takes, and a CPU whose
-TLB shootdown waits on that CPU's acknowledgement spins as long, masked.
+preemption off for as long as the device takes. A disk's bind after boot
+spins inside the Ring 3 tick's pass, which runs with interrupts masked, and
+`time::DEAF_CPU` (5 s), past which a CPU waiting on a TLB acknowledgement
+panics, is held above `CALL_AFTER_BREAK` (4.75 s), the longest one call spins
+once its transport has broken.
 
-**Its bound outruns the TLB-ack tripwire.** `time::DEAF_CPU` (5 s), past which
-a CPU waiting on an acknowledgement panics, is held above `CALL_AFTER_BREAK`
-(4.75 s), the longest a disk call spins once its transport has broken. But that
-bound opens at the wait that broke, and `transfer_blocks` starts a batch while
-the operation's 2 s `block::OPERATION` has any left: a batch that starts at
-1.99 s and breaks runs its ladder to 6.74 s after the operation began, all of
-it with `IF` clear. That is arithmetic on the declared constants; no boot has
-been seen to do it.
+Its "On the T14" readings below were taken while a syscall still ran with
+interrupts masked from entry to exit.
 
 ## On the T14
 
@@ -101,11 +98,9 @@ whose step 10 moves the whole xHCI to usbd and deletes the kernel's driver. It
 owes both exits.
 
 **Exit**, two:
-- **The tripwire's, short of step 10**: the whole of one operation's `IF`-clear
-  spin is under `DEAF_CPU` by construction, because the call's bound opens
-  where the operation opens, or a later batch starts only while a whole call's
-  bound is still inside it; and a staged boot whose first batch spends most of
-  the budget and whose next batch breaks shows the operation ending inside
-  `DEAF_CPU`.
+- **The tripwire's, short of step 10**: no disk wait spins with `IF` clear,
+  because the tick's pass leaves the interrupt gate (step 2 of
+  `issues/toyos-beats-linuxs-latency-on-the-t14.md`), and
+  `kernel/src/drivers/xhci/mod.rs` holds no bound under `DEAF_CPU`.
 - **The spin's**: the tree has no `kernel/src/drivers/xhci`, so no kernel wait
   is a USB device's. Step 10 meets both.