Stage 6 step 2, built: the longest interrupts-off and preemption-off windows per CPU, and a held window the T14 reads back

issues/build/the-control-registers-verdicts-doc-comment-sits-on-the-census-verdict.md

@@ -0,0 +1,18 @@
+---
+status: open
+kind: tooling
+opened: 2026-10-01
+---
+
+# The control registers verdict's doc comment sits on the census verdict
+
+In `tests/toyos.rs`, the doc block that starts "Every CPU's `CR0` and `CR4`,
+against what a CPU running this kernel must hold" runs straight into
+"The interrupt census adds up, …". Both paragraphs render as the doc of
+`irq_census`. `control_regs`, which the first block describes, has no doc
+comment. Nothing checks that a doc comment names the item it is attached to,
+so rustdoc and a reader both credit the census verdict with the
+control-register rules.
+
+**Exit:** the block sits on `control_regs`, and `irq_census` carries only its
+own doc.

issues/kernel/a-process-lengthens-an-interrupts-off-walk-by-the-threads-it-parks-on-one-ring.md

@@ -31,11 +31,47 @@ holder, decides how long a CPU runs with interrupts masked:
   registrations sweep them four at a time.
 
 Nothing caps N: a thread costs its process a 128 KiB kernel stack
-(`kernel/src/process.rs`) and no count. Before #634 every one of these
-walks ran with interrupts open, under preemption off. By reading, not
-measured: no instrument in the tree reads an interrupts-off window.
-
-**Exit**: step 2's interrupts-off window, read on the T14 while one process
-parks 256 threads in `submit` on one ring and a sibling thread completes into
-it, is no longer at step 1's head than at stage 6's first commit under the
-same load.
+(`kernel/src/process.rs`) and no count. The first walk starts in
+`SYS_INBOX_SUBMIT`, and a syscall runs with interrupts masked from entry to
+exit (`issues/kernel/syscall-preemption-is-incidental.md`), with #634 and
+without it: #634 did not mask it, and reverting #634 does not shorten it.
+The second runs in the device's handler since #634.
+
+How long the second is has not been read. The first has, at one size: the
+three T14 boots of #649 at `8b73eba69` (comment 5959415453, readbacks
+`649-r5/1-head`, `649-r5/2-report-halved`, whose kernel prints half of every
+span, and `649-r5/3-idle-halt-counted`) ran it with N = 256
+(`test_rs_ring_park_herd`) and read it from the herd's own report, which
+spans the runner's spawn of the herd as well. The longest `irqs_off_ns` on
+any CPU there is 1032292 (`windowscase/kernel.log:428`), 2 × 519358 (`:426`)
+and 1175943 (`:430`). The third is cpu7's, the CPU that spawned the herd
+(`issues/kernel/the-cpu-that-spawns-a-toybox-applet-reads-1-4-ms-of-interrupts-and-preemption-off-on-the-t14.md`),
+and the other seven read 257646 to 364953 in that boot.
+
+The three at `0aa8d4c88` (comment 5960575031, readbacks `649-r6/1-head`,
+`649-r6/2-report-halved` and `649-r6/3-idle-halt-counted`, the same two
+mutated kernels) read the same report of the same load at 1980239
+(`windowscase/kernel.log:430`), 2 × 5075290 (`:422`) and 4725822 (`:416`):
+
+- 1980239 is cpu7's, which spawned the herd (`:395`). The other seven read
+  969922 to 1044783.
+- 2 × 5075290 is one of eight: every CPU reads 2 × 5014552 to 2 × 5075290
+  (`:416` to `:430`), beside `tlb: shootdowns=282 wait=131633us max=10038us`
+  (`:431`). It is the first reading of
+  `issues/kernel/some-t14-boots-carry-an-interrupts-off-window-of-9-ms-or-more.md`.
+- 4725822 is cpu0's. The other seven read 544283 to 1018355.
+
+So one size reads 1.03 to 10.15 ms over six boots, and which boot carried
+one of the machine's own events decides a single reading. None of the six
+took an interrupt a handler #634 changed serves
+(`userdev=0 sound=0 dmafault=0 hda=0` on every CPU).
+
+**Exit**: the interrupts-off window step 2's instrument reads on the T14
+under N threads parked in `submit` on one ring, a sibling thread completing
+into it, does not grow with N: read at two sizes on one kernel, each size the
+median of at least five boots, each boot from the load's own report, with
+the tail reported beside it. `ring_park_herd`'s sibling waits on the kernel's
+roster, which is itself a masked walk over every thread of the machine (3,619
+to 4,111 `SYS_SYSINFO` calls a run in #649's fourteen T14 boots, and 768 to
+847 ms of syscall wall in a run of 0.9 s), so that reading has to tell the
+two walks apart.

issues/kernel/cpu0-holds-interrupts-and-preemption-off-for-6-5-ms-on-the-t14.md

@@ -0,0 +1,58 @@
+---
+status: open
+kind: defect
+opened: 2026-10-02
+---
+
+# cpu0 holds interrupts and preemption off for 6.5 ms on the T14
+
+Read by the `mask-windows` kernel (`kernel/src/windows.rs`) on LENOVO
+20W0003AMZ, BIOS N34ET71W (1.71), in the eight `mask_windows` boots of #649
+at `72f16e39a`: four as that head stands and four with stage 6 step 1 (#634)
+reverted on it. Each boot printed one report, at `test_rs_ring_park_herd`'s
+exit, so a line is its CPU's longest window from the moment it joined the
+scheduler to that exit.
+
+cpu0's line, in the seven boots where no longer window covers it:
+
+| boot | `irqs_off_ns` | `preempt_off_ns` |
+|---|---|---|
+| 1, reverted | 6551791 | 6551707 |
+| 2 | 6540754 | 6540671 |
+| 3, reverted | 6549549 | 6549484 |
+| 5, reverted | 6540326 | 6540262 |
+| 6 | 6589053 | 6588992 |
+| 7, reverted | 6590095 | 6590031 |
+| 8 | 6587342 | 6587281 |
+
+- The two kinds differ by 61 to 84 ns in every row, so one section holds
+  both.
+- It is cpu0's alone. In the six of those boots where no CPU carries a
+  longer one, the other seven CPUs' interrupts-off lines read 251,452 to
+  4,720,498 ns.
+- It is there with step 1 and without it.
+- In all eight logs cpu0 says `sched: cpu=0 ready=0 … current=None trips=1`
+  at 1.155 or 1.156 s, as the CPUs join the scheduler, and nothing more
+  before init's first line at 1.161 or 1.162 s.
+
+The fourth boot's cpu0 reads 9644415 and 9643939, the event
+`issues/kernel/some-t14-boots-carry-an-interrupts-off-window-of-9-ms-or-more.md`
+records.
+
+Nothing names what opens it. A report carries a span and no address, and one
+report a boot cannot say whether the window recurs. What will: the record
+keeping, beside each CPU's longest span, the address its opening hook was
+called from, printed in the report and resolved through the kernel's symbols.
+
+It opens before the boot's first job ends. The three boots of #649 at
+`8b73eba69` (comment 5959415453, readbacks `649-r5/1-head`,
+`649-r5/2-report-halved`, whose kernel prints half of every span, and
+`649-r5/3-idle-halt-counted`) print a report at each of four exits. cpu0's
+`irqs_off_ns` in the first, at `test_rs_idle_span`'s exit 1.746 s into the
+boot (`windowscase/kernel.log:348` in each), reads 6533222, 2 × 3273547 and
+6508165. In the three reports after it cpu0 reads at most 943904, 2 × 472938
+and 345229.
+
+**Exit**: the section that opens the window is named by that reading on the
+T14, and cpu0's longest window there no longer includes it, or this file is
+replaced by the bound the section is held to and the derivation of it.

issues/kernel/some-t14-boots-carry-an-interrupts-off-window-of-9-ms-or-more.md

@@ -0,0 +1,94 @@
+---
+status: open
+kind: defect
+opened: 2026-10-02
+---
+
+# Some T14 boots carry an interrupts-off window of 9 ms or more
+
+Read by the `mask-windows` kernel (`kernel/src/windows.rs`) on LENOVO
+20W0003AMZ, BIOS N34ET71W (1.71), in the eight `mask_windows` boots of #649
+at `72f16e39a`: four as that head stands and four with stage 6 step 1 (#634)
+reverted on it, one job each, `test_rs_ring_park_herd`. The longest window
+that recurs in every boot is cpu0's 6.5 ms
+(`issues/kernel/cpu0-holds-interrupts-and-preemption-off-for-6-5-ms-on-the-t14.md`).
+Three readings are longer, each in some boots and not in others, on both
+arms:
+
+- **Every CPU at once, the fourth boot** (step 1 in). The herd's report:
+
+  | CPU | `irqs_off_ns` | `preempt_off_ns` |
+  |---|---|---|
+  | cpu0 | 9644415 | 9643939 |
+  | cpu1 | 9603617 | 9588515 |
+  | cpu2 | 9627971 | 9625674 |
+  | cpu3 | 9641999 | 9620791 |
+  | cpu4 | 9651846 | 9635137 |
+  | cpu5 | 9623545 | 9612295 |
+  | cpu6 | 9624969 | 9621345 |
+  | cpu7 | 9614836 | 9614716 |
+
+  The same report's shootdown line reads
+  `tlb: shootdowns=280 wait=125911us max=9575us`, where the other seven
+  boots read at most `wait=49119us` and at most `max=1421us`, and cpu0's
+  census reads `nmi=2`, where the other seven read `nmi=1`.
+- **One CPU, the seventh boot** (step 1 reverted). cpu7 reads 9260193 and
+  9260124. cpu0 reads its 6590095, the other six 920,455 to 1,404,001, and
+  the shootdown line `max=1386us`.
+- **One CPU, after the herd's report, in four boots of the eight.** The stop
+  prints a report too, of which the black box keeps the lines of cpu2 to
+  cpu7. In it one CPU reads
+
+  | boot | CPU | `irqs_off_ns` | `preempt_off_ns` |
+  |---|---|---|---|
+  | 1, reverted | cpu7 | 10677997 | 10677829 |
+  | 2 | cpu7 | 11267384 | 11267320 |
+  | 5, reverted | cpu2 | 10214820 | 10214656 |
+  | 6 | cpu7 | 10874427 | 10874364 |
+
+  and in the other four cpu7 reads 1,424,134 to 1,532,702 ns and no kept
+  line more. What that report spans is the herd's teardown, which its own
+  report precedes, the spawn of `reboot` on cpu7 and the stop.
+
+By reading, not measured: every CPU waiting out one CPU's masked window
+inside a shootdown would read as the fourth boot does.
+
+Nothing names what opens any of them. A report carries a span and no address,
+and no time: eight equal spans are not shown to be one event. What will: the
+record keeping, beside each CPU's longest span, the address its opening hook
+was called from and the counter it closed at, printed in the report.
+
+The third reading falls on the `reboot` and stop side of the herd's
+teardown. The three boots of #649 at `8b73eba69` (comment 5959415453,
+readbacks `649-r5/1-head`, `649-r5/2-report-halved`, whose kernel prints half
+of every span, and `649-r5/3-idle-halt-counted`) took a report at `echo`'s
+exit, after the herd's and before `reboot` was spawned. Its longest
+`irqs_off_ns` is cpu7's in each: 1499808, 2 × 748872 and 1479558
+(`windowscase/kernel.log:456`). The stop's report after it, in the six lines
+the black box keeps, reads cpu7 11522834 and cpu4 11645411 in the first boot
+(`649-r5/1-head/windowscase/loader.log:57`, `:63`), and at most cpu7's
+2 × 3799327 in the second and cpu7's 1305656 in the third.
+
+The three boots at `0aa8d4c88` (comment 5960575031, readbacks
+`649-r6/1-head`, `649-r6/2-report-halved` and `649-r6/3-idle-halt-counted`,
+the same two mutated kernels) print a report at three exits, `idle_span`'s,
+`pwd`'s and the herd's, and at the stop:
+
+- **The first reading recurred, inside the herd's own report**, which opens
+  at `pwd`'s exit. In `2-report-halved` every CPU reads 2 × 5014552 to
+  2 × 5075290 (`windowscase/kernel.log:416` to `:430`), beside
+  `tlb: shootdowns=282 wait=131633us max=10038us` (`:431`): the shootdown's
+  `max` is beside it again, 9575us then and 10038us now, and cpu0's second
+  NMI is not: `nmi=1` (`:415`). That is two boots of #649's fourteen.
+- **One CPU, before the first job's exit.** cpu7's line of the first report,
+  `idle_span`'s (`:362`), reads 11492028 in `1-head`, 2 × 1308074 in
+  `2-report-halved` and 8456658 in `3-idle-halt-counted`, beside
+  `max=1655us`, `max=1451us` and `max=1353us` (`:363`). The same line read
+  2327848, 2 × 1831727 and 3630933 at `8b73eba69`.
+- **Nothing after the herd's report.** The stop's report, in the six lines
+  the black box keeps, reads at most cpu7's 1396887, 2 × 758960 and 1470910
+  (`windowscase/loader.log:57`).
+
+**Exit**: each of the three, and the one-CPU reading before the first job's
+exit, is named by that reading on the T14, and is removed, or this file is
+replaced by the bound it is held to and the derivation of it.

issues/kernel/the-cpu-that-spawns-a-toybox-applet-reads-1-4-ms-of-interrupts-and-preemption-off-on-the-t14.md

@@ -0,0 +1,53 @@
+---
+status: open
+kind: defect
+opened: 2026-10-02
+---
+
+# The CPU that spawns a toybox applet reads 1.4 ms of interrupts and preemption off on the T14
+
+Read by the `mask-windows` kernel (`kernel/src/windows.rs`) on LENOVO
+20W0003AMZ, BIOS N34ET71W (1.71), in the three `mask_windows` boots of #649
+at `8b73eba69` (comment 5959415453, readbacks `649-r5/1-head`,
+`649-r5/2-report-halved`, whose kernel prints half of every span, and
+`649-r5/3-idle-halt-counted`, whose kernel counts the idle halt as a
+preemption-off window).
+
+test-runner spawned every job from cpu7: each `spawn:` record of a job is
+cpu7's. Two reports a boot span the spawn of a toybox applet and of nothing
+else, `pwd`'s and `echo`'s. cpu7's line in each
+(`windowscase/kernel.log:388` and `:456`):
+
+| boot | report | `irqs_off_ns` | `preempt_off_ns` |
+|---|---|---|---|
+| `1-head` | `pwd`'s | 1408843 | 1408557 |
+| `1-head` | `echo`'s | 1499808 | 1499590 |
+| `2-report-halved` | `pwd`'s | 2 × 737408 | 2 × 737277 |
+| `2-report-halved` | `echo`'s | 2 × 748872 | 2 × 748704 |
+| `3-idle-halt-counted` | `pwd`'s | 1379642 | 536504022, a halt |
+| `3-idle-halt-counted` | `echo`'s | 1479558 | 1479398 |
+
+- The two kinds differ by 160 to 336 ns in the five rows that have both, so
+  one section holds both.
+- No other CPU reads as much in `echo`'s report: 947018 at most.
+- It is the floor under a load's reading on that CPU. A load's own report
+  spans the runner's spawn of it, and in the herd's cpu7 reads 1019488,
+  2 × 514392 and 1175943, the last the longest line of its report by three
+  times.
+- It is not all that line reads. In the three boots at `0aa8d4c88` (comment
+  5960575031, readbacks `649-r6/1-head`, `649-r6/2-report-halved` and
+  `649-r6/3-idle-halt-counted`, the same two mutated kernels) `pwd`'s report
+  reads cpu7 at 1511452, 2 × 740441 and 1360995
+  (`windowscase/kernel.log:388`), and the herd's reads it at 1980239 in
+  `1-head` (`:430`): 468787 past that boot's applet spawn, which the spawn
+  does not account for and nothing names.
+
+By reading, not measured: it is `SYS_SPAWN`, which runs with interrupts
+masked from entry to exit like every syscall
+(`issues/kernel/syscall-preemption-is-incidental.md`), and whose own record
+reads `total=1ms` for each of these applets. A report carries a span and no
+address, so nothing names it.
+
+**Exit**: the section is named on the T14 by the address its opening hook was
+called from, and the spawning CPU's longest window no longer includes it, or
+this file is replaced by the bound it is held to and the derivation of it.

issues/kernel/the-kernel-is-small-interrupts-post-and-threads-wait.md

@@ -148,8 +148,9 @@ times:
    device's `Watch` and ends its interrupt; the thread waiting on that watch
    does the work, and no step creates a kernel thread. `irq_ring`, the driver
    list in `drain_irqs` and the idle loop's device checks are gone by step 5.
-   Each step measures the kernel's lines, and from step 2 the longest
-   interrupts-off and preemption-off windows, against stage 6's first commit.
+   Each step measures the kernel's lines against stage 6's first commit, and
+   the longest interrupts-off and preemption-off windows against the readings
+   step 2's exit puts in the tree, by that exit's rule.
    Steps 3 and 4 do not land alone: they land with #592's i8042 stage and
    with usbd.
    1. **Interrupts post.** A post is legal in a handler: the watches a handler
@@ -167,12 +168,23 @@ times:
    2. **The windows, measured**: the longest interrupts-off and preemption-off
       windows per CPU, reported beside the IRQ census and fed by each
       architecture's masking primitives and entries, the number the ARM
-      track's stage 4 owes as well. Applied to stage 6's first commit for
-      the baseline. **Exit**: both windows read on the T14, which is x86
-      metal, at stage 6's start and at step 1's head, and neither is longer
-      at step 1's head than at the start, under the load
-      `issues/kernel/a-process-lengthens-an-interrupts-off-walk-by-the-threads-it-parks-on-one-ring.md`
-      names as well.
+      track's stage 4 owes as well. Built: the `mask-windows` kernel and the
+      T14's `mask_windows` row. The exit's first half is met: the instrument
+      reads back a window of known length on the T14, which is x86 metal, and
+      the row's judge refuses a boot that reads it back shorter than it was
+      held or at more than twice that. **Exit**, open: neither window is
+      longer with step 1 than
+      with it reverted on the tree that carries the instrument, the patch
+      posted on #649 being what reverted means. That comparison is the median
+      of at least five interleaved boots an arm, each read from the load's own
+      report with a report taken as the load starts, against a tolerance
+      stated before measuring, the reverted arm's own spread, with the tail
+      reported beside it, under a load in which a handler step 1 changed posts
+      into a watch that reaches a ring with parked threads
+      (`issues/kernel/a-process-lengthens-an-interrupts-off-walk-by-the-threads-it-parks-on-one-ring.md`'s
+      second bullet). `ring_park_herd` is not that load: its walks start in
+      `SYS_INBOX_SUBMIT`, which runs with interrupts masked on both arms, and
+      it reaches no handler step 1 changed.
    3. **The i8042's thread is ps2server's** (#592's i8042 stage): `irq_ring`'s
       `I8042`, `keyboard_controller::service` and the idle loop's
       `verdict_due` go with the kernel's driver. **Exit**: that stage's.
@@ -185,7 +197,7 @@ times:
       report on the panel and holding it there, a device the pass reaches
       (owner, 2026-09-30). **Exit**: `drain_irqs` and the
       idle loop's device checks are gone, both windows are measured against
-      stage 6's start, and the exits of
+      step 2's readings by its rule, and the exits of
       `issues/kernel/an-irq-watchs-freeing-cancel-compiles-in-a-handler.md`
       and
       `issues/kernel/nothing-fails-when-a-devices-release-or-close-stops-answering-its-polls.md`

kernel/Cargo.toml

@@ -129,6 +129,12 @@ debug-wait = []
 # Not a boot parameter, and the one that could not be: it forwards to a
 # dependency's cargo feature, which no runtime value can reach.
 sched-check = ["toyos-sched/check"]
+# The longest interrupts-off and preemption-off window each CPU closed since the
+# last report, printed beside the IRQ census (`src/windows.rs`). Its own build
+# because it reads the counter at every masking transition and every preempt
+# count crossing zero, on every entry and exit: a kernel being measured for
+# anything else must not carry it.
+mask-windows = []
 # The stray-write tripwire on the per-CPU `CpuSched` record: a shadow copy of its
 # bytes, taken when the driver's exclusive region is left and compared when it is
 # next entered, so a word that changed while nothing was allowed to change it is

kernel/src/arch/aarch64/cpu.rs

@@ -67,14 +67,24 @@ pub unsafe fn write_thread_pointer(tp: u64) {
 
 /// Unmask interrupts on this CPU: `DAIF.I` and `DAIF.F`.
 pub fn enable_interrupts() {
+    #[cfg(feature = "mask-windows")]
+    if !interrupts_enabled() {
+        crate::windows::irqs_unmasking();
+    }
     // SAFETY: writes two `DAIF` bits; a compiler barrier, so no access moves across it.
     unsafe { asm!("msr daifclr, #3", options(nostack)) };
 }
 
 /// Mask interrupts on this CPU.
 pub fn disable_interrupts() {
+    #[cfg(feature = "mask-windows")]
+    let were_open = interrupts_enabled();
     // SAFETY: writes two `DAIF` bits; a compiler barrier, so no access moves across it.
     unsafe { asm!("msr daifset, #3", options(nostack)) };
+    #[cfg(feature = "mask-windows")]
+    if were_open {
+        crate::windows::irqs_masked();
+    }
 }
 
 /// Whether this CPU takes interrupts: `DAIF.I` clear.