A wake to another CPU that is busy is drained at its next preemption point and runs within a bound

issues/a-wake-whose-waker-parks-next-is-forwarded-to-an-idle-cpu.md

@@ -0,0 +1,27 @@
+---
+status: open
+kind: finding
+opened: 2026-10-12
+---
+
+# A wake whose waker parks next is forwarded to an idle CPU
+
+`CpuSched::place` (`kernel/pure/sched/cpu.rs`) hands a fair wake that would
+queue behind the running fair task to an idle sibling. When the waker is about
+to park on that same CPU — a request answered and waited on, a thread started
+and then joined — the CPU is free a moment later, and the forward pays an IPI
+and a cold cache where a local switch after the park was free. Nothing tells
+`place` that the running task is about to park.
+
+`acpi_mediated` met it on a guest: a thread started from a busy CPU went to the
+idle one, so it stages its asker differently since PR #874. On metal the
+spawn-then-wait row reads no change past one boot's spread: `spawn_cost`'s
+median went from 1,025 to 1,029 µs on the T14 and from 2,054 to 2,086 µs on
+the AMD bench, base to head, one boot each, read from the second judgement's
+`testcases` readbacks.
+
+Owned by stage S3 of the scheduler design (#872), whose 1:1 pair rule decides
+where a pair's wakes go.
+
+**Exit**: S3 decides this case, with a simulator scenario of a wake whose waker
+parks next and a metal row that prices the forward against the local switch.

issues/a-woken-thread-waits-out-a-sibling-spinners-quantum-when-every-cpu-is-busy.md

@@ -0,0 +1,40 @@
+---
+status: open
+kind: defect
+opened: 2026-10-12
+---
+
+# A woken thread waits out a sibling spinner's quantum when every CPU is busy
+
+With every CPU running a spinner of the woken thread's own process, a wake that
+lands where it cannot be forwarded and cuts nothing waits for the pick, and the
+pick serves the fair band's frozen insertion-time keys: a queued sibling whose
+key went stale when it was queued sits below the woken thread's, and two
+threads of one process tie at the running one's. With the busy-CPU wake change
+(PR #874) the PR's metal scout found phase 2's pair together on one CPU on both
+benches at both judgements, which base's spawn rotation never did; how it got
+there is not in the logs. One boot per tree each:
+
+- **Phase 2** (a ping-pong pair and eight spinners of its process, eight CPUs):
+  base 500 rounds in 5 s, head 250, on the T14 and the AMD bench, at both
+  judgements. Every head wake was posted to the waker's own CPU and waited one
+  quantum: max 10,003,344 ns on the T14 and 10,004,238 ns on the AMD bench.
+- **Phase 3** (sixteen spinners): the AMD bench's woken p99 rose from
+  ≤ 10,485,759 ns at base to ≤ 15,728,639 ns at the first judgement and
+  ≤ 12,582,911 ns at the second, at equal rounds. The T14's rose from
+  ≤ 16,777,215 ns to ≤ 29,360,127 ns at the first, and was lower than base's
+  at the second.
+- **Phase 5** (sixteen spinners of a second process): one T14 wake of 9,946
+  waited 10,002,218 ns, against the protected run's 1 ms; the code's
+  mechanism is the same pick, serving a queued spinner whose stale key sits
+  below the woken thread's.
+
+The logs: [first judgement](https://github.com/ToyOSOrg/evidence/tree/cfb4aa7c18348380fcb873e99809db20cf18af22/metal/judgement),
+[second](https://github.com/ToyOSOrg/evidence/tree/d63b878f86d09c3aefcad27fd30eb3485fa81e5b/r2/metal/judgement).
+
+Owned by stage S4 of the scheduler design (#872), per-thread EEVDF, which
+deletes the frozen insertion keys: the orchestrator's application of the
+owner's ruling (#885) on PR #874 files it there.
+
+**Exit**: S4's re-run of K1–K4 on both benches, which reads phases 2, 3 and 5
+against the base numbers above.

issues/an-ordinary-wake-posted-on-its-own-cpu-waits-for-that-cpus-next-pass.md

@@ -0,0 +1,36 @@
+---
+status: open
+kind: defect
+opened: 2026-10-12
+---
+
+# An ordinary wake posted on its own CPU waits for that CPU's next pass
+
+`park::deliver` (`kernel/pure/sched/park.rs`) posts an ordinary wake
+(`Urgency::Normal`) whose target CPU is the one the poster runs on without
+ringing that CPU's doorbell: the CPU's own next pass drains it. A device
+interrupt that posts a wake sets `need_resched` before its EOI, so the CPU it
+interrupted runs that pass on the way out. A thread that wakes a task parked
+on its own CPU and keeps running is what is left: the woken task stays in that
+CPU's mailbox until the thread blocks or its quantum ends, and no idle CPU can
+take it meanwhile. That is `main`'s behaviour for these wakes, kept by
+PR #874 where it kicks every other busy target. A thread in the real-time band
+woken without a boost is an ordinary wake too.
+
+Kicking the poster's own CPU, as PR #874's third round did, ran a pass inside
+the poster's call that placed every task the call woke onto that CPU. On the
+T14 a 64-waiter `wake_storm_cost` storm then cost its waker 132,867 to 153,899
+cycles in the three boots that carried that kick, against 52,910 to 74,328 in
+50 boots of trees without it, and one of the three crossed the row's linear
+bound (PR #874's fourth round). An 8-vCPU guest showed the pass's time going to the
+process's fair-share lock, which the storm's other waiters were taking as
+they re-parked.
+
+Owned by stage S3 of the scheduler design (#872), which decides where a
+pair's wakes go: a wake whose waker parks next is cheapest left here
+(`issues/a-wake-whose-waker-parks-next-is-forwarded-to-an-idle-cpu.md`).
+
+**Exit**: a wake posted on its own CPU by a poster that keeps running reaches
+a CPU within the bound `wake_onto_busy` holds for other CPUs' wakes, on the
+T14 and the AMD bench, and `wake_storm_cost` stays inside its linear bound on
+both.

issues/interactive-mix-crosses-the-fairness-bound-at-two-cpus.md

@@ -0,0 +1,45 @@
+---
+status: open
+kind: defect
+opened: 2026-10-12
+---
+
+# `interactive_mix` at two CPUs and sixteen hogs crosses invariant I5's derived bound
+
+`kernel/sim`'s invariant I5 bounds the service spread between two equal shares
+over one contention window by
+`lag_spread + (ΣT + 1) × (QUANTUM + max KernelSection + 2 × RUN_CHUNK)`
+(`kernel/sim/src/invariants.rs`, `check_fairness`). `interactive_mix(2, 16)` —
+one sleeper thread against a sixteen-thread hog process on two CPUs — crosses
+it on `main` as well as with the busy-CPU wake change (PR #874).
+
+Measured with I5's ceiling lifted (`with_fair_allowance(u64::MAX)`), seeds
+0..2000 alternating the seeded and PCT streams as `sim/tests/policy.rs` does,
+by a scratch test ([at `6e323ccf0`](https://github.com/ToyOSOrg/evidence/blob/b036b48d71bb81456b770307ccbd9532d80eb175/sim-investigation/scratch-tests/zz_scratch3.rs),
+[its output](https://github.com/ToyOSOrg/evidence/blob/b036b48d71bb81456b770307ccbd9532d80eb175/sim-investigation/i5-2000.txt);
+[again at `17c40459d`](https://github.com/ToyOSOrg/evidence/blob/d6d8e852b6e7e249e45d6218c252583ee624b03c/r3/sim/zz_scratch.rs), [its output](https://github.com/ToyOSOrg/evidence/blob/d6d8e852b6e7e249e45d6218c252583ee624b03c/r3/sim/scratch-at-head.log)):
+
+| tree | runs crossing | worst spread | bound in force there |
+|---|---|---|---|
+| `main` at `177e72a9e` | 90 / 2000 | 250 ms | 216 ms |
+| PR #874 at `6e323ccf0` | 196 / 2000 | 254 ms | 217 ms |
+| PR #874 at `17c40459d` | 197 / 2000 | 254 ms | 217 ms |
+
+The policy suite's forty seeds miss every crossing on `main`, and two of them
+(24 and 36) cross with the change, which is why `scenarios::interactive_mix`
+now carries a recorded allowance at that shape (`INTERACTIVE_I5_SAMPLE`, the
+worst spread plus a quarter, as `FAIRNESS_SAMPLE` does). `Outcome::fair_over_bound`
+still reports every crossing of the derived bound.
+
+What a crossing run shows (`interactive_mix(2, 16)`, seed 24, from a step
+timeline): the sleeper's first wake waits behind the eight hogs on its CPU,
+whose keys are still the spawn's zero, and each later quantum end on that CPU
+hands it to a hog whose key went stale when it was queued, while the sleeper
+counts as runnable throughout. The spread is the fair band's insertion-time
+keys, the same staleness the bound's `ΣT + 1` term carries, past that term.
+
+Owned by stage S4 of the scheduler design (#872), per-thread EEVDF, which
+deletes the frozen insertion keys and I5's staleness term with them.
+
+**Exit**: `interactive_mix(2, 16)` meets I5's derived bound over 2,000 seeds,
+and `INTERACTIVE_I5_SAMPLE` is deleted.

issues/no-watchdog-arms-on-an-amd-machine.md

@@ -0,0 +1,22 @@
+---
+status: open
+kind: defect
+opened: 2026-10-12
+---
+
+# No watchdog arms on an AMD machine
+
+The loader and the kernel arm only an Intel PCH's TCO timer
+(`kernel/src/arch/x86_64/watchdog.rs`, `toyos_tco`). On the AMD bench both say
+there is none — `watchdog: no function on bus 0 carries a TCO block this
+loader knows` and `watchdog: no PCI function here carries a TCO block this
+kernel knows — not armed` (PR #874's whole-suite run on that bench) — so a
+frozen boot there waits for a hand on the power button, and the
+`loader_watchdog_arms` metal row is red there. AMD's fusion controller hub
+carries a watchdog timer of its own; its register map is unread here.
+
+Owned by `issues/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md`,
+whose ruling is a watchdog on every machine.
+
+**Exit**: an AMD machine's boot arms its chipset's watchdog, and a row on the
+AMD bench reads it armed.

issues/the-hard-lockup-detector-samples-no-amd-cpu.md

@@ -0,0 +1,27 @@
+---
+status: open
+kind: defect
+opened: 2026-10-12
+---
+
+# The hard-lockup detector samples no AMD CPU
+
+`kernel/src/hardlockup/mod.rs` arms a counter only where CPUID leaf 0AH
+states Intel's architectural performance monitoring. An AMD CPU states none
+there, so on the AMD bench the boot says `hard lockup: 5000 ms, but CPUID
+states no architectural performance counter on this machine, so nothing
+samples a cpu that stops taking interrupts` (PR #874's whole-suite run on that
+bench), and the `hard_lockup_ends_a_deaf_cpu` metal row is red there. Its
+probe sends the deaf CPU the NMI from another CPU instead, which the row does
+not count as the detector.
+
+AMD CPUs have core performance counters of their own, which CPUID
+Fn8000_0001_ECX states and the local APIC's performance-counter LVT delivers
+(AMD64 Architecture Programmer's Manual Vol. 2, "Performance Monitoring
+Counters"); the AMD bench's leaf is unread here.
+
+Owned by `issues/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md`,
+whose next step arms the detector on every boot.
+
+**Exit**: the detector samples every CPU of an AMD machine from its own
+counter, and `hard_lockup_ends_a_deaf_cpu` passes on the AMD bench.

issues/the-kernel-brings-up-at-most-eight-cpus.md

@@ -0,0 +1,27 @@
+---
+status: assigned
+kind: defect
+opened: 2026-10-12
+---
+
+# The kernel brings up at most eight CPUs
+
+Every per-CPU structure in the kernel is a `static` array of `MAX_CPUS = 8`,
+declared three times (`kernel/src/sched/mod.rs`, `kernel/src/smp_roster.rs`,
+`kernel/src/shootdown.rs`) and pinned equal by `const` assertions. The roster
+refuses an id at or above it, so a machine whose firmware names more CPUs runs
+on the first eight in MADT order and leaves the rest off. The ABI carries the
+same number twice: `MAX_LOG_SHARDS = 8` is `LogCursor::next`'s width, which the
+trace diary's cursor shares, and `TraceRecord::cpu` is a `u16`.
+
+**Evidence.** The AMD bench's kernel log posted on PR #852
+(https://github.com/ToyOSOrg/ToyOS/pull/852#issuecomment-6107031099):
+`ACPI: MADT cpus=[0, 1, …, 15]`, then
+`SMP: roster full at 8 CPUs; ignoring further MADT entries`, then
+`SMP: 8 of 16 MADT cpus online`. Half its CPUs run nothing.
+
+**Owner:** the orchestrator, which holds the design round in PR #865.
+
+**Exit:** `git grep -n 'MAX_CPUS\|MAX_LOG_SHARDS' -- kernel sdk` finds
+nothing, and the AMD bench's `smp_roster_and_tsc_trail` row logs
+`SMP: 16 of 16 MADT cpus online`.

issues/the-simulator-rotates-spawn-placement-unlike-the-kernel.md

@@ -0,0 +1,27 @@
+---
+status: open
+kind: tooling
+opened: 2026-10-12
+---
+
+# The simulator rotates spawn placement from where the last spawn landed; the kernel from a counter
+
+Both call `CpuHandles::place(start, now)`, the least-loaded scan, but they
+choose `start` by different rules:
+
+- the kernel, `kernel/src/sched/driver.rs`'s `placement`: a global counter,
+  advanced by one on every spawn of any process, wherever the spawn landed;
+- the simulator, `kernel/sim/src/vm.rs`'s `Vm::spawn`: one past the CPU the
+  previous spawn landed on.
+
+On a machine where the loads tie the two agree; wherever a spawn lands
+somewhere other than its start, the next start differs, and so does every
+placement after it. So which CPUs a spawn burst fills — the arrangement the
+rest of a run is measured on — is a different sequence in the model than on
+metal. PR #874's metal judgement turned on exactly that arrangement: its
+scout's pair shared one CPU with a spinner of its own process on both
+benches, which a scratch simulator scenario of the same phase (8 CPUs, 8
+spinners, 6 seeds per tree) did not reach once.
+
+**Exit**: the simulator chooses `start` by the kernel's rule, from one
+definition both call.

issues/the-simulators-gate-command-waits-for-overlong-pass-to-abort.md

@@ -0,0 +1,23 @@
+---
+status: open
+kind: tooling
+opened: 2026-10-12
+---
+
+# `kernel-sim gate` waits for `overlong_pass` to abort, and it records instead
+
+`kernel/sim/src/main.rs`'s `gate` runs `overlong_pass` through
+`sweep::abort_gate` and prints `NOT CAUGHT — the harness proves nothing`
+unless the core aborts. `scenarios::overlong_pass`'s own doc says the
+opposite: "It must be recorded, not aborted" — the pass-cost recorder
+(`cpu::PassCosts`) replaced the assert, and `sim/tests/scenarios.rs`'s
+`a_pass_that_overruns_its_budget_is_recorded` holds the recording. So the
+command that `kernel/sim/src/lib.rs` names as the full exit criterion exits 1
+on a tree whose every gate holds.
+
+Measured on `main` at `177e72a9e`: `kernel-sim gate 10` prints
+`overlong_pass: NOT CAUGHT` and every other negative gate caught; PR #874's
+head prints the same at `gate 10000`.
+
+**Exit**: `kernel-sim gate` judges `overlong_pass` by its recorded pass costs,
+as the test does, and exits 0 on a tree whose gates hold.

issues/the-work-conservation-cases-were-staged-for-a-push-threshold-of-two.md

@@ -0,0 +1,33 @@
+---
+status: open
+kind: tooling
+opened: 2026-10-12
+---
+
+# The work-conservation cases were staged for a push threshold of two
+
+`kernel/sim/tests/topology.rs` and `scenarios::group_imbalance` came with the
+simulator's I15 under `PUSH_THRESHOLD` 2 and a busy victim that drained a
+probe at its quantum's end. At threshold 1, with every ordinary message
+kicking a busy target (PR #874), two of their claims stop being shown:
+
+- **I15's bound is not shown to constrain.** Each probe kicks the victim, and
+  the extra pass pushes the next sleeper, so the lopsided machine's cursor walk
+  ends long before the derivation's one push a round. Over 200 seeds the worst
+  stretch is 5 rounds against a bound of 10 on the T14's shape and 6 against
+  18 on the AMD bench's ([at `17c40459d`](https://github.com/ToyOSOrg/evidence/blob/d6d8e852b6e7e249e45d6218c252583ee624b03c/r3/sim/scratch-at-head.log)). The bound is still the derivation's worst case; no
+  sampled schedule comes near it, so
+  `the_push_keeps_an_idle_cpu_beside_work_inside_its_bound` no longer asserts
+  that the worst stretch is more than half the bound.
+- **`group_imbalance` no longer stages an idle CPU whose CCX holds nothing a
+  probe takes.** Each of cpu8's CCX neighbours runs one task and queues one,
+  and one queued task is now stealable, so cpu8 can be served from its own
+  CCX. A neighbour that looks loaded and has nothing stealable needs load the
+  fair band does not express.
+
+Owned by stage S2 of the scheduler design (#872), which replaces the push and
+the victim choice and derives I15's bound for them.
+
+**Exit**: S2's tests assert that a sampled schedule comes within half of I15's
+bound at both benches' shapes, and `group_imbalance` stages CCX load no probe
+takes at the shipped threshold.

kernel/loom/Cargo.toml

@@ -225,11 +225,11 @@ serial-try-lock-then-some = []
 # `tests/loom_mailbox.rs` asserts exactly that.
 no-preempt-guard = []
 # The negative control for the sleep handshake's kick edge.
-# It makes `mailbox.rs`'s `Doorbell::ring` read the doorbell's bits `Relaxed`
-# instead of `AcqRel`, so a producer can miss a consumer's freshly published
-# `SLEEPING` bit and elide the IPI that would have woken it, and
-# `loom_sleep.rs` must red under it — the one command that proves the model
-# has teeth:
+# It makes `mailbox.rs`'s `Doorbell::ring` read-modify-write the doorbell's
+# bits `Relaxed` instead of `AcqRel`, so a producer that coalesces into a kick
+# the consumer has already spent can leave its message out of that consumer's
+# drain, and `loom_sleep.rs` must red under it — the one command that proves
+# the model has teeth:
 #
 #   cargo test -p kernel-loom --features doorbell-kick-relaxed \
 #     --test loom_sleep

kernel/loom/src/sched_model.rs

@@ -161,6 +161,10 @@ impl Kicker for Kicks {
     fn kick(&self, _target: CpuId) {
         self.count.fetch_add(1, Ordering::AcqRel);
     }
+
+    fn runs_on(&self, _cpu: CpuId) -> bool {
+        false
+    }
 }
 
 /// Loom explores every interleaving of the modelled threads; the preemption

kernel/loom/tests/loom_mailbox.rs

@@ -9,9 +9,8 @@
 //! * `irq`    — CPU0 IRQ context: pushes (interrupting `local`'s push in some
 //!   interleavings), then at IRQ exit runs a pass *if* preemption
 //!   is enabled.
-//! * `remote` — another CPU: pushes and rings with `Urgency::Normal`, so a
-//!   busy target gets no IPI — its message is the one
-//!   that ends up behind the unlinked suffix.
+//! * `remote` — another CPU: pushes and rings with `Urgency::Normal`; its
+//!   message is the one that ends up behind the unlinked suffix.
 //!
 //! Two properties are checked.
 //!

kernel/loom/tests/loom_push.rs

@@ -153,9 +153,9 @@ fn a_cpu_that_halts_without_seeing_the_surplus_was_pushed() {
 /// The other direction, and the one that says the fence is not simply making
 /// everything look asleep: the push does not ring a CPU that is awake.
 ///
-/// A ring on a busy target sets the kick edge with no message behind it, which
-/// costs that target's next idle disposition a spurious extra pass — harmless,
-/// and exactly what the push is allowed to cost. What it may not do is ring a
+/// A ring on a busy target kicks it with no message behind it, which costs
+/// that target an interrupt and a spurious extra pass — harmless, and exactly
+/// what the push is allowed to cost. What it may not do is ring a
 /// CPU that never armed a sleep at all, because then the count this policy is
 /// priced by is measuring something that is not a wake.
 #[test]

kernel/loom/tests/loom_retire.rs

@@ -356,6 +356,10 @@ struct Silent;
 
 impl Kicker for Silent {
     fn kick(&self, _target: CpuId) {}
+
+    fn runs_on(&self, _cpu: CpuId) -> bool {
+        false
+    }
 }
 
 impl Machine for Silent {