Tracks: the T14 runs its tests in sessions of a booted ToyOS, and every machine ships a watchdog and a hard-lockup detector

issues/boot-media/the-loader-does-only-what-must-precede-the-handover.md

@@ -224,8 +224,9 @@ Each stage lands on its own, in this order.
      label `ready`, and the service writes one byte to it once it serves. A
      read end that closes with no byte is a service that never came up.
    - Init runs `update --good` once every service `[boot] up` names has
-     written its byte; `update` holds the `slots` claim's table and writes the
-     running slot's good flag and nothing else.
+     written its byte, and never for an image that names no `[boot] up`;
+     `update` holds the `slots` claim's table and writes the running slot's
+     good flag and nothing else.
    - An image whose `system.toml` grants no program the `slots` claim is never
      good, so each boot spends a try and it boots three times. Today only
      `system.toml` and `tests/updatecase/system.toml` grant it.
@@ -309,13 +310,35 @@ Each stage lands on its own, in this order.
    boot manager booting the entry the loader wrote.
 
 6. **The T14 bench.**
+   - It opens with a timed `ssh t14 update < image` of a session image of
+     `issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md`, since
+     an image switch costs that write and a reset: ToyOS has written this
+     stick at 0.2 s to 9.5 s per MiB.
    - It is built from #539's pieces that stage 6 takes above, and the bench
      path of `src/metal.rs`. `--via-ubuntu` stays as the old path.
+   - Every T14 image is signed with the bench key the T14 trusts, and a loader
+     reaches the T14 as an image does: ToyOS receives it over ssh and writes it
+     to the stick, and the running loader tries it once and keeps the old one
+     as the fallback (owner rulings). A stick that boots neither loader costs
+     a hand and `diag/flash.sh`.
+   - The host's first exec on an image is `update --good`, and a session image
+     names no `[boot] up`. A death goes by `update --once`, the kept slot boots
+     after it, and the host fetches the record over `sftp`.
    - The bench reads the loader's file off the ESP.
    - The tested pass's `loader.log` is kept as `loader-previous.log` by a
      rename (`SetInfo`), not by #539's copy.
    - It runs over the cable that
-     `issues/hardware/the-t14-answers-only-through-a-usb-stick.md` owns.
+     `issues/hardware/the-t14-answers-only-through-a-usb-stick.md` owns, and
+     waits on
+     `issues/isolation/a-reset-stops-xhci-and-leaves-every-claimed-pci-function-armed.md`,
+     `issues/hardware/the-t14-hung-after-rebooting-with-its-i219-faulted.md`,
+     `issues/panic-path/a-fatal-event-stands-down-both-bounds-and-may-leave-nothing-to-end-the-machine.md`,
+     `issues/hardware/most-t14-leases-land-one-dhcp-retry-late.md`,
+     `issues/diagnostics/a-swaps-redial-asks-again-with-no-event-to-wait-on.md`,
+     `issues/hardware/the-t14-redial-re-asks-mdns-after-every-refusal.md`,
+     `issues/diagnostics/a-first-dial-turned-away-before-a-line-is-waited-on-to-its-callers-bound.md`
+     and
+     `issues/diagnostics/a-netd-that-dies-while-serving-leaves-the-hosts-stream-silent.md`.
    - #539's issues `a-loader-change-reaches-a-machine-only-by-writing-its-stick`,
      `the-bench-reads-no-quiescent-log-volume`,
      `the-bench-runs-with-no-bound-on-its-own-boot`,
@@ -324,13 +347,26 @@ Each stage lands on its own, in this order.
      as far as it is true of what lands.
 
    **Exit**: `bench_loop_drives_a_toyos_machine` passes in QEMU. On the T14,
-   with Ubuntu never started, three things hold: a kernel change boots; a slot
-   with a flipped byte, no signature or a lower security version is refused
-   and the other boots; and a slot that dies falls back on its own.
+   with Ubuntu never started: a whole run, its sessions and every boot that
+   goes by `--once` (`deadlinewedge`, `hardlockup`, `usbload` and
+   `foreignrecord` among them), gives the verdicts a per-boot run gave at the
+   commit this stage branches from; a kernel change boots; a slot with a
+   flipped byte, no signature or a lower security version is refused and the
+   other boots; a slot that dies falls back on its own, and one whose `sshd`
+   refuses the host's key is never marked good; a `--once` image that panics
+   returns the machine to its session with its record judged; and a loader
+   sent over ssh boots once, and one that brings no slot to good leaves the
+   old one booting.
 
 7. **Ubuntu leaves the loop.** Delete `toyos-metal`'s `--via-ubuntu` path,
-   `--metal-via-ubuntu` and `bootloader/src/bootnext.rs`. After a reset the
-   firmware comes back to the loader because ToyOS's entry is first (stage 5).
+   `--metal-via-ubuntu`, `bootloader/src/bootnext.rs` and test-runner's
+   job-list mode. After a reset the firmware comes back to the loader because
+   ToyOS's entry is first (stage 5).
+   - `issues/build/a-hung-boots-log-partition-is-wiped-by-the-next-runs-flash.md`,
+     `issues/build/the-sudoers-rendering-has-no-host-side-judge.md`,
+     `issues/build/the-metal-loop-writes-the-readback-volume-into-a-directory-it-has-not-made.md`
+     and `issues/hardware/the-t14-stopped-answering-ssh-between-two-lan-boots.md`
+     close here, each only as far as it is true of what lands.
 
    **Exit**: no path in `src/metal*.rs` reaches Ubuntu. On the T14, a panic's
    reset reaches the loader with no `BootNext` set.
@@ -358,11 +394,13 @@ Each stage lands on its own, in this order.
      `watchdog_quiet`'s loader half. `watchdog_armed` judges the kernel's
      read-back, and `loader_watchdog_arms` becomes the kernel's row.
 
-   **Exit**: `bootloader/src` holds no TCO access. On q35, `watchdog_armed`
-   passes on the kernel's lines (counting, `no_reboot=0`, `timeout=0`), and
-   `watchdog_resets` passes. A boot that hangs right after the arm, before
-   `mm::init`, is reset by the TCO; moving the arm back after `pci::enumerate`
-   makes that test fail.
+   **Exit**: `bootloader/src` holds no TCO access. On the T14, `watchdog_armed`
+   passes on the kernel's lines (counting, `no_reboot=0`, `timeout=0`). Once
+   the reading of
+   `issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md` has
+   seen the TCO reset the T14, a boot there that hangs right after the arm,
+   before `mm::init`, is reset by the TCO; moving the arm back after
+   `pci::enumerate` makes that row fail.
 
 9. **The crash report belongs to the kernel.** The loader hands the last
    boot's record to the kernel instead of decoding it. The kernel logs it, and
@@ -373,8 +411,7 @@ Each stage lands on its own, in this order.
      `DROPPED_OPENS_WITH`'s count.
    - The report pass goes, taking with it `end_this_pass`, the chain
      constants, `loaderlog`'s chain lines, `armed_at`'s `GetTime`, everything
-     else in `bootloader/src/blackbox.rs` except the claim and the arm, and the
-     chained-pass item of `issues/hardware/the-t14-boots-toyos-unattended.md`.
+     else in `bootloader/src/blackbox.rs` except the claim and the arm.
    - `ENDS_AT_CHAIN` goes, so this stage rewrites the exit of
      `issues/diagnostics/a-wedged-reports-newest-records-were-cut-by-the-loaders-own-log-file.md`:
      a `deadlinewedge` rerun whose `/log` carries the `WEDGED` report with its

issues/build/hard-lockup-bound-ms-is-read-by-nothing-but-its-own-assertion.md

@@ -19,10 +19,12 @@ alone.
 
 ## Owner
 
-`issues/hardware/the-t14-boots-toyos-unattended.md`, the track that holds the
-hard-lockup detector and its bound.
+`issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md`, the
+track that arms the hard-lockup detector on every boot.
 
 ## What would close it
 
-The constant and its assertion deleted, with whatever `toyos-tco`'s version
-owes for it; `hard_lockup_bound_ms`'s own test already holds the half.
+That track's first step: a boot that names no `boot-deadline=` arms the
+detector at `HARD_LOCKUP_BOUND_MS`, whose doc then says so in place of "the one
+a T14 boot runs under". The constant is not deleted, since that step would
+declare it again.

issues/build/the-update-rig-the-guest-cut-deleted-is-still-cited.md

@@ -0,0 +1,37 @@
+---
+status: open
+kind: tooling
+opened: 2026-10-03
+---
+
+# The update rig the guest cut deleted is still cited
+
+#660 (`06788146b`) deleted `tests/common/update.rs` with its `Rig`,
+`tests/common/fwvars.rs` and `tests/updatecase/system.toml`. `git grep -n
+'fwvars\|updatecase\|common/update\.rs\|Rig::\|vars::plant\|vars::live'` finds
+nine lines in three files, each planning or describing a test on them:
+
+- `issues/boot-media/the-loader-does-only-what-must-precede-the-handover.md`,
+  six lines. Stage 3's exit takes `fwvars::live` as
+  `update_floor_is_the_images_own`'s oracle. Stage 5 says
+  `tests/updatecase/system.toml` grants `slots` and that two tests run
+  `updatecase`; one of its negative controls signs with `Rig::update` and
+  reads the floor with `fwvars::live`, which its oracles name too.
+- `issues/boot-media/the-loader-never-sets-the-firmwares-memory-overwrite-request.md`,
+  two lines: its exit's `mor_is_set_where_defined` plants and reads the vars
+  store with `vars::plant` and `vars::live`.
+- `issues/build/the-kernel-console-split-does-not-re-arm-across-a-guest-reset.md`,
+  one line: the reset in place it describes is `Rig::boot`'s.
+
+What each should name is the tier its test has now, and stage A of
+`issues/build/the-guest-suite-runs-only-what-no-cheaper-tier-reaches.md` gives
+it: `update_floor_is_the_images_own`, `update_refusals_boot_the_other_slot`
+and `update_grant_refuses_a_stray_partition` are host tests there and guest
+tests in the loader track, and the tree holds no `update_*` test today.
+
+**Owner**: that stage A, whose pull request lands those tests and with them
+the oracle and the rig a sentence can name.
+
+**Exit**: the `git grep` above finds this file alone: each sentence names the
+tier that holds its test and an oracle, a rig or a config the tree has, or has
+gone with the test it described.

issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md

@@ -0,0 +1,96 @@
+---
+status: open
+kind: track
+opened: 2026-09-30
+---
+
+# A frozen ToyOS waits for a hand on the power button
+
+No shipped image arms a hardware watchdog or the hard-lockup detector: the
+kernel arms the chipset's TCO only on the `watchdog` parameter
+(`arch::watchdog::init`), feeds it from the scheduler pass
+(`kernel/src/sched/driver.rs:512`), and arms the detector only through
+`boot-deadline=` (`kernel/src/deadline.rs:168-183`).
+
+**The owner's rulings:**
+
+- A userland `watchdogd` feeds the watchdog on every machine, with no new
+  syscall; the detector is armed on every boot; nothing ships for tests alone.
+- **A panic's panel holds as it does today** and feeds the watchdog while it
+  holds, after a key too (`panic_console::hold_the_panel`).
+- **`watchdogd` reaches the chipset's timer through the general port claim
+  #592 brings, its `isa` claim, as the ACPI server will**, and not through a
+  device class of its own: one way of doing it and not two (2026-10-03). That
+  ruling does not settle two things, and nothing here decides them:
+  - #592's grant covers ports below `0x100`, and the T14's TCO is at `0x400`
+    (`issues/hardware/an-armed-tco-has-never-reset-the-t14.md`).
+  - #592 refuses a claim on ports the kernel drives, and under this track the
+    kernel still arms the timer, feeds it from the panel and disarms it.
+
+**The track's own lines, which no ruling carries:**
+
+- **The scheduler pass feeds from the kernel's arm until the timer is first
+  claimed, and never after**, so a `watchdogd` that ends with no successor
+  holding the claim resets the machine. After that first claim the panel's
+  hold is the one feed the kernel keeps. A successor's claim that meets its
+  predecessor's release still in flight
+  (`issues/kernel/deferred-release-outlives-its-syscall.md`) is refused, so
+  `watchdogd` waits on that defect.
+- A job holding `device` can mint the claim where no `watchdogd` holds it,
+  which
+  `issues/isolation/every-job-test-runner-starts-holds-its-whole-system-capability.md`
+  fences.
+- `watchdogd` feeds from a thread under `rt`, until
+  `issues/kernel/cpu-time-is-a-band-and-not-a-reservation.md` gives it a
+  reservation.
+- **Every guest runs with `-action watchdog=none`.** q35's TCO counts
+  `QEMU_CLOCK_VIRTUAL`, which a loaded host advances while it starves a guest,
+  and its second expiry does what `-action watchdog=` says (QEMU 11.1.1,
+  `hw/acpi/ich9_tco.c:61-70,244`). A TCO reset, and a feeding kernel that
+  `watchdog_fed` finds not reset, are judged on the T14.
+- The T14's PCH TCO is its one reset that needs nothing of the kernel (no BMC,
+  no AMT, a battery no switch cuts), and no armed TCO has reset it yet
+  (`issues/hardware/an-armed-tco-has-never-reset-the-t14.md`).
+
+**Next: the detector on every boot**, at the half of `boot-deadline=` it takes
+today or at `toyos_tco::HARD_LOCKUP_BOUND_MS` where a boot names none. The
+constant then has a reader, which closes
+`issues/build/hard-lockup-bound-ms-is-read-by-nothing-but-its-own-assertion.md`.
+
+**Exit**: a host test holds the bound of a boot that names no `boot-deadline=`
+to `HARD_LOCKUP_BOUND_MS`, and putting the arm back behind `deadline::start`
+fails to build. `Armed`'s `(0, _)` arm, which nothing reaches, is gone.
+
+**Then, once loader stage 7 has taken Ubuntu**, whose `iTCO_wdt_probe` clears
+`SECOND_TO_STS` (Linux 6.12, `drivers/watchdog/iTCO_wdt.c:545-560`), out of the
+loop: **the reading**, `watchdog_resets`' metal row in
+`issues/build/the-guest-suite-runs-only-what-no-cheaper-tier-reaches.md`. The
+owner's ruling (2026-10-03): one T14 boot may stop feeding the chipset's
+watchdog on purpose, "only if its quick i dont want tests doing nothing for a
+long time". So the row's wait is bounded by the watchdog's own bound and fails
+loudly past it, with no long idle wait. One T14 boot is armed with `watchdog`
+and `wedge-before-reset`, under a deadline past its job list and
+`toyos_tco::BOUND_MS`, which `toyos_tco::STAGED_BOUND_MS` is not. The fed
+control is `watchdog_fed`, once
+`issues/hardware/watchdog-fed-ends-its-boot-before-the-bound-it-judges.md`
+closes.
+
+**Exit**: the next armed kernel's `TCO2_STS` read, in `arch::watchdog`'s
+`arm`, says the last boot ended in a TCO reset, and the page still reads
+`ARMED` where the deadline would have sealed a record; the armed boot after
+that says no TCO reset. That closes
+`issues/hardware/tco2-sts-clearing-is-verified-on-qemu-only.md`. Otherwise the
+T14 exit below waits on the TCO defect.
+
+**Last: `watchdogd` ships**, after loader stage 8
+(`issues/boot-media/the-loader-does-only-what-must-precede-the-handover.md`)
+has made the kernel's arm the only one. `toyos_tco::PARAM` goes, and with it
+`watchdog_quiet`, `loader_watchdog_arms`' control arm and the
+`testcases-watchdog` boot: the kernel arms every boot whose chipset has a
+`toyos_tco` row, q35 included, and one with none says it is unwatched.
+
+**Exit**, on the T14: a boot whose `watchdogd` ends with no successor, and one
+wedged by `wedge-before-reset` with `watchdogd` feeding, each end in the TCO's
+reset, and the boot after each says so; a kernel whose scheduler pass feeds on
+once the claim has gone keeps the first up. The boot after a held panic says
+no TCO reset.

issues/hardware/most-t14-leases-land-one-dhcp-retry-late.md

@@ -0,0 +1,32 @@
+---
+status: open
+kind: defect
+opened: 2026-09-29
+---
+
+# Most T14 leases land one DHCP retry late
+
+Over the LAN boots in the readbacks of three T14 runs, the I219's link came up
+2.72–2.82 s after its driver every time. Of the 16 leases, 5 landed 3.2–3.6 s
+after netd came up and 11 at 13.1–13.4 s, so the first lease of a boot, and
+with it `sshd`, arrived at 9.1–9.3 s of boot on 4 of the 13 boots and at
+19.0–19.1 s on the other 9.
+
+The ten-second step is smoltcp 0.12's default `RetryConfig::discover_timeout`,
+which netd keeps. netd restarts
+discovery when the link comes up (`userland/netd/src/dhcp.rs:42-61`), and on
+the late boots the lease came from the DISCOVER sent one timeout after that
+one. What became of the first, whether it left the machine and whether an
+OFFER came back, is unmeasured. RFC 2131 §4.1 puts the first retransmission
+at 4 s, randomized by ±1 s.
+
+The network track owns it: netd leaves smoltcp in stage 5 of
+`issues/design-debt/toyos-has-its-own-network-stack.md`, whose stage 3 is
+`toyos-dhcp`. Stage 6 of
+`issues/boot-media/the-loader-does-only-what-must-precede-the-handover.md`
+waits on it.
+
+**Exit**: netd logs each DISCOVER it sends and each OFFER it receives, and a
+T14 boot's log shows what became of the DISCOVER sent as the link came up; on
+every boot of a T14 run an unanswered DISCOVER is sent again within RFC 2131
+§4.1's 4 ± 1 s.