From bf7edfa08ba5014d7e0753e195b44595f617affa Mon Sep 17 00:00:00 2001 From: japabu Date: Tue, 29 Sep 2026 21:15:19 +0200 Subject: [PATCH 01/11] The T14 runs its tests in sessions: the track that takes Ubuntu out of the loop Every metal arm is its own boot today. Each one is a flash over ssh to Ubuntu, `BootNext`, three POSTs, and a readback. Three full runs of 25-27 boots spent 191-218 s in their kernels, out of 2683-2957 s of boot cycles (sums over t14rig/budget.txt). issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md designs the loop that replaces this. - A session is one image. Its tests run one after another, one `exec` each, as `test-runner ` over ToyOS's own sshd on the I219. init's launcher gives each test a fresh process holding the runner's row. - After each test the runner checks that nothing is left over: processes, device claims, files, and `must_be_clean` over the test's slice of the stream. - The kernel deadline becomes a lease the host renews. - Another image costs one `update` and one reset. A death costs an `update --once` and two resets (three until the loader track's stage 9). Folded here: - issues/hardware/the-t14-boots-toyos-unattended.md is deleted. Its umbrella (flash, `BootNext`, read the log partition) is what this track replaces. Its chained-pass item becomes stage 2's `loader-previous.log` rename. Its TCO, AP-IDT and measurement items already have their own files. The sentinel's remaining gap (before `clock::init`) is stated in kernel/src/deadline.rs's header. Its one citation, loader track stage 9, goes. - The loader track's stages 6 (the T14 bench) and 7 (Ubuntu leaves the loop) are now this track's stage 2. The bench pieces from #539 and #539's five issues move with them. The no-Ubuntu track's stage 2 points here for taking Ubuntu out of `toyos-metal`. Evidence, from commands run on the three readback sets under target/metal/ of toyos-fsd, toyos-metaljudges and toyos-perfstate: - POST was 8.8-21.7 s (median 9.3 s) and the loader 1.6-7.8 s (median 2.2 s), over 78 kernel `boot:` lines. - The I219's link came up 2.7-2.8 s after its driver. - 4 of 14 leases landed 3.2-3.6 s after netd started, and 10 landed at 13.3-13.4 s. smoltcp retries after 10 s (userland/netd/src/dhcp.rs). - Every talking and swapping boot dropped 9-24 frames with no transmit descriptor free (`TX_RING` is 16). `boot-deadline=` is a whole-boot bound (kernel/src/deadline.rs), and `hardlockup::start` is reached only from `deadline::start`. A boot that stays up has neither bound today. Co-Authored-By: Claude Opus 5.5 --- ...oes-only-what-must-precede-the-handover.md | 46 ++-- ...e-machine-updates-itself-without-ubuntu.md | 3 +- .../the-t14-boots-toyos-unattended.md | 119 ---------- ...4-reboots-through-ubuntu-for-every-test.md | 207 ++++++++++++++++++ 4 files changed, 224 insertions(+), 151 deletions(-) delete mode 100644 issues/hardware/the-t14-boots-toyos-unattended.md create mode 100644 issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md diff --git a/issues/boot-media/the-loader-does-only-what-must-precede-the-handover.md b/issues/boot-media/the-loader-does-only-what-must-precede-the-handover.md index 9ae21c395e..e34db8bf74 100644 --- a/issues/boot-media/the-loader-does-only-what-must-precede-the-handover.md +++ b/issues/boot-media/the-loader-does-only-what-must-precede-the-handover.md @@ -28,7 +28,8 @@ PR #539 does not land. Its pieces: `update_boot_next_boots_the_entry_once`'s read-only half; the harness's `stick_readonly`; and `update_trial_writes_nothing_of_the_kept_slot`, rewritten for priorities; -- stage 6 takes the bench: `src/metalbench.rs`, `tests/common/bench.rs`, +- stage 2 of `issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md` + takes what it uses of the bench: `src/metalbench.rs`, `tests/common/bench.rs`, `tests/bench*case`, `--bench-image` with `build::bench_image`, `src/image.rs`' `update_of`, `tests/common/metal.rs`' `Reach`, `stage` and bench `invocation`, `--metal-via-ubuntu`, toybox `date`, `Ssh::probe` and @@ -248,8 +249,8 @@ Each stage lands on its own, in this order. differs from `LAST_LAYOUT` (stage 4's derived value, pinned as a literal) and `assert!(LAYOUT != LAST_LAYOUT)` builds. This stage is an ABI change. - `update --boot-first` writes only the request. The loader writes its own - `HD(…)/File(…)` entry and puts it first. Once stage 7 deletes - `bootnext.rs`, that is the only boot-variable write. + `HD(…)/File(…)` entry and puts it first. Once `bootnext.rs` is deleted + (stage 7), that is the only boot-variable write. - This stage deletes the floor issue's "a floor that never rises" bullet and its exit's second half. @@ -306,32 +307,16 @@ Each stage lands on its own, in this order. format's bytes and not by `toyos_update::slots::Table::decode`; and OVMF's boot manager booting the entry the loader wrote. -6. **The T14 bench.** - - It is built from #539's pieces that stage 6 takes above, and the bench - path of `src/metal.rs`. `--via-ubuntu` stays as the old path. - - The bench reads the loader's file off the ESP. - - The tested pass's `loader.log` is kept as `loader-previous.log` by a - rename (`SetInfo`), not by #539's copy. - - It runs over the cable that - `issues/hardware/the-t14-answers-only-through-a-usb-stick.md` owns. - - #539's issues `a-loader-change-reaches-a-machine-only-by-writing-its-stick`, - `the-bench-reads-no-quiescent-log-volume`, - `the-bench-runs-with-no-bound-on-its-own-boot`, - `the-bench-sometimes-comes-back-two-minutes-late` and - `the-benchs-cable-is-read-by-the-driver-under-test` land here, each only - as far as it is true of what lands. - - **Exit**: `bench_loop_drives_a_toyos_machine` passes in QEMU. On the T14, - with Ubuntu never started, three things hold: a kernel change boots; a slot - with a flipped byte, no signature or a lower security version is refused - and the other boots; and a slot that dies falls back on its own. - -7. **Ubuntu leaves the loop.** Delete `toyos-metal`'s `--via-ubuntu` path, - `--metal-via-ubuntu` and `bootloader/src/bootnext.rs`. After a reset the - firmware comes back to the loader because ToyOS's entry is first (stage 5). - - **Exit**: no path in `src/metal*.rs` reaches Ubuntu. On the T14, a panic's - reset reaches the loader with no `BootNext` set. +6. **The T14 bench** and 7. **Ubuntu leaves the loop** are stage 2 of + `issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md`, which + drives the T14 by sessions of tests over the cable and deletes + `bootloader/src/bootnext.rs`. #539's issues + `a-loader-change-reaches-a-machine-only-by-writing-its-stick`, + `the-bench-reads-no-quiescent-log-volume`, + `the-bench-runs-with-no-bound-on-its-own-boot`, + `the-bench-sometimes-comes-back-two-minutes-late` and + `the-benchs-cable-is-read-by-the-driver-under-test` land there, each only + as far as it is true of what lands. 8. **The kernel arms the TCO before anything unbounded.** The loader's arm is the only bound today from `ExitBootServices` to `deadline::start` on @@ -371,8 +356,7 @@ Each stage lands on its own, in this order. `DROPPED_OPENS_WITH`'s count. - The report pass goes, taking with it `end_this_pass`, the chain constants, `loaderlog`'s chain lines, `armed_at`'s `GetTime`, everything - else in `bootloader/src/blackbox.rs` except the claim and the arm, and the - chained-pass item of `issues/hardware/the-t14-boots-toyos-unattended.md`. + else in `bootloader/src/blackbox.rs` except the claim and the arm. - `ENDS_AT_CHAIN` goes, so this stage rewrites the exit of `issues/diagnostics/a-wedged-reports-newest-records-were-cut-by-the-loaders-own-log-file.md`: a `deadlinewedge` rerun whose `/log` carries the `WEDGED` report with its diff --git a/issues/boot-media/the-machine-updates-itself-without-ubuntu.md b/issues/boot-media/the-machine-updates-itself-without-ubuntu.md index d4559130bc..758b1101ba 100644 --- a/issues/boot-media/the-machine-updates-itself-without-ubuntu.md +++ b/issues/boot-media/the-machine-updates-itself-without-ubuntu.md @@ -42,7 +42,8 @@ The machine boots the stick and installs onto its own NVMe, then takes `ssh t14 update < image` for every change after. It waits on stage 5 of `issues/boot-media/the-loader-does-only-what-must-precede-the-handover.md`, whose `update --boot-first` puts the NVMe loader's entry first; taking Ubuntu -out of `toyos-metal`'s loop is that track's too. +out of `toyos-metal`'s loop is stage 2 of +`issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md`. **Exit**: with the stick pulled, the T14 boots ToyOS off its NVMe, and a kernel change sent with `ssh t14 update < image` boots at the next reset. diff --git a/issues/hardware/the-t14-boots-toyos-unattended.md b/issues/hardware/the-t14-boots-toyos-unattended.md deleted file mode 100644 index 7e5de03080..0000000000 --- a/issues/hardware/the-t14-boots-toyos-unattended.md +++ /dev/null @@ -1,119 +0,0 @@ ---- -status: open -kind: track -opened: 2026-09-03 ---- - -# The T14 boots ToyOS unattended and reports through a log partition - -The T14 stopped being a GitHub Actions runner (owner ruling, 2026-09-03), so -what the tracker owes "on hardware" is owed to this loop: a driver on this Mac -flashes the stick left plugged into the machine, sets one boot, reboots, and -reads the verdict off a log partition. **There is no serial channel** — the -16550 loopback reads `0xFF` -(`issues/hardware/a-metal-session-runs-a-pre-flash-gate-first.md`, which is also -the loop's admission check: no image is flashed that has not passed it) — so the -log partition and the screen are the only two channels there are. - -What is left to build: - -- **The chipset watchdog is armed and does not count**, so nothing this tree - arms can end a wedged boot; a panicking kernel ends one by its own bound - instead, and a kernel that wedges without panicking after `clock::init` is now - ended by `kernel/src/deadline.rs` (below) — before it, still nothing. - `issues/hardware/an-armed-tco-has-never-reset-the-t14.md` carries the - registers and the datasheet. **Exit**: a T14 boot that resets itself on an - armed TCO. - - **The software answer was designed and is not built, and the first reason is - a number this loop does not have yet.** The design evaluated was a sentinel: - an AP started early becomes a CPU spinning on the TSC, the BSP publishes each - boot phase it reaches, and a phase missed within a bound seals a record into - the black box and writes the FADT reset register. Four findings, taken by - reading the code: - - 1. **The bound cannot be derived.** It has to be wider than the slowest - healthy phase on this machine and narrower than a wait for a hand, and - nothing in this tree has measured a T14 phase duration. A bound guessed - wrong resets a healthy laptop in a loop, which is worse than the present - state. This loop's boot-fact run is what publishes those numbers, so the - design is sequenced behind it rather than blocked on a ruling. - 2. **A sentinel is a CPU outside the roster, and the roster is modelled.** - `Roster::begin_attempt`/`commit` hands out dense ids that `boot_aps` fills - in attempt order, `smp_failed_ap_leaves_no_hole` is the gate that exists - because a hole in them is a defect, and `kernel-loom/tests/smp_bringup.rs` - is what decides whether a second committer is sound. A CPU taking an id - before `boot_aps` runs is a change to that protocol and needs the model - extended first. - 3. **It cannot start before the machine has a clock without a second - bring-up path.** `boot_aps`' SDM §8.4.4.1 delays and its 100 ms start - budget are spun on `clock::nanos_since_boot`, which answers zero until - `clock::init` — so an AP started before that never leaves the delay. - `clock::cpuid_tsc_hz` exists for the panic path's version of this problem - and would serve, at the cost of a second decider for how long an - INIT-SIPI wait is. - 4. **The reset it would take is not lock-free.** `acpi::reboot` opens with - `serial::flush_final`, and a `BackendGuard` masks interrupts for its whole - life — so a BSP wedged inside one holds the lock a sentinel would spin on, - on exactly the boots the sentinel exists for. A `reset_now` that writes the - decoded port and nothing else is small and separable from the rest. - - What no design of this shape can cover is the span before `apic::init`: no AP - can be started before the BSP's own LAPIC is enabled, and `acpi::init_reset` - — which decodes the register any of this would write — runs inside it. That - floor is inherent rather than an argument against the design. - - **The half of it that does not need an AP is built** (`kernel/src/deadline.rs`): - a bound armed off the parameter line as `boot-deadline=` and polled from - the timer interrupt entry, in both rings, on every CPU — two atomics and a - `rdtsc`, no lock, no allocation. On expiry it seals a `WEDGED` record naming - the bound, the uptime, the boot phase and **the tail of the log ring** into - the black box, and writes the reset register. That closes findings 1 and 4 - for everything after `clock::init`: the bound is derived rather than guessed - (`toyos_tco::WEDGE_BOUND_MS`, twice the runner's own, so a job the runner is - about to end is not a wedge), and `acpi::reset_now` is the lock-free write - finding 4 asked for — `reboot` now goes through it, so there is one writer of - that port. It ends a wedge whose every CPU has stopped taking scheduler - passes: measured on QEMU as `boot_deadline_ends_a_wedge`, 18 s with the arm - against a machine that never came back without it, twice at 66 s. - - **The second of the two seams that were left is closed, and not by a - sentinel.** A machine on which no CPU takes an interrupt is now - `kernel/src/hardlockup`: the local APIC's performance-counter LVT in NMI - delivery mode, armed on every CPU whose CPUID states an architectural PMU, - sampling that CPU's own interrupt count about once a second and sealing a - record — the CPU, its `rip` and `rsp` off the NMI frame, the lock it is - spinning on, a line for every CPU — from the stuck CPU's own NMI, then - `reset_now`. Its bound is half the deadline's and off the same parameter - (`toyos_tco::hard_lockup_bound_ms`), so the two compose rather than race. That - is the state run 22 sat in past 420 s with the deadline armed and unfired, and - it wanted a *sample* rather than a poll — no CPU outside the roster would have - helped, since a sentinel spinning on the TSC still has to be given a CPU that - runs, and the CPUs that were running were the deaf ones. - - **What is left for a sentinel is finding 3's span alone**: before - `clock::init` there is no TSC period to convert either bound with, and before - `apic::init` there is no LVT to arm — so a kernel that dies in early bring-up - still needs a hand, and only a CPU started that early reaches it. Finding 2 - (the roster is modelled and a CPU taking an id outside it is a protocol - change) stands unchanged and is what that would cost. -- **An AP loads its IDT before its control registers**, so a fault in that span - triple-faults the machine — - `issues/kernel/an-ap-loads-the-idt-before-its-control-registers.md`, whose - exit is one of the two orderings it names, judged by `smp_bringup` and the - SMP suite. -- **A loader pass overwrites the pass before it.** Every pass writes - `loader.log`, so on a chained boot the second pass's three lines replace the - first pass's record of the boot it is reporting on — measured on run 10, where - the surviving file was 228 bytes and the boot's own watchdog read-back was - gone. **Exit**: a chained run whose stick carries every pass, each named by - the pass that wrote it. -- **The measurements owed on hardware are this loop's jobs**, by record: - `issues/kernel/the-split-window-tlb-cost-is-unpriced.md`, - `issues/kernel/ap-control-registers-inherit-init.md`, - `issues/kernel/ap-tsc-trail-is-assumed-and-never-checked.md`, - `issues/audio/hda-ring-fix-unverified-on-metal.md`, and the - IOMMU track's three hardware-only answers — isolation scopes and reserved - regions, the 2× cost bar, and the compatibility-format question in - `issues/kernel/qemu-passes-compatibility-format-interrupts.md` - (`issues/kernel/the-iommu-refuses-nothing-yet.md` states the first two). diff --git a/issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md b/issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md new file mode 100644 index 0000000000..3f2cf13fcb --- /dev/null +++ b/issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md @@ -0,0 +1,207 @@ +--- +status: open +kind: track +opened: 2026-09-29 +--- + +# The T14 reboots through Ubuntu for every test + +Every metal arm is a boot of its own. The host flashes the stick over ssh to +Ubuntu, sets `BootNext`, and reads the log partition back after the loader's +report pass has reset the machine into Ubuntu. That is one flash and three +POSTs per boot. Three full runs of 25–27 boots spent 191–218 s in their +kernels, out of 2683–2957 s of boot cycles. The owner's direction is that the +tests run inside ToyOS one after another, with no reboot per test and no +Ubuntu. + +- **The transport is the LAN.** The host reaches ToyOS's own `sshd` on the + I219 at `toyos-t14.local` through `tests/ssh-client-host`. The T14 has no + serial channel, and nothing in the tree drives USB in device mode. + `logd`'s stream (TCP 41337) stays open for the whole session. The host cuts + it per test at the runner's markers. +- **A session is one image**: a kernel build, a parameter line and a config. + Every test of that image runs in it, one after another, with no reboot. A test + is one `exec` of `test-runner `, which `sshd` starts through init's + `launcher`. Each test is therefore a fresh process holding test-runner's + manifest row: init builds its namespace and claims, and they go when it + exits. It prints the `===TEST_START/END===` markers QEMU's shared block + already reads off serial. +- **A test leaves the machine as it found it, and the runner checks that.** + After each test the runner checks four things: + - no process the test started is alive (`roster`); + - every device claim has the holder it had before (`inventory`); + - `/tmp`, the session home and `/state` list as they did, which is the only + file fence until per-program views land; + - the kernel's records in the test's slice pass `must_be_clean`. + + A leftover reds that test by name. The session then resets before the next + test. +- **The bound is a lease.** `boot-deadline=` is armed from boot, and + `hardlockup` is armed only through it (`deadline::start`). A boot that stays + up therefore has neither. The deadline becomes a lease: + - each test's launch renews it for twice the runner's bound on that test, + as `WEDGE_BOUND_MS` is twice `JOB_BOUND_MS`, and the host renews it + between tests; + - a lapse seals `WEDGED` and resets the machine; + - an image that reaches good retires the lease its parameter started, so a + machine the host has let go of idles rather than resets; + - `hardlockup` is armed on every boot. +- **Another image costs one reset, and a death costs two.** + - For another image, the host runs `update < image` into the idle slot and + reboots: one POST, no Ubuntu. + - A panic, wedge, reset or loader test goes in with `update --once`. Its end + resets the machine and the loader boots the kept slot. The host then + fetches the record over `sftp`: from `loader.log` until the loader track's + stage 9, and from `/log` after it. Until that stage the report pass costs a + third POST. + - A session image's `[boot] up` names netd and sshd, so an image the host + cannot reach is never good and its slot falls back. +- **What no detector catches stops the run and waits for a hand.** That is the + span before `clock::init`, a machine that powers itself off, firmware that + does not come back, or a fatal event that stood both bounds down. The host + declares the machine lost when nothing has answered within the lease and a + POST allowance, derived as `return_secs` is. It keeps what the stream + carried. An armed TCO has never reset this machine, and whether it counts is + undecided (`issues/hardware/an-armed-tco-has-never-reset-the-t14.md`), so + nothing here rests on it. +- **QEMU needs no resident runner.** Its shared block is already one boot for + many tests, and each of its machine tests has its own boot as its subject. + It does take the per-test check. The KVM and CI lanes do not change. + +Measured from those three runs' readbacks: + +- POST (power-on to loader) took 8.8–21.7 s, median 9.3 s over 78 boots. The + loader took 1.6–7.8 s, median 2.2 s. +- The I219's link came up 2.7–2.8 s after its driver on every boot. +- 4 of 14 DHCP leases landed 3.2–3.6 s after netd started. The other 10 + landed at 13.3–13.4 s: the first DISCOVER was lost and smoltcp retries after + 10 s. So the lease, and with it `sshd`, arrives at 9.1–9.3 s or at + 19.0–19.1 s of boot. +- Every talking and swapping boot dropped 9–24 frames because no transmit + descriptor was free (`toyos_i219::TX_RING` is 16). A frame the stream loses + waits for TCP's retransmit. + +## Stages + +1. **One session on today's rig.** + - `test-runner` gains a one-test mode. It renews the lease, runs the job, + kills it at its bound, runs the check, and exits without rebooting. + - The deadline becomes the lease and `hardlockup` is armed on every boot + (owner question 1). + - The host flashes one session image. It is `tests/testcases` with the + talking boot's netd, sshd and streaming `logd`. The host runs every member + over the cable, hands the machine back with `reboot`, and reads the stick + as today. + - The session takes the base boots: the C corpus, the shared block (without + `shared-debug`), `testcases` with its `mkdir` and `readdir` boots, and the + talking and swapping boots. + - `mkdir_cap` and `readdir_bound` clean up after themselves or red. A test + whose premise is a fresh boot says so in its row and runs first. + - QEMU's shared block judges each member with `must_be_clean` too. + - The shared lists' chunking into boots goes, and so do the LAN hold jobs. + - Waits on `issues/kernel/deferred-release-outlives-its-syscall.md`, because + a claim that is not back before the next launch reds the check. Also waits + on the transmit drops above. + - Closes `issues/hardware/a-swap-on-the-t14-lives-inside-a-metal-boots-bound.md`. + + **Exit**: a T14 run folds those boots into one and judges every member per + test. Its verdicts equal the per-boot rig's at the same head, which is the + oracle. Each of the following reds only the test that staged it: a child + left alive, a file left in `/tmp`, and a `NEVER_CLEAN` line. A job that + spins past its bound is killed and red by name. With the renewal skipped, + the machine resets at the lease and not at the boot's deadline. An e1000e + QEMU guest runs the same session end to end. + +2. **Ubuntu leaves the loop.** This stage is the loader track's stages 6 and 7. + - It waits on: + - that track's stage 5 (tries, the good flag, `update --once` and + `--boot-first`); + - `issues/isolation/a-reset-stops-xhci-and-leaves-every-claimed-pci-function-armed.md` + and `issues/hardware/the-t14-hung-after-rebooting-with-its-i219-faulted.md`, + because every reset is now taken with netd holding the host's only + channel; + - `issues/panic-path/a-fatal-event-stands-down-both-bounds-and-may-leave-nothing-to-end-the-machine.md`; + - host redials that wait on an event: + `issues/diagnostics/a-swaps-redial-asks-again-with-no-event-to-wait-on.md`, + `issues/hardware/the-t14-redial-re-asks-mdns-after-every-refusal.md`, + `issues/diagnostics/a-first-dial-turned-away-before-a-line-is-waited-on-to-its-callers-bound.md` + and + `issues/diagnostics/a-netd-that-dies-while-serving-leaves-the-hosts-stream-silent.md`; + - owner question 2. + - The stick is written once and put first with `update --boot-first`. From + then on every image goes by `update`, and a machine that cannot boot its + stick is recovered by hand. + - An image that reaches good retires its boot's lease. + - Every test row names its place: a session's image, a `--once` image, or + QEMU with what it needs there. `METAL_ONLY`, `QemuOnly` and the name-keyed + dispatch go, and a metal row can be redlisted like any other. + - The loader keeps the pass before it as `loader-previous.log`, by a rename + (`SetInfo`), so the kept slot's pass does not erase a death's report. + It also writes the machine's SMBIOS line (Timing, below). + - A multi-arm stick (the loader booting the next armed kernel instead of + Ubuntu) is not built. Slots and `--once` do the same with what stays. + - Deleted, with line counts at `afa84aee7`: + - `src/metal.rs` (3378 lines) except its arm gate; + - `src/icmp.rs` (290); + - `bootloader/src/bootnext.rs` (192); + - `tests/common/metal.rs`' stick readback, boot batching and per-boot run; + - test-runner's job-list mode; + - `metaltalk`'s `converse` and `judge`; + - every timing row for firmware, Ubuntu, the stick or the router. + - Closes + `issues/build/a-hung-boots-log-partition-is-wiped-by-the-next-runs-flash.md`, + `issues/build/the-sudoers-rendering-has-no-host-side-judge.md`, + `issues/build/the-metal-loop-writes-the-readback-volume-into-a-directory-it-has-not-made.md` + and `issues/hardware/the-t14-stopped-answering-ssh-between-two-lan-boots.md`. + - Ubuntu stays on the NVMe, never booted, until + `issues/hardware/linuxs-readings-of-the-t14-and-the-tcg-model-are-not-committed.md` + lets it go. The NVMe install is stage 2 of + `issues/boot-media/the-machine-updates-itself-without-ubuntu.md`, and a + session does not care which disk holds its slots. + + **Exit**: a whole T14 run with Ubuntu never started gives the per-boot rig's + verdicts at the same head. No path in `src/metal*.rs` reaches Ubuntu. A + panic's reset reaches the loader with no `BootNext` set. A `--once` image + that panics returns the machine to its session, and its record is judged. + A slot with a flipped byte, no signature or a lower security version is + refused and the other slot boots. A slot that dies falls back on its own. + +3. **Sessions merge.** A run holds one session per kernel build, parameter + line and config that some test needs. + - Actuators that leave the machine as it is share an image where their + authors say so, as `SELFTESTS` does. + - A config that differs by one program's row joins the base session. + - The QEMU registrations the T14 can run move into sessions and cost no + boot. + + **Exit**: the host's plan names every reset in a run by what forces it: a + kernel build, a parameter no session carries, or a death. + +## Timing + +The per-machine record (PR #630) is keyed per test, and its rows time only +ToyOS's own work: the runner's time for each test, and each image's kernel to +`Boot: complete`. Nothing that times firmware, the router or ssh gets a row. The +boot under test names its machine. Before `ExitBootServices`, the loader reads +SMBIOS type 1's vendor and product and type 0's BIOS version from the UEFI +configuration table and writes them as one `loader.log` line. The judge +compares that line byte for byte with `tests/metal/-.toml`, +and the Ubuntu query goes in stage 2. + +## Owner questions + +1. **The lease is an ABI change.** It needs a kernel operation that renews the + deadline to at most `WEDGE_BOUND_MS` from now and retires it. The operation + sits behind a capability that only test-runner's row and init hold, and the + build refuses it elsewhere, as it does `swap` and `slots`. `hardlockup` + would also be armed without `boot-deadline=`. *Recommended: yes.* +2. **One key signs the T14's images.** `update` never replaces the ESP loader, + so the T14 installs only what its loader's key signed, and every checkout + signs with its own throwaway key. *Recommended:* a bench key minted once + with `--signing-key-new`, kept outside every checkout, and not the owner's + own key. +3. **A loader change without a flash.** *Recommended:* a later stage in which + `update` writes a signed loader to a second ESP file. The running loader + verifies it and chain-loads it once, and only a loader that booted a good + slot becomes the one firmware boots. Until then a loader change is proven in + QEMU and reaches the T14 by hand. From 95ac0bb63ab2d0314040c904544bbc9cc6f2e2e7 Mon Sep 17 00:00:00 2001 From: japabu Date: Tue, 29 Sep 2026 23:29:17 +0200 Subject: [PATCH 02/11] Answer the review of #631: the host judges each test's window, init holds the lease The track is rewritten around the review's six blockers, the questions go to files of their own, and the review's REMOVEs are deleted. - A test's records reach the host on one path each. The markers and the test's own output travel over the session's one exec channel: sshd pipes a program's stdio (userland/sshd/src/main.rs:234-239), init keeps a launch caller's pipes (userland/init/src/main.rs:1593-1604), and a job inherits the runner's stdout (userland/test-runner/src/main.rs:276-277). The kernel's records are to be built: after each test run_one reads the log on its own LogTail cursor, on the logread its row already holds, and writes that window's records with the ring's drop count before ===TEST_END===. The host judges with Serial::must_be_clean, so no list is copied into ToyOS, and a builtin's window is as exact as a spawned job's. Cutting logd's stream at the kernel's spawn:/exit: records was weighed and not taken: a builtin runs inside the runner and has neither. - The check covers /log, less the files bootlog::split_listing gives logd and the loader. A test whose premise is a fresh boot is declared and runs first in a fresh session, and stage 1's exit adds a second session in reverse order and negative controls for a claim and for files in the home, /state and /log. - An image is good only once the host has reached it: the host's first exec is update --good, and a session image names no [boot] up. Stage 2's exit adds an image whose sshd does not authorize the host's key. - The lease's right is init's alone; the runner asks init on a port no job inherits, and stage 1's exit adds a job that calls the renewal and is refused. The runner's un-narrowed SysCap duplicate to every job is filed as issues/isolation/every-job-test-runner-starts-holds-its-whole-system-capability.md. - Stage 2 starts with the measurement its shape rests on, a timed `ssh t14 update < image`, and its estimate is bounded by measured stick writes; the 25-35 s switch and every estimate not derived are gone. - No one-test mode: the runner's stdin loop is the session's loop, over one exec channel, because sshd aliases a second channel's input. Owner questions, each a kind: question file cited by the stage it blocks: the lease (stage 1), the hard-lockup detector on every boot (no stage), the bench signing key (stage 2, with the --signing-key-new claim corrected: it mints only the owner's key and refuses to replace it), and how a loader change reaches the T14 without Ubuntu (stage 2, both orders stated). Stage 1 no longer waits on the transmit drops, for which no reason was given. Only members that claim a device wait on deferred release. The Timing section and its SMBIOS line go, as outside this track's fence. The loader track's bench list gives --metal-via-ubuntu a disposition (deleted), and the reworded bootnext.rs line is restored. Evidence, from commands run for this commit: - Stick writes: metalprobe's usbwrite moves 314,368 B (userland/metalprobe/src/usb.rs's BYTES). Its exit code, the span in us, read 63,452 in toyos-fsd's target/metal/metaldevicecase/kernel.log (4.72 MiB/s) and 2,834,265 in toyos-perfstate's (108.3 KiB/s), beside issues/kernel/the-vfs-lock-is-held-across-a-usb-write-for-thirty-seconds.md's 108.6 KiB/s. That is 0.21-9.45 s per MiB (bc). - ROOT: the GPT of toyos-fsd's target/metal/shared/image.img gives slot A's ROOT 55,574,528 B (53 MiB), testcases' 34,603,008 B and lantalkcase's 30,408,704 B. 53 MiB takes 11.2 s at the fast end and 8.35 min at the slow end. - DHCP: the lease lines in the LAN boots of toyos-fsd, toyos-perfstate and toyos-metaljudges' target/metal/lan*/kernel.log, 16 leases over 13 boots. 5 landed 3,218-3,639 ms after netd came up and 11 at 13,148-13,383 ms; each I219 link came up 2,723-2,822 ms after its driver; a boot's first lease was at 9.105-9.342 s of boot on 4 boots and 18.990-19.116 s on 9. smoltcp-0.12.0/src/socket/dhcpv4.rs:134 sets discover_timeout to 10 s, and netd never sets a RetryConfig. Filed as issues/hardware/most-t14-leases-land-one-dhcp-retry-late.md, which the network track owns and stage 2 waits on. Co-Authored-By: Claude Opus 5.5 --- ...es-the-t14-without-ubuntu-is-the-owners.md | 31 ++ ...oes-only-what-must-precede-the-handover.md | 11 +- ...ges-signed-by-a-bench-key-is-the-owners.md | 28 ++ ...ost-t14-leases-land-one-dhcp-retry-late.md | 31 ++ ...4-reboots-through-ubuntu-for-every-test.md | 286 ++++++++---------- ...tarts-holds-its-whole-system-capability.md | 27 ++ ...-the-hard-lockup-detector-is-the-owners.md | 26 ++ ...y-renew-the-boot-deadline-is-the-owners.md | 29 ++ 8 files changed, 298 insertions(+), 171 deletions(-) create mode 100644 issues/boot-media/how-a-loader-change-reaches-the-t14-without-ubuntu-is-the-owners.md create mode 100644 issues/boot-media/whether-the-t14-takes-images-signed-by-a-bench-key-is-the-owners.md create mode 100644 issues/hardware/most-t14-leases-land-one-dhcp-retry-late.md create mode 100644 issues/isolation/every-job-test-runner-starts-holds-its-whole-system-capability.md create mode 100644 issues/kernel/whether-every-boot-arms-the-hard-lockup-detector-is-the-owners.md create mode 100644 issues/kernel/whether-init-may-renew-the-boot-deadline-is-the-owners.md diff --git a/issues/boot-media/how-a-loader-change-reaches-the-t14-without-ubuntu-is-the-owners.md b/issues/boot-media/how-a-loader-change-reaches-the-t14-without-ubuntu-is-the-owners.md new file mode 100644 index 0000000000..9ed2912c04 --- /dev/null +++ b/issues/boot-media/how-a-loader-change-reaches-the-t14-without-ubuntu-is-the-owners.md @@ -0,0 +1,31 @@ +--- +status: owner +kind: question +opened: 2026-09-29 +--- + +# How a loader change reaches the T14 without Ubuntu is the owner's + +`update` writes a slot and never the ESP's loader. Once stage 2 of +`issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md` takes +Ubuntu out of the loop, a loader change reaches the T14 only as its stick +pulled and flashed on the Mac through `diag/flash.sh`, whose `diskutil`, +`plutil` and `dd` the rules refuse +(`issues/build/the-owners-flash-script-runs-diskutil.md`). The loader track's +stages 8 and 9 +(`issues/boot-media/the-loader-does-only-what-must-precede-the-handover.md`) +both change the loader after that point. Two orders: + +- **The loader updates itself first, and Ubuntu stays until then.** `update` + writes a signed loader to a second ESP file; the running loader verifies it + and chain-loads it once, and only a loader that has booted a good slot + becomes the one the firmware boots. Stage 2 waits on it, and until it lands + a loader change is flashed through Ubuntu as today. +- **Ubuntu goes at stage 2, and loader changes are flashed by hand until a + self-update exists.** Every loader change until then is the stick pulled and + written on the Mac. + +*Recommended:* the first, because the second puts a hand and a script the +rules refuse in the path of every loader change. + +**Exit**: the owner rules. diff --git a/issues/boot-media/the-loader-does-only-what-must-precede-the-handover.md b/issues/boot-media/the-loader-does-only-what-must-precede-the-handover.md index e34db8bf74..bb56c41b74 100644 --- a/issues/boot-media/the-loader-does-only-what-must-precede-the-handover.md +++ b/issues/boot-media/the-loader-does-only-what-must-precede-the-handover.md @@ -29,14 +29,15 @@ PR #539 does not land. Its pieces: `stick_readonly`; and `update_trial_writes_nothing_of_the_kept_slot`, rewritten for priorities; - stage 2 of `issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md` - takes what it uses of the bench: `src/metalbench.rs`, `tests/common/bench.rs`, + takes the bench: `src/metalbench.rs`, `tests/common/bench.rs`, `tests/bench*case`, `--bench-image` with `build::bench_image`, `src/image.rs`' `update_of`, `tests/common/metal.rs`' `Reach`, `stage` and - bench `invocation`, `--metal-via-ubuntu`, toybox `date`, `Ssh::probe` and + bench `invocation`, toybox `date`, `Ssh::probe` and `Ssh::fetch` with `ssh-client-host`'s `probe` and `fetch`, `build::AUTHORIZED_ON_ROOT`, and `src/bootlog.rs`' `LOADER_PREVIOUS_LOG`, `MOUNTED_FROM_MEMORY` and `BOOT_PARAMETER`; -- deleted: `update --boot-next ` with `slots::Next::Esp`, `Guid::parse`, +- deleted: `--metal-via-ubuntu`, because that stage takes Ubuntu out of the + loop; `update --boot-next ` with `slots::Next::Esp`, `Guid::parse`, `bootvars.rs`' `BootNext` and entry-after-its-own writes, and `update_boot_next_boots_the_entry_once`'s other half; the panic handler's fall to the next boot entry, with the two issues #539 filed about it, @@ -249,8 +250,8 @@ Each stage lands on its own, in this order. differs from `LAST_LAYOUT` (stage 4's derived value, pinned as a literal) and `assert!(LAYOUT != LAST_LAYOUT)` builds. This stage is an ABI change. - `update --boot-first` writes only the request. The loader writes its own - `HD(…)/File(…)` entry and puts it first. Once `bootnext.rs` is deleted - (stage 7), that is the only boot-variable write. + `HD(…)/File(…)` entry and puts it first. Once stage 7 deletes + `bootnext.rs`, that is the only boot-variable write. - This stage deletes the floor issue's "a floor that never rises" bullet and its exit's second half. diff --git a/issues/boot-media/whether-the-t14-takes-images-signed-by-a-bench-key-is-the-owners.md b/issues/boot-media/whether-the-t14-takes-images-signed-by-a-bench-key-is-the-owners.md new file mode 100644 index 0000000000..fe5ef73c40 --- /dev/null +++ b/issues/boot-media/whether-the-t14-takes-images-signed-by-a-bench-key-is-the-owners.md @@ -0,0 +1,28 @@ +--- +status: owner +kind: question +opened: 2026-09-29 +--- + +# Whether the T14 takes images signed by a bench key is the owner's + +The owner's ruling is that the machine installs nothing the owner did not sign +(`issues/boot-media/the-machine-updates-itself-without-ubuntu.md:9-13`). Once +the T14 takes every image by `update`, which stage 2 of +`issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md` makes it do +and which never replaces the loader, it boots only what its stick's loader was +built to verify. Every checkout signs with a throwaway key of its own +(`src/signing.rs`' `THROWAWAY_FILE`), so the stick would take images from the +one checkout that flashed it; signing test images with the owner's key would +put that key in every agent's build. + +*Recommended:* a bench key, made once and kept outside every checkout, which +the T14's loader verifies and every checkout signs its T14 images with: this +test laptop's one exception to the ruling. Nothing in the tree makes one: +`--signing-key-new` mints the owner's key at `owner_key_path()` and refuses to +replace it (`src/main.rs:143-144`, `src/signing.rs:195-236`), so a bench key +needs a mint and a signing path of its own. + +Stage 2 of that track waits on this. + +**Exit**: the owner rules. diff --git a/issues/hardware/most-t14-leases-land-one-dhcp-retry-late.md b/issues/hardware/most-t14-leases-land-one-dhcp-retry-late.md new file mode 100644 index 0000000000..21229e308c --- /dev/null +++ b/issues/hardware/most-t14-leases-land-one-dhcp-retry-late.md @@ -0,0 +1,31 @@ +--- +status: open +kind: defect +opened: 2026-09-29 +--- + +# Most T14 leases land one DHCP retry late + +Over the LAN boots in the readbacks of three T14 runs, the I219's link came up +2.72–2.82 s after its driver every time. Of the 16 leases, 5 landed 3.2–3.6 s +after netd came up and 11 at 13.1–13.4 s, so the first lease of a boot, and +with it `sshd`, arrived at 9.1–9.3 s of boot on 4 of the 13 boots and at +19.0–19.1 s on the other 9. + +The ten-second step is smoltcp 0.12's default `RetryConfig::discover_timeout` +(`smoltcp-0.12.0/src/socket/dhcpv4.rs:134`), which netd keeps. netd restarts +discovery when the link comes up (`userland/netd/src/dhcp.rs:42-61`), and on +the late boots the lease came from the DISCOVER sent one timeout after that +one. What became of the first, whether it left the machine and whether an +OFFER came back, is unmeasured. RFC 2131 §4.1 puts the first retransmission +at 4 s, randomized by ±1 s. + +The network track owns it: netd leaves smoltcp in stage 5 of +`issues/design-debt/toyos-has-its-own-network-stack.md`, whose stage 3 is +`toyos-dhcp`. Stage 2 of +`issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md` waits on +it. + +**Exit**: a capture of a T14 boot shows what became of the DISCOVER sent as +the link came up, and on every boot of a T14 run an unanswered DISCOVER is +sent again within RFC 2131 §4.1's 4 ± 1 s. diff --git a/issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md b/issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md index 3f2cf13fcb..c2a109c5c1 100644 --- a/issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md +++ b/issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md @@ -6,148 +6,133 @@ opened: 2026-09-29 # The T14 reboots through Ubuntu for every test -Every metal arm is a boot of its own. The host flashes the stick over ssh to -Ubuntu, sets `BootNext`, and reads the log partition back after the loader's -report pass has reset the machine into Ubuntu. That is one flash and three -POSTs per boot. Three full runs of 25–27 boots spent 191–218 s in their -kernels, out of 2683–2957 s of boot cycles. The owner's direction is that the -tests run inside ToyOS one after another, with no reboot per test and no -Ubuntu. - -- **The transport is the LAN.** The host reaches ToyOS's own `sshd` on the - I219 at `toyos-t14.local` through `tests/ssh-client-host`. The T14 has no - serial channel, and nothing in the tree drives USB in device mode. - `logd`'s stream (TCP 41337) stays open for the whole session. The host cuts - it per test at the runner's markers. -- **A session is one image**: a kernel build, a parameter line and a config. - Every test of that image runs in it, one after another, with no reboot. A test - is one `exec` of `test-runner `, which `sshd` starts through init's - `launcher`. Each test is therefore a fresh process holding test-runner's - manifest row: init builds its namespace and claims, and they go when it - exits. It prints the `===TEST_START/END===` markers QEMU's shared block - already reads off serial. -- **A test leaves the machine as it found it, and the runner checks that.** - After each test the runner checks four things: - - no process the test started is alive (`roster`); - - every device claim has the holder it had before (`inventory`); - - `/tmp`, the session home and `/state` list as they did, which is the only - file fence until per-program views land; - - the kernel's records in the test's slice pass `must_be_clean`. - - A leftover reds that test by name. The session then resets before the next - test. -- **The bound is a lease.** `boot-deadline=` is armed from boot, and - `hardlockup` is armed only through it (`deadline::start`). A boot that stays - up therefore has neither. The deadline becomes a lease: - - each test's launch renews it for twice the runner's bound on that test, - as `WEDGE_BOUND_MS` is twice `JOB_BOUND_MS`, and the host renews it - between tests; - - a lapse seals `WEDGED` and resets the machine; - - an image that reaches good retires the lease its parameter started, so a - machine the host has let go of idles rather than resets; - - `hardlockup` is armed on every boot. -- **Another image costs one reset, and a death costs two.** - - For another image, the host runs `update < image` into the idle slot and - reboots: one POST, no Ubuntu. - - A panic, wedge, reset or loader test goes in with `update --once`. Its end - resets the machine and the loader boots the kept slot. The host then - fetches the record over `sftp`: from `loader.log` until the loader track's - stage 9, and from `/log` after it. Until that stage the report pass costs a - third POST. - - A session image's `[boot] up` names netd and sshd, so an image the host - cannot reach is never good and its slot falls back. -- **What no detector catches stops the run and waits for a hand.** That is the - span before `clock::init`, a machine that powers itself off, firmware that - does not come back, or a fatal event that stood both bounds down. The host - declares the machine lost when nothing has answered within the lease and a - POST allowance, derived as `return_secs` is. It keeps what the stream - carried. An armed TCO has never reset this machine, and whether it counts is - undecided (`issues/hardware/an-armed-tco-has-never-reset-the-t14.md`), so - nothing here rests on it. -- **QEMU needs no resident runner.** Its shared block is already one boot for - many tests, and each of its machine tests has its own boot as its subject. - It does take the per-test check. The KVM and CI lanes do not change. - -Measured from those three runs' readbacks: - -- POST (power-on to loader) took 8.8–21.7 s, median 9.3 s over 78 boots. The - loader took 1.6–7.8 s, median 2.2 s. -- The I219's link came up 2.7–2.8 s after its driver on every boot. -- 4 of 14 DHCP leases landed 3.2–3.6 s after netd started. The other 10 - landed at 13.3–13.4 s: the first DISCOVER was lost and smoltcp retries after - 10 s. So the lease, and with it `sshd`, arrives at 9.1–9.3 s or at - 19.0–19.1 s of boot. -- Every talking and swapping boot dropped 9–24 frames because no transmit - descriptor was free (`toyos_i219::TX_RING` is 16). A frame the stream loses - waits for TCP's retransmit. +Every metal arm is a boot of its own: a flash over ssh to Ubuntu, `BootNext`, +and the log partition read back once the loader's report pass has reset the +machine into Ubuntu, three POSTs in all. The owner's direction is that the +tests run one after another in a ToyOS that stays booted: no reboot per test, +and no Ubuntu. + +- **A session is one image** (a kernel build, a parameter line and a config) + booted once and driven over one exec channel to its `sshd`: `test-runner`, + launched by init under its own row, whose stdin loop is the session's + (`userland/test-runner/src/main.rs:140-145,229-249`). The host writes + `run ` and waits for `===TEST_END===`; one channel, because `sshd` + aliases a second one's input + (`issues/isolation/sshd-holds-one-channel-and-does-not-say-so.md`). +- **The host judges each test** with `Serial::must_be_clean` + (`tests/common/serial.rs:341`) and the exit code; ToyOS copies no list. + - The markers and the test's own output come over the channel: `sshd` pipes + a program's stdio (`userland/sshd/src/main.rs:234-239`), init keeps a + launch caller's pipes (`userland/init/src/main.rs:1593-1604`), and a job + inherits the runner's stdout (`userland/test-runner/src/main.rs:276-277`). + None of it reaches `/log`, so a test that kills the machine leaves the + host what crossed before it died; the kernel's account is the black box's. + - The kernel's records, to be built: after each test `run_one` reads the log + on its own cursor (`toyos::log::LogTail`, on its row's `logread`) and + writes the test's records over the session before its `===TEST_END===`, + with how many the ring dropped; a drop reds the test. Every record lands + in one window, a builtin's too. A runner init started at boot, as QEMU's + shared block's is, writes none: its stdout is a log ring, and the console + carries the records already. +- **A test leaves the machine as it found it.** After each test `run_one` + writes the processes alive (`roster`), each claim and its holder + (`inventory`), and the listings of `/tmp`, the session home, `/state` and + `/log`; the host compares each with the session's opening one, `/log` less + what `bootlog::split_listing` gives `logd` and the loader. A leftover reds + its test and gives the next one a fresh session. A test whose premise is a + fresh boot, as `audio_idle_suspend`'s is, says so in its row and runs first + in a fresh session. +- **The bound is a lease.** `boot-deadline=` arms it; `run_one` asks init to + renew it for `WEDGE_BOUND_MS`, twice the `JOB_BOUND_MS` it kills a job at, + and `quit` asks init to retire it. Only init holds the renewal, and no job + holds the port the runner asks on. A boot no host reaches lapses and resets. + `hardlockup` is armed only through `boot-deadline=`; arming it on every boot + is `issues/kernel/whether-every-boot-arms-the-hard-lockup-detector-is-the-owners.md`. +- **Another image costs one reset, and a death costs two.** Another image goes + by `update < image` and a reboot. A panic, wedge, reset or loader test goes + by `update --once`; the loader then boots the kept slot and the host fetches + the record over `sftp`: `loader.log`, after a third POST for the loader's + report pass, until the loader track's stage 9, and `/log` after it. +- **What no detector catches waits for a hand**: the span before + `clock::init`, a machine that powers itself off, firmware that does not come + back, or a fatal event that stood both bounds down. The host declares the + machine lost when nothing answers within the lease and a POST allowance, + derived as `return_secs` is; nothing rests on the TCO + (`issues/hardware/an-armed-tco-has-never-reset-the-t14.md`). ## Stages 1. **One session on today's rig.** - - `test-runner` gains a one-test mode. It renews the lease, runs the job, - kills it at its bound, runs the check, and exits without rebooting. - - The deadline becomes the lease and `hardlockup` is armed on every boot - (owner question 1). - - The host flashes one session image. It is `tests/testcases` with the - talking boot's netd, sshd and streaming `logd`. The host runs every member - over the cable, hands the machine back with `reboot`, and reads the stick - as today. - - The session takes the base boots: the C corpus, the shared block (without + - `run_one` gains the bound, the renewal, the kernel's records and the + readings; QEMU's shared block runs it too, lease included. + - The host flashes one session image, `tests/testcases` with the talking + boot's netd, sshd and streaming `logd`, runs every member over the + channel, hands the machine back with `reboot`, and reads the stick as + today. It takes the base boots: the C corpus, the shared block (without `shared-debug`), `testcases` with its `mkdir` and `readdir` boots, and the - talking and swapping boots. - - `mkdir_cap` and `readdir_bound` clean up after themselves or red. A test - whose premise is a fresh boot says so in its row and runs first. - - QEMU's shared block judges each member with `must_be_clean` too. - - The shared lists' chunking into boots goes, and so do the LAN hold jobs. - - Waits on `issues/kernel/deferred-release-outlives-its-syscall.md`, because - a claim that is not back before the next launch reds the check. Also waits - on the transmit drops above. + talking and swapping boots. Chunking the shared lists into boots goes, and + so do the LAN hold jobs. + - `mkdir_cap` and `readdir_bound` clean up after themselves or red. + - Waits on + `issues/kernel/whether-init-may-renew-the-boot-deadline-is-the-owners.md`. + A member that claims a device also waits on + `issues/kernel/deferred-release-outlives-its-syscall.md`, whose fix is + `issues/kernel/every-wait-in-this-kernel-is-a-spin.md`'s, because a claim + not back by the check reds it. - Closes `issues/hardware/a-swap-on-the-t14-lives-inside-a-metal-boots-bound.md`. - **Exit**: a T14 run folds those boots into one and judges every member per - test. Its verdicts equal the per-boot rig's at the same head, which is the - oracle. Each of the following reds only the test that staged it: a child - left alive, a file left in `/tmp`, and a `NEVER_CLEAN` line. A job that - spins past its bound is killed and red by name. With the renewal skipped, - the machine resets at the lease and not at the boot's deadline. An e1000e - QEMU guest runs the same session end to end. - -2. **Ubuntu leaves the loop.** This stage is the loader track's stages 6 and 7. - - It waits on: - - that track's stage 5 (tries, the good flag, `update --once` and - `--boot-first`); + **Exit**: a T14 run folds those boots into one session and judges every + member per test. Its verdicts equal the per-boot rig's at the same head, + and a second session in reverse order gives the same verdicts. Each of + these reds only the test that staged it: a child left alive, a claim left + held, a file left in `/tmp`, the home, `/state` or `/log`, and a + `NEVER_CLEAN` line. A job that spins past its bound is killed and red by + name, and one that calls the renewal is refused. With the renewals + skipped, the machine resets when its boot's lease runs out. An e1000e QEMU + guest runs the same session end to end. + +2. **Ubuntu leaves the loop**, which is the loader track's stages 6 and 7. + - First, the measurement its shape rests on: `ssh t14 update < image` timed + for a session image on today's rig. An image switch costs that write and a + reset. ToyOS has written this stick at between about 108 KiB/s and + 4.7 MiB/s, 0.2 s to 9.5 s per MiB: 11 s to 8.4 min for the 53 MiB ROOT a + `shared` image carries. + - Waits on the loader track's stage 5 (tries, the good flag, `update --once` + and `--boot-first`), and on: - `issues/isolation/a-reset-stops-xhci-and-leaves-every-claimed-pci-function-armed.md` and `issues/hardware/the-t14-hung-after-rebooting-with-its-i219-faulted.md`, because every reset is now taken with netd holding the host's only channel; - `issues/panic-path/a-fatal-event-stands-down-both-bounds-and-may-leave-nothing-to-end-the-machine.md`; - - host redials that wait on an event: - `issues/diagnostics/a-swaps-redial-asks-again-with-no-event-to-wait-on.md`, + - `issues/diagnostics/a-swaps-redial-asks-again-with-no-event-to-wait-on.md`, `issues/hardware/the-t14-redial-re-asks-mdns-after-every-refusal.md`, `issues/diagnostics/a-first-dial-turned-away-before-a-line-is-waited-on-to-its-callers-bound.md` and - `issues/diagnostics/a-netd-that-dies-while-serving-leaves-the-hosts-stream-silent.md`; - - owner question 2. + `issues/diagnostics/a-netd-that-dies-while-serving-leaves-the-hosts-stream-silent.md`, + so that every host redial waits on an event; + - `issues/hardware/most-t14-leases-land-one-dhcp-retry-late.md`, because + every reset now reaches the host through a lease; + - `issues/boot-media/whether-the-t14-takes-images-signed-by-a-bench-key-is-the-owners.md` + and + `issues/boot-media/how-a-loader-change-reaches-the-t14-without-ubuntu-is-the-owners.md`. - The stick is written once and put first with `update --boot-first`. From then on every image goes by `update`, and a machine that cannot boot its - stick is recovered by hand. - - An image that reaches good retires its boot's lease. + stick costs a hand and `diag/flash.sh` + (`issues/build/the-owners-flash-script-runs-diskutil.md`). + - An image is good only once the host has reached it: the host's first exec + on it is `update --good`, and a session image names no `[boot] up`, so + init never marks it good. An image the host cannot reach lapses, spends + its tries and falls back. - Every test row names its place: a session's image, a `--once` image, or - QEMU with what it needs there. `METAL_ONLY`, `QemuOnly` and the name-keyed - dispatch go, and a metal row can be redlisted like any other. + QEMU. `METAL_ONLY`, `QemuOnly` and the name-keyed dispatch go, and a metal + row can be redlisted like any other. - The loader keeps the pass before it as `loader-previous.log`, by a rename (`SetInfo`), so the kept slot's pass does not erase a death's report. - It also writes the machine's SMBIOS line (Timing, below). - - A multi-arm stick (the loader booting the next armed kernel instead of - Ubuntu) is not built. Slots and `--once` do the same with what stays. - - Deleted, with line counts at `afa84aee7`: - - `src/metal.rs` (3378 lines) except its arm gate; - - `src/icmp.rs` (290); - - `bootloader/src/bootnext.rs` (192); - - `tests/common/metal.rs`' stick readback, boot batching and per-boot run; - - test-runner's job-list mode; - - `metaltalk`'s `converse` and `judge`; - - every timing row for firmware, Ubuntu, the stick or the router. + - Deleted: `src/metal.rs` except its arm gate, `src/icmp.rs`, + `bootloader/src/bootnext.rs`, `tests/common/metal.rs`' stick readback, + boot batching and per-boot run, test-runner's job-list mode, `metaltalk`'s + `converse` and `judge`, and every timing row for firmware, Ubuntu, the + stick or the router. - Closes `issues/build/a-hung-boots-log-partition-is-wiped-by-the-next-runs-flash.md`, `issues/build/the-sudoers-rendering-has-no-host-side-judge.md`, @@ -155,53 +140,22 @@ Measured from those three runs' readbacks: and `issues/hardware/the-t14-stopped-answering-ssh-between-two-lan-boots.md`. - Ubuntu stays on the NVMe, never booted, until `issues/hardware/linuxs-readings-of-the-t14-and-the-tcg-model-are-not-committed.md` - lets it go. The NVMe install is stage 2 of - `issues/boot-media/the-machine-updates-itself-without-ubuntu.md`, and a - session does not care which disk holds its slots. + lets it go. - **Exit**: a whole T14 run with Ubuntu never started gives the per-boot rig's - verdicts at the same head. No path in `src/metal*.rs` reaches Ubuntu. A - panic's reset reaches the loader with no `BootNext` set. A `--once` image - that panics returns the machine to its session, and its record is judged. - A slot with a flipped byte, no signature or a lower security version is - refused and the other slot boots. A slot that dies falls back on its own. + **Exit**: a whole T14 run with Ubuntu never started gives the verdicts a + per-boot run gave at the commit stage 2 branches from. No path in + `src/metal*.rs` reaches Ubuntu. A panic's reset reaches the loader with no + `BootNext` set. A `--once` image that panics returns the machine to its + session, and its record is judged. A slot with a flipped byte, no signature + or a lower security version is refused and the other slot boots. A slot + that dies falls back on its own, and so does one whose `sshd` does not + authorize the host's key. 3. **Sessions merge.** A run holds one session per kernel build, parameter - line and config that some test needs. - - Actuators that leave the machine as it is share an image where their - authors say so, as `SELFTESTS` does. - - A config that differs by one program's row joins the base session. - - The QEMU registrations the T14 can run move into sessions and cost no - boot. + line and config that some test needs. Actuators that leave the machine as + it is share an image where their authors say so, as `SELFTESTS` does; a + config that differs by one program's row joins the base session; and the + QEMU registrations the T14 can run move into sessions. **Exit**: the host's plan names every reset in a run by what forces it: a kernel build, a parameter no session carries, or a death. - -## Timing - -The per-machine record (PR #630) is keyed per test, and its rows time only -ToyOS's own work: the runner's time for each test, and each image's kernel to -`Boot: complete`. Nothing that times firmware, the router or ssh gets a row. The -boot under test names its machine. Before `ExitBootServices`, the loader reads -SMBIOS type 1's vendor and product and type 0's BIOS version from the UEFI -configuration table and writes them as one `loader.log` line. The judge -compares that line byte for byte with `tests/metal/-.toml`, -and the Ubuntu query goes in stage 2. - -## Owner questions - -1. **The lease is an ABI change.** It needs a kernel operation that renews the - deadline to at most `WEDGE_BOUND_MS` from now and retires it. The operation - sits behind a capability that only test-runner's row and init hold, and the - build refuses it elsewhere, as it does `swap` and `slots`. `hardlockup` - would also be armed without `boot-deadline=`. *Recommended: yes.* -2. **One key signs the T14's images.** `update` never replaces the ESP loader, - so the T14 installs only what its loader's key signed, and every checkout - signs with its own throwaway key. *Recommended:* a bench key minted once - with `--signing-key-new`, kept outside every checkout, and not the owner's - own key. -3. **A loader change without a flash.** *Recommended:* a later stage in which - `update` writes a signed loader to a second ESP file. The running loader - verifies it and chain-loads it once, and only a loader that booted a good - slot becomes the one firmware boots. Until then a loader change is proven in - QEMU and reaches the T14 by hand. diff --git a/issues/isolation/every-job-test-runner-starts-holds-its-whole-system-capability.md b/issues/isolation/every-job-test-runner-starts-holds-its-whole-system-capability.md new file mode 100644 index 0000000000..21612ed279 --- /dev/null +++ b/issues/isolation/every-job-test-runner-starts-holds-its-whole-system-capability.md @@ -0,0 +1,27 @@ +--- +status: open +kind: defect +opened: 2026-09-29 +--- + +# Every job test-runner starts holds test-runner's whole system capability + +`run_one` endows each job it spawns a `SysCap::duplicate` of test-runner's own +capability (`userland/test-runner/src/main.rs:292-295`), and a duplicate +carries every right the original does (`toyos/src/syscap.rs:63-70`). The job +also inherits test-runner's whole namespace +(`userland/test-runner/src/main.rs:86-92`). On `tests/testcases` the row grants +`device`, `dup`, `logread`, `power` and `roster` +(`tests/testcases/system.toml:41`), so every test binary may mint a device +claim, read every kernel record, reset or power off the machine, list every +process, and hand all of it to a child of its own. A right or a name added to +that row reaches every job the same way. + +The binaries have no `[programs]` rows, so nothing declares what any of them +needs: `test_rs_audio_idle_suspend` reads `roster` off the duplicate +(`tests/toyos-rust-tests/src/bin/audio_idle_suspend.rs:7-11`), and +`endowment_denied` narrows `power` away from it. + +**Exit**: a job holds only the rights and names its test is declared to need, +narrowed by test-runner (`SysCap::narrowed`, `toyos/src/syscap.rs:121-128`), and +a job that asks for anything else is refused. diff --git a/issues/kernel/whether-every-boot-arms-the-hard-lockup-detector-is-the-owners.md b/issues/kernel/whether-every-boot-arms-the-hard-lockup-detector-is-the-owners.md new file mode 100644 index 0000000000..f830d53016 --- /dev/null +++ b/issues/kernel/whether-every-boot-arms-the-hard-lockup-detector-is-the-owners.md @@ -0,0 +1,26 @@ +--- +status: owner +kind: question +opened: 2026-09-29 +--- + +# Whether every boot arms the hard-lockup detector is the owner's + +`kernel/src/hardlockup` ends a machine one of whose CPUs has taken no interrupt, +with `IF` clear, for its bound: it seals a `WEDGED` record naming that CPU's +`pc` and `sp`, and resets the machine. It is armed only through +`boot-deadline=` (`deadline::start`, `kernel/src/deadline.rs:168-183`), so a +boot without that parameter, the owner's own machine included, has none, and +a CPU frozen there is a hand on the power button. + +Arming it on every boot, at `toyos_tco::HARD_LOCKUP_BOUND_MS` where no +deadline names a bound, costs one NMI per busy CPU per second +(`SAMPLE_NS`, `kernel/src/hardlockup/mod.rs:75-80`; a halted CPU takes none), +and turns such a freeze on the owner's machine into a reset with a record. It +changes no ABI. No stage of +`issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md` waits on +it, since every session image carries `boot-deadline=`. + +*Recommended: yes.* + +**Exit**: the owner rules. diff --git a/issues/kernel/whether-init-may-renew-the-boot-deadline-is-the-owners.md b/issues/kernel/whether-init-may-renew-the-boot-deadline-is-the-owners.md new file mode 100644 index 0000000000..3269ec872a --- /dev/null +++ b/issues/kernel/whether-init-may-renew-the-boot-deadline-is-the-owners.md @@ -0,0 +1,29 @@ +--- +status: owner +kind: question +opened: 2026-09-29 +--- + +# Whether init may renew the boot deadline is the owner's + +`boot-deadline=` arms a bound from boot that nothing on the machine can hold +off (`kernel/src/deadline.rs`), so a boot meant to stay up for a session of +tests is reset at it. Stage 1 of +`issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md` needs it to +be a lease, and that is a new kernel call, so an ABI change: + +- It renews the deadline to at most `WEDGE_BOUND_MS` from now, or retires it. + Its right is init's alone: no manifest name grants it, so no row can hold + it. test-runner asks init over a port the build lets only test-runner's row + receive, and hands its jobs a namespace without that port. +- It is refused once `deadline::stand_down` has run + (`kernel/src/deadline.rs:79-81`, called from `kernel/src/panic.rs:352`). A + renewal that raced a panic would otherwise arm the deadline again and seal + `WEDGED` over the panic report. +- A lapse is recorded as the deadline expiring, with when it was last + renewed, and not as a wedge: a host that went away stops renewing as surely + as a wedged machine does. + +*Recommended: yes.* + +**Exit**: the owner rules. From 11c0efd0aad215fbefb2b10a72ce3ca631b0194c Mon Sep 17 00:00:00 2001 From: japabu Date: Wed, 30 Sep 2026 13:25:38 +0200 Subject: [PATCH 03/11] The owner's rulings go into the T14 session track; the bound is the chipset's TCO The owner answered three of the four questions 95ac0bb63 filed. Each ruling is now written into the track, and its question file is deleted. - The bound (was whether-init-may-renew-the-boot-deadline-is-the-owners): no new syscall. The chipset's TCO becomes a device class held by one userland service, watchdogd, which feeds it only while the host renews a lease over the session. The owner's reason: metal farms reset a hung machine from outside, with a networked power switch, a BMC or AMT. The T14 has none of these: its i5-1135G7 has no vPro, so no AMT, it has no BMC, and its battery defeats a power switch. On the T14's Ubuntu, `modprobe iTCO_wdt` found a version 6 TCO at TCOBASE 0x0400, the port toyos-tco already names. Whether an unfed TCO resets the T14 is not measured. So that measurement is stage 1's first step, and the stage stops if it fails. - No new syscall: the claim's holder reaches the TCO's registers through the allow-list SYS_DEVICE_REG_READ/WRITE already gives a claim, as soundd reaches HDA's (kernel/src/syscall/device.rs:55-77). - The kernel feeds the timer from its scheduler pass (kernel/src/sched/driver.rs:512) until init mints the claim, and never after. A dead service therefore resets the machine. - The service feeds from the real-time band. A userland feeder that a test saturating every CPU could starve for toyos_tco::BOUND_MS (9.6 s) would reset a healthy machine. - The service is named watchdogd because `watchdog` is already the kernel parameter (toyos_tco::PARAM). - A session image drops boot-deadline=, because a whole-boot bound would end the session. That also drops hardlockup, which only that parameter arms, so the track links the question that stays with the owner. - A TCO reset seals nothing, unlike the deadline's WEDGED record. The host therefore holds logd's stream open for the session and keeps its tail. - The bench key (was whether-the-t14-takes-images-signed-by-a-bench-key- is-the-owners): done. The owner made it with `TOYOS_SIGNING_KEY= cargo run -- --signing-key-new`. `stat -f '%N %Sp %z bytes'` reads /Users/jan/.config/toyos/t14-bench-signing-key -rw------- 399 bytes. This corrects 95ac0bb63: its message and question file said --signing-key-new mints only the owner's key and so cannot make a bench key. It mints wherever TOYOS_SIGNING_KEY points (src/signing.rs:195-236). - A loader change (was how-a-loader-change-reaches-the-t14-without-ubuntu- is-the-owners): nothing is flashed by hand. ToyOS receives a new loader over ssh, as update receives an image, and writes it to the stick. The running loader tries it once and keeps the old one as the fallback, and Ubuntu leaves the loop only once that exists. It is stage 2's first build. The frozen-CPU detector on every boot stays with the owner, who asked what it is. Its question file now says so in plain terms and names Linux's nmi_watchdog and Windows' bug check 0x101 as the same mechanism. It now also says a session image without it loses the record that names the CPU, not the reset. The paragraph on another image and on deaths moves to stage 2, because `update --once` exists only from there. The track is 185 lines and 1654 words, of which 57 lines are prose before the stages (`wc -l`, `wc -w`). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L --- ...es-the-t14-without-ubuntu-is-the-owners.md | 31 --- ...ges-signed-by-a-bench-key-is-the-owners.md | 28 --- ...4-reboots-through-ubuntu-for-every-test.md | 176 ++++++++++-------- ...-the-hard-lockup-detector-is-the-owners.md | 37 ++-- ...y-renew-the-boot-deadline-is-the-owners.md | 29 --- 5 files changed, 124 insertions(+), 177 deletions(-) delete mode 100644 issues/boot-media/how-a-loader-change-reaches-the-t14-without-ubuntu-is-the-owners.md delete mode 100644 issues/boot-media/whether-the-t14-takes-images-signed-by-a-bench-key-is-the-owners.md delete mode 100644 issues/kernel/whether-init-may-renew-the-boot-deadline-is-the-owners.md diff --git a/issues/boot-media/how-a-loader-change-reaches-the-t14-without-ubuntu-is-the-owners.md b/issues/boot-media/how-a-loader-change-reaches-the-t14-without-ubuntu-is-the-owners.md deleted file mode 100644 index 9ed2912c04..0000000000 --- a/issues/boot-media/how-a-loader-change-reaches-the-t14-without-ubuntu-is-the-owners.md +++ /dev/null @@ -1,31 +0,0 @@ ---- -status: owner -kind: question -opened: 2026-09-29 ---- - -# How a loader change reaches the T14 without Ubuntu is the owner's - -`update` writes a slot and never the ESP's loader. Once stage 2 of -`issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md` takes -Ubuntu out of the loop, a loader change reaches the T14 only as its stick -pulled and flashed on the Mac through `diag/flash.sh`, whose `diskutil`, -`plutil` and `dd` the rules refuse -(`issues/build/the-owners-flash-script-runs-diskutil.md`). The loader track's -stages 8 and 9 -(`issues/boot-media/the-loader-does-only-what-must-precede-the-handover.md`) -both change the loader after that point. Two orders: - -- **The loader updates itself first, and Ubuntu stays until then.** `update` - writes a signed loader to a second ESP file; the running loader verifies it - and chain-loads it once, and only a loader that has booted a good slot - becomes the one the firmware boots. Stage 2 waits on it, and until it lands - a loader change is flashed through Ubuntu as today. -- **Ubuntu goes at stage 2, and loader changes are flashed by hand until a - self-update exists.** Every loader change until then is the stick pulled and - written on the Mac. - -*Recommended:* the first, because the second puts a hand and a script the -rules refuse in the path of every loader change. - -**Exit**: the owner rules. diff --git a/issues/boot-media/whether-the-t14-takes-images-signed-by-a-bench-key-is-the-owners.md b/issues/boot-media/whether-the-t14-takes-images-signed-by-a-bench-key-is-the-owners.md deleted file mode 100644 index fe5ef73c40..0000000000 --- a/issues/boot-media/whether-the-t14-takes-images-signed-by-a-bench-key-is-the-owners.md +++ /dev/null @@ -1,28 +0,0 @@ ---- -status: owner -kind: question -opened: 2026-09-29 ---- - -# Whether the T14 takes images signed by a bench key is the owner's - -The owner's ruling is that the machine installs nothing the owner did not sign -(`issues/boot-media/the-machine-updates-itself-without-ubuntu.md:9-13`). Once -the T14 takes every image by `update`, which stage 2 of -`issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md` makes it do -and which never replaces the loader, it boots only what its stick's loader was -built to verify. Every checkout signs with a throwaway key of its own -(`src/signing.rs`' `THROWAWAY_FILE`), so the stick would take images from the -one checkout that flashed it; signing test images with the owner's key would -put that key in every agent's build. - -*Recommended:* a bench key, made once and kept outside every checkout, which -the T14's loader verifies and every checkout signs its T14 images with: this -test laptop's one exception to the ruling. Nothing in the tree makes one: -`--signing-key-new` mints the owner's key at `owner_key_path()` and refuses to -replace it (`src/main.rs:143-144`, `src/signing.rs:195-236`), so a bench key -needs a mint and a signing path of its own. - -Stage 2 of that track waits on this. - -**Exit**: the owner rules. diff --git a/issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md b/issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md index c2a109c5c1..44ef1426f1 100644 --- a/issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md +++ b/issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md @@ -8,32 +8,31 @@ opened: 2026-09-29 Every metal arm is a boot of its own: a flash over ssh to Ubuntu, `BootNext`, and the log partition read back once the loader's report pass has reset the -machine into Ubuntu, three POSTs in all. The owner's direction is that the -tests run one after another in a ToyOS that stays booted: no reboot per test, -and no Ubuntu. +machine into Ubuntu. The owner's direction is that the tests run one after +another in a ToyOS that stays booted: no reboot per test, and no Ubuntu. - **A session is one image** (a kernel build, a parameter line and a config) booted once and driven over one exec channel to its `sshd`: `test-runner`, launched by init under its own row, whose stdin loop is the session's (`userland/test-runner/src/main.rs:140-145,229-249`). The host writes - `run ` and waits for `===TEST_END===`; one channel, because `sshd` + `run ` and waits for `===TEST_END===`, on one channel because `sshd` aliases a second one's input (`issues/isolation/sshd-holds-one-channel-and-does-not-say-so.md`). - **The host judges each test** with `Serial::must_be_clean` (`tests/common/serial.rs:341`) and the exit code; ToyOS copies no list. - - The markers and the test's own output come over the channel: `sshd` pipes - a program's stdio (`userland/sshd/src/main.rs:234-239`), init keeps a - launch caller's pipes (`userland/init/src/main.rs:1593-1604`), and a job - inherits the runner's stdout (`userland/test-runner/src/main.rs:276-277`). - None of it reaches `/log`, so a test that kills the machine leaves the - host what crossed before it died; the kernel's account is the black box's. + - The markers and the test's own output come over the channel and never + reach `/log`: `sshd` pipes a program's stdio + (`userland/sshd/src/main.rs:234-239`), init keeps a launch caller's pipes + (`userland/init/src/main.rs:1593-1604`), and a job inherits the runner's + stdout (`userland/test-runner/src/main.rs:276-277`). - The kernel's records, to be built: after each test `run_one` reads the log on its own cursor (`toyos::log::LogTail`, on its row's `logread`) and - writes the test's records over the session before its `===TEST_END===`, - with how many the ring dropped; a drop reds the test. Every record lands - in one window, a builtin's too. A runner init started at boot, as QEMU's - shared block's is, writes none: its stdout is a log ring, and the console - carries the records already. + writes the test's records before its `===TEST_END===`, with how many the + ring dropped; a drop reds the test. Every record lands in one window, a + builtin's too. + - A death's own account is the black box's where it sealed one; after a + reset that sealed nothing, the host has the tail of `logd`'s stream + (`toyos_logstream::PORT`), held open for the session. - **A test leaves the machine as it found it.** After each test `run_one` writes the processes alive (`roster`), each claim and its holder (`inventory`), and the listings of `/tmp`, the session home, `/state` and @@ -42,61 +41,86 @@ and no Ubuntu. its test and gives the next one a fresh session. A test whose premise is a fresh boot, as `audio_idle_suspend`'s is, says so in its row and runs first in a fresh session. -- **The bound is a lease.** `boot-deadline=` arms it; `run_one` asks init to - renew it for `WEDGE_BOUND_MS`, twice the `JOB_BOUND_MS` it kills a job at, - and `quit` asks init to retire it. Only init holds the renewal, and no job - holds the port the runner asks on. A boot no host reaches lapses and resets. - `hardlockup` is armed only through `boot-deadline=`; arming it on every boot - is `issues/kernel/whether-every-boot-arms-the-hard-lockup-detector-is-the-owners.md`. -- **Another image costs one reset, and a death costs two.** Another image goes - by `update < image` and a reboot. A panic, wedge, reset or loader test goes - by `update --once`; the loader then boots the kept slot and the host fetches - the record over `sftp`: `loader.log`, after a third POST for the loader's - report pass, until the loader track's stage 9, and `/log` after it. -- **What no detector catches waits for a hand**: the span before - `clock::init`, a machine that powers itself off, firmware that does not come - back, or a fatal event that stood both bounds down. The host declares the - machine lost when nothing answers within the lease and a POST allowance, - derived as `return_secs` is; nothing rests on the TCO - (`issues/hardware/an-armed-tco-has-never-reset-the-t14.md`). +- **The bound is the chipset's TCO watchdog, fed from userland while the host + renews a lease (owner ruling).** The T14 has no BMC or AMT (its i5-1135G7 + has no vPro) and a battery no power switch cuts, so the TCO is its one reset + that needs nothing of the kernel. It becomes a device class of its own, held + by a `watchdogd` service that reaches its registers through a claim's + allow-list (`kernel/src/syscall/device.rs:55-77`), with no new syscall. The + kernel feeds it from the scheduler pass (`kernel/src/sched/driver.rs:512`) + until the claim is minted; then only the service does, from the real-time + band, which no job's load starves, and only while its lease is live. The + lease starts at `WEDGE_BOUND_MS`, `run_one` renews it for as long on a port + no job's namespace carries, and `quit` retires it by halting the timer. A + frozen kernel, a dead service, a dead runner or a silent host stops the + feeding, the chipset resets the machine within `toyos_tco::BOUND_MS` + (9.6 s), and the next boot says so + (`kernel/src/arch/x86_64/watchdog.rs:92-99`). Only a machine that powers + itself off, or firmware that does not come back, waits for a hand; the host + declares it lost when nothing answers within the lease, `BOUND_MS` and a + POST allowance, derived as `return_secs` is. ## Stages 1. **One session on today's rig.** - - `run_one` gains the bound, the renewal, the kernel's records and the - readings; QEMU's shared block runs it too, lease included. + - First, the measurement the bound rests on: a T14 boot armed with + `watchdog` and `tco-starve` (which `src/metal.rs`' `FLASHABLE` must first + admit) is reset by the chipset within `toyos_tco::BOUND_MS` of its last + feed, and the boot after it reads `TCO_SECOND_TO_STS`: the exit of + `issues/hardware/an-armed-tco-has-never-reset-the-t14.md`. If it is not + reset, the stage stops and the bound goes back to the owner with that + run's registers: nothing else resets this machine without the kernel, + and renewing the kernel's own deadline is the syscall this ruling + declined. + - The build grants the TCO's class to `watchdogd`'s row alone and its lease + port to test-runner's alone, as it does `swap`. A session image carries + the `watchdog` parameter and a `watchdogd` row but no `boot-deadline=`, + whose whole-boot bound would end the session, so `judge_arms` + (`src/metal.rs:897-908`) takes that row as its bound, and `hardlockup` + runs only if every boot arms it + (`issues/kernel/whether-every-boot-arms-the-hard-lockup-detector-is-the-owners.md`). + - QEMU's shared block runs the same `run_one`; its runner's stdout is a log + ring, so the console carries the kernel's records and it writes none. - The host flashes one session image, `tests/testcases` with the talking - boot's netd, sshd and streaming `logd`, runs every member over the - channel, hands the machine back with `reboot`, and reads the stick as - today. It takes the base boots: the C corpus, the shared block (without - `shared-debug`), `testcases` with its `mkdir` and `readdir` boots, and the - talking and swapping boots. Chunking the shared lists into boots goes, and - so do the LAN hold jobs. - - `mkdir_cap` and `readdir_bound` clean up after themselves or red. - - Waits on - `issues/kernel/whether-init-may-renew-the-boot-deadline-is-the-owners.md`. - A member that claims a device also waits on - `issues/kernel/deferred-release-outlives-its-syscall.md`, whose fix is - `issues/kernel/every-wait-in-this-kernel-is-a-spin.md`'s, because a claim + boot's netd, sshd and streaming `logd`, runs every member, hands the + machine back with `reboot`, and reads the stick as today. It folds the + base boots: the C corpus, the shared block (without `shared-debug`), + `testcases` with its `mkdir` and `readdir` boots, and the talking and + swapping boots; `mkdir_cap` and `readdir_bound` clean up after themselves + or red. + - A member that claims a device waits on + `issues/kernel/deferred-release-outlives-its-syscall.md` (fixed under + `issues/kernel/every-wait-in-this-kernel-is-a-spin.md`), because a claim not back by the check reds it. - Closes `issues/hardware/a-swap-on-the-t14-lives-inside-a-metal-boots-bound.md`. **Exit**: a T14 run folds those boots into one session and judges every - member per test. Its verdicts equal the per-boot rig's at the same head, - and a second session in reverse order gives the same verdicts. Each of - these reds only the test that staged it: a child left alive, a claim left - held, a file left in `/tmp`, the home, `/state` or `/log`, and a - `NEVER_CLEAN` line. A job that spins past its bound is killed and red by - name, and one that calls the renewal is refused. With the renewals - skipped, the machine resets when its boot's lease runs out. An e1000e QEMU + member per test with the per-boot rig's verdicts at the same head, and a + second session in reverse order gives the same verdicts. Each of these reds + only the test that staged it: a child left alive, a claim left held, a file + left in `/tmp`, the home, `/state` or `/log`, and a `NEVER_CLEAN` line. A + job that spins past its bound is killed and red by name; a job finds no + lease port in its namespace, and its claim of the TCO is refused. Stopped + renewals, a dropped host connection and a kernel wedged by + `wedge-before-reset` each end in the chipset's reset within the lease and + `BOUND_MS`, and the next boot says the TCO did it. In QEMU, a `watchdogd` + that exits without halting the timer resets the machine, and an e1000e guest runs the same session end to end. 2. **Ubuntu leaves the loop**, which is the loader track's stages 6 and 7. - First, the measurement its shape rests on: `ssh t14 update < image` timed - for a session image on today's rig. An image switch costs that write and a - reset. ToyOS has written this stick at between about 108 KiB/s and - 4.7 MiB/s, 0.2 s to 9.5 s per MiB: 11 s to 8.4 min for the 53 MiB ROOT a - `shared` image carries. + for a session image on today's rig, since an image switch costs that + write and a reset. ToyOS has written this stick at between about + 108 KiB/s and 4.7 MiB/s, 0.2 s to 9.5 s per MiB: 11 s to 8.4 min for the + 53 MiB ROOT a `shared` image carries. + - Then a loader reaches the T14 as an image does (owner ruling): ToyOS + receives it over ssh as `update` receives an image and writes it to the + stick, and the running loader tries it once and keeps the old one as the + fallback. Ubuntu leaves the loop only once this works. + - Every T14 image is signed with the bench key the T14 trusts (owner + ruling), made once outside every checkout at + `~/.config/toyos/t14-bench-signing-key` by `--signing-key-new` with + `TOYOS_SIGNING_KEY` naming that path. - Waits on the loader track's stage 5 (tries, the good flag, `update --once` and `--boot-first`), and on: - `issues/isolation/a-reset-stops-xhci-and-leaves-every-claimed-pci-function-armed.md` @@ -111,28 +135,27 @@ and no Ubuntu. `issues/diagnostics/a-netd-that-dies-while-serving-leaves-the-hosts-stream-silent.md`, so that every host redial waits on an event; - `issues/hardware/most-t14-leases-land-one-dhcp-retry-late.md`, because - every reset now reaches the host through a lease; - - `issues/boot-media/whether-the-t14-takes-images-signed-by-a-bench-key-is-the-owners.md` - and - `issues/boot-media/how-a-loader-change-reaches-the-t14-without-ubuntu-is-the-owners.md`. - - The stick is written once and put first with `update --boot-first`. From - then on every image goes by `update`, and a machine that cannot boot its - stick costs a hand and `diag/flash.sh` + every reset now reaches the host through a lease. + - The stick is written once and put first with `update --boot-first`; from + then on every image goes by `update`, and a stick that boots neither + loader costs a hand and `diag/flash.sh` (`issues/build/the-owners-flash-script-runs-diskutil.md`). + - Another image costs one reset: `update < image` and a reboot. A death + costs two: a panic, wedge, reset or loader test goes by `update --once`, + the kept slot boots after it, and the host fetches the record over + `sftp`, from `loader.log` until the loader track's stage 9 (whose report + pass costs a third POST) and from `/log` after it. - An image is good only once the host has reached it: the host's first exec on it is `update --good`, and a session image names no `[boot] up`, so init never marks it good. An image the host cannot reach lapses, spends its tries and falls back. - - Every test row names its place: a session's image, a `--once` image, or - QEMU. `METAL_ONLY`, `QemuOnly` and the name-keyed dispatch go, and a metal - row can be redlisted like any other. + - Every test row names its place, a session's image, a `--once` image or + QEMU, so `METAL_ONLY`, `QemuOnly` and the name-keyed dispatch go. - The loader keeps the pass before it as `loader-previous.log`, by a rename (`SetInfo`), so the kept slot's pass does not erase a death's report. - - Deleted: `src/metal.rs` except its arm gate, `src/icmp.rs`, - `bootloader/src/bootnext.rs`, `tests/common/metal.rs`' stick readback, - boot batching and per-boot run, test-runner's job-list mode, `metaltalk`'s - `converse` and `judge`, and every timing row for firmware, Ubuntu, the - stick or the router. + - Deleted: `src/metal.rs` but its arm gate, `src/icmp.rs`, + `bootloader/src/bootnext.rs`, the stick readback and batching of + `tests/common/metal.rs`, and test-runner's job-list mode. - Closes `issues/build/a-hung-boots-log-partition-is-wiped-by-the-next-runs-flash.md`, `issues/build/the-sudoers-rendering-has-no-host-side-judge.md`, @@ -149,12 +172,13 @@ and no Ubuntu. session, and its record is judged. A slot with a flipped byte, no signature or a lower security version is refused and the other slot boots. A slot that dies falls back on its own, and so does one whose `sshd` does not - authorize the host's key. + authorize the host's key. A loader sent over ssh boots once, and one that + brings no slot to good leaves the old one booting. 3. **Sessions merge.** A run holds one session per kernel build, parameter - line and config that some test needs. Actuators that leave the machine as - it is share an image where their authors say so, as `SELFTESTS` does; a - config that differs by one program's row joins the base session; and the + line and config that some test needs: actuators that leave the machine as + it is share an image where their authors say so, as `SELFTESTS` does, a + config that differs by one program's row joins the base session, and the QEMU registrations the T14 can run move into sessions. **Exit**: the host's plan names every reset in a run by what forces it: a diff --git a/issues/kernel/whether-every-boot-arms-the-hard-lockup-detector-is-the-owners.md b/issues/kernel/whether-every-boot-arms-the-hard-lockup-detector-is-the-owners.md index f830d53016..96fb1647d0 100644 --- a/issues/kernel/whether-every-boot-arms-the-hard-lockup-detector-is-the-owners.md +++ b/issues/kernel/whether-every-boot-arms-the-hard-lockup-detector-is-the-owners.md @@ -6,20 +6,31 @@ opened: 2026-09-29 # Whether every boot arms the hard-lockup detector is the owner's -`kernel/src/hardlockup` ends a machine one of whose CPUs has taken no interrupt, -with `IF` clear, for its bound: it seals a `WEDGED` record naming that CPU's -`pc` and `sp`, and resets the machine. It is armed only through -`boot-deadline=` (`deadline::start`, `kernel/src/deadline.rs:168-183`), so a -boot without that parameter, the owner's own machine included, has none, and -a CPU frozen there is a hand on the power button. +What it is: a performance counter on each CPU raises an NMI about once a second +of that CPU's busy time, and the NMI checks whether the CPU has taken any other +interrupt since the last one. A CPU that has taken none for +`toyos_tco::HARD_LOCKUP_BOUND_MS` with interrupts masked is frozen: it will +never run a thread again, and nothing else on the machine says so. The detector +(`kernel/src/hardlockup`) then seals a `WEDGED` record naming that CPU, its `pc` +and `sp` and the lock it spins on, and resets the machine. Linux runs the same +detector on every boot by default (`nmi_watchdog`, +`Documentation/admin-guide/lockup-watchdogs.rst`) and panics on it only where +`hardlockup_panic` is set; Windows ends such a machine with bug check 0x101, +`CLOCK_WATCHDOG_TIMEOUT`. -Arming it on every boot, at `toyos_tco::HARD_LOCKUP_BOUND_MS` where no -deadline names a bound, costs one NMI per busy CPU per second -(`SAMPLE_NS`, `kernel/src/hardlockup/mod.rs:75-80`; a halted CPU takes none), -and turns such a freeze on the owner's machine into a reset with a record. It -changes no ABI. No stage of -`issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md` waits on -it, since every session image carries `boot-deadline=`. +It is armed only through `boot-deadline=` (`deadline::start`, +`kernel/src/deadline.rs:168-183`), so a boot without that parameter, the +owner's own machine included, has none, and a CPU frozen there is a hand on the +power button. Arming it on every boot, at `HARD_LOCKUP_BOUND_MS` where no +deadline names a bound, costs one NMI per busy CPU per second (`SAMPLE_NS`, +`kernel/src/hardlockup/mod.rs:75-80`; a halted CPU takes none), and turns such +a freeze into a reset with a record. It changes no ABI. + +A session image of +`issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md` names no +`boot-deadline=`, so without this it has no detector: a CPU that locks up there +still ends in the chipset's reset once the host stops renewing, but with no +record naming it. No stage waits on it. *Recommended: yes.* diff --git a/issues/kernel/whether-init-may-renew-the-boot-deadline-is-the-owners.md b/issues/kernel/whether-init-may-renew-the-boot-deadline-is-the-owners.md deleted file mode 100644 index 3269ec872a..0000000000 --- a/issues/kernel/whether-init-may-renew-the-boot-deadline-is-the-owners.md +++ /dev/null @@ -1,29 +0,0 @@ ---- -status: owner -kind: question -opened: 2026-09-29 ---- - -# Whether init may renew the boot deadline is the owner's - -`boot-deadline=` arms a bound from boot that nothing on the machine can hold -off (`kernel/src/deadline.rs`), so a boot meant to stay up for a session of -tests is reset at it. Stage 1 of -`issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md` needs it to -be a lease, and that is a new kernel call, so an ABI change: - -- It renews the deadline to at most `WEDGE_BOUND_MS` from now, or retires it. - Its right is init's alone: no manifest name grants it, so no row can hold - it. test-runner asks init over a port the build lets only test-runner's row - receive, and hands its jobs a namespace without that port. -- It is refused once `deadline::stand_down` has run - (`kernel/src/deadline.rs:79-81`, called from `kernel/src/panic.rs:352`). A - renewal that raced a panic would otherwise arm the deadline again and seal - `WEDGED` over the panic report. -- A lapse is recorded as the deadline expiring, with when it was last - renewed, and not as a wedge: a host that went away stops renewing as surely - as a wedged machine does. - -*Recommended: yes.* - -**Exit**: the owner rules. From 361a5c4f08b9801f43e08d0cda38a7ff25efea9c Mon Sep 17 00:00:00 2001 From: japabu Date: Wed, 30 Sep 2026 15:30:25 +0200 Subject: [PATCH 04/11] The watchdog ships on every machine, and the T14's sessions use it as it ships The owner's ruling on the session's bound: the hardware watchdog is a production feature, worth shipping because it resets a frozen ToyOS, and no kernel functionality is written for tests alone. - New: issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md, a small track for that watchdog. No shipped image arms one today: the kernel arms the TCO only on the `watchdog` parameter, which a shipped image never carries (src/build.rs:1518-1520 says a --kernel-param makes an image not a shipping one). watchdogd claims the machine's hardware watchdog through a device class of its own and that class's register allow-list, with no syscall, and feeds it from a thread in the real-time band. The TCO line is the one `modprobe iTCO_wdt` printed on the T14's Ubuntu (11c0efd0a); the 30 s heartbeat is from the owner's brief, and this branch did not read it. - Moved there from the session track's stage 1: the measurement the whole bound rests on. It is the watchdog track's first stage, and it runs only on the owner's go-ahead, because he stopped it once. It now names the job that holds the boot past the bound, because the shutdown disarms the timer (kernel/src/syscall/machine.rs:88). - Deleted from the session track, as the test-only parts of the bound: the lease the host renewed, the renewal port, the runner renewing, "only while its lease is live", `quit` halting the timer, and the exit's controls for them (a job finding no lease port, stopped renewals and a dropped host connection each resetting the machine). What stands is the ruling's own: a stuck test is killed over the session, a frozen ToyOS is reset by the watchdog, and a ToyOS that still runs but that the host cannot reach stops the run and waits for a hand. - Deleted as false once the lease went: stage 2's "an image the host cannot reach lapses, spends its tries and falls back", and the exit's claim that a slot whose sshd does not authorize the host's key falls back on its own. Such a slot is fed by its watchdogd; the exit now asks only that it is never marked good. - Stage 1's first step is now concrete, and needs neither the watchdog nor the owner: the `testcases` arm's members in one session, under the boot-deadline= every metal image already carries. Folding more boots into a session waits on the watchdog track's stage 2, because a session longer than that bound needs the watchdog in its place. - The hard-lockup question loses the one clause that leaned on the host's renewals. Its question and recommendation are unchanged. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L --- ...os-waits-for-a-hand-on-the-power-button.md | 58 +++++++++++ ...4-reboots-through-ubuntu-for-every-test.md | 95 +++++++++---------- ...-the-hard-lockup-detector-is-the-owners.md | 4 +- 3 files changed, 102 insertions(+), 55 deletions(-) create mode 100644 issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md diff --git a/issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md b/issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md new file mode 100644 index 0000000000..3f45e88e1c --- /dev/null +++ b/issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md @@ -0,0 +1,58 @@ +--- +status: open +kind: track +opened: 2026-09-30 +--- + +# A frozen ToyOS waits for a hand on the power button + +No shipped image arms a hardware watchdog. The kernel arms the chipset's TCO +only on the `watchdog` boot parameter +(`kernel/src/arch/x86_64/watchdog.rs:48-51`), which no shipped image carries, +and feeds it from the scheduler pass (`kernel/src/sched/driver.rs:512`), which +proves only that some CPU still reaches one. The owner's ruling: the watchdog +is a production feature, shipped to every machine. `watchdogd`, a userland +service, claims the machine's hardware watchdog and feeds it while the system +is healthy; if ToyOS freezes or its userland stops being scheduled, the +hardware resets the machine. + +- **No syscall.** The watchdog becomes a device class of its own, and + `watchdogd` reaches its registers through that class's allow-list in + `sys_device_reg` (`kernel/src/syscall/device.rs:55-77`), as soundd reaches + HDA's. The build grants the class to `watchdogd`'s row alone, as it does + `swap`. +- **Fed from the real-time band.** `watchdogd` feeds from a thread of its own + in the real-time band (`syscap = ["rt"]`), so a saturated fair band does not + starve it; `issues/kernel/cpu-time-is-a-band-and-not-a-reservation.md` + replaces that precedence with a reservation, which `watchdogd` then holds. +- **Nothing in the kernel is added for tests only** (owner ruling): a test + harness uses this watchdog as it ships. +- On the T14 it is the PCH's TCO, which Linux's `iTCO_wdt` finds as "Intel PCH + TCO device (Version=6, TCOBASE=0x0400)" and runs at a 30 s heartbeat. The + T14 has no BMC or AMT (its i5-1135G7 has no vPro) and a battery no power + switch cuts, so the TCO is its one reset that needs nothing of the kernel. +- A panic stops every feed, so an armed chipset ends the panic panel's hold + within its bound, before `toyos_tco::PANIC_BOUND_MS` and whatever key is + pressed (`kernel/src/drivers/panic_console/mod.rs:716-725`). + +## Stages + +1. **The measurement, on the owner's go-ahead.** A T14 boot arms the TCO and + stops feeding it: `watchdog` with `tco-starve`, which `src/metal.rs`' + `FLASHABLE` must first admit, and a job that holds the boot past the bound, + since the shutdown disarms the timer (`kernel/src/syscall/machine.rs:88`). + The machine resets within `toyos_tco::BOUND_MS` of its last feed with no + hand on it, which is the exit of + `issues/hardware/an-armed-tco-has-never-reset-the-t14.md`. If it does not, + the track stops and goes back to the owner with that run's `loader.log`. + +2. **`watchdogd` ships.** The shipped image arms the watchdog and starts + `watchdogd`. The kernel feeds the timer from the scheduler pass until the + claim is minted, and never after. A machine whose chipset `toyos_tco` has no + row refuses the claim as absent, and the boot says it is unwatched. + + **Exit**: in QEMU, a `watchdogd` that stops feeding and a kernel wedged by + `wedge-before-reset` each end in the chipset's reset; a fair-band load that + resets a machine whose `watchdogd` lacks `rt` leaves one with `rt` fed; and + the build refuses the class to any other row. On the T14, the wedged kernel + ends in the same reset. diff --git a/issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md b/issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md index 44ef1426f1..5941a96845 100644 --- a/issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md +++ b/issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md @@ -41,53 +41,48 @@ another in a ToyOS that stays booted: no reboot per test, and no Ubuntu. its test and gives the next one a fresh session. A test whose premise is a fresh boot, as `audio_idle_suspend`'s is, says so in its row and runs first in a fresh session. -- **The bound is the chipset's TCO watchdog, fed from userland while the host - renews a lease (owner ruling).** The T14 has no BMC or AMT (its i5-1135G7 - has no vPro) and a battery no power switch cuts, so the TCO is its one reset - that needs nothing of the kernel. It becomes a device class of its own, held - by a `watchdogd` service that reaches its registers through a claim's - allow-list (`kernel/src/syscall/device.rs:55-77`), with no new syscall. The - kernel feeds it from the scheduler pass (`kernel/src/sched/driver.rs:512`) - until the claim is minted; then only the service does, from the real-time - band, which no job's load starves, and only while its lease is live. The - lease starts at `WEDGE_BOUND_MS`, `run_one` renews it for as long on a port - no job's namespace carries, and `quit` retires it by halting the timer. A - frozen kernel, a dead service, a dead runner or a silent host stops the - feeding, the chipset resets the machine within `toyos_tco::BOUND_MS` - (9.6 s), and the next boot says so - (`kernel/src/arch/x86_64/watchdog.rs:92-99`). Only a machine that powers - itself off, or firmware that does not come back, waits for a hand; the host - declares it lost when nothing answers within the lease, `BOUND_MS` and a - POST allowance, derived as `return_secs` is. +- **A session runs under the production watchdog as it ships** + (`issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md`), + and nothing in the kernel is added for tests. A stuck test is killed over + the session: past its bound the host has the runner end it, and it reds by + name. A frozen ToyOS is reset by the watchdog. A ToyOS that still runs but + that the host cannot reach keeps its watchdog fed, so the run stops there + and waits for a hand, as it does for a machine that powers itself off or + firmware that does not come back; the host says so once nothing has + answered within the watchdog's bound and a POST allowance, derived as + `return_secs` is. ## Stages 1. **One session on today's rig.** - - First, the measurement the bound rests on: a T14 boot armed with - `watchdog` and `tco-starve` (which `src/metal.rs`' `FLASHABLE` must first - admit) is reset by the chipset within `toyos_tco::BOUND_MS` of its last - feed, and the boot after it reads `TCO_SECOND_TO_STS`: the exit of - `issues/hardware/an-armed-tco-has-never-reset-the-t14.md`. If it is not - reset, the stage stops and the bound goes back to the owner with that - run's registers: nothing else resets this machine without the kernel, - and renewing the kernel's own deadline is the syscall this ruling - declined. - - The build grants the TCO's class to `watchdogd`'s row alone and its lease - port to test-runner's alone, as it does `swap`. A session image carries - the `watchdog` parameter and a `watchdogd` row but no `boot-deadline=`, - whose whole-boot bound would end the session, so `judge_arms` - (`src/metal.rs:897-908`) takes that row as its bound, and `hardlockup` - runs only if every boot arms it + - First, and next to build: the `testcases` arm's members in one session, + under the `boot-deadline=` every metal image already carries + (`tests/common/metal.rs:805-823`). It builds `run_one`'s kernel records; + a `tests/ssh-client-host` mode that relays one exec channel's input and + output as they arrive, which the metal loop opens as `test-runner` during + the boot, writing `run ` for each member in the arm's order and + keeping each window in the readback; and the session image, + `tests/testcases` with `tests/lantalkcase`'s netd, sshd and streaming + `logd` rows, in which that exec, not `[boot] start`, starts + `test-runner`. The loop hands the machine back with `reboot` and reads + the stick as today. Its exit: every registration that rides `testcases` + gives, off the stick, the verdict the per-boot arm gives at the same + head, the host's judgement of each member's window agrees with it, and a + member patched to fault reds on its window by the kernel's record of the + fault. + - A session outlives that bound once stage 2 of + `issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md` + ships `watchdogd`. A session image then starts it as the shipped image + does and names no `boot-deadline=`, whose whole-boot bound would end the + session, so `judge_arms` (`src/metal.rs:909-936`) takes `watchdogd`'s row + as its bound, and `hardlockup` runs only if every boot arms it (`issues/kernel/whether-every-boot-arms-the-hard-lockup-detector-is-the-owners.md`). - QEMU's shared block runs the same `run_one`; its runner's stdout is a log ring, so the console carries the kernel's records and it writes none. - - The host flashes one session image, `tests/testcases` with the talking - boot's netd, sshd and streaming `logd`, runs every member, hands the - machine back with `reboot`, and reads the stick as today. It folds the - base boots: the C corpus, the shared block (without `shared-debug`), - `testcases` with its `mkdir` and `readdir` boots, and the talking and - swapping boots; `mkdir_cap` and `readdir_bound` clean up after themselves - or red. + - One session image then folds the base boots: the C corpus, the shared + block (without `shared-debug`), `testcases` with its `mkdir` and + `readdir` boots, and the talking and swapping boots; `mkdir_cap` and + `readdir_bound` clean up after themselves or red. - A member that claims a device waits on `issues/kernel/deferred-release-outlives-its-syscall.md` (fixed under `issues/kernel/every-wait-in-this-kernel-is-a-spin.md`), because a claim @@ -99,13 +94,10 @@ another in a ToyOS that stays booted: no reboot per test, and no Ubuntu. second session in reverse order gives the same verdicts. Each of these reds only the test that staged it: a child left alive, a claim left held, a file left in `/tmp`, the home, `/state` or `/log`, and a `NEVER_CLEAN` line. A - job that spins past its bound is killed and red by name; a job finds no - lease port in its namespace, and its claim of the TCO is refused. Stopped - renewals, a dropped host connection and a kernel wedged by - `wedge-before-reset` each end in the chipset's reset within the lease and - `BOUND_MS`, and the next boot says the TCO did it. In QEMU, a `watchdogd` - that exits without halting the timer resets the machine, and an e1000e - guest runs the same session end to end. + job that spins past its bound is killed over the session and red by name, + and the next member runs. In QEMU, an e1000e guest runs the same session + end to end, and one whose link is cut under a session stops the run and + says it waits for a hand. 2. **Ubuntu leaves the loop**, which is the loader track's stages 6 and 7. - First, the measurement its shape rests on: `ssh t14 update < image` timed @@ -147,8 +139,7 @@ another in a ToyOS that stays booted: no reboot per test, and no Ubuntu. pass costs a third POST) and from `/log` after it. - An image is good only once the host has reached it: the host's first exec on it is `update --good`, and a session image names no `[boot] up`, so - init never marks it good. An image the host cannot reach lapses, spends - its tries and falls back. + init never marks it good. - Every test row names its place, a session's image, a `--once` image or QEMU, so `METAL_ONLY`, `QemuOnly` and the name-keyed dispatch go. - The loader keeps the pass before it as `loader-previous.log`, by a rename @@ -171,9 +162,9 @@ another in a ToyOS that stays booted: no reboot per test, and no Ubuntu. `BootNext` set. A `--once` image that panics returns the machine to its session, and its record is judged. A slot with a flipped byte, no signature or a lower security version is refused and the other slot boots. A slot - that dies falls back on its own, and so does one whose `sshd` does not - authorize the host's key. A loader sent over ssh boots once, and one that - brings no slot to good leaves the old one booting. + that dies falls back on its own, and one whose `sshd` does not authorize + the host's key is never marked good. A loader sent over ssh boots once, and + one that brings no slot to good leaves the old one booting. 3. **Sessions merge.** A run holds one session per kernel build, parameter line and config that some test needs: actuators that leave the machine as diff --git a/issues/kernel/whether-every-boot-arms-the-hard-lockup-detector-is-the-owners.md b/issues/kernel/whether-every-boot-arms-the-hard-lockup-detector-is-the-owners.md index 96fb1647d0..8e8008d447 100644 --- a/issues/kernel/whether-every-boot-arms-the-hard-lockup-detector-is-the-owners.md +++ b/issues/kernel/whether-every-boot-arms-the-hard-lockup-detector-is-the-owners.md @@ -28,9 +28,7 @@ a freeze into a reset with a record. It changes no ABI. A session image of `issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md` names no -`boot-deadline=`, so without this it has no detector: a CPU that locks up there -still ends in the chipset's reset once the host stops renewing, but with no -record naming it. No stage waits on it. +`boot-deadline=`, so without this it has no detector. No stage waits on it. *Recommended: yes.* From f3e64a01a6b04501861eeec0b610390baac3de09 Mon Sep 17 00:00:00 2001 From: japabu Date: Wed, 30 Sep 2026 16:48:58 +0200 Subject: [PATCH 05/11] Answer the third review of #631: the first session deletes seven or eight boots, and the watchdog's measurement is a latch The session track goes to one screen. Its first step names the boots it deletes: `shared` and `ccorpus` with their chunks, `testcases`, `testcases-mkdir` and `testcases-readdir`, which share a config, a parameter line and the shipping kernel. From the runner's `spawn:` to `reboot`'s in each boot's `kernel.log`, summed with `bc`, their lists took 42.887 s on the #536 branch's T14 run (eight boots) and 58.386 s on toyos-metaltimings' (seven, no `shared-3`). `loader_watchdog_arms`' control arm rides the session boot. The leftover check and a reverse-order session are in the step's exit. Judges read the stick where they read it today, except `syscall_cost`, whose line is the job's own stdout and so crosses only the channel, and `log_poll_outlives_a_close`, whose anchor on the job before it would misfire in reverse order. The claim check waits on the deferred-release defect, since a released claim can still read held when the check runs. Ubuntu's exit goes back to the loader track's stages 6 and 7, which own the bench, carrying what round 2 had added in the session track: the timed `update` first, the owner's rulings on the bench key and on a loader reaching the T14 as an image does, `update --good` as the host's first exec, deaths by `update --once`, the waits, and the job-list mode's deletion. Stage 5 now says init never marks good an image with no `[boot] up`. The watchdog track: - Stage 1's reading is the `TCO2_STS` latch, read by the next armed boot's loader and kernel, because a TCO reset seals no record and the report pass after it prints no TCO line. `tco-starve` must join `FLASHABLE` and `WEDGE_ARMS`. The measurement runs on the T14 only on the owner's go-ahead. - The fed control is `watchdog_fed`, filed as a defect: its boot is `reboot` alone, spawned 1.196 s and 1.489 s into the kernel's clock on two T14 runs, so it ends inside one 9.6 s bound and a kernel that never fed passes it. - The fair-band arm is deleted: QEMU gives no timing verdict. - The panic panel holds as today, 60 s or for good after a key, because its hold feeds the watchdog; a panic that never reaches the panel is reset. - The kernel refuses the class to every other `device` holder, since any holder of that right may mint a claim. - `toyos_tco::PARAM` goes after loader stage 8, and every q35 guest is armed. - The negative path is a real wedge, `wedge-before-reset`, not a knob. The DHCP defect's exit reads netd's own log instead of a capture, and loses its citation of smoltcp's source. The eight REMOVEs are deleted. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L --- ...oes-only-what-must-precede-the-handover.md | 76 ++++-- ...e-machine-updates-itself-without-ubuntu.md | 3 +- ...os-waits-for-a-hand-on-the-power-button.md | 90 ++++--- ...ost-t14-leases-land-one-dhcp-retry-late.md | 17 +- ...4-reboots-through-ubuntu-for-every-test.md | 220 +++++------------- ...nds-its-boot-before-the-bound-it-judges.md | 21 ++ ...tarts-holds-its-whole-system-capability.md | 4 +- 7 files changed, 207 insertions(+), 224 deletions(-) create mode 100644 issues/hardware/watchdog-fed-ends-its-boot-before-the-bound-it-judges.md diff --git a/issues/boot-media/the-loader-does-only-what-must-precede-the-handover.md b/issues/boot-media/the-loader-does-only-what-must-precede-the-handover.md index bb56c41b74..858fb15ab4 100644 --- a/issues/boot-media/the-loader-does-only-what-must-precede-the-handover.md +++ b/issues/boot-media/the-loader-does-only-what-must-precede-the-handover.md @@ -28,16 +28,14 @@ PR #539 does not land. Its pieces: `update_boot_next_boots_the_entry_once`'s read-only half; the harness's `stick_readonly`; and `update_trial_writes_nothing_of_the_kept_slot`, rewritten for priorities; -- stage 2 of `issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md` - takes the bench: `src/metalbench.rs`, `tests/common/bench.rs`, +- stage 6 takes the bench: `src/metalbench.rs`, `tests/common/bench.rs`, `tests/bench*case`, `--bench-image` with `build::bench_image`, `src/image.rs`' `update_of`, `tests/common/metal.rs`' `Reach`, `stage` and - bench `invocation`, toybox `date`, `Ssh::probe` and + bench `invocation`, `--metal-via-ubuntu`, toybox `date`, `Ssh::probe` and `Ssh::fetch` with `ssh-client-host`'s `probe` and `fetch`, `build::AUTHORIZED_ON_ROOT`, and `src/bootlog.rs`' `LOADER_PREVIOUS_LOG`, `MOUNTED_FROM_MEMORY` and `BOOT_PARAMETER`; -- deleted: `--metal-via-ubuntu`, because that stage takes Ubuntu out of the - loop; `update --boot-next ` with `slots::Next::Esp`, `Guid::parse`, +- deleted: `update --boot-next ` with `slots::Next::Esp`, `Guid::parse`, `bootvars.rs`' `BootNext` and entry-after-its-own writes, and `update_boot_next_boots_the_entry_once`'s other half; the panic handler's fall to the next boot entry, with the two issues #539 filed about it, @@ -224,8 +222,9 @@ Each stage lands on its own, in this order. label `ready`, and the service writes one byte to it once it serves. A read end that closes with no byte is a service that never came up. - Init runs `update --good` once every service `[boot] up` names has - written its byte; `update` holds the `slots` claim's table and writes the - running slot's good flag and nothing else. + written its byte, and never for an image that names no `[boot] up`; + `update` holds the `slots` claim's table and writes the running slot's + good flag and nothing else. - An image whose `system.toml` grants no program the `slots` claim is never good, so each boot spends a try and it boots three times. Today only `system.toml` and `tests/updatecase/system.toml` grant it. @@ -308,16 +307,59 @@ Each stage lands on its own, in this order. format's bytes and not by `toyos_update::slots::Table::decode`; and OVMF's boot manager booting the entry the loader wrote. -6. **The T14 bench** and 7. **Ubuntu leaves the loop** are stage 2 of - `issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md`, which - drives the T14 by sessions of tests over the cable and deletes - `bootloader/src/bootnext.rs`. #539's issues - `a-loader-change-reaches-a-machine-only-by-writing-its-stick`, - `the-bench-reads-no-quiescent-log-volume`, - `the-bench-runs-with-no-bound-on-its-own-boot`, - `the-bench-sometimes-comes-back-two-minutes-late` and - `the-benchs-cable-is-read-by-the-driver-under-test` land there, each only - as far as it is true of what lands. +6. **The T14 bench.** + - It opens with a timed `ssh t14 update < image` of a session image of + `issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md`, since + an image switch costs that write and a reset: ToyOS has written this + stick at 0.2 s to 9.5 s per MiB, 11 s to 8.4 min for a 53 MiB ROOT. + - It is built from #539's pieces that stage 6 takes above, and the bench + path of `src/metal.rs`. `--via-ubuntu` stays as the old path. + - Every T14 image is signed with the bench key the T14 trusts, and a loader + reaches the T14 as an image does: ToyOS receives it over ssh and writes it + to the stick, and the running loader tries it once and keeps the old one + as the fallback (owner rulings). + - The host's first exec on an image is `update --good`, and a session image + names no `[boot] up`. A death goes by `update --once`, the kept slot boots + after it, and the host fetches the record over `sftp`. + - The bench reads the loader's file off the ESP. + - The tested pass's `loader.log` is kept as `loader-previous.log` by a + rename (`SetInfo`), not by #539's copy. + - It runs over the cable that + `issues/hardware/the-t14-answers-only-through-a-usb-stick.md` owns, and + waits on + `issues/isolation/a-reset-stops-xhci-and-leaves-every-claimed-pci-function-armed.md`, + `issues/hardware/the-t14-hung-after-rebooting-with-its-i219-faulted.md`, + `issues/panic-path/a-fatal-event-stands-down-both-bounds-and-may-leave-nothing-to-end-the-machine.md`, + `issues/hardware/most-t14-leases-land-one-dhcp-retry-late.md`, + `issues/diagnostics/a-swaps-redial-asks-again-with-no-event-to-wait-on.md`, + `issues/hardware/the-t14-redial-re-asks-mdns-after-every-refusal.md`, + `issues/diagnostics/a-first-dial-turned-away-before-a-line-is-waited-on-to-its-callers-bound.md` + and + `issues/diagnostics/a-netd-that-dies-while-serving-leaves-the-hosts-stream-silent.md`. + - #539's issues `a-loader-change-reaches-a-machine-only-by-writing-its-stick`, + `the-bench-reads-no-quiescent-log-volume`, + `the-bench-runs-with-no-bound-on-its-own-boot`, + `the-bench-sometimes-comes-back-two-minutes-late` and + `the-benchs-cable-is-read-by-the-driver-under-test` land here, each only + as far as it is true of what lands. + + **Exit**: `bench_loop_drives_a_toyos_machine` passes in QEMU. On the T14, + with Ubuntu never started: a run of sessions gives the verdicts a per-boot + run gave at the commit this stage branches from; a kernel change boots; a + slot with a flipped byte, no signature or a lower security version is + refused and the other boots; a slot that dies falls back on its own, and + one whose `sshd` refuses the host's key is never marked good; a `--once` + image that panics returns the machine to its session with its record + judged; and a loader sent over ssh that brings no slot to good leaves the + old one booting. + +7. **Ubuntu leaves the loop.** Delete `toyos-metal`'s `--via-ubuntu` path, + `--metal-via-ubuntu`, `bootloader/src/bootnext.rs` and test-runner's + job-list mode. After a reset the firmware comes back to the loader because + ToyOS's entry is first (stage 5). + + **Exit**: no path in `src/metal*.rs` reaches Ubuntu. On the T14, a panic's + reset reaches the loader with no `BootNext` set. 8. **The kernel arms the TCO before anything unbounded.** The loader's arm is the only bound today from `ExitBootServices` to `deadline::start` on diff --git a/issues/boot-media/the-machine-updates-itself-without-ubuntu.md b/issues/boot-media/the-machine-updates-itself-without-ubuntu.md index 758b1101ba..d4559130bc 100644 --- a/issues/boot-media/the-machine-updates-itself-without-ubuntu.md +++ b/issues/boot-media/the-machine-updates-itself-without-ubuntu.md @@ -42,8 +42,7 @@ The machine boots the stick and installs onto its own NVMe, then takes `ssh t14 update < image` for every change after. It waits on stage 5 of `issues/boot-media/the-loader-does-only-what-must-precede-the-handover.md`, whose `update --boot-first` puts the NVMe loader's entry first; taking Ubuntu -out of `toyos-metal`'s loop is stage 2 of -`issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md`. +out of `toyos-metal`'s loop is that track's too. **Exit**: with the stick pulled, the T14 boots ToyOS off its NVMe, and a kernel change sent with `ssh t14 update < image` boots at the next reset. diff --git a/issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md b/issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md index 3f45e88e1c..a0b6ad9f28 100644 --- a/issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md +++ b/issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md @@ -10,49 +10,69 @@ No shipped image arms a hardware watchdog. The kernel arms the chipset's TCO only on the `watchdog` boot parameter (`kernel/src/arch/x86_64/watchdog.rs:48-51`), which no shipped image carries, and feeds it from the scheduler pass (`kernel/src/sched/driver.rs:512`), which -proves only that some CPU still reaches one. The owner's ruling: the watchdog -is a production feature, shipped to every machine. `watchdogd`, a userland -service, claims the machine's hardware watchdog and feeds it while the system -is healthy; if ToyOS freezes or its userland stops being scheduled, the -hardware resets the machine. +proves only that some CPU still reaches one; and no armed TCO has yet reset the +T14 (`issues/hardware/an-armed-tco-has-never-reset-the-t14.md`). The owner's +ruling: the watchdog is a production feature, shipped to every machine. +`watchdogd`, a userland service, claims the machine's hardware watchdog and +feeds it while the system is healthy; if ToyOS freezes or its userland stops +being scheduled, the hardware resets the machine. - **No syscall.** The watchdog becomes a device class of its own, and `watchdogd` reaches its registers through that class's allow-list in `sys_device_reg` (`kernel/src/syscall/device.rs:55-77`), as soundd reaches - HDA's. The build grants the class to `watchdogd`'s row alone, as it does - `swap`. + HDA's. +- **Only `watchdogd` holds it.** Any holder of `device` may mint a claim + (`kernel/src/syscall/device.rs:123-141`), test-runner and every job it starts + among them (`tests/testcases/system.toml:41`), and the first mint ends the + kernel's feed: so the kernel refuses the class to every other holder, which + a build refusing it to other rows does not do. - **Fed from the real-time band.** `watchdogd` feeds from a thread of its own - in the real-time band (`syscap = ["rt"]`), so a saturated fair band does not - starve it; `issues/kernel/cpu-time-is-a-band-and-not-a-reservation.md` + under `rt`; `issues/kernel/cpu-time-is-a-band-and-not-a-reservation.md` replaces that precedence with a reservation, which `watchdogd` then holds. - **Nothing in the kernel is added for tests only** (owner ruling): a test harness uses this watchdog as it ships. -- On the T14 it is the PCH's TCO, which Linux's `iTCO_wdt` finds as "Intel PCH - TCO device (Version=6, TCOBASE=0x0400)" and runs at a 30 s heartbeat. The - T14 has no BMC or AMT (its i5-1135G7 has no vPro) and a battery no power - switch cuts, so the TCO is its one reset that needs nothing of the kernel. -- A panic stops every feed, so an armed chipset ends the panic panel's hold - within its bound, before `toyos_tco::PANIC_BOUND_MS` and whatever key is - pressed (`kernel/src/drivers/panic_console/mod.rs:716-725`). +- The T14 has no BMC or AMT (its i5-1135G7 has no vPro) and a battery no power + switch cuts, so its PCH's TCO is its one reset that needs nothing of the + kernel. +- **A panic's panel holds as it does today**, for `toyos_tco::PANIC_BOUND_MS` + or for good once a key is pressed: its hold feeds the watchdog for as long + as it holds, the hold after a key included, which halts its CPU today + (`kernel/src/drivers/panic_console/mod.rs:710-725`). A panic that never + reaches the panel stops every feed, and the watchdog resets the machine. ## Stages -1. **The measurement, on the owner's go-ahead.** A T14 boot arms the TCO and - stops feeding it: `watchdog` with `tco-starve`, which `src/metal.rs`' - `FLASHABLE` must first admit, and a job that holds the boot past the bound, - since the shutdown disarms the timer (`kernel/src/syscall/machine.rs:88`). - The machine resets within `toyos_tco::BOUND_MS` of its last feed with no - hand on it, which is the exit of - `issues/hardware/an-armed-tco-has-never-reset-the-t14.md`. If it does not, - the track stops and goes back to the owner with that run's `loader.log`. - -2. **`watchdogd` ships.** The shipped image arms the watchdog and starts - `watchdogd`. The kernel feeds the timer from the scheduler pass until the - claim is minted, and never after. A machine whose chipset `toyos_tco` has no - row refuses the claim as absent, and the boot says it is unwatched. - - **Exit**: in QEMU, a `watchdogd` that stops feeding and a kernel wedged by - `wedge-before-reset` each end in the chipset's reset; a fair-band load that - resets a machine whose `watchdogd` lacks `rt` leaves one with `rt` fed; and - the build refuses the class to any other row. On the T14, the wedged kernel - ends in the same reset. +1. **The measurement, run on the T14 only on the owner's go-ahead.** Two boots. + The first is armed with `watchdog` and `tco-starve`, which `src/metal.rs`' + `FLASHABLE` and `WEDGE_ARMS` must first name, and a job holds it past the + bound, since the shutdown disarms the timer + (`kernel/src/syscall/machine.rs:88`); its `boot-deadline=` ends it if the + TCO does not. A TCO reset seals no record, so the reading is the second, the + next armed boot: its loader's `TCO2_STS` and its kernel's + `watchdog: the last boot ended in a TCO reset` + (`kernel/src/arch/x86_64/watchdog.rs:92-99`), which the report pass in + between does not print. That line is also the reading + `issues/hardware/tco2-sts-clearing-is-verified-on-qemu-only.md` waits on. + The fed control is `watchdog_fed`, once + `issues/hardware/watchdog-fed-ends-its-boot-before-the-bound-it-judges.md` + holds its boot past the bound; the positive control is q35's starved guest, + `watchdog_resets` (`tests/common/power.rs:515-536,694-701`). + + **Exit**: the armed boot after the starved one reads `TCO_SECOND_TO_STS` + set. If it reads it clear, the track stops and goes back to the owner with + both boots' readbacks. + +2. **`watchdogd` ships**, after the loader track's stage 8 + (`issues/boot-media/the-loader-does-only-what-must-precede-the-handover.md`) + has made the kernel's arm the only one. `toyos_tco::PARAM` goes: the kernel + arms on every boot whose chipset `toyos_tco` has a row, q35 included, and + feeds from the scheduler pass until the claim is minted and never after. + With the parameter go `watchdog_quiet`, `loader_watchdog_arms`' control arm + and the `testcases-watchdog` boot, since every boot is then armed. A machine + with no row refuses the claim as absent, and the boot says it is unwatched. + + **Exit**: in QEMU, a kernel wedged by `wedge-before-reset` with `watchdogd` + feeding ends in the chipset's reset; a job holding `device` is refused the + class; and a whole QEMU run resets no guest it did not stage. On the T14, + the boot after a wedged one reads the latch set, and the boot after a panic + whose panel held reads it clear. diff --git a/issues/hardware/most-t14-leases-land-one-dhcp-retry-late.md b/issues/hardware/most-t14-leases-land-one-dhcp-retry-late.md index 21229e308c..67fd30a095 100644 --- a/issues/hardware/most-t14-leases-land-one-dhcp-retry-late.md +++ b/issues/hardware/most-t14-leases-land-one-dhcp-retry-late.md @@ -12,8 +12,8 @@ after netd came up and 11 at 13.1–13.4 s, so the first lease of a boot, and with it `sshd`, arrived at 9.1–9.3 s of boot on 4 of the 13 boots and at 19.0–19.1 s on the other 9. -The ten-second step is smoltcp 0.12's default `RetryConfig::discover_timeout` -(`smoltcp-0.12.0/src/socket/dhcpv4.rs:134`), which netd keeps. netd restarts +The ten-second step is smoltcp 0.12's default `RetryConfig::discover_timeout`, +which netd keeps. netd restarts discovery when the link comes up (`userland/netd/src/dhcp.rs:42-61`), and on the late boots the lease came from the DISCOVER sent one timeout after that one. What became of the first, whether it left the machine and whether an @@ -22,10 +22,11 @@ at 4 s, randomized by ±1 s. The network track owns it: netd leaves smoltcp in stage 5 of `issues/design-debt/toyos-has-its-own-network-stack.md`, whose stage 3 is -`toyos-dhcp`. Stage 2 of -`issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md` waits on -it. +`toyos-dhcp`. Stage 6 of +`issues/boot-media/the-loader-does-only-what-must-precede-the-handover.md` +waits on it. -**Exit**: a capture of a T14 boot shows what became of the DISCOVER sent as -the link came up, and on every boot of a T14 run an unanswered DISCOVER is -sent again within RFC 2131 §4.1's 4 ± 1 s. +**Exit**: netd logs each DISCOVER it sends and each OFFER it receives, and a +T14 boot's log shows what became of the DISCOVER sent as the link came up; on +every boot of a T14 run an unanswered DISCOVER is sent again within RFC 2131 +§4.1's 4 ± 1 s. diff --git a/issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md b/issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md index 5941a96845..c8bd305aa9 100644 --- a/issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md +++ b/issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md @@ -6,171 +6,71 @@ opened: 2026-09-29 # The T14 reboots through Ubuntu for every test -Every metal arm is a boot of its own: a flash over ssh to Ubuntu, `BootNext`, -and the log partition read back once the loader's report pass has reset the -machine into Ubuntu. The owner's direction is that the tests run one after -another in a ToyOS that stays booted: no reboot per test, and no Ubuntu. +A T14 boot is a flash through Ubuntu, `BootNext`, and a stick read back after +the loader's report pass resets into Ubuntu. The owner's direction: the tests +run one after another in a ToyOS that stays booted, driven by the host over +ssh, with no reboot between them and no Ubuntu. -- **A session is one image** (a kernel build, a parameter line and a config) - booted once and driven over one exec channel to its `sshd`: `test-runner`, - launched by init under its own row, whose stdin loop is the session's - (`userland/test-runner/src/main.rs:140-145,229-249`). The host writes - `run ` and waits for `===TEST_END===`, on one channel because `sshd` - aliases a second one's input - (`issues/isolation/sshd-holds-one-channel-and-does-not-say-so.md`). -- **The host judges each test** with `Serial::must_be_clean` - (`tests/common/serial.rs:341`) and the exit code; ToyOS copies no list. - - The markers and the test's own output come over the channel and never - reach `/log`: `sshd` pipes a program's stdio - (`userland/sshd/src/main.rs:234-239`), init keeps a launch caller's pipes - (`userland/init/src/main.rs:1593-1604`), and a job inherits the runner's - stdout (`userland/test-runner/src/main.rs:276-277`). - - The kernel's records, to be built: after each test `run_one` reads the log - on its own cursor (`toyos::log::LogTail`, on its row's `logread`) and - writes the test's records before its `===TEST_END===`, with how many the - ring dropped; a drop reds the test. Every record lands in one window, a - builtin's too. - - A death's own account is the black box's where it sealed one; after a - reset that sealed nothing, the host has the tail of `logd`'s stream - (`toyos_logstream::PORT`), held open for the session. -- **A test leaves the machine as it found it.** After each test `run_one` - writes the processes alive (`roster`), each claim and its holder - (`inventory`), and the listings of `/tmp`, the session home, `/state` and - `/log`; the host compares each with the session's opening one, `/log` less - what `bootlog::split_listing` gives `logd` and the loader. A leftover reds - its test and gives the next one a fresh session. A test whose premise is a - fresh boot, as `audio_idle_suspend`'s is, says so in its row and runs first - in a fresh session. -- **A session runs under the production watchdog as it ships** - (`issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md`), - and nothing in the kernel is added for tests. A stuck test is killed over - the session: past its bound the host has the runner end it, and it reds by - name. A frozen ToyOS is reset by the watchdog. A ToyOS that still runs but - that the host cannot reach keeps its watchdog fed, so the run stops there - and waits for a hand, as it does for a machine that powers itself off or - firmware that does not come back; the host says so once nothing has - answered within the watchdog's bound and a POST allowance, derived as - `return_secs` is. +- **One exec channel**, to `test-runner`'s stdin loop, since `sshd` aliases a + second channel's input (`issues/isolation/sshd-holds-one-channel-and-does-not-say-so.md`); + `tests/ssh-client-host`'s `pipe` relays it once it writes output as it + arrives (`tests/ssh-client-host/src/main.rs:251-263`). +- **A test's window** is what crosses between its markers: the job's output, + never `/log` (`userland/sshd/src/main.rs:234-239`, + `userland/init/src/main.rs:2050-2063`, `userland/test-runner/src/main.rs:276-277`), + and the kernel's records, which `run_one` reads on its own + `toyos::log::LogTail` and writes before `===TEST_END===`; a record the ring + dropped reds the test. +- **A leftover reds its test.** After each test `run_one` writes the processes + alive and the listings of `/tmp`, `/state` and `/log`, which the host holds to + the session's first, `/log` less `bootlog::split_listing`'s files. Claims join + once `issues/kernel/deferred-release-outlives-its-syscall.md` closes, since a + released claim can read held until then. A test whose premise is a fresh + boot, as `audio_idle_suspend`'s is, says so and runs first. +- **A stuck test** is killed by the runner's deadline thread made per job + (`userland/test-runner/src/main.rs:153-194`), and reds by name. ## Stages -1. **One session on today's rig.** - - First, and next to build: the `testcases` arm's members in one session, - under the `boot-deadline=` every metal image already carries - (`tests/common/metal.rs:805-823`). It builds `run_one`'s kernel records; - a `tests/ssh-client-host` mode that relays one exec channel's input and - output as they arrive, which the metal loop opens as `test-runner` during - the boot, writing `run ` for each member in the arm's order and - keeping each window in the readback; and the session image, - `tests/testcases` with `tests/lantalkcase`'s netd, sshd and streaming - `logd` rows, in which that exec, not `[boot] start`, starts - `test-runner`. The loop hands the machine back with `reboot` and reads - the stick as today. Its exit: every registration that rides `testcases` - gives, off the stick, the verdict the per-boot arm gives at the same - head, the host's judgement of each member's window agrees with it, and a - member patched to fault reds on its window by the kernel's record of the - fault. - - A session outlives that bound once stage 2 of - `issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md` - ships `watchdogd`. A session image then starts it as the shipped image - does and names no `boot-deadline=`, whose whole-boot bound would end the - session, so `judge_arms` (`src/metal.rs:909-936`) takes `watchdogd`'s row - as its bound, and `hardlockup` runs only if every boot arms it - (`issues/kernel/whether-every-boot-arms-the-hard-lockup-detector-is-the-owners.md`). - - QEMU's shared block runs the same `run_one`; its runner's stdout is a log - ring, so the console carries the kernel's records and it writes none. - - One session image then folds the base boots: the C corpus, the shared - block (without `shared-debug`), `testcases` with its `mkdir` and - `readdir` boots, and the talking and swapping boots; `mkdir_cap` and - `readdir_bound` clean up after themselves or red. - - A member that claims a device waits on - `issues/kernel/deferred-release-outlives-its-syscall.md` (fixed under - `issues/kernel/every-wait-in-this-kernel-is-a-spin.md`), because a claim - not back by the check reds it. - - Closes `issues/hardware/a-swap-on-the-t14-lives-inside-a-metal-boots-bound.md`. +1. **Next to build: one session in place of seven or eight boots.** `shared` + and `ccorpus` with their chunks, `testcases`, `testcases-mkdir` and + `testcases-readdir` share a config, a parameter line and the shipping + kernel; from the runner's `spawn:` to `reboot`'s their lists took 42.9 s and + 58.4 s on two T14 runs, inside the 120 s `boot-deadline=` after a lease as + late as 19.1 s. The session image is `tests/testcases` with + `tests/lantalkcase`'s netd, sshd and streaming `logd`, `test-runner` started + by the exec; its label takes its `tests/metal-profile.toml` rows + (`tests/common/metal.rs:671-679`), and `loader_watchdog_arms`'s control arm + rides it. The loop writes `run ` per member, then `run reboot`, and + reads the stick. Every judge reads the stick as today but `syscall_cost`, + whose lines are the job's own, and `log_poll_outlives_a_close`, whose `echo` + record needs no anchor on the job before it: those read the window. + `mkdir_cap` and `readdir_bound` remove what they made. - **Exit**: a T14 run folds those boots into one session and judges every - member per test with the per-boot rig's verdicts at the same head, and a - second session in reverse order gives the same verdicts. Each of these reds - only the test that staged it: a child left alive, a claim left held, a file - left in `/tmp`, the home, `/state` or `/log`, and a `NEVER_CLEAN` line. A - job that spins past its bound is killed over the session and red by name, - and the next member runs. In QEMU, an e1000e guest runs the same session - end to end, and one whose link is cut under a session stops the run and - says it waits for a hand. - -2. **Ubuntu leaves the loop**, which is the loader track's stages 6 and 7. - - First, the measurement its shape rests on: `ssh t14 update < image` timed - for a session image on today's rig, since an image switch costs that - write and a reset. ToyOS has written this stick at between about - 108 KiB/s and 4.7 MiB/s, 0.2 s to 9.5 s per MiB: 11 s to 8.4 min for the - 53 MiB ROOT a `shared` image carries. - - Then a loader reaches the T14 as an image does (owner ruling): ToyOS - receives it over ssh as `update` receives an image and writes it to the - stick, and the running loader tries it once and keeps the old one as the - fallback. Ubuntu leaves the loop only once this works. - - Every T14 image is signed with the bench key the T14 trusts (owner - ruling), made once outside every checkout at - `~/.config/toyos/t14-bench-signing-key` by `--signing-key-new` with - `TOYOS_SIGNING_KEY` naming that path. - - Waits on the loader track's stage 5 (tries, the good flag, `update --once` - and `--boot-first`), and on: - - `issues/isolation/a-reset-stops-xhci-and-leaves-every-claimed-pci-function-armed.md` - and `issues/hardware/the-t14-hung-after-rebooting-with-its-i219-faulted.md`, - because every reset is now taken with netd holding the host's only - channel; - - `issues/panic-path/a-fatal-event-stands-down-both-bounds-and-may-leave-nothing-to-end-the-machine.md`; - - `issues/diagnostics/a-swaps-redial-asks-again-with-no-event-to-wait-on.md`, - `issues/hardware/the-t14-redial-re-asks-mdns-after-every-refusal.md`, - `issues/diagnostics/a-first-dial-turned-away-before-a-line-is-waited-on-to-its-callers-bound.md` - and - `issues/diagnostics/a-netd-that-dies-while-serving-leaves-the-hosts-stream-silent.md`, - so that every host redial waits on an event; - - `issues/hardware/most-t14-leases-land-one-dhcp-retry-late.md`, because - every reset now reaches the host through a lease. - - The stick is written once and put first with `update --boot-first`; from - then on every image goes by `update`, and a stick that boots neither - loader costs a hand and `diag/flash.sh` - (`issues/build/the-owners-flash-script-runs-diskutil.md`). - - Another image costs one reset: `update < image` and a reboot. A death - costs two: a panic, wedge, reset or loader test goes by `update --once`, - the kept slot boots after it, and the host fetches the record over - `sftp`, from `loader.log` until the loader track's stage 9 (whose report - pass costs a third POST) and from `/log` after it. - - An image is good only once the host has reached it: the host's first exec - on it is `update --good`, and a session image names no `[boot] up`, so - init never marks it good. - - Every test row names its place, a session's image, a `--once` image or - QEMU, so `METAL_ONLY`, `QemuOnly` and the name-keyed dispatch go. - - The loader keeps the pass before it as `loader-previous.log`, by a rename - (`SetInfo`), so the kept slot's pass does not erase a death's report. - - Deleted: `src/metal.rs` but its arm gate, `src/icmp.rs`, - `bootloader/src/bootnext.rs`, the stick readback and batching of - `tests/common/metal.rs`, and test-runner's job-list mode. - - Closes - `issues/build/a-hung-boots-log-partition-is-wiped-by-the-next-runs-flash.md`, - `issues/build/the-sudoers-rendering-has-no-host-side-judge.md`, - `issues/build/the-metal-loop-writes-the-readback-volume-into-a-directory-it-has-not-made.md` - and `issues/hardware/the-t14-stopped-answering-ssh-between-two-lan-boots.md`. - - Ubuntu stays on the NVMe, never booted, until - `issues/hardware/linuxs-readings-of-the-t14-and-the-tcg-model-are-not-committed.md` - lets it go. + **Exit**: every registration those boots carried gives the per-boot verdict + at the same head, each window's judgement agrees, and a second session in + reverse order gives the same verdicts. Staged in one member, each of these + reds it alone and the next runs: a fault, by the kernel's record in its + window; a child left alive; a file left behind; a `NEVER_CLEAN` line; a job + spinning past its bound. - **Exit**: a whole T14 run with Ubuntu never started gives the verdicts a - per-boot run gave at the commit stage 2 branches from. No path in - `src/metal*.rs` reaches Ubuntu. A panic's reset reaches the loader with no - `BootNext` set. A `--once` image that panics returns the machine to its - session, and its record is judged. A slot with a flipped byte, no signature - or a lower security version is refused and the other slot boots. A slot - that dies falls back on its own, and one whose `sshd` does not authorize - the host's key is never marked good. A loader sent over ssh boots once, and - one that brings no slot to good leaves the old one booting. +2. **Ubuntu leaves the loop** in stages 6 and 7 of + `issues/boot-media/the-loader-does-only-what-must-precede-the-handover.md`, + which bring each session image by `update` and each death's record back + over `sftp`. -3. **Sessions merge.** A run holds one session per kernel build, parameter - line and config that some test needs: actuators that leave the machine as - it is share an image where their authors say so, as `SELFTESTS` does, a - config that differs by one program's row joins the base session, and the - QEMU registrations the T14 can run move into sessions. +3. **Sessions outlive the boot deadline** once stage 2 of + `issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md` + ships `watchdogd`. An image names no `boot-deadline=`, `judge_arms` + (`src/metal.rs:894-921`) takes `watchdogd`'s row as its bound, and `pipe` + bounds each window, not its whole run (`tests/ssh-client-host/src/main.rs:64`). + Whether such an image has a hard-lockup detector is + `issues/kernel/whether-every-boot-arms-the-hard-lockup-detector-is-the-owners.md`. + A ToyOS that runs but that the host cannot reach keeps its watchdog fed, so + the run stops and says it waits for a hand once nothing has answered within + the watchdog's bound and a POST allowance. A run holds one session per + kernel build, parameter line and config some test needs. - **Exit**: the host's plan names every reset in a run by what forces it: a - kernel build, a parameter no session carries, or a death. + **Exit**: the host's plan names every reset in a run by what forces it, and + in QEMU an e1000e guest whose link is cut under a session stops the run and + says it waits for a hand. diff --git a/issues/hardware/watchdog-fed-ends-its-boot-before-the-bound-it-judges.md b/issues/hardware/watchdog-fed-ends-its-boot-before-the-bound-it-judges.md new file mode 100644 index 0000000000..7f44fe76fd --- /dev/null +++ b/issues/hardware/watchdog-fed-ends-its-boot-before-the-bound-it-judges.md @@ -0,0 +1,21 @@ +--- +status: open +kind: tooling +opened: 2026-09-30 +--- + +# `watchdog_fed` ends its boot before the bound it judges + +`watchdog_fed` holds that an armed T14 boot runs its list to its own stop, +which a chipset reset anywhere in it would have cut short +(`tests/toyos.rs:1740-1749`). Its boot, `testcases-watchdog`, carries no job of +its own and neither does `loader_watchdog_arms`' arm on it +(`tests/toyos.rs:1667-1679`), so its list is `reboot` alone: on two T14 runs +the runner spawned `reboot` 1.196 s and 1.489 s into the kernel's clock, and the +shutdown disarms the timer (`kernel/src/syscall/machine.rs:88`), inside one +`toyos_tco::BOUND_MS` of 9,600 ms. A kernel that never fed the timer passes it. + +**Exit**: `watchdog_fed`'s boot holds past `toyos_tco::BOUND_MS` before its +`reboot`, and the same boot armed with `tco-starve` reds it once stage 1 of +`issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md` has +seen the TCO reset the T14. diff --git a/issues/isolation/every-job-test-runner-starts-holds-its-whole-system-capability.md b/issues/isolation/every-job-test-runner-starts-holds-its-whole-system-capability.md index 21612ed279..2b0d3f8164 100644 --- a/issues/isolation/every-job-test-runner-starts-holds-its-whole-system-capability.md +++ b/issues/isolation/every-job-test-runner-starts-holds-its-whole-system-capability.md @@ -8,7 +8,7 @@ opened: 2026-09-29 `run_one` endows each job it spawns a `SysCap::duplicate` of test-runner's own capability (`userland/test-runner/src/main.rs:292-295`), and a duplicate -carries every right the original does (`toyos/src/syscap.rs:63-70`). The job +carries every right the original does (`toyos/src/syscap.rs:64-71`). The job also inherits test-runner's whole namespace (`userland/test-runner/src/main.rs:86-92`). On `tests/testcases` the row grants `device`, `dup`, `logread`, `power` and `roster` @@ -23,5 +23,5 @@ needs: `test_rs_audio_idle_suspend` reads `roster` off the duplicate `endowment_denied` narrows `power` away from it. **Exit**: a job holds only the rights and names its test is declared to need, -narrowed by test-runner (`SysCap::narrowed`, `toyos/src/syscap.rs:121-128`), and +narrowed by test-runner (`SysCap::narrowed`, `toyos/src/syscap.rs:134-141`), and a job that asks for anything else is refused. From 6489e255b06e7d03a6f8a71ab2e661eec49c6ddc Mon Sep 17 00:00:00 2001 From: japabu Date: Thu, 1 Oct 2026 04:35:16 +0200 Subject: [PATCH 06/11] Answer the fourth review of #631: two sessions in place of six boots, the detector on every boot, and the watchdog class refused by right Designed against #638 (23 boots a full T14 run, lists sized at `JOB_BOUND_MS` less a tenth over 860/800/260 ms allowances) as if landed. The session track: - The fold is six boots of #638's run: shared, shared-2, ccorpus, testcases, testcases-mkdir, testcases-readdir. Each `Boot parameter:` line in toyos-tight/target/metal (b2ccd1c84) carries only root=, boot-deadline=120000, boot-slot=A and blackbox=. 220 members there: 62+18+130+8+1+1 `===TEST_END` lines. - Prices: #638's 860 ms a shipping member and 260 ms a corpus case; the other three lists at twice their slowest span over ten full T14 runs (536-metal-full, 536r21, 590r6-m1, 590r6-m2, 616, 616r3, 630-full1, 630-full2, 630r4, 638-metal-full): testcases 11.861 s (590r6-m1), mkdir 2.106 s (536r21), readdir_bound 5.043 s (638, the only run without its /home arm). - Per-job bound 14.1 s: twice mutual_kill's 7.041 s (630-full2), the slowest member of those runs once readdir_bound's /home arm (24.8 s on the two 536 runs) is set aside. - Budget: 120000 - 12000 - 19100 - 14100 = 74800 ms. 80 x 860 = 68800; 130 x 260 + 23722 + 4212 + 10086 = 71820. Two sessions for six boots: 23 -> 19. - The fence is every root `/` has. Of the 89 folded binaries, 14 name /home and hierarchy_paths writes /apps; the file-left-behind control is staged in /home. - A window's copy of the kernel's records is the exec-started runner's only: on QEMU's serial runner a copy would double every record on the console, where 4 must_be_clean_apart_from sites and 55 `.matches(..).count()` sites in 10 files count lines. - device_claim_lifetime, the one folded member that mints a claim, runs last until the deferred-release defect closes. - The QEMU e1000e session arm is restored, and the last stage waits on the detector on every boot. It answers the swap-bound issue. The watchdog track: - The owner ruled the hard-lockup detector armed on every boot, so its question file is deleted and the arm is the track's next step. All 23 boots of #638's run say `hard lockup:` (60000 ms, 5000 ms on the three staged boots); only hardlockup's loader.log carries the lockup record. - Refusal by right: a new Rights::WATCHDOG that no syscap name grants, demanded by sys_device_claim for the class. Declared as an ABI change. - The kernel feeds while no claim is held, so watchdogd's exit or swap hands the timer back; watchdogd waits on the deferred-release defect. - QEMU: q35's TCO counts QEMU_CLOCK_VIRTUAL and its second expiry calls watchdog_perform_action (QEMU v11.1.1 hw/acpi/ich9_tco.c:61-70,244); tests/qtest/tco-test.c:325-347 tests the `none` action. Every guest that stages no reset runs with -action watchdog=none. - The T14 reading moves to the loader's report pass right after the reset: Linux v6.12 drivers/watchdog/iTCO_wdt.c:545-560 clears SECOND_TO_STS in iTCO_wdt_probe. The starved boot's deadline must outlast tco-starve's 5 s plus the 9.6 s bound, which #638's 10 s STAGED_BOUND_MS does not. - The go-ahead is an owner question file of its own. The loader track: stage 6's exit asks a whole run, every --once boot included, and a loader sent over ssh to boot once; a stick booting neither loader costs a hand and diag/flash.sh; stage 7 closes the four Ubuntu-loop issues; the 53 MiB ROOT clause goes. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L --- ...oes-only-what-must-precede-the-handover.md | 26 ++-- ...os-waits-for-a-hand-on-the-power-button.md | 134 +++++++++--------- ...4-reboots-through-ubuntu-for-every-test.md | 113 ++++++++------- ...-starve-the-t14s-watchdog-is-the-owners.md | 30 ++++ ...-the-hard-lockup-detector-is-the-owners.md | 35 ----- 5 files changed, 177 insertions(+), 161 deletions(-) create mode 100644 issues/hardware/whether-a-boot-may-starve-the-t14s-watchdog-is-the-owners.md delete mode 100644 issues/kernel/whether-every-boot-arms-the-hard-lockup-detector-is-the-owners.md diff --git a/issues/boot-media/the-loader-does-only-what-must-precede-the-handover.md b/issues/boot-media/the-loader-does-only-what-must-precede-the-handover.md index 858fb15ab4..6b248ab7fd 100644 --- a/issues/boot-media/the-loader-does-only-what-must-precede-the-handover.md +++ b/issues/boot-media/the-loader-does-only-what-must-precede-the-handover.md @@ -311,13 +311,14 @@ Each stage lands on its own, in this order. - It opens with a timed `ssh t14 update < image` of a session image of `issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md`, since an image switch costs that write and a reset: ToyOS has written this - stick at 0.2 s to 9.5 s per MiB, 11 s to 8.4 min for a 53 MiB ROOT. + stick at 0.2 s to 9.5 s per MiB. - It is built from #539's pieces that stage 6 takes above, and the bench path of `src/metal.rs`. `--via-ubuntu` stays as the old path. - Every T14 image is signed with the bench key the T14 trusts, and a loader reaches the T14 as an image does: ToyOS receives it over ssh and writes it to the stick, and the running loader tries it once and keeps the old one - as the fallback (owner rulings). + as the fallback (owner rulings). A stick that boots neither loader costs + a hand and `diag/flash.sh`. - The host's first exec on an image is `update --good`, and a session image names no `[boot] up`. A death goes by `update --once`, the kept slot boots after it, and the host fetches the record over `sftp`. @@ -344,19 +345,26 @@ Each stage lands on its own, in this order. as far as it is true of what lands. **Exit**: `bench_loop_drives_a_toyos_machine` passes in QEMU. On the T14, - with Ubuntu never started: a run of sessions gives the verdicts a per-boot - run gave at the commit this stage branches from; a kernel change boots; a - slot with a flipped byte, no signature or a lower security version is - refused and the other boots; a slot that dies falls back on its own, and - one whose `sshd` refuses the host's key is never marked good; a `--once` - image that panics returns the machine to its session with its record - judged; and a loader sent over ssh that brings no slot to good leaves the + with Ubuntu never started: a whole run, its sessions and every boot that + goes by `--once` (`deadlinewedge`, `hardlockup`, `usbload` and + `foreignrecord` among them), gives the verdicts a per-boot run gave at the + commit this stage branches from; a kernel change boots; a slot with a + flipped byte, no signature or a lower security version is refused and the + other boots; a slot that dies falls back on its own, and one whose `sshd` + refuses the host's key is never marked good; a `--once` image that panics + returns the machine to its session with its record judged; and a loader + sent over ssh boots once, and one that brings no slot to good leaves the old one booting. 7. **Ubuntu leaves the loop.** Delete `toyos-metal`'s `--via-ubuntu` path, `--metal-via-ubuntu`, `bootloader/src/bootnext.rs` and test-runner's job-list mode. After a reset the firmware comes back to the loader because ToyOS's entry is first (stage 5). + - `issues/build/a-hung-boots-log-partition-is-wiped-by-the-next-runs-flash.md`, + `issues/build/the-sudoers-rendering-has-no-host-side-judge.md`, + `issues/build/the-metal-loop-writes-the-readback-volume-into-a-directory-it-has-not-made.md` + and `issues/hardware/the-t14-stopped-answering-ssh-between-two-lan-boots.md` + close here, each only as far as it is true of what lands. **Exit**: no path in `src/metal*.rs` reaches Ubuntu. On the T14, a panic's reset reaches the loader with no `BootNext` set. diff --git a/issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md b/issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md index a0b6ad9f28..60fd8531a9 100644 --- a/issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md +++ b/issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md @@ -6,73 +6,79 @@ opened: 2026-09-30 # A frozen ToyOS waits for a hand on the power button -No shipped image arms a hardware watchdog. The kernel arms the chipset's TCO -only on the `watchdog` boot parameter -(`kernel/src/arch/x86_64/watchdog.rs:48-51`), which no shipped image carries, -and feeds it from the scheduler pass (`kernel/src/sched/driver.rs:512`), which -proves only that some CPU still reaches one; and no armed TCO has yet reset the -T14 (`issues/hardware/an-armed-tco-has-never-reset-the-t14.md`). The owner's -ruling: the watchdog is a production feature, shipped to every machine. -`watchdogd`, a userland service, claims the machine's hardware watchdog and -feeds it while the system is healthy; if ToyOS freezes or its userland stops -being scheduled, the hardware resets the machine. +No shipped image arms a hardware watchdog or the hard-lockup detector: the +kernel arms the chipset's TCO only on the `watchdog` parameter +(`kernel/src/arch/x86_64/watchdog.rs:48-51`), feeds it from the scheduler pass +(`kernel/src/sched/driver.rs:512`), and arms the detector only through +`boot-deadline=` (`kernel/src/deadline.rs:168-183`). The owner's rulings: a +userland `watchdogd` feeds the watchdog on every machine, with no new syscall; +the detector is armed on every boot; nothing in the kernel exists for tests +alone. -- **No syscall.** The watchdog becomes a device class of its own, and - `watchdogd` reaches its registers through that class's allow-list in - `sys_device_reg` (`kernel/src/syscall/device.rs:55-77`), as soundd reaches - HDA's. -- **Only `watchdogd` holds it.** Any holder of `device` may mint a claim - (`kernel/src/syscall/device.rs:123-141`), test-runner and every job it starts - among them (`tests/testcases/system.toml:41`), and the first mint ends the - kernel's feed: so the kernel refuses the class to every other holder, which - a build refusing it to other rows does not do. -- **Fed from the real-time band.** `watchdogd` feeds from a thread of its own - under `rt`; `issues/kernel/cpu-time-is-a-band-and-not-a-reservation.md` - replaces that precedence with a reservation, which `watchdogd` then holds. -- **Nothing in the kernel is added for tests only** (owner ruling): a test - harness uses this watchdog as it ships. -- The T14 has no BMC or AMT (its i5-1135G7 has no vPro) and a battery no power - switch cuts, so its PCH's TCO is its one reset that needs nothing of the - kernel. -- **A panic's panel holds as it does today**, for `toyos_tco::PANIC_BOUND_MS` - or for good once a key is pressed: its hold feeds the watchdog for as long - as it holds, the hold after a key included, which halts its CPU today - (`kernel/src/drivers/panic_console/mod.rs:710-725`). A panic that never - reaches the panel stops every feed, and the watchdog resets the machine. +- **Refused by right.** `watchdogd` reaches the TCO through a device class of + its own, as soundd reaches HDA (`kernel/src/syscall/device.rs:55-77`). Any + `device` holder may mint a claim (`:123-158`), every test job among them + (`tests/testcases/system.toml:41`), so `sys_device_claim` demands a new + `Rights::WATCHDOG` for the class, which no `syscap` name grants: only init + holds it, and mints the claim for the row whose `devices` names the class. + The bit and the class are an ABI change. +- **The kernel feeds while no claim is held**, so a `watchdogd` that ends + hands the timer back until `restart` brings it up; a release that outlives + its syscall (`issues/kernel/deferred-release-outlives-its-syscall.md`) would + leave nobody feeding, so `watchdogd` waits on it. +- `watchdogd` feeds from a thread under `rt`, until + `issues/kernel/cpu-time-is-a-band-and-not-a-reservation.md` gives it a + reservation. +- **A panic's panel holds as it does today** and feeds the watchdog while it + holds, after a key too (`kernel/src/drivers/panic_console/mod.rs:710-725`). +- **QEMU judges no reset it did not stage.** q35's TCO counts + `QEMU_CLOCK_VIRTUAL`, which a loaded host advances while it starves a guest, + and its second expiry does what `-action watchdog=` says (QEMU 11.1.1, + `hw/acpi/ich9_tco.c:61-70,244`). Every guest but one that stages a reset runs + with `-action watchdog=none`; that a feeding kernel is not reset is + `watchdog_fed`'s verdict, on the T14 (`tests/toyos.rs:2034-2038`). +- The T14's PCH TCO is its one reset that needs nothing of the kernel (no BMC, + no AMT, a battery no switch cuts), and no armed TCO has reset it yet + (`issues/hardware/an-armed-tco-has-never-reset-the-t14.md`). -## Stages +**Next: the detector on every boot**, at the half of `boot-deadline=` it takes +today or at `toyos_tco::HARD_LOCKUP_BOUND_MS` where a boot names none, since a +CPU frozen with interrupts off is a hand on the power button while `watchdogd` +on the other CPUs feeds. All 23 boots of a full T14 run were armed, and only +the staged `hardlockup` boot fired. -1. **The measurement, run on the T14 only on the owner's go-ahead.** Two boots. - The first is armed with `watchdog` and `tco-starve`, which `src/metal.rs`' - `FLASHABLE` and `WEDGE_ARMS` must first name, and a job holds it past the - bound, since the shutdown disarms the timer - (`kernel/src/syscall/machine.rs:88`); its `boot-deadline=` ends it if the - TCO does not. A TCO reset seals no record, so the reading is the second, the - next armed boot: its loader's `TCO2_STS` and its kernel's - `watchdog: the last boot ended in a TCO reset` - (`kernel/src/arch/x86_64/watchdog.rs:92-99`), which the report pass in - between does not print. That line is also the reading - `issues/hardware/tco2-sts-clearing-is-verified-on-qemu-only.md` waits on. - The fed control is `watchdog_fed`, once - `issues/hardware/watchdog-fed-ends-its-boot-before-the-bound-it-judges.md` - holds its boot past the bound; the positive control is q35's starved guest, - `watchdog_resets` (`tests/common/power.rs:515-536,694-701`). +**Exit**: a boot that names no `boot-deadline=` says `hard lockup:` with +`HARD_LOCKUP_BOUND_MS`, and with the arm back behind `deadline::start` it says +it has no bound. - **Exit**: the armed boot after the starved one reads `TCO_SECOND_TO_STS` - set. If it reads it clear, the track stops and goes back to the owner with - both boots' readbacks. +**Then, on the owner's yes** to +`issues/hardware/whether-a-boot-may-starve-the-t14s-watchdog-is-the-owners.md`: +**the reading.** One T14 boot armed with `watchdog` and `tco-starve`, which +`src/metal.rs`' `FLASHABLE` and `WEDGE_ARMS` must first name, under a deadline +past `tco-starve`'s 5 s and `toyos_tco::BOUND_MS`, which +`toyos_tco::STAGED_BOUND_MS` is not. A TCO reset seals no record, and Ubuntu's +`iTCO_wdt_probe` clears `SECOND_TO_STS` (Linux 6.12, +`drivers/watchdog/iTCO_wdt.c:545-560`), so the reading is the loader's report +pass right after the reset, which reads `TCO2_STS` with the page until loader +stage 8 takes the TCO out of the loader. The fed control is `watchdog_fed`, +once `issues/hardware/watchdog-fed-ends-its-boot-before-the-bound-it-judges.md` +closes; the positive control is q35's `watchdog_resets` +(`tests/common/power.rs:515-536`), read through the pass after its reset. -2. **`watchdogd` ships**, after the loader track's stage 8 - (`issues/boot-media/the-loader-does-only-what-must-precede-the-handover.md`) - has made the kernel's arm the only one. `toyos_tco::PARAM` goes: the kernel - arms on every boot whose chipset `toyos_tco` has a row, q35 included, and - feeds from the scheduler pass until the claim is minted and never after. - With the parameter go `watchdog_quiet`, `loader_watchdog_arms`' control arm - and the `testcases-watchdog` boot, since every boot is then armed. A machine - with no row refuses the claim as absent, and the boot says it is unwatched. +**Exit**: that pass finds no record on the page, which the deadline behind +the starve would have sealed, and reads `TCO_SECOND_TO_STS` set; the pass after +`deadlinewedge` finds its record and reads the latch clear. Otherwise the T14 +exit below waits on the TCO defect. - **Exit**: in QEMU, a kernel wedged by `wedge-before-reset` with `watchdogd` - feeding ends in the chipset's reset; a job holding `device` is refused the - class; and a whole QEMU run resets no guest it did not stage. On the T14, - the boot after a wedged one reads the latch set, and the boot after a panic - whose panel held reads it clear. +**Last: `watchdogd` ships**, after loader stage 8 +(`issues/boot-media/the-loader-does-only-what-must-precede-the-handover.md`) +has made the kernel's arm the only one. `toyos_tco::PARAM` goes, and with it +`watchdog_quiet`, `loader_watchdog_arms`' control arm and the +`testcases-watchdog` boot: the kernel arms every boot whose chipset has a +`toyos_tco` row, q35 included, and one with none says it is unwatched. + +**Exit**: in QEMU, a kernel wedged by `wedge-before-reset` with `watchdogd` +feeding ends in the chipset's reset, and a job holding `device` on an image +with no `watchdogd` is refused the class `PermissionDenied`, which it mints with +the right check taken out of `sys_device_claim`. On the T14 the boot after a +wedge the TCO ended says so, and the boot after a held panic does not. diff --git a/issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md b/issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md index c8bd305aa9..e9a55b7b10 100644 --- a/issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md +++ b/issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md @@ -16,61 +16,68 @@ ssh, with no reboot between them and no Ubuntu. `tests/ssh-client-host`'s `pipe` relays it once it writes output as it arrives (`tests/ssh-client-host/src/main.rs:251-263`). - **A test's window** is what crosses between its markers: the job's output, - never `/log` (`userland/sshd/src/main.rs:234-239`, - `userland/init/src/main.rs:2050-2063`, `userland/test-runner/src/main.rs:276-277`), - and the kernel's records, which `run_one` reads on its own - `toyos::log::LogTail` and writes before `===TEST_END===`; a record the ring - dropped reds the test. -- **A leftover reds its test.** After each test `run_one` writes the processes - alive and the listings of `/tmp`, `/state` and `/log`, which the host holds to - the session's first, `/log` less `bootlog::split_listing`'s files. Claims join - once `issues/kernel/deferred-release-outlives-its-syscall.md` closes, since a - released claim can read held until then. A test whose premise is a fresh - boot, as `audio_idle_suspend`'s is, says so and runs first. -- **A stuck test** is killed by the runner's deadline thread made per job - (`userland/test-runner/src/main.rs:153-194`), and reds by name. + which never reaches `/log` (`userland/sshd/src/main.rs:234-239`), and, from a + runner the exec starts, the kernel's records, read on its own + `toyos::log::LogTail` and written before `===TEST_END===`; a record the ring + dropped reds the test. QEMU's serial runner writes no copy, since its output + reaches the console beside the ring and judges count kernel lines there + (`tests/common/blockd.rs:579`, `tests/common/iommu.rs:2161`). +- **A leftover reds its test.** After each test the runner lists the processes + alive and every root `/` has, whole (`kernel/src/vfs.rs:83-86`), `/home` and + `/apps` among them, and the host holds both to the session's first, `/log` + less `bootlog::split_listing`'s files. Claims join once + `issues/kernel/deferred-release-outlives-its-syscall.md` closes; until then a + member that mints one, as `device_claim_lifetime` does, runs last. A member + whose premise is a fresh boot, as `audio_idle_suspend`'s is, runs first. +- **A stuck test** is killed by the runner's deadline thread + (`userland/test-runner/src/main.rs:147-194`), made per job: 14.1 s, twice + `mutual_kill`'s 7.041 s, the slowest any of these members has taken on the + T14. It reds by name, and the next member runs. -## Stages +**Next: two sessions in place of six boots.** `shared`, `shared-2`, `ccorpus`, +`testcases`, `testcases-mkdir` and `testcases-readdir` share a config, a +parameter line and the shipping kernel. The session image is `tests/testcases` +with `tests/lantalkcase`'s netd, sshd and streaming `logd`, under the same +120 s `boot-deadline=`. A member is priced at the allowance behind +`SharedBoot::members`' counts, 860 ms a shipping Rust member and 260 ms a C +corpus case, and the three other lists at twice their slowest on the T14: 23.7 s +for `testcases`' eight, 4.2 s for `mkdir_cap`, 10.1 s for `readdir_bound`. A +session holds 74.8 s: the bound less a tenth, less a lease as late as 19.1 s +(`issues/hardware/most-t14-leases-land-one-dhcp-retry-late.md`), less one +per-job bound. `shared`'s 80 members price at 68.8 s and the rest at 71.8 s, so +a full run goes from 23 boots to 19. The loop writes `run ` per member, +then `run reboot`, and reads the stick, where every judge reads as today but +`syscall_cost`, whose lines are the job's own, and `log_poll_outlives_a_close`, +whose `echo` record needs no anchor on the job before it: those read the +window. +`loader_watchdog_arms`' control arm rides it, and `mkdir_cap` and +`readdir_bound` remove what they made. -1. **Next to build: one session in place of seven or eight boots.** `shared` - and `ccorpus` with their chunks, `testcases`, `testcases-mkdir` and - `testcases-readdir` share a config, a parameter line and the shipping - kernel; from the runner's `spawn:` to `reboot`'s their lists took 42.9 s and - 58.4 s on two T14 runs, inside the 120 s `boot-deadline=` after a lease as - late as 19.1 s. The session image is `tests/testcases` with - `tests/lantalkcase`'s netd, sshd and streaming `logd`, `test-runner` started - by the exec; its label takes its `tests/metal-profile.toml` rows - (`tests/common/metal.rs:671-679`), and `loader_watchdog_arms`'s control arm - rides it. The loop writes `run ` per member, then `run reboot`, and - reads the stick. Every judge reads the stick as today but `syscall_cost`, - whose lines are the job's own, and `log_poll_outlives_a_close`, whose `echo` - record needs no anchor on the job before it: those read the window. - `mkdir_cap` and `readdir_bound` remove what they made. +**Exit**: every registration those boots carried gives the per-boot verdict at +the same head, each window's judgement agrees, and the sessions in reverse +order, fixed members in place, give the same verdicts; in QEMU an e1000e guest +runs a session end to end over `ssh`. Staged in one member, each of these reds +it alone and the next runs: a fault, by the kernel's record in its window; a +child left alive; a file left in `/home`; a `NEVER_CLEAN` line; a job spinning +past its bound. - **Exit**: every registration those boots carried gives the per-boot verdict - at the same head, each window's judgement agrees, and a second session in - reverse order gives the same verdicts. Staged in one member, each of these - reds it alone and the next runs: a fault, by the kernel's record in its - window; a child left alive; a file left behind; a `NEVER_CLEAN` line; a job - spinning past its bound. +**Then: Ubuntu leaves the loop** in stages 6 and 7 of +`issues/boot-media/the-loader-does-only-what-must-precede-the-handover.md`, +which bring each session image by `update` and each death's record back over +`sftp`. -2. **Ubuntu leaves the loop** in stages 6 and 7 of - `issues/boot-media/the-loader-does-only-what-must-precede-the-handover.md`, - which bring each session image by `update` and each death's record back - over `sftp`. +**Last: sessions outlive the boot deadline**, once +`issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md` has +armed the hard-lockup detector on every boot and shipped `watchdogd`. An image +names no `boot-deadline=`, `judge_arms` (`src/metal.rs:899-926`) takes +`watchdogd`'s row as its bound, and `pipe` bounds each window rather than its +whole run (`tests/ssh-client-host/src/main.rs:64`). A ToyOS that runs but that +the host cannot reach keeps its watchdog fed, so the run says it waits for a +hand once nothing has answered within the watchdog's bound and a POST +allowance. A run holds one session per kernel build, parameter line and config +some test needs, and a swap leaves a boot's bound +(`issues/hardware/a-swap-on-the-t14-lives-inside-a-metal-boots-bound.md`). -3. **Sessions outlive the boot deadline** once stage 2 of - `issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md` - ships `watchdogd`. An image names no `boot-deadline=`, `judge_arms` - (`src/metal.rs:894-921`) takes `watchdogd`'s row as its bound, and `pipe` - bounds each window, not its whole run (`tests/ssh-client-host/src/main.rs:64`). - Whether such an image has a hard-lockup detector is - `issues/kernel/whether-every-boot-arms-the-hard-lockup-detector-is-the-owners.md`. - A ToyOS that runs but that the host cannot reach keeps its watchdog fed, so - the run stops and says it waits for a hand once nothing has answered within - the watchdog's bound and a POST allowance. A run holds one session per - kernel build, parameter line and config some test needs. - - **Exit**: the host's plan names every reset in a run by what forces it, and - in QEMU an e1000e guest whose link is cut under a session stops the run and - says it waits for a hand. +**Exit**: the host's plan names every reset in a run by what forces it, and in +QEMU an e1000e guest whose link is cut under a session stops the run and says +it waits for a hand. diff --git a/issues/hardware/whether-a-boot-may-starve-the-t14s-watchdog-is-the-owners.md b/issues/hardware/whether-a-boot-may-starve-the-t14s-watchdog-is-the-owners.md new file mode 100644 index 0000000000..1b5b5307b1 --- /dev/null +++ b/issues/hardware/whether-a-boot-may-starve-the-t14s-watchdog-is-the-owners.md @@ -0,0 +1,30 @@ +--- +status: owner +kind: question +opened: 2026-10-01 +--- + +# Whether a boot may starve the T14's watchdog is the owner's + +The question: may one test boot on the T14 stop feeding the chipset's watchdog +on purpose, so that the chipset resets the laptop? + +The watchdog is what will reset a frozen ToyOS on every machine (owner ruling), +and on the T14 it is the only reset that needs nothing of the kernel. Nobody has +seen it reset this laptop: runs 3 to 5 armed it, and each needed a hand on the +power button (`issues/hardware/an-armed-tco-has-never-reset-the-t14.md`). On +QEMU's model of the chipset, `watchdog_resets` sees it reset the guest. + +- **What a yes risks**: about 15 s into that boot the laptop resets, as it does + at the end of every test boot, but by the chipset's timer rather than by + ToyOS. What this laptop's firmware does after such a reset has never been + seen. If the chipset does not reset it, the boot's own deadline ends it at + 120 s, as it ends the `deadlinewedge` boot in every run. +- **What a no stops**: the reading in + `issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md` and + that track's T14 exit, which needs the same reset. `watchdogd` then ships to + the T14 with its reset unproven there. + +*Recommended: yes.* + +**Exit**: the owner rules. diff --git a/issues/kernel/whether-every-boot-arms-the-hard-lockup-detector-is-the-owners.md b/issues/kernel/whether-every-boot-arms-the-hard-lockup-detector-is-the-owners.md deleted file mode 100644 index 8e8008d447..0000000000 --- a/issues/kernel/whether-every-boot-arms-the-hard-lockup-detector-is-the-owners.md +++ /dev/null @@ -1,35 +0,0 @@ ---- -status: owner -kind: question -opened: 2026-09-29 ---- - -# Whether every boot arms the hard-lockup detector is the owner's - -What it is: a performance counter on each CPU raises an NMI about once a second -of that CPU's busy time, and the NMI checks whether the CPU has taken any other -interrupt since the last one. A CPU that has taken none for -`toyos_tco::HARD_LOCKUP_BOUND_MS` with interrupts masked is frozen: it will -never run a thread again, and nothing else on the machine says so. The detector -(`kernel/src/hardlockup`) then seals a `WEDGED` record naming that CPU, its `pc` -and `sp` and the lock it spins on, and resets the machine. Linux runs the same -detector on every boot by default (`nmi_watchdog`, -`Documentation/admin-guide/lockup-watchdogs.rst`) and panics on it only where -`hardlockup_panic` is set; Windows ends such a machine with bug check 0x101, -`CLOCK_WATCHDOG_TIMEOUT`. - -It is armed only through `boot-deadline=` (`deadline::start`, -`kernel/src/deadline.rs:168-183`), so a boot without that parameter, the -owner's own machine included, has none, and a CPU frozen there is a hand on the -power button. Arming it on every boot, at `HARD_LOCKUP_BOUND_MS` where no -deadline names a bound, costs one NMI per busy CPU per second (`SAMPLE_NS`, -`kernel/src/hardlockup/mod.rs:75-80`; a halted CPU takes none), and turns such -a freeze into a reset with a record. It changes no ABI. - -A session image of -`issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md` names no -`boot-deadline=`, so without this it has no detector. No stage waits on it. - -*Recommended: yes.* - -**Exit**: the owner rules. From 589618636b822513736f9014d388e2e56a5c6dcf Mon Sep 17 00:00:00 2001 From: japabu Date: Thu, 1 Oct 2026 06:30:16 +0200 Subject: [PATCH 07/11] Answer the fifth review of #631: the reading is the kernel's own TCO2_STS read, init mints the watchdog class, and every guest arm goes to metal or a host test BLOCKER 1. No loader pass that reports reads TCO2_STS: after a TCO reset harvest ends the pass before watchdog::arm, and that arm reads the register only on the `watchdog` parameter. The reading now waits for loader stage 7 and is the kernel's own read in arch::watchdog's `arm` on the next armed boot; nothing clears the latch in between once Ubuntu's iTCO_wdt_probe is out of the loop. Its control is the armed boot after that one reading it clear, which closes tco2-sts-clearing-is-verified-on-qemu-only.md, the link round 3 carried. The boot wedges with `wedge-before-reset` instead of starving with `tco-starve`, which #660 deletes; both its arms are already in FLASHABLE. BLOCKER 2. No system.toml row outside tests/ grants `device`, so Rights::WATCHDOG and its PermissionDenied guest control refused only test jobs. Both go: init mints the class for watchdogd's row as it mints blockd's, and the isolation issue this branch files fences the jobs. BLOCKER 3. The kernel feeds from its arm until the class is first claimed and never after, so a watchdogd that ends with no successor resets the machine. The T14 exit stages that death, and a kernel that feeds on keeps it up. BLOCKER 4. The QEMU e1000e session, the QEMU link cut, watchdog_resets read through the loader pass and the QEMU wedge reset are gone. The link cut is a host test over a transport that goes quiet; watchdog_resets is the T14 reading, as the guest cut files it. Of the five staged controls a fault and a spinning job are T14 rows and the other three host tests over a recorded window. Every guest runs with -action watchdog=none, so loader stage 8's exit moves its two q35 TCO resets to the T14 behind the reading. NOTEs. The host names each member's bound on `run` and asks for the listing with `list`; QEMU's harness sends neither. device_claim_lifetime and endowment_denied each end a session. audio_idle_suspend waits for soundd's inspect to read `suspended` instead of resting on a fresh boot. The detector's exit is a host test of the bound plus a build that fails with the arm back behind deadline::start; the unreachable `(0, _)` arm goes, and HARD_LOCKUP_BOUND_MS gains the reader #638's issue says it lacks. The session track now plans for the guest cut's load: a metal row is a session member unless it ends the machine or judges a boot. REMOVEs deleted: the false "slowest" clause, the boot-count arithmetic, the ruling's reasoning and its run count, "at 120 s" and "stage 1 of". Line citations into files #660 rewrites became names. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L --- ...oes-only-what-must-precede-the-handover.md | 12 +- ...os-waits-for-a-hand-on-the-power-button.md | 89 ++++++++------- ...4-reboots-through-ubuntu-for-every-test.md | 103 +++++++++--------- ...nds-its-boot-before-the-bound-it-judges.md | 17 ++- ...-starve-the-t14s-watchdog-is-the-owners.md | 21 ++-- 5 files changed, 120 insertions(+), 122 deletions(-) diff --git a/issues/boot-media/the-loader-does-only-what-must-precede-the-handover.md b/issues/boot-media/the-loader-does-only-what-must-precede-the-handover.md index 6b248ab7fd..30537fb6b0 100644 --- a/issues/boot-media/the-loader-does-only-what-must-precede-the-handover.md +++ b/issues/boot-media/the-loader-does-only-what-must-precede-the-handover.md @@ -392,11 +392,13 @@ Each stage lands on its own, in this order. `watchdog_quiet`'s loader half. `watchdog_armed` judges the kernel's read-back, and `loader_watchdog_arms` becomes the kernel's row. - **Exit**: `bootloader/src` holds no TCO access. On q35, `watchdog_armed` - passes on the kernel's lines (counting, `no_reboot=0`, `timeout=0`), and - `watchdog_resets` passes. A boot that hangs right after the arm, before - `mm::init`, is reset by the TCO; moving the arm back after `pci::enumerate` - makes that test fail. + **Exit**: `bootloader/src` holds no TCO access. On the T14, `watchdog_armed` + passes on the kernel's lines (counting, `no_reboot=0`, `timeout=0`). Once + the reading of + `issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md` has + seen the TCO reset the T14, a boot there that hangs right after the arm, + before `mm::init`, is reset by the TCO; moving the arm back after + `pci::enumerate` makes that row fail. 9. **The crash report belongs to the kernel.** The loader hands the last boot's record to the kernel instead of decoding it. The kernel logs it, and diff --git a/issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md b/issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md index 60fd8531a9..cbe9c993a7 100644 --- a/issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md +++ b/issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md @@ -8,67 +8,65 @@ opened: 2026-09-30 No shipped image arms a hardware watchdog or the hard-lockup detector: the kernel arms the chipset's TCO only on the `watchdog` parameter -(`kernel/src/arch/x86_64/watchdog.rs:48-51`), feeds it from the scheduler pass +(`arch::watchdog::init`), feeds it from the scheduler pass (`kernel/src/sched/driver.rs:512`), and arms the detector only through `boot-deadline=` (`kernel/src/deadline.rs:168-183`). The owner's rulings: a userland `watchdogd` feeds the watchdog on every machine, with no new syscall; -the detector is armed on every boot; nothing in the kernel exists for tests -alone. +the detector is armed on every boot; nothing ships for tests alone. -- **Refused by right.** `watchdogd` reaches the TCO through a device class of - its own, as soundd reaches HDA (`kernel/src/syscall/device.rs:55-77`). Any - `device` holder may mint a claim (`:123-158`), every test job among them - (`tests/testcases/system.toml:41`), so `sys_device_claim` demands a new - `Rights::WATCHDOG` for the class, which no `syscap` name grants: only init - holds it, and mints the claim for the row whose `devices` names the class. - The bit and the class are an ABI change. -- **The kernel feeds while no claim is held**, so a `watchdogd` that ends - hands the timer back until `restart` brings it up; a release that outlives - its syscall (`issues/kernel/deferred-release-outlives-its-syscall.md`) would - leave nobody feeding, so `watchdogd` waits on it. +- `watchdogd` reaches the TCO through a device class of its own, as soundd + reaches HDA (`kernel/src/syscall/device.rs:55-77`), and init mints its claim + for `watchdogd`'s row as it mints blockd's (`system.toml:182-186`). The class + is an ABI change. A job holding `device` can mint it where no `watchdogd` + holds it, which + `issues/isolation/every-job-test-runner-starts-holds-its-whole-system-capability.md` + fences. +- **The kernel feeds from its arm until the class is first claimed, and never + after**, so a `watchdogd` that ends with no successor holding the class + resets the machine. A successor's claim that meets its predecessor's release + still in flight (`issues/kernel/deferred-release-outlives-its-syscall.md`) is + refused, so `watchdogd` waits on that defect. - `watchdogd` feeds from a thread under `rt`, until `issues/kernel/cpu-time-is-a-band-and-not-a-reservation.md` gives it a reservation. - **A panic's panel holds as it does today** and feeds the watchdog while it holds, after a key too (`kernel/src/drivers/panic_console/mod.rs:710-725`). -- **QEMU judges no reset it did not stage.** q35's TCO counts +- **Every guest runs with `-action watchdog=none`.** q35's TCO counts `QEMU_CLOCK_VIRTUAL`, which a loaded host advances while it starves a guest, and its second expiry does what `-action watchdog=` says (QEMU 11.1.1, - `hw/acpi/ich9_tco.c:61-70,244`). Every guest but one that stages a reset runs - with `-action watchdog=none`; that a feeding kernel is not reset is - `watchdog_fed`'s verdict, on the T14 (`tests/toyos.rs:2034-2038`). + `hw/acpi/ich9_tco.c:61-70,244`). A TCO reset, and a feeding kernel that + `watchdog_fed` finds not reset, are judged on the T14. - The T14's PCH TCO is its one reset that needs nothing of the kernel (no BMC, no AMT, a battery no switch cuts), and no armed TCO has reset it yet (`issues/hardware/an-armed-tco-has-never-reset-the-t14.md`). **Next: the detector on every boot**, at the half of `boot-deadline=` it takes -today or at `toyos_tco::HARD_LOCKUP_BOUND_MS` where a boot names none, since a -CPU frozen with interrupts off is a hand on the power button while `watchdogd` -on the other CPUs feeds. All 23 boots of a full T14 run were armed, and only -the staged `hardlockup` boot fired. +today or at `toyos_tco::HARD_LOCKUP_BOUND_MS` where a boot names none. The +constant then has a reader, which closes +`issues/build/hard-lockup-bound-ms-is-read-by-nothing-but-its-own-assertion.md`. -**Exit**: a boot that names no `boot-deadline=` says `hard lockup:` with -`HARD_LOCKUP_BOUND_MS`, and with the arm back behind `deadline::start` it says -it has no bound. +**Exit**: a host test holds the bound of a boot that names no `boot-deadline=` +to `HARD_LOCKUP_BOUND_MS`, and putting the arm back behind `deadline::start` +fails to build. `Armed`'s `(0, _)` arm, which nothing reaches, is gone. **Then, on the owner's yes** to -`issues/hardware/whether-a-boot-may-starve-the-t14s-watchdog-is-the-owners.md`: -**the reading.** One T14 boot armed with `watchdog` and `tco-starve`, which -`src/metal.rs`' `FLASHABLE` and `WEDGE_ARMS` must first name, under a deadline -past `tco-starve`'s 5 s and `toyos_tco::BOUND_MS`, which -`toyos_tco::STAGED_BOUND_MS` is not. A TCO reset seals no record, and Ubuntu's -`iTCO_wdt_probe` clears `SECOND_TO_STS` (Linux 6.12, -`drivers/watchdog/iTCO_wdt.c:545-560`), so the reading is the loader's report -pass right after the reset, which reads `TCO2_STS` with the page until loader -stage 8 takes the TCO out of the loader. The fed control is `watchdog_fed`, -once `issues/hardware/watchdog-fed-ends-its-boot-before-the-bound-it-judges.md` -closes; the positive control is q35's `watchdog_resets` -(`tests/common/power.rs:515-536`), read through the pass after its reset. +`issues/hardware/whether-a-boot-may-starve-the-t14s-watchdog-is-the-owners.md`, +and once loader stage 7 has taken Ubuntu, whose `iTCO_wdt_probe` clears +`SECOND_TO_STS` (Linux 6.12, `drivers/watchdog/iTCO_wdt.c:545-560`), out of the +loop: **the reading**, `watchdog_resets`' metal row in +`issues/build/the-guest-suite-runs-only-what-no-cheaper-tier-reaches.md`. One +T14 boot is armed with `watchdog` and `wedge-before-reset`, under a deadline +past its job list and `toyos_tco::BOUND_MS`, which `toyos_tco::STAGED_BOUND_MS` +is not. The fed control is `watchdog_fed`, once +`issues/hardware/watchdog-fed-ends-its-boot-before-the-bound-it-judges.md` +closes. -**Exit**: that pass finds no record on the page, which the deadline behind -the starve would have sealed, and reads `TCO_SECOND_TO_STS` set; the pass after -`deadlinewedge` finds its record and reads the latch clear. Otherwise the T14 -exit below waits on the TCO defect. +**Exit**: the next armed kernel's `TCO2_STS` read, in `arch::watchdog`'s +`arm`, says the last boot ended in a TCO reset, and the page still reads +`ARMED` where the deadline would have sealed a record; the armed boot after +that says no TCO reset. That closes +`issues/hardware/tco2-sts-clearing-is-verified-on-qemu-only.md`. Otherwise the +T14 exit below waits on the TCO defect. **Last: `watchdogd` ships**, after loader stage 8 (`issues/boot-media/the-loader-does-only-what-must-precede-the-handover.md`) @@ -77,8 +75,7 @@ has made the kernel's arm the only one. `toyos_tco::PARAM` goes, and with it `testcases-watchdog` boot: the kernel arms every boot whose chipset has a `toyos_tco` row, q35 included, and one with none says it is unwatched. -**Exit**: in QEMU, a kernel wedged by `wedge-before-reset` with `watchdogd` -feeding ends in the chipset's reset, and a job holding `device` on an image -with no `watchdogd` is refused the class `PermissionDenied`, which it mints with -the right check taken out of `sys_device_claim`. On the T14 the boot after a -wedge the TCO ended says so, and the boot after a held panic does not. +**Exit**, on the T14: a boot whose `watchdogd` ends with no successor, and one +wedged by `wedge-before-reset` with `watchdogd` feeding, each end in the TCO's +reset, and the boot after each says so; a kernel that feeds on once the claim +has gone keeps the first up. The boot after a held panic says no TCO reset. diff --git a/issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md b/issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md index e9a55b7b10..7da69082dc 100644 --- a/issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md +++ b/issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md @@ -6,78 +6,77 @@ opened: 2026-09-29 # The T14 reboots through Ubuntu for every test -A T14 boot is a flash through Ubuntu, `BootNext`, and a stick read back after -the loader's report pass resets into Ubuntu. The owner's direction: the tests -run one after another in a ToyOS that stays booted, driven by the host over -ssh, with no reboot between them and no Ubuntu. +The owner's direction: the tests run one after another in a ToyOS that stays +booted, driven by the host over ssh, with no reboot between them and no Ubuntu. +The T14 is the integration tier: +`issues/build/the-guest-suite-runs-only-what-no-cheaper-tier-reaches.md` leaves +the shared boot to `shared_metal` and `c_corpus_metal` and adds metal rows. +- **A metal row is a member** of the session its kernel build, parameter line + and config name, and costs no boot, unless it ends the machine (a panic, a + wedge, a reset) or judges a boot (the loader, an update, a file read back + after a reboot). Its registration says which. - **One exec channel**, to `test-runner`'s stdin loop, since `sshd` aliases a - second channel's input (`issues/isolation/sshd-holds-one-channel-and-does-not-say-so.md`); + second channel's input (`issues/isolation/sshd-holds-one-channel-and-does-not-say-so.md`). `tests/ssh-client-host`'s `pipe` relays it once it writes output as it arrives (`tests/ssh-client-host/src/main.rs:251-263`). - **A test's window** is what crosses between its markers: the job's output, - which never reaches `/log` (`userland/sshd/src/main.rs:234-239`), and, from a - runner the exec starts, the kernel's records, read on its own - `toyos::log::LogTail` and written before `===TEST_END===`; a record the ring - dropped reds the test. QEMU's serial runner writes no copy, since its output - reaches the console beside the ring and judges count kernel lines there - (`tests/common/blockd.rs:579`, `tests/common/iommu.rs:2161`). -- **A leftover reds its test.** After each test the runner lists the processes - alive and every root `/` has, whole (`kernel/src/vfs.rs:83-86`), `/home` and - `/apps` among them, and the host holds both to the session's first, `/log` - less `bootlog::split_listing`'s files. Claims join once + which never reaches `/log` (`userland/sshd/src/main.rs:234-239`), and the + kernel's records, which a runner the exec starts reads on its own + `toyos::log::LogTail` and writes before `===TEST_END===`. A record the ring + dropped reds the test. QEMU's serial runner writes no copy: its console + already carries the ring. +- **A leftover reds its test.** After each member the host sends `list`, and + the runner prints the processes alive and every root `/` has, whole + (`kernel/src/vfs.rs:83-86`). The host holds both to the session's first, + `/log` less `bootlog::split_listing`'s files. Claims join once `issues/kernel/deferred-release-outlives-its-syscall.md` closes; until then a - member that mints one, as `device_claim_lifetime` does, runs last. A member - whose premise is a fresh boot, as `audio_idle_suspend`'s is, runs first. -- **A stuck test** is killed by the runner's deadline thread - (`userland/test-runner/src/main.rs:147-194`), made per job: 14.1 s, twice - `mutual_kill`'s 7.041 s, the slowest any of these members has taken on the - T14. It reds by name, and the next member runs. + member that mints one runs last, so `device_claim_lifetime` ends one session + and `endowment_denied` the other. +- **A stuck test** is killed at the bound the host's `run` names, as + `--bound-ms=` names the job list's (`userland/test-runner/src/main.rs:101-107`): + 14.1 s, twice `mutual_kill`'s 7.041 s. It reds by name and the next member + runs. QEMU's harness names no bound and sends no `list`; its own ceiling ends + the guest. **Next: two sessions in place of six boots.** `shared`, `shared-2`, `ccorpus`, `testcases`, `testcases-mkdir` and `testcases-readdir` share a config, a parameter line and the shipping kernel. The session image is `tests/testcases` with `tests/lantalkcase`'s netd, sshd and streaming `logd`, under the same -120 s `boot-deadline=`. A member is priced at the allowance behind -`SharedBoot::members`' counts, 860 ms a shipping Rust member and 260 ms a C -corpus case, and the three other lists at twice their slowest on the T14: 23.7 s -for `testcases`' eight, 4.2 s for `mkdir_cap`, 10.1 s for `readdir_bound`. A -session holds 74.8 s: the bound less a tenth, less a lease as late as 19.1 s +120 s `boot-deadline=`. A session holds 74.8 s of members, priced as +`SharedBoot::members` prices them and a list at twice its slowest on the T14: +the bound less a tenth, less a lease as late as 19.1 s (`issues/hardware/most-t14-leases-land-one-dhcp-retry-late.md`), less one -per-job bound. `shared`'s 80 members price at 68.8 s and the rest at 71.8 s, so -a full run goes from 23 boots to 19. The loop writes `run ` per member, -then `run reboot`, and reads the stick, where every judge reads as today but -`syscall_cost`, whose lines are the job's own, and `log_poll_outlives_a_close`, -whose `echo` record needs no anchor on the job before it: those read the -window. -`loader_watchdog_arms`' control arm rides it, and `mkdir_cap` and -`readdir_bound` remove what they made. +per-job bound. Every judge reads the stick as today but `syscall_cost` and +`log_poll_outlives_a_close`, which read the window. `loader_watchdog_arms`' +control arm rides a session, `mkdir_cap` and `readdir_bound` remove what they +made, and `audio_idle_suspend` waits for soundd's `inspect` to read `suspended` +(`userland/soundd/src/inspect.rs:21-33`), not for a boot no client has reached. -**Exit**: every registration those boots carried gives the per-boot verdict at -the same head, each window's judgement agrees, and the sessions in reverse -order, fixed members in place, give the same verdicts; in QEMU an e1000e guest -runs a session end to end over `ssh`. Staged in one member, each of these reds -it alone and the next runs: a fault, by the kernel's record in its window; a -child left alive; a file left in `/home`; a `NEVER_CLEAN` line; a job spinning -past its bound. +**Exit**, on the T14: every registration those boots carried gives the +per-boot verdict at the same head, each window's judgement agrees, and the +sessions in reverse order, claims last, give the same verdicts. A fault, by the +kernel's record in its window, and a job spinning past its bound, each staged +in one member, red it alone and the next runs. A host test over a recorded +window reds only the member that left a child alive, a file in `/home` or a +`NEVER_CLEAN` line. **Then: Ubuntu leaves the loop** in stages 6 and 7 of -`issues/boot-media/the-loader-does-only-what-must-precede-the-handover.md`, -which bring each session image by `update` and each death's record back over -`sftp`. +`issues/boot-media/the-loader-does-only-what-must-precede-the-handover.md`: +a session image comes by `update`, a row that boots on its own by +`update --once`, and each death's record over `sftp`. **Last: sessions outlive the boot deadline**, once `issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md` has armed the hard-lockup detector on every boot and shipped `watchdogd`. An image names no `boot-deadline=`, `judge_arms` (`src/metal.rs:899-926`) takes `watchdogd`'s row as its bound, and `pipe` bounds each window rather than its -whole run (`tests/ssh-client-host/src/main.rs:64`). A ToyOS that runs but that -the host cannot reach keeps its watchdog fed, so the run says it waits for a -hand once nothing has answered within the watchdog's bound and a POST -allowance. A run holds one session per kernel build, parameter line and config -some test needs, and a swap leaves a boot's bound +whole run (`tests/ssh-client-host/src/main.rs:64`). A ToyOS the host cannot +reach keeps its watchdog fed, so the run says it waits for a hand once nothing +has answered within the watchdog's bound and a POST allowance. A swap then +leaves a boot's bound (`issues/hardware/a-swap-on-the-t14-lives-inside-a-metal-boots-bound.md`). -**Exit**: the host's plan names every reset in a run by what forces it, and in -QEMU an e1000e guest whose link is cut under a session stops the run and says -it waits for a hand. +**Exit**: the host's plan names every reset in a run by what forces it, and a +host test whose transport goes quiet under a session finds the run stopped, +saying it waits for a hand. diff --git a/issues/hardware/watchdog-fed-ends-its-boot-before-the-bound-it-judges.md b/issues/hardware/watchdog-fed-ends-its-boot-before-the-bound-it-judges.md index 7f44fe76fd..3d8acda7a5 100644 --- a/issues/hardware/watchdog-fed-ends-its-boot-before-the-bound-it-judges.md +++ b/issues/hardware/watchdog-fed-ends-its-boot-before-the-bound-it-judges.md @@ -7,15 +7,14 @@ opened: 2026-09-30 # `watchdog_fed` ends its boot before the bound it judges `watchdog_fed` holds that an armed T14 boot runs its list to its own stop, -which a chipset reset anywhere in it would have cut short -(`tests/toyos.rs:1740-1749`). Its boot, `testcases-watchdog`, carries no job of -its own and neither does `loader_watchdog_arms`' arm on it -(`tests/toyos.rs:1667-1679`), so its list is `reboot` alone: on two T14 runs -the runner spawned `reboot` 1.196 s and 1.489 s into the kernel's clock, and the -shutdown disarms the timer (`kernel/src/syscall/machine.rs:88`), inside one +which a chipset reset anywhere in it would have cut short. Its boot, +`testcases-watchdog`, carries no job of its own and neither does +`loader_watchdog_arms`' arm on it, so its list is `reboot` alone: on two T14 +runs the runner spawned `reboot` 1.196 s and 1.489 s into the kernel's clock, +and the shutdown disarms the timer (`arch::watchdog::disarm`), inside one `toyos_tco::BOUND_MS` of 9,600 ms. A kernel that never fed the timer passes it. **Exit**: `watchdog_fed`'s boot holds past `toyos_tco::BOUND_MS` before its -`reboot`, and the same boot armed with `tco-starve` reds it once stage 1 of -`issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md` has -seen the TCO reset the T14. +`reboot`, and the same boot on a kernel whose scheduler pass never feeds reds +it once `issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md` +has seen the TCO reset the T14. diff --git a/issues/hardware/whether-a-boot-may-starve-the-t14s-watchdog-is-the-owners.md b/issues/hardware/whether-a-boot-may-starve-the-t14s-watchdog-is-the-owners.md index 1b5b5307b1..22a8ca99ef 100644 --- a/issues/hardware/whether-a-boot-may-starve-the-t14s-watchdog-is-the-owners.md +++ b/issues/hardware/whether-a-boot-may-starve-the-t14s-watchdog-is-the-owners.md @@ -12,18 +12,19 @@ on purpose, so that the chipset resets the laptop? The watchdog is what will reset a frozen ToyOS on every machine (owner ruling), and on the T14 it is the only reset that needs nothing of the kernel. Nobody has seen it reset this laptop: runs 3 to 5 armed it, and each needed a hand on the -power button (`issues/hardware/an-armed-tco-has-never-reset-the-t14.md`). On -QEMU's model of the chipset, `watchdog_resets` sees it reset the guest. +power button (`issues/hardware/an-armed-tco-has-never-reset-the-t14.md`). -- **What a yes risks**: about 15 s into that boot the laptop resets, as it does - at the end of every test boot, but by the chipset's timer rather than by - ToyOS. What this laptop's firmware does after such a reset has never been - seen. If the chipset does not reset it, the boot's own deadline ends it at - 120 s, as it ends the `deadlinewedge` boot in every run. +- **What a yes risks**: the laptop resets, as it does at the end of every test + boot, but by the chipset's timer rather than by ToyOS. What this laptop's + firmware does after such a reset has never been seen. If the chipset does not + reset it, the boot's own deadline ends it, as it ends the `deadlinewedge` + boot in every run. - **What a no stops**: the reading in - `issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md` and - that track's T14 exit, which needs the same reset. `watchdogd` then ships to - the T14 with its reset unproven there. + `issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md`, + and the T14 exits of that track and of stage 8 of + `issues/boot-media/the-loader-does-only-what-must-precede-the-handover.md`, + which need the same reset. `watchdogd` then ships to the T14 with its reset + unproven there. *Recommended: yes.* From 269a77f6919d06ad8d015e05b7ccbb8c7f8883ad Mon Sep 17 00:00:00 2001 From: japabu Date: Thu, 1 Oct 2026 06:32:24 +0200 Subject: [PATCH 08/11] issues: name judge_arms and hold_the_panel rather than lines #638 moves #638 shifts src/metal.rs' judge_arms to line 909 and panic_console's hold_the_panel to 716, and this branch lands after it, so both tracks name the functions. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L --- ...en-toyos-waits-for-a-hand-on-the-power-button.md | 2 +- ...the-t14-reboots-through-ubuntu-for-every-test.md | 13 ++++++------- 2 files changed, 7 insertions(+), 8 deletions(-) diff --git a/issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md b/issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md index cbe9c993a7..524a4c93c5 100644 --- a/issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md +++ b/issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md @@ -30,7 +30,7 @@ the detector is armed on every boot; nothing ships for tests alone. `issues/kernel/cpu-time-is-a-band-and-not-a-reservation.md` gives it a reservation. - **A panic's panel holds as it does today** and feeds the watchdog while it - holds, after a key too (`kernel/src/drivers/panic_console/mod.rs:710-725`). + holds, after a key too (`panic_console::hold_the_panel`). - **Every guest runs with `-action watchdog=none`.** q35's TCO counts `QEMU_CLOCK_VIRTUAL`, which a loaded host advances while it starves a guest, and its second expiry does what `-action watchdog=` says (QEMU 11.1.1, diff --git a/issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md b/issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md index 7da69082dc..c194794000 100644 --- a/issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md +++ b/issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md @@ -69,13 +69,12 @@ a session image comes by `update`, a row that boots on its own by **Last: sessions outlive the boot deadline**, once `issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md` has armed the hard-lockup detector on every boot and shipped `watchdogd`. An image -names no `boot-deadline=`, `judge_arms` (`src/metal.rs:899-926`) takes -`watchdogd`'s row as its bound, and `pipe` bounds each window rather than its -whole run (`tests/ssh-client-host/src/main.rs:64`). A ToyOS the host cannot -reach keeps its watchdog fed, so the run says it waits for a hand once nothing -has answered within the watchdog's bound and a POST allowance. A swap then -leaves a boot's bound -(`issues/hardware/a-swap-on-the-t14-lives-inside-a-metal-boots-bound.md`). +names no `boot-deadline=`, `metal::judge_arms` takes `watchdogd`'s row as its +bound, and `pipe` bounds each window rather than its whole run +(`tests/ssh-client-host/src/main.rs:64`). A ToyOS the host cannot reach keeps +its watchdog fed, so the run says it waits for a hand once nothing has answered +within the watchdog's bound and a POST allowance. A swap then leaves a boot's +bound (`issues/hardware/a-swap-on-the-t14-lives-inside-a-metal-boots-bound.md`). **Exit**: the host's plan names every reset in a run by what forces it, and a host test whose transport goes quiet under a session finds the run stopped, From 81dbc52b903dcc70c3b873b465787e08e637e7f4 Mon Sep 17 00:00:00 2001 From: japabu Date: Sat, 3 Oct 2026 10:38:46 +0200 Subject: [PATCH 09/11] issues: the session track and the isolation issue cite what main has today Each citation the branch's files make was checked against 4adc1c091. Every name a track or issue cites exists there (toyos_tco::STAGED_BOUND_MS, RUST_MEMBER_MS, C_MEMBER_MS, SharedBoot's `members`, metal::judge_arms, panic_console::hold_the_panel, arch::watchdog's `arm` and its TCO2_STS read, bootlog::split_listing, toyos::log::LogTail, vfs::ROOT_ENTRIES, Armed's (0, _) arm, every boot and row name), but for these, which this commit fixes: - `device_claim_lifetime` is gone (51cc87fcc, #639). `endowment_denied` is now the one session member that mints a claim (`git grep -l device_claim` over the Rust tests, test-runner and the C corpus finds it alone), so the track says it runs last in its session rather than that the two end one each. - `log_poll_outlives_a_close` is gone (ad6dc0781, #639). Of the judges on the six folded boots, `syscall_cost` alone reads the job's own output; the audio judges read soundd's lines between the kernel's `spawn:` records, which reach the stick. - The guest-suite track does not itself leave the shared boot to the T14: #660 did, and `shared_metal` and `c_corpus_metal` are its only runs outside `--debug`. The track now says the first and cites the issue for the rows it adds. - The prices name #638's allowances, toyos_tco::RUST_MEMBER_MS and C_MEMBER_MS, rather than "as SharedBoot::members prices them": `members` is a count derived from them. - Line citations into test-runner and toyos/src/syscap.rs moved: `--bound-ms=` is main.rs:78-83, run_one's duplicate 248-250, the namespace's inheritance 62-68, SysCap::duplicate 63-70 and SysCap::narrowed 133-139. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_013UDZQ6fSKw14e4w2TKTRfm --- ...4-reboots-through-ubuntu-for-every-test.md | 27 ++++++++++--------- ...tarts-holds-its-whole-system-capability.md | 8 +++--- 2 files changed, 18 insertions(+), 17 deletions(-) diff --git a/issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md b/issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md index c194794000..66dae43deb 100644 --- a/issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md +++ b/issues/hardware/the-t14-reboots-through-ubuntu-for-every-test.md @@ -8,9 +8,10 @@ opened: 2026-09-29 The owner's direction: the tests run one after another in a ToyOS that stays booted, driven by the host over ssh, with no reboot between them and no Ubuntu. -The T14 is the integration tier: -`issues/build/the-guest-suite-runs-only-what-no-cheaper-tier-reaches.md` leaves -the shared boot to `shared_metal` and `c_corpus_metal` and adds metal rows. +The T14 is the integration tier: the shared boot runs only as `shared_metal` +and `c_corpus_metal`, and +`issues/build/the-guest-suite-runs-only-what-no-cheaper-tier-reaches.md` adds +metal rows. - **A metal row is a member** of the session its kernel build, parameter line and config name, and costs no boot, unless it ends the machine (a panic, a @@ -31,10 +32,9 @@ the shared boot to `shared_metal` and `c_corpus_metal` and adds metal rows. (`kernel/src/vfs.rs:83-86`). The host holds both to the session's first, `/log` less `bootlog::split_listing`'s files. Claims join once `issues/kernel/deferred-release-outlives-its-syscall.md` closes; until then a - member that mints one runs last, so `device_claim_lifetime` ends one session - and `endowment_denied` the other. + member that mints one runs last in its session, as `endowment_denied` does. - **A stuck test** is killed at the bound the host's `run` names, as - `--bound-ms=` names the job list's (`userland/test-runner/src/main.rs:101-107`): + `--bound-ms=` names the job list's (`userland/test-runner/src/main.rs:78-83`): 14.1 s, twice `mutual_kill`'s 7.041 s. It reds by name and the next member runs. QEMU's harness names no bound and sends no `list`; its own ceiling ends the guest. @@ -43,14 +43,15 @@ the shared boot to `shared_metal` and `c_corpus_metal` and adds metal rows. `testcases`, `testcases-mkdir` and `testcases-readdir` share a config, a parameter line and the shipping kernel. The session image is `tests/testcases` with `tests/lantalkcase`'s netd, sshd and streaming `logd`, under the same -120 s `boot-deadline=`. A session holds 74.8 s of members, priced as -`SharedBoot::members` prices them and a list at twice its slowest on the T14: -the bound less a tenth, less a lease as late as 19.1 s +120 s `boot-deadline=`. A session holds 74.8 s of members, priced at +`toyos_tco::RUST_MEMBER_MS` a Rust member, `toyos_tco::C_MEMBER_MS` a C case +and a list at twice its slowest on the T14: the bound less a tenth, less a +lease as late as 19.1 s (`issues/hardware/most-t14-leases-land-one-dhcp-retry-late.md`), less one -per-job bound. Every judge reads the stick as today but `syscall_cost` and -`log_poll_outlives_a_close`, which read the window. `loader_watchdog_arms`' -control arm rides a session, `mkdir_cap` and `readdir_bound` remove what they -made, and `audio_idle_suspend` waits for soundd's `inspect` to read `suspended` +per-job bound. Every judge reads the stick as today but `syscall_cost`, which +reads the window. `loader_watchdog_arms`' control arm rides a session, +`mkdir_cap` and `readdir_bound` remove what they made, and `audio_idle_suspend` +waits for soundd's `inspect` to read `suspended` (`userland/soundd/src/inspect.rs:21-33`), not for a boot no client has reached. **Exit**, on the T14: every registration those boots carried gives the diff --git a/issues/isolation/every-job-test-runner-starts-holds-its-whole-system-capability.md b/issues/isolation/every-job-test-runner-starts-holds-its-whole-system-capability.md index 2b0d3f8164..4fdc5f270a 100644 --- a/issues/isolation/every-job-test-runner-starts-holds-its-whole-system-capability.md +++ b/issues/isolation/every-job-test-runner-starts-holds-its-whole-system-capability.md @@ -7,10 +7,10 @@ opened: 2026-09-29 # Every job test-runner starts holds test-runner's whole system capability `run_one` endows each job it spawns a `SysCap::duplicate` of test-runner's own -capability (`userland/test-runner/src/main.rs:292-295`), and a duplicate -carries every right the original does (`toyos/src/syscap.rs:64-71`). The job +capability (`userland/test-runner/src/main.rs:248-250`), and a duplicate +carries every right the original does (`toyos/src/syscap.rs:63-70`). The job also inherits test-runner's whole namespace -(`userland/test-runner/src/main.rs:86-92`). On `tests/testcases` the row grants +(`userland/test-runner/src/main.rs:62-68`). On `tests/testcases` the row grants `device`, `dup`, `logread`, `power` and `roster` (`tests/testcases/system.toml:41`), so every test binary may mint a device claim, read every kernel record, reset or power off the machine, list every @@ -23,5 +23,5 @@ needs: `test_rs_audio_idle_suspend` reads `roster` off the duplicate `endowment_denied` narrows `power` away from it. **Exit**: a job holds only the rights and names its test is declared to need, -narrowed by test-runner (`SysCap::narrowed`, `toyos/src/syscap.rs:134-141`), and +narrowed by test-runner (`SysCap::narrowed`, `toyos/src/syscap.rs:133-139`), and a job that asks for anything else is refused. From 10d98a786bf009f9173ed7736aeffd69a3ac137f Mon Sep 17 00:00:00 2001 From: japabu Date: Sat, 3 Oct 2026 11:12:05 +0200 Subject: [PATCH 10/11] issues: the watchdog track names the panel's hold as the kernel's one feed after the claim; the update rig's stale citations have a file Round 5's NOTEs 1 and 2 on #631. The watchdog track said the kernel feeds "until the class is first claimed, and never after" and, three bullets on, that a panic's panel "feeds the watchdog while it holds". The panel is kernel code feeding after the claim, so the two contradicted as written, and an implementer who made any kernel feed after the claim unrepresentable could not meet the last exit's held panic. The feed that stops at the claim is the scheduler pass's (kernel/src/sched/driver.rs:512), and the track now says so; the panel's bullet names its hold as the one feed the kernel keeps once the class is claimed; the last exit's mutation is a kernel whose scheduler pass feeds on. issues/build/the-update-rig-the-guest-cut-deleted-is-still-cited.md is new. #660 (06788146b) deleted tests/common/update.rs, tests/common/fwvars.rs and tests/updatecase/system.toml, and nine lines in three issue files still plan or describe tests on them: six in the loader track's stages 3 and 5, which this branch does not touch, two in the memory-overwrite issue's exit and one in the console-split issue. `git grep -n 'fwvars\|updatecase\|common/update\.rs\|Rig::\|vars::plant\|vars::live'` finds those nine and, with this commit, the new file. They are filed and not fixed here: with the path struck each sentence still names an oracle, a rig or a config that is gone, and what it should name is the tier stage A of the guest-suite track gives its test, so that stage owns the file. No `update_*` test is in the tree at this head (`git grep 'fn update_\|"update_'` outside issues/ finds no test or row). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_013UDZQ6fSKw14e4w2TKTRfm --- ...ig-the-guest-cut-deleted-is-still-cited.md | 38 +++++++++++++++++++ ...os-waits-for-a-hand-on-the-power-button.md | 19 ++++++---- 2 files changed, 49 insertions(+), 8 deletions(-) create mode 100644 issues/build/the-update-rig-the-guest-cut-deleted-is-still-cited.md diff --git a/issues/build/the-update-rig-the-guest-cut-deleted-is-still-cited.md b/issues/build/the-update-rig-the-guest-cut-deleted-is-still-cited.md new file mode 100644 index 0000000000..8c8e8a77cb --- /dev/null +++ b/issues/build/the-update-rig-the-guest-cut-deleted-is-still-cited.md @@ -0,0 +1,38 @@ +--- +status: open +kind: tooling +opened: 2026-10-03 +--- + +# The update rig the guest cut deleted is still cited + +#660 (`06788146b`) deleted `tests/common/update.rs` with its `Rig`, +`tests/common/fwvars.rs` and `tests/updatecase/system.toml`. `git grep -n +'fwvars\|updatecase\|common/update\.rs\|Rig::\|vars::plant\|vars::live'` finds +nine lines in three files, each planning or describing a test on them: + +- `issues/boot-media/the-loader-does-only-what-must-precede-the-handover.md`, + six lines. Stage 3's exit takes `fwvars::live` as + `update_floor_is_the_images_own`'s oracle. Stage 5 says + `tests/updatecase/system.toml` grants `slots` and that two tests run + `updatecase`; one of its negative controls signs with `Rig::update` and + reads the floor with `fwvars::live`, which its oracles name too. +- `issues/boot-media/the-loader-never-sets-the-firmwares-memory-overwrite-request.md`, + two lines: its exit's `mor_is_set_where_defined` plants and reads the vars + store with `vars::plant` and `vars::live`. +- `issues/build/the-kernel-console-split-does-not-re-arm-across-a-guest-reset.md`, + one line: the reset in place it describes is `Rig::boot`'s. + +Striking a path leaves its sentence naming an oracle, a rig or a config that +is gone. What each should name is the tier its test has now, and stage A of +`issues/build/the-guest-suite-runs-only-what-no-cheaper-tier-reaches.md` gives +it: `update_floor_is_the_images_own`, `update_refusals_boot_the_other_slot` +and `update_grant_refuses_a_stray_partition` are host tests there and guest +tests in the loader track, and the tree holds no `update_*` test today. + +**Owner**: that stage A, whose pull request lands those tests and with them +the oracle and the rig a sentence can name. + +**Exit**: the `git grep` above finds this file alone: each sentence names the +tier that holds its test and an oracle, a rig or a config the tree has, or has +gone with the test it described. diff --git a/issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md b/issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md index 524a4c93c5..c92789c90e 100644 --- a/issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md +++ b/issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md @@ -21,16 +21,18 @@ the detector is armed on every boot; nothing ships for tests alone. holds it, which `issues/isolation/every-job-test-runner-starts-holds-its-whole-system-capability.md` fences. -- **The kernel feeds from its arm until the class is first claimed, and never - after**, so a `watchdogd` that ends with no successor holding the class - resets the machine. A successor's claim that meets its predecessor's release - still in flight (`issues/kernel/deferred-release-outlives-its-syscall.md`) is - refused, so `watchdogd` waits on that defect. +- **The scheduler pass feeds from the kernel's arm until the class is first + claimed, and never after**, so a `watchdogd` that ends with no successor + holding the class resets the machine. A successor's claim that meets its + predecessor's release still in flight + (`issues/kernel/deferred-release-outlives-its-syscall.md`) is refused, so + `watchdogd` waits on that defect. - `watchdogd` feeds from a thread under `rt`, until `issues/kernel/cpu-time-is-a-band-and-not-a-reservation.md` gives it a reservation. - **A panic's panel holds as it does today** and feeds the watchdog while it - holds, after a key too (`panic_console::hold_the_panel`). + holds, after a key too (`panic_console::hold_the_panel`): the one feed the + kernel keeps once the class is claimed. - **Every guest runs with `-action watchdog=none`.** q35's TCO counts `QEMU_CLOCK_VIRTUAL`, which a loaded host advances while it starves a guest, and its second expiry does what `-action watchdog=` says (QEMU 11.1.1, @@ -77,5 +79,6 @@ has made the kernel's arm the only one. `toyos_tco::PARAM` goes, and with it **Exit**, on the T14: a boot whose `watchdogd` ends with no successor, and one wedged by `wedge-before-reset` with `watchdogd` feeding, each end in the TCO's -reset, and the boot after each says so; a kernel that feeds on once the claim -has gone keeps the first up. The boot after a held panic says no TCO reset. +reset, and the boot after each says so; a kernel whose scheduler pass feeds on +once the claim has gone keeps the first up. The boot after a held panic says +no TCO reset. From 523509c9fafa13e5c6c622c2a0036dbb64b8df1c Mon Sep 17 00:00:00 2001 From: japabu Date: Sat, 3 Oct 2026 11:32:19 +0200 Subject: [PATCH 11/11] issues: the watchdog track takes the owner's two rulings of 2026-10-03, and the question he answered is closed The sixth review of #631 found the track's device class stated as decided in the list under "The owner's rulings", where it was this branch's own mechanism and nobody had ruled on it. The owner has since ruled on it, and on the question this branch had filed for him. How watchdogd reaches the chipset's timer. His ruling: through the general port-claim mechanism #592 brings, its `isa` claim, as the ACPI server will, and not through a device class of its own; one way of doing it instead of two. The track's device-class line, its "init mints its claim as it mints blockd's" and its "the class is an ABI change" are replaced by that ruling, and the lines that named "the class" name the claim. The track's list mixed his lines with the branch's. It is now two lists: the owner's rulings (a userland watchdogd on every machine with no new syscall, the detector on every boot, nothing for tests alone, the panel's hold, the port claim), and the track's own lines, which no ruling carries (the scheduler pass feeding until the first claim, the deferred-release wait, the `device` holder the isolation issue fences, the `rt` thread, `-action watchdog=none`, and the T14's one reset). "The one feed the kernel keeps" was the branch's gloss on the panel's hold and moves to the track's own list. The ruling leaves two things unsettled, and the track says they are open instead of deciding them. Both are read from #592's head 34ef28c8d: - `IO_PORTS` is 0x100 (kernel/src/arch/x86_64/percpu.rs:27), and a compile-time assert holds every `GRANTABLE` port below it (arch/x86_64/pio.rs:33-43). The T14's TCO is at 0x400 (run 5's loader.log, in issues/hardware/an-armed-tco-has-never-reset-the-t14.md). - A `Grantable` row carries `kernel_drives`, "which no claim shares", and `mint` refuses such a row with `ClaimError::KernelDriven` (kernel/src/isa.rs:44-45,98-100). Under this track the kernel still arms the TCO (`arch::watchdog::init`), feeds it from the panel's hold and disarms it (kernel/src/syscall/machine.rs:68). Whether a boot may starve the T14's watchdog. His answer is yes, on one condition, in his words: "only if its quick i dont want tests doing nothing for a long time". The track's reading step carries the ruling and the condition: the row's wait is bounded by the watchdog's own bound and fails loudly past it, with no long idle wait. The step no longer waits on him, only on loader stage 7. issues/hardware/whether-a-boot-may-starve-the-t14s-watchdog-is-the-owners.md is deleted, as issues/README.md closes an answered question. It never reached main: this branch filed it in 6489e255b and deletes it here. What it recorded beside the question: a yes risks a reset by the chipset's timer in place of ToyOS's own, on a laptop whose firmware has never been seen after one, with the boot's deadline ending the boot if the chipset does not; a no would have stopped the track's reading and the T14 exits of the track and of loader stage 8, and shipped watchdogd to the T14 with its reset unproven there. Its one citation, in the track's reading step, goes with it. The review's REMOVE in the update-rig issue: the sentence saying why this branch filed the stale citations and did not strike them. 10d98a786's message carries that. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_013UDZQ6fSKw14e4w2TKTRfm --- ...ig-the-guest-cut-deleted-is-still-cited.md | 3 +- ...os-waits-for-a-hand-on-the-power-button.md | 56 +++++++++++-------- ...-starve-the-t14s-watchdog-is-the-owners.md | 31 ---------- 3 files changed, 35 insertions(+), 55 deletions(-) delete mode 100644 issues/hardware/whether-a-boot-may-starve-the-t14s-watchdog-is-the-owners.md diff --git a/issues/build/the-update-rig-the-guest-cut-deleted-is-still-cited.md b/issues/build/the-update-rig-the-guest-cut-deleted-is-still-cited.md index 8c8e8a77cb..52745ef69b 100644 --- a/issues/build/the-update-rig-the-guest-cut-deleted-is-still-cited.md +++ b/issues/build/the-update-rig-the-guest-cut-deleted-is-still-cited.md @@ -23,8 +23,7 @@ nine lines in three files, each planning or describing a test on them: - `issues/build/the-kernel-console-split-does-not-re-arm-across-a-guest-reset.md`, one line: the reset in place it describes is `Rig::boot`'s. -Striking a path leaves its sentence naming an oracle, a rig or a config that -is gone. What each should name is the tier its test has now, and stage A of +What each should name is the tier its test has now, and stage A of `issues/build/the-guest-suite-runs-only-what-no-cheaper-tier-reaches.md` gives it: `update_floor_is_the_images_own`, `update_refusals_boot_the_other_slot` and `update_grant_refuses_a_stray_partition` are host tests there and guest diff --git a/issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md b/issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md index c92789c90e..aeda14821e 100644 --- a/issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md +++ b/issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md @@ -10,29 +10,39 @@ No shipped image arms a hardware watchdog or the hard-lockup detector: the kernel arms the chipset's TCO only on the `watchdog` parameter (`arch::watchdog::init`), feeds it from the scheduler pass (`kernel/src/sched/driver.rs:512`), and arms the detector only through -`boot-deadline=` (`kernel/src/deadline.rs:168-183`). The owner's rulings: a -userland `watchdogd` feeds the watchdog on every machine, with no new syscall; -the detector is armed on every boot; nothing ships for tests alone. +`boot-deadline=` (`kernel/src/deadline.rs:168-183`). -- `watchdogd` reaches the TCO through a device class of its own, as soundd - reaches HDA (`kernel/src/syscall/device.rs:55-77`), and init mints its claim - for `watchdogd`'s row as it mints blockd's (`system.toml:182-186`). The class - is an ABI change. A job holding `device` can mint it where no `watchdogd` - holds it, which - `issues/isolation/every-job-test-runner-starts-holds-its-whole-system-capability.md` - fences. -- **The scheduler pass feeds from the kernel's arm until the class is first +**The owner's rulings:** + +- A userland `watchdogd` feeds the watchdog on every machine, with no new + syscall; the detector is armed on every boot; nothing ships for tests alone. +- **A panic's panel holds as it does today** and feeds the watchdog while it + holds, after a key too (`panic_console::hold_the_panel`). +- **`watchdogd` reaches the chipset's timer through the general port claim + #592 brings, its `isa` claim, as the ACPI server will**, and not through a + device class of its own: one way of doing it and not two (2026-10-03). That + ruling does not settle two things, and nothing here decides them: + - #592's grant covers ports below `0x100`, and the T14's TCO is at `0x400` + (`issues/hardware/an-armed-tco-has-never-reset-the-t14.md`). + - #592 refuses a claim on ports the kernel drives, and under this track the + kernel still arms the timer, feeds it from the panel and disarms it. + +**The track's own lines, which no ruling carries:** + +- **The scheduler pass feeds from the kernel's arm until the timer is first claimed, and never after**, so a `watchdogd` that ends with no successor - holding the class resets the machine. A successor's claim that meets its + holding the claim resets the machine. After that first claim the panel's + hold is the one feed the kernel keeps. A successor's claim that meets its predecessor's release still in flight (`issues/kernel/deferred-release-outlives-its-syscall.md`) is refused, so `watchdogd` waits on that defect. +- A job holding `device` can mint the claim where no `watchdogd` holds it, + which + `issues/isolation/every-job-test-runner-starts-holds-its-whole-system-capability.md` + fences. - `watchdogd` feeds from a thread under `rt`, until `issues/kernel/cpu-time-is-a-band-and-not-a-reservation.md` gives it a reservation. -- **A panic's panel holds as it does today** and feeds the watchdog while it - holds, after a key too (`panic_console::hold_the_panel`): the one feed the - kernel keeps once the class is claimed. - **Every guest runs with `-action watchdog=none`.** q35's TCO counts `QEMU_CLOCK_VIRTUAL`, which a loaded host advances while it starves a guest, and its second expiry does what `-action watchdog=` says (QEMU 11.1.1, @@ -51,15 +61,17 @@ constant then has a reader, which closes to `HARD_LOCKUP_BOUND_MS`, and putting the arm back behind `deadline::start` fails to build. `Armed`'s `(0, _)` arm, which nothing reaches, is gone. -**Then, on the owner's yes** to -`issues/hardware/whether-a-boot-may-starve-the-t14s-watchdog-is-the-owners.md`, -and once loader stage 7 has taken Ubuntu, whose `iTCO_wdt_probe` clears +**Then, once loader stage 7 has taken Ubuntu**, whose `iTCO_wdt_probe` clears `SECOND_TO_STS` (Linux 6.12, `drivers/watchdog/iTCO_wdt.c:545-560`), out of the loop: **the reading**, `watchdog_resets`' metal row in -`issues/build/the-guest-suite-runs-only-what-no-cheaper-tier-reaches.md`. One -T14 boot is armed with `watchdog` and `wedge-before-reset`, under a deadline -past its job list and `toyos_tco::BOUND_MS`, which `toyos_tco::STAGED_BOUND_MS` -is not. The fed control is `watchdog_fed`, once +`issues/build/the-guest-suite-runs-only-what-no-cheaper-tier-reaches.md`. The +owner's ruling (2026-10-03): one T14 boot may stop feeding the chipset's +watchdog on purpose, "only if its quick i dont want tests doing nothing for a +long time". So the row's wait is bounded by the watchdog's own bound and fails +loudly past it, with no long idle wait. One T14 boot is armed with `watchdog` +and `wedge-before-reset`, under a deadline past its job list and +`toyos_tco::BOUND_MS`, which `toyos_tco::STAGED_BOUND_MS` is not. The fed +control is `watchdog_fed`, once `issues/hardware/watchdog-fed-ends-its-boot-before-the-bound-it-judges.md` closes. diff --git a/issues/hardware/whether-a-boot-may-starve-the-t14s-watchdog-is-the-owners.md b/issues/hardware/whether-a-boot-may-starve-the-t14s-watchdog-is-the-owners.md deleted file mode 100644 index 22a8ca99ef..0000000000 --- a/issues/hardware/whether-a-boot-may-starve-the-t14s-watchdog-is-the-owners.md +++ /dev/null @@ -1,31 +0,0 @@ ---- -status: owner -kind: question -opened: 2026-10-01 ---- - -# Whether a boot may starve the T14's watchdog is the owner's - -The question: may one test boot on the T14 stop feeding the chipset's watchdog -on purpose, so that the chipset resets the laptop? - -The watchdog is what will reset a frozen ToyOS on every machine (owner ruling), -and on the T14 it is the only reset that needs nothing of the kernel. Nobody has -seen it reset this laptop: runs 3 to 5 armed it, and each needed a hand on the -power button (`issues/hardware/an-armed-tco-has-never-reset-the-t14.md`). - -- **What a yes risks**: the laptop resets, as it does at the end of every test - boot, but by the chipset's timer rather than by ToyOS. What this laptop's - firmware does after such a reset has never been seen. If the chipset does not - reset it, the boot's own deadline ends it, as it ends the `deadlinewedge` - boot in every run. -- **What a no stops**: the reading in - `issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md`, - and the T14 exits of that track and of stage 8 of - `issues/boot-media/the-loader-does-only-what-must-precede-the-handover.md`, - which need the same reset. `watchdogd` then ships to the T14 with its reset - unproven there. - -*Recommended: yes.* - -**Exit**: the owner rules.