Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions issues/build/a-metal-failure-drops-every-row-its-boot-measured.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
---
status: open
kind: tooling
opened: 2026-09-30
---

# A metal failure drops every row its boot measured

`tests/common/metal.rs`'s `judge_readbacks` puts boot labels in `failed`, and
`Readback::measured` records a number with no test attached. One red test or
shared member riding a boot therefore keeps every number that boot measured
out of the record, and a number never recorded is never judged. A runner that
carries every test in one session would drop every row of the session for one
red test.

It already costs rows. On the T14's full run at `0d2dda66f`, `testcases`,
`shared`, `lancase` and `lantalkcase` each carried a known red, and
`tests/metal/lenovo-20w0003amz.toml` holds no row of theirs, so nothing
judges their numbers while those reds stand.

## Exit condition

Each number is recorded against its owner, and only that owner fails.
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
---
status: open
kind: tooling
opened: 2026-09-29
---

# A metal run asks Ubuntu which machine it is, and the boot under test says nothing

`src/metal.rs`'s `run` asks the operating system the T14 runs between boots
for `metaltimings::Machine::QUERY` over ssh before the flash, and writes the
answer into the readback's `boot.txt` under three keys (`VENDOR_KEY`,
`PRODUCT_KEY`, `BIOS_KEY`); `tests/common/metal.rs` reads it back with
`metal::machine`, and `metaltimings::Record::load` picks the machine's record
by it. Both T14 runs of `wt/toyos-metaltimings` named the machine this way:
`machine LENOVO 20W0003AMZ, BIOS N34ET71W (1.71 )`.

A resident runner, tests inside a ToyOS that stays up on the machine with no
Ubuntu and no reboot per test, has nothing to ask: the ssh read,
`Refusal::Machine`, the three keys and `metal::machine` all go with Ubuntu.
The record compares BIOS strings byte for byte, so identity has one reader:
two that trim differently fail every run as a firmware change.

## Exit condition

The boot names its machine: the loader reads SMBIOS type 1's manufacturer and
product and type 0's BIOS version from the UEFI configuration table before
`ExitBootServices` and writes them as one `loader.log` line, the judge reads
the machine from the readback's `loader.log`, the ssh read, `Refusal::Machine`,
the three keys and `metal::machine` are deleted, and every record under
`tests/metal/` names its machine in the loader's own strings.
20 changes: 20 additions & 0 deletions issues/build/a-metal-timing-record-never-tightens.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
---
status: open
kind: tooling
opened: 2026-09-29
---

# A metal timing record never tightens

`src/metaltimings.rs`'s `judge` adds a name its machine has not recorded and
never moves one it has, so a run cannot loosen the record that judges it. The
same rule keeps a speed-up out: a number that halves keeps its old ceiling,
and a regression that later doubles it back is inside that ceiling and passes.
Only deleting the row re-records it.

## Exit condition

A recorded number that a run measures well under its record is either taken
as the new record by a rule that a noisy run cannot ratchet into a false red,
or reported by name so the row is deleted, and the choice is in
`src/metaltimings.rs`'s module doc.
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,7 @@ the T14, and the reason is the loop rather than the kernel.
`Boot: complete (Nms)` *and* a trailing `Rebooting.`. A boot whose subject is
a panic correctly writes neither the second word nor anything after it, so
the loop refuses it — the readback is still written, and
`tests/common/metal.rs`'s `Mode::Drive` tolerates the non-zero exit, but the
boot-level rows `tests/metal-profile.toml` prices are then judged against a
log the loop has already called unfit.
`tests/common/metal.rs`'s `Mode::Drive` tolerates the non-zero exit.

2. `test-late-panic` fires in `kernel_main` after `spawn_init` and before
`enter_idle_loop`, so `logd` has not run: that boot writes no `/log` file at
Expand Down

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,7 @@ reaches no file
(`issues/diagnostics/the-cable-judge-reads-three-netd-records-that-cannot-arrive-on-the-t14.md`):
the kernel's `exit: netd pid=N code=N` record and a file netd writes itself are
the two words of a process that cross. It is the `lan_lease_report` metal row,
a third image flashed to the stick, a third boot of the machine and six rows of
`tests/metal-profile.toml`.
a third image flashed to the stick and a third boot of the machine.

## Owner

Expand All @@ -32,7 +31,7 @@ which the shipping `lancase` arm carries netd's own lines about the lease and
the probe answers a question already answered. Then the arm is:
`tests/lanleasecase/system.toml`, its row in `src/build.rs`'s `ALL_CONFIGS`,
the `lan_lease_report` metal row in `tests/toyos.rs` with `LANLEASECASE` and
`lan::leased_on_metal`, the six `tests/metal-profile.toml` rows,
`lan::leased_on_metal`,
`userland/netd/src/report.rs` and `toyos-i219/src/lease.rs`'s report lines —
and netd's `--exit-with-lease` with `tests/e1000leasecase` and the
`lan_lease_report` QEMU registration, the arm that proves the channel.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,7 @@ readback could say about the run-22 class of hang.

**Exit condition**: the loop names that case as itself — a readback with no
`logd` file and no harvested report reported as "wedged before its first durable
record", distinct from "no `Boot: complete`" and from "no readback at all" — and
a metal-profile row that says which of the three a boot is.
record", distinct from "no `Boot: complete`" and from "no readback at all".

Off the path of whoever finds this: it is `src/metal.rs`'s verdict and belongs
to the driver, not to the kernel side that produced the boot.
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,9 @@ and times reading it back as "a cache miss the stick has to answer" — true
while the kernel's write-back dropped a closed file from its cache. `/log` is
served by `/system/bin/fsd` now, whose block cache (`userland/fsd/src/cache.rs`)
keeps clean blocks until it holds `CLEAN_LIMIT` of them and drops nothing at a
close. So the timed read is fsd's memory, and the `usbread` span
`tests/metal-profile.toml` prices is no longer the stick's.
close. So the timed read is fsd's memory, and the `usbread` span is no longer
the stick's.

**Exit**: the timed read is one the stick answers — a file fsd has not read
since it started, or a read through the partition claim that bypasses the
file server — with the profile row re-measured on the T14.
file server.
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,7 @@ opened: 2026-09-13
`--provoke-message`, which writes one enabled cause to `ICS` so the kernel's
`pcidev: slot N took its first message` record says whether delivery works at
all. It is the `lan_message_delivery` metal row, a second image flashed to the
stick, a second boot of the machine and six rows of `tests/metal-profile.toml`
(`boot.lanicscase.{complete_ms,back_secs,stick_secs,panel_max_us,panel_us}`,
`list.lanicscase.job_ms`).
stick and a second boot of the machine.

**A count of no messages is two facts** — a part nothing made speak and a
message that reached no CPU — and only this arm separates them. On a card that
Expand All @@ -30,8 +28,7 @@ The shipping `lancase` arm recording `pcidev: slot N took its first message`
without the actuator. Then the actuator has no question left and the arm is
four files: `tests/lanicscase/system.toml`, its row in `src/build.rs`'s
`ALL_CONFIGS`, `tests/toyos.rs`'s `lan_message_delivery` row, its `METAL_ONLY`
entry and `LANICSCASE` with the judge `lan::provoked_on_metal`, and the six
`tests/metal-profile.toml` rows.
entry and `LANICSCASE` with the judge `lan::provoked_on_metal`.

Metal run 57 recorded `pcidev: slot 0 took its first message on vector 0x28`
after the I219's hand-over on that slot and vector, so the second fact is read:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -107,16 +107,10 @@ runner asks for no reboot; the deadline was the only bound left, and it fired
## The instrument that measures this already exists

`src/metal.rs:1591-1597`'s `deadline_lateness_ms` computes exactly
`reached − bound` out of the `DEADLINE_EXPIRED` line (`src/bootlog.rs:28`);
`tests/common/metal.rs:274-275` reads it off every readback and `:895-919`
hands it to `profile.judge` as `boot.<label>.deadline_lateness_ms`;
`tests/metal-profile.toml:409-414` prices that row for `deadlinewedge` —
ceiling 10000 ms, "the widest true bound before [the timer period] has been"
measured, `measured = 61`. 132859 is 13x that ceiling, on a boot the profile
does not name: `lancase` is not among the labels the file prices, and is not
in this tree. The arithmetic above is the instrument's own, done by hand
because the run was `toyos-metal` invoked directly and not the harness. No
second instrument is owed; a cause is.
`reached − bound` out of the `DEADLINE_EXPIRED` line (`src/bootlog.rs:28`).
The arithmetic above is the instrument's own, done by hand because the run
was `toyos-metal` invoked directly and not the harness. No second instrument
is owed; a cause is.

## What is known and what is not

Expand Down Expand Up @@ -148,5 +142,4 @@ second instrument is owed; a cause is.
than the poll's one call site.

**Exit condition**: the cause of a `poll` that ran 132859 ms past its bound is
named with evidence and either removed or priced, so that a T14 expiry's
lateness is held to the row that already exists.
named with evidence and either removed or priced.
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
---
status: open
kind: defect
opened: 2026-09-29
---

# `test_rs_mutual_kill` panicked on the T14: stdio slot 1 "holds a region that is not a log ring"

The full metal run of `wt/toyos-metaltimings` at `db5e59bc4` failed
`test_rs_mutual_kill` in the `shared-2` boot; the run of the same branch at
`09fe6121e` passed it. `git diff --stat 09fe6121e db5e59bc4` names one file,
`tests/metal/lenovo-20w0003amz.toml`, which the host harness reads and no image
carries. Unexplained: nothing below says why.

## What the log shows

The job was spawned at 3.045 s as pid 28 and spawned its children, pid 29 to
156, each ending `code=137` or `code=0`. The two exit records written just
before the panic:

[2026-09-29 18:49:34 10.010 cpu4] exit: test_rs_mutual_kill pid=155 code=0 cpu=10ms
[2026-09-29 18:49:34 10.010 cpu5] exit: test_rs_mutual_kill pid=156 code=137 cpu=10ms

Then pid 28 itself, under the runner's name:

{2026-09-29 18:49:34 10.011 error pid=28 test-runner} thread 'main' (1) panicked at /Users/jan/Dev/jan/toyos-phdr/toyos/src/log/stdio.rs:209:33:
{2026-09-29 18:49:34 10.011 error pid=28 test-runner} stdio: slot 1 holds a region that is not a log ring
...
11: 0x1000006e6be - toyos[db68fa2b5c494fe7]::log::stdio::target::sink
12: 0x1000006e7c2 - toyos[db68fa2b5c494fe7]::log::stdio::target::write
13: 0x100000568d8 - <alloc[58e89b8cb86be8e3]::io::buffered::bufwriter::BufWriter<std[cb898f2ab581ea90]::io::stdio::StdoutRaw>>::flush_buf
...
18: 0x10000063bc3 - std[cb898f2ab581ea90]::io::stdio::_print
19: 0x10000028f12 - mutual_kill[652ee4c06bb687b]::main
[2026-09-29 18:49:34 10.085 cpu5] exit: test_rs_mutual_kill pid=28 code=101 cpu=5840ms

`toyos/src/log/stdio.rs:209` is `sink`'s panic on the first write to a
stream whose slot `ask` refused, with `NotARing`'s words (`:276`). The path is
the sysroot's source, which is keyed by content and so names whichever
worktree built that key first.

## Exit condition

Why pid 28's slot 1 answered `NotARing` on its first `println!` is named with
evidence and removed, and `test_rs_mutual_kill` passes on a T14 run.
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ record this issue is about.
## Why it is not free to answer

The record is a fixed line rendered by `toyos-quiesce` and read back by
`src/metal.rs` and the harness, and the profile prices two numbers out of it.
`src/metal.rs` and the harness.
Naming a thread means the record carries a pid, a tid and the class of wait
that thread was in — which the sweep does not collect today, because it counts
rather than remembers.
Expand All @@ -81,5 +81,4 @@ rather than remembers.

The record naming the last thread to stop, and a boot of `tests/metalcase`
whose stop time can then be attributed to it rather than inferred. Until the
record names one, its `in N ms` explains nothing about itself, and no profile
row prices it.
record names one, its `in N ms` explains nothing about itself.
9 changes: 3 additions & 6 deletions kernel/src/hardlockup/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -72,12 +72,9 @@ pub mod probe;
/// line and nothing links the two crates (`src/bootlog.rs`).
pub const LOCKED_UP: &str = "a cpu locked up with interrupts off";

/// How often an armed CPU samples itself, in nanoseconds of unhalted time.
///
/// A second: the bound is measured in tens of them, so a sample period this
/// long costs one NMI per CPU per second and puts the detection within one
/// period of the bound. It is also the period the report's ages are quoted at.
const SAMPLE_NS: u64 = 1_000_000_000;
/// Declared beside the bound it samples, so the host judging a lockup's
/// lateness reads the same period.
const SAMPLE_NS: u64 = toyos_tco::HARD_LOCKUP_SAMPLE_NS;

/// The bound in TSC ticks, or 0 for a boot that armed none. Written on the BSP
/// before any AP exists.
Expand Down
5 changes: 5 additions & 0 deletions src/bootlog.rs
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,10 @@ pub const JOB_DEADLINE_SAID: &str =
/// it has: a wedged boot's `logd` wrote nothing.
pub const DEADLINE_EXPIRED: &str = "the boot deadline expired";

/// What the kernel logs as it arms that deadline, in `kernel/src/deadline.rs`:
/// the record whose time the bound is counted from.
pub const DEADLINE_ARMED: &str = "boot deadline: ";

/// What the `wedge-before-reset` actuator says before it stops every CPU, in
/// `kernel/src/deadline.rs`. The witness that a deadline ended a wedge and not
/// a boot merely slower than its bound, which is what makes that control one.
Expand Down Expand Up @@ -708,6 +712,7 @@ mod tests {
("kernel/src/arch/x86_64/smp.rs", format!("log!(\"{AP_BRINGUP}")),
("kernel/src/process.rs", format!("THREAD_NAME_LEN: usize = {NAME_LEN}")),
("kernel/src/deadline.rs", format!("EXPIRED: &str = \"{DEADLINE_EXPIRED}\"")),
("kernel/src/deadline.rs", format!("\"{DEADLINE_ARMED}{{ms}} ms")),
("kernel/src/deadline.rs", format!("WEDGE_STAGED: &str = \"{WEDGE_STAGED}\"")),
("kernel/src/deadline.rs", format!("\"{WEDGE_ARRIVED_DEAF}\"")),
("kernel/src/usb_gate.rs", format!("USB_WEDGE_STAGED: &str = \"{USB_WEDGE_STAGED}\"")),
Expand Down
2 changes: 1 addition & 1 deletion src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -31,9 +31,9 @@ pub mod licence;
pub mod metal;
pub mod metaldevices;
pub mod metalimage;
pub mod metalprofile;
pub mod metalswap;
pub mod metaltalk;
pub mod metaltimings;
pub mod redlist;
pub mod release;
pub mod sdkversion;
Expand Down
Loading
Loading