Every QEMU ceiling is at most three times what its test takes; wedges end in seconds; two LAN boots ride the talking boot

[Japabu]

Head e3094a94a, on 75dc056f1, which merges origin/main at 73282fa93 (#648, #680). git diff --shortstat origin/main...HEAD: 38 files, +717 −704. Shipping code (netd, toyos-i219's lib.rs and regs.rs, toyos-tco) +23 −63; toyos-i219's model and tests +2 −57; src/ production +112 −14 and its tests +109 −58; tests/ +169 −351; issues/ +302 −161.

Between 75dc056f1 and this head

git diff --stat 75dc056f1 e3094a94a:

 ...-talking-boots-reboot-outruns-its-log-stream.md | 45 ----------------------
 ...ds-i219-drops-a-transmit-burst-past-its-ring.md |  5 +--
 src/metal.rs                                       | 14 +------
 3 files changed, 4 insertions(+), 60 deletions(-)

Test code and issues only. Both hunks in src/metal.rs lie inside its #[cfg(test)] mod tests, in coming_back_is_ssh_answering_and_not_its_port. Nothing the loop does at run time changed, nor anything the image or the guest suite builds, so the T14 readings, the image and the guest suite at 75dc056f1 stand for this head.

Every QEMU ceiling is at most three times what its test takes

The rule is stated once, at tests/common/qemu.rs's budget_smp: a ceiling is at most three times the slowest its test was measured to take, and it is a guard against a wedge, never a verdict on duration.

• The width no longer multiplies a ceiling. WIDTH, set_width and the free budget are deleted. At the default 12 wide every number in the source was twelve times itself: virt_irq_storm's 180 s was 36 minutes.
• The measurement is five whole runs of cargo test --test toyos-build at bf28c1e38, 12 wide, on the dev host at load average 40 to 59 on 14 cores (guest-m2.log to guest-m6.log): 26, 25, 26, 26 and 26 passed in 229.2, 66.8, 37.1, 32.9 and 34.1 s. A wait is bounded by three times its whole test, the one number the suite prints.

ceiling	was	is	the tests that wait through it, slowest pass in s
BOOT_CEILING	10 s × width, 120 s	63	nested_nmi_is_loud 21, the slowest test that is one boot and nothing after it
GUEST_WEDGED, await_guest's total, now paid out through QemuInstance::budget	300	141	iommu_virtio_platform 47, the slowest of the sixteen tests that wait through await_guest
virt_selftest's drain	180	36	virt_irq_storm 12, virt_timer_floor 9
virt_user_mode's drain	180	30	10
virt_early_panic's two waits	30, 10	6, 6	2
virt_early_fault's two waits	30, 10	6, 6	2
screen_fatal_behind_a_painter's second wait	20	87	29
screen_fatal_halt_composited's first wait	30	96	32
screen_panic_muted's wait	30	30	19
virt_el2_drop's drain	10	10	7
screen_fatal_behind_a_painter's first wait	30	30	29
screen_fatal_halt_composited's second wait	40	40	32

• The two raised waits were each under their own test's slowest run (20 s against 29 s, 30 s against 32 s), so how much room the wait itself had is unmeasured; they are at the rule's bound. Every measured run passed inside the old numbers too.
• The four kept ones are at or above their test's slowest run and under three times it, and every measured run passed inside them.
• REFERENCE_BOOT_MS is 1424, the largest fastest boot of the five runs (1419, 1424, 1048, 1055, 750 ms), so the measuring host pays every ceiling at 1×.
• The backstop of ceiling_verdict is (ceiling * 2).max(ceiling + GUEST_QUIET), and a kernel that died is named at it. On main it was ceiling.max(GUEST_WEDGED), which gave GUEST_WEDGED two readers: this floor and await_guest's total. GUEST_WEDGED is await_guest's alone now. On today's suite only --debug reaches ceiling_verdict, through run_test; their deletion is filed (Issues, below).
• VERDICT_POLL (100 ms) is the read loop's poll, and the walk in tests/checks/qemu.rs that holds the backstop steps by it, so the walk moves with the poll.
• Unchanged: GUEST_QUIET (15 s): the silence that makes a wait a stall, the wait for QEMU's own stop after a guest's last word, and virt_fatal_halts_the_others_first's wait for the other vCPUs to halt. Also the QMP plumbing's bounds, pacing drains and --debug's 60 s.
• On CI's runners the same tests took 1 to 9 s in three of CI's guest / suite runs (37064140924, 37063415370, 37061831501; 1 wide, 4 cores), each at or under the dev host's slowest. virt_timer_preempts printed 76 to 100 s there: its time includes the build of the job's binary, which no ceiling covers.

Wedges end in seconds

• QEMU, measured. The wedge is wedge-irq-storm.patch (in the negative-control comment): the irq-storm selftest loses every tick, so its flood never ends and it says nothing. Its test's verdict is the ceiling alone.

  arm	command	result
  75dc056f1, alone	cargo test --test toyos-build -- virt_irq_storm	EXIT=1, FAIL virt_irq_storm: irq-storm never reported, 37 s
  75dc056f1, whole suite 12 wide	cargo test --test toyos-build	EXIT=1, 25 passed and that one red at 38 s, 45.4 s in all
  the whole change reverted onto origin/main at 73282fa93, alone (638-r5/at-head/mutations/wedge-main.log)	cargo test --test toyos-build -- virt_irq_storm	EXIT=1, the same red at 181 s

  Alone the width is 1. In a whole run main multiplies the 180 s by 12: 2160 s (derived, not run).

• The T14. toyos_tco::STAGED_BOUND_MS is 10 s, and metal::bound_for arms the three images that stage their own wedge with it (deadlinewedge, hardlockup, usbload) where every other image keeps WEDGE_BOUND_MS, 120 s. The lockup bound is half of it, 5 s. The full T14 run at b2ccd1c84 passed all three under this bound.

Two LAN boots ride the talking boot

• lan_dhcp_lease and lan_message_delivery ride tests/lantalkcase, which now names the I219's PCI function, so the loop still pings the address Ubuntu held. tests/lancase and tests/lanicscase are deleted with their ALL_CONFIGS rows and lanicscase's machine-record rows.
• --provoke-message is deleted from netd, from toyos-i219 (provoke_message, its two tests, the model's ICS register) and from src/build.rs's gate, which is one flag now. At b2ccd1c84 the T14 recorded pcidev: slot 0 took its first message on vector 0x28 on lantalkcase with no actuator armed, which was the exit of the lanicscase issue.
• tests/lanleasecase stays. netd keeps --exit-with-lease for it, and deleting the probe takes userland/netd/src/report.rs, toyos_i219::lease and toyos_i219::phy::Outcome with the driver tests that assert its codes: the I219 driver crate, past this brief. issues/diagnostics/netds-lease-probe-answers-a-question-its-lines-already-answer.md names the deletion as its exit.
• So lan_hold keeps its name: it still holds lanleasecase and dump_nmi_probe's boot.

The T14 loop

• wait_on polls ssh's port once a second and asks ssh true only once the port accepts. It slept 5 s before every probe.
• Going down is a dial given CONNECT_SECS (10 s), what ssh's own ConnectTimeout gives, so a SYN lost while Ubuntu is still up does not read as down; coming back, a dial gets one poll.
• SharedBoot::members follows the runner's bound. toyos_tco::RUST_MEMBER_MS (860) and C_MEMBER_MS (260), beside JOB_BOUND_MS, are what one shipping Rust member and one C case are allowed of it, and metal::members_fitting gives shared and ccorpus as many as fit in JOB_BOUND_MS less a tenth: 62 and 207, where main has 38 and 90. Each allowance is twice the slowest per-member mean of five full T14 runs taken before The guest suite keeps the 21 tests only a booted machine answers; the rest are metal, host or tracked #660 changed the lists. shared-debug keeps main's 18: its list is five members.
• readdir_bound's /home arm is deleted: on the file servers read_dir("/home") answers NotFound at any count, so 32,769 creates proved nothing. Filed as issues/filesystem/listing-home-answers-not-found-and-its-bound-has-no-guest-test.md.
• The machine record gains boot.shared.* (1154 / 3608 / 21429, the T14's reading at b2ccd1c84) and loses the rows of ccorpus-2 and lanicscase.

Carried from #609

• metaltalk::hand_back fires reboot once the stream has carried this boot's Boot: complete, or after CARRIED_WAIT (30 s). This is the exit of issues/build/the-talking-boots-reboot-outruns-its-log-stream.md, deleted here (Issues, below).
• metal::said_refusal: a failed toyos-metal run's verdict carries the refusal it ended on. Since A test and a build each say when they start and how long they took, the suite's last line splits building from testing, and every statement on stderr opens with the UTC time of day #680 the driver's statement opens with the printer's time-of-day stamp, so the reader finds REFUSAL_HEAD past it (printer::unstamped) on the statement's first line, the one the stamp is on.
• issues/hardware/netds-i219-drops-a-transmit-burst-past-its-ring.md (Issues, below).

Issues

• Closed: the-lanicscase-boot-is-a-second-t14-flash-for-one-question.md (its exit, above) and the-cable-judge-reads-three-netd-records-that-cannot-arrive-on-the-t14.md (netd's lease record reached the stick at b2ccd1c84), and the-talking-boots-reboot-outruns-its-log-stream.md: its exit, two consecutive T14 runs passing lan_talk, is the runs at 4859e8d71 (comment 5962753432) and 75dc056f1 (comment 5966698095), both with hand_back. Its rule is hand_back's doc, and its one citation, in issues/hardware/netds-i219-drops-a-transmit-burst-past-its-ring.md, goes with it; git grep finds neither its path nor its slug.
• Renamed: the lanleasecase issue, whose slug said a third flash.
• Filed here, each with its owner:
  • issues/build/a-suites-first-run-after-the-fork-pin-moves-races-its-own-checkout.md (the first run after the merge: EXIT=1, 18 of 26 red, guest-m1.log), held by the toolchain stage of issues/build/the-tooling-is-a-review-prompt-and-three-workflows.md. A test and a build each say when they start and how long they took, the suite's last line splits building from testing, and every statement on stderr opens with the UTC time of day #680 filed the same unlocked fork_checkout reddening a new worktree's first run, issues/build/a-worktrees-first-wide-run-reds-on-the-fork-checkout-it-is-still-making.md, which this one names.
  • issues/kernel/virt-mask-windows-read-nine-censuses-and-eight-windows-lines-for-one-cpu.md (the one red of the five measuring runs, on main's own test, at load average 43 to 59), held by the windows instrument's author.
  • issues/build/run-test-and-its-ceiling-verdict-serve-only-debugs-run.md: The guest suite keeps the 21 tests only a booted machine answers; the rest are metal, host or tracked #660's cut left run_test, run_test_hooked, run_test_paced and ceiling_verdict one caller, --debug's run; held by issues/build/the-guest-suite-runs-only-what-no-cheaper-tier-reaches.md.
  • issues/build/host-scale-reads-host-speed-off-boots-that-stop-at-different-markers.md: on the dev host, one run after the other at e5daa61ff, virt_early_panic alone (HVF, to EARLY PANIC:) read a fastest boot of 479 ms and virt_smp alone (TCG, to SCTLR_EL1=) 1180 ms, both paid at 1.00×, and machine_shutdown alone (x86-64, TCG, to ===READY===) 3316 ms, paid at 2.33×: the scale reads which boots a run held; held by the tooling track.
  • issues/hardware/netds-i219-drops-a-transmit-burst-past-its-ring.md, held by stage 2 of issues/hardware/the-lan-is-not-yet-production-grade.md.
  • issues/build/hard-lockup-bound-ms-is-read-by-nothing-but-its-own-assertion.md, held by issues/hardware/the-t14-boots-toyos-unattended.md, the track that holds the detector and its bound.

Gates

head	command	exit	log
e3094a94a	cargo run -- --ci host	0, "Host: 67 step(s), all green"	638-r6/host.log
e3094a94a	cargo test --test toyos-checks	0, 31 passed	638-r6/checks.log
75dc056f1	cargo run -- --build-only	0	638-r5/at-head/build-only.log
75dc056f1	cargo test --test toyos-build	0, 26 passed in 44.5 s, 12 wide from load average 3.8	638-r5/at-head/guest.log

The last two stand for this head (Between 75dc056f1 and this head, above). Logs are under /Users/jan/.claude/jobs/2280e09e/tmp/scratchpad/orch/; 4859e8d71's and the measuring runs' under …/orch/638-recut/.

Negative controls

Each is a checked patch, built (--no-run, EXIT=0), run, and reversed, with the tree clean after it (638-r5/at-head/mutations/run.log). The patches and their run at 75dc056f1 are in comment 5963627444. The one whose test this round changed, wait_on with let answered = listening;, ran again at e3094a94a (638-r6/mutations/run.log): unmutated EXIT=0, built EXIT=0, mutated EXIT=101 at the first assertion (left: Ok(0), right: Err(Silent { what: "come back", secs: 1 })), tree clean before and after.

patch	test it reds	exit
hand_back without its wait	the_hand_back_waits_for_the_boot_record_the_judge_reads	101
wait_for_boot answering at once	the same	101
said_refusal keeping its first line only	a_refusal_is_read_back_off_the_drivers_stderr	101
said_refusal looking for REFUSAL_HEAD at the start of the stamped line	the same	101
wait_on with let answered = listening;	coming_back_is_ssh_answering_and_not_its_port	101
bound_for arming every image with 120 s	every_arm_that_stops_this_machine_is_cleared_and_judged_as_one	101
backstop ceiling + GUEST_QUIET - 1 s	serial_vocabulary: "silent from 4.2s under a 5s ceiling"	101
backstop ceiling + 6 s	serial_vocabulary: "silent from 0ns"	101
backstop ceiling * 2	serial_vocabulary: "silent from 0ns"	101
backstop ceiling.max(300 s), main's	a_stall_stays_red	101
no death named at the backstop	serial_vocabulary: "a kernel death at 5.2s"	101

Oracles

• Ceilings: the five whole-suite logs, and three of CI's on other hardware.
• The LAN fold and hand_back: the T14's own records at 4859e8d71 (below) and at b2ccd1c84.
• The staged bound and the shared boots' sizes: the T14 in Drive mode at 75dc056f1 (below).
• The port poll: a real loopback listener, reached through the port_accepts the loop dials the T14 with.
• said_refusal: the T14's own stderr from metal: the LAN lease rides the talking boot, and two T14 flashes go #609's run at 7d2e15c0, with the printer's stamp A test and a build each say when they start and how long they took, the suite's last line splits building from testing, and every statement on stderr opens with the UTC time of day #680 puts on the driver's statement.

The T14

Read at 4859e8d71 (comment 5962753432)

The orchestrator ran three boots from the staged request, each image's sha256 checked before it was flashed, each toyos-metal exit 0, then the two judges from the clean worktree at that head. No row armed the TCO watchdog, a staged wedge or the hard-lockup probe.

boot	image sha256	toyos-metal
lantalkcase (--nic 0000:00:1f.6 --talk …)	c01bdbdda51f554281b3729a9519689e020d57094720699a25a5634c67682814	exit 0; talk: the stream carried `Boot: complete` (1153 ms), 822 ms after it opened
lanleasecase (--nic 0000:00:1f.6)	5efcceb3e0455de666fe68617cf8f6b4b72fbb4a88069b926ea57acf101970a7	exit 0
testcases-readdir	b51844becf2b8f8704067c1afde7dc4969082dc0f1f00092ba8e9cc7a61e62aa	exit 0

• cargo test --test toyos-build -- --metal --metal-readback <dir> lan_, judged: exit 1, 3 passed, 1 failed, 2 boot(s). PASS lan_talk (415 lines over the cable, Boot: complete among them in the stick's order; echo answered byte for byte, status 0, 73 ms after the stream opened; reboot taken). PASS lan_message_delivery (slot 0 on vector 0x28 at 1.193 s, its first message at 8.485 s). PASS lan_lease_report (netd exit 83, leased 192.168.1.48/24, 5 sent, 14 received). FAIL lan_dhcp_lease, main's red, both findings the one issues/hardware/the-benchs-router-leases-toyos-another-address-than-ubuntu.md records: the boot leased .48, the host pinged .46, and that ping was answered on the far side of the reset.
• … readdir_bound, judged: exit 0, PASS readdir_bound, 1 passed, 0 failed, 1 boot(s).

Read at 75dc056f1, Drive mode (comment 5966698095)

The owner approved these, watchdog rows included. The orchestrator ran each as cargo test --test toyos-build -- --metal [filter] from the clean worktree at the head, without --metal-readback, the stick's log partition saved before each run's first flash; no judging left a row in tests/metal.

run	exit	read
boot_deadline_ends_a_wedge	0	the boot deadline expired: a bound of 10000 ms, reached at 10062 ms; back in 51 s; PASS
hard_lockup_ends_a_deaf_cpu	0	cpu7 has taken no interrupt for 5000 ms, with IF clear at every sample in that span. Its bound is 5000 ms.; back in 59 s; PASS
usb_reset_records_the_phase_it_cut	0	deadline at 10066 ms; the controller had that device's data endpoint Running with 252 TRB(s) it had not reached on the ring; back in 54 s; PASS
unfiltered	1	270 passed, 2 failed, 25 boot(s) in 2273 s

• The shared boots at this head's sizes, each one boot with every member run: shared 62 (61 ms per member), shared-2 19 (26 ms), shared-debug 5 (31 ms), ccorpus 136 (25 ms), no ccorpus-2.
• PASS in the unfiltered run: the three rows above again, loader_watchdog_arms, watchdog_fed, lan_talk, lan_message_delivery, lan_lease_report, readdir_bound, hda_tone.
• The two reds are main's and recorded, and there is no other: lan_dhcp_lease (issues/hardware/the-benchs-router-leases-toyos-another-address-than-ubuntu.md) and hda_client_stall (issues/audio/hda-client-stall-reads-one-resume-where-its-judge-wants-two.md).

Unsure

• The ceilings were measured on the dev host, where x86-64 guests and the EL2 guests are emulated. CI pays them out through host_scale, which reads the run's fastest boot; filed above, that is an early-panic boot on nearly any host, so a whole run pays 1× and an AArch64 guest on a 4-core runner is widened only by vcpus/cores. The three CI runs above say the margin is there today.
• A wait is bounded by three times its whole test because nothing measures the wait alone. The four kept waits and BOOT_CEILING have the least room: screen_fatal_behind_a_painter's first wait is 30 s against a 29 s test.
• The shared boots' allowances were measured on lists The guest suite keeps the 21 tests only a booted machine answers; the rest are metal, host or tracked #660 has since changed. One unfiltered run at 75dc056f1 read shared at 62 members in 61 ms per member; one boot is all that says so.
• The allowances sit in toyos-tco, beside the bound they divide, and only the harness reads them.
• The hard-lockup issue is held by the in-tree track that holds the detector. Tracks: the T14 runs its tests in sessions of a booted ToyOS, and every machine ships a watchdog and a hard-lockup detector #631, not landed, deletes that track for one that plans HARD_LOCKUP_BOUND_MS's reader, so its merge moves the citation.
• lan_dhcp_lease is main's red on the T14, so after the fold no green row says the T14 leased an address except through lan_talk and lan_lease_report.
• CI's guest / suite bound is 60 minutes against 18 m 11 s for the slowest of six of CI's gate runs, 3.3 times. It is untouched: the job builds the tree, and its TCG arm has no run to measure.

🤖 Generated with Claude Code

https://claude.ai/code/session_013UDZQ6fSKw14e4w2TKTRfm

[Japabu]

Orchestrator runs at b2ccd1c (logs orch/logs/638-*.log):

job	command	exit
whole	cargo test --test toyos-build --	0
T14 full	cargo test --test toyos-build -- --metal	1 — [metal] 260 passed, 3 failed, 23 boot(s), wall 2101 s

The same full T14 run on main-equivalent code (#630's re-record at 0d2dda6): 27 boots, wall 2899 s.

The three T14 reds:

• lan_dhcp_lease: "this boot leased 192.168.1.49 and the host pinged 192.168.1.46 …". This is main's red, filed as issues/hardware/the-benchs-router-leases-toyos-another-address-than-ubuntu.md.
• hda_client_stall: "soundd resumed 1 time(s) — the second stream did not find a suspended daemon". This is main's red.
• hda_tone: "soundd filled 104 period(s) the tone client had not covered, on a client that keeps its ring full". It passed at 0d2dda6 and at Stage 6 step 1: a device's handler posts its watch; irq_ring's UserDev and Audio go #634's fb0fc7b, and it was filed as a T14 underrun by Metal judges read the channel their line crosses on #616.

The run added three rows to the machine record, boot.shared.{complete_ms,panel_max_us,panel_us} = 1154 / 3608 / 21429. They are not committed yet; the patch is kept by the orchestrator.

[Japabu]

Round 1, head b2ccd1c84.

Readiness. CI host passed at this head (run 36739037448). The whole guest suite passed 498 of 498 (638-whole.log). The full T14 run gave 260 passed, 3 failed, over 23 boots.

Growth. +414 / −881 over 44 files:

• Shipping code (netd, toyos-i219's lib.rs): +11 / −67.
• The metal loop (src/metal.rs): +75 / −12. The port poll, bound_for and the wait after a refusal account for it, and I accept that growth.
• The rest is the harness, configs and issues.

BLOCKER

• tests/common/qemu.rs:564 — the backstop is now ceiling * 2, and it fires before the kernel-death arm and the silence arm, which both wait for GUEST_QUIET (15 s). That happens whenever the guest went quiet later than 2×ceiling − 15 s, and so at once for every ceiling under 7.5 s: the 5 s shared-member default, the 2 s C members, ipc_hostile_peer's 1 s and locale_detect_unrecognized's 2 s.
  • What goes wrong: a kernel panic or a silent wedge in any of them now reports timed out after 10s, with the guest still talking 10s ago … it was working and did not finish, and the summary counts it under "N of those reds are the ceiling". The panic is not named.
  • Before this branch: the 300 s floor named the panic first, unless it came after 285 s.
  • Patch: let backstop = (ceiling * 2).max(ceiling + GUEST_QUIET);
  • Test: add to tests/checks/qemu.rs ceiling_self_check. It is red at b2ccd1c84 and green with the patch:
    • ceiling_verdict(Some(KERNEL), Duration::from_secs(11), Duration::from_secs(5), Duration::from_secs(10), 40) must be None;
    • ceiling_verdict(Some(KERNEL), Duration::from_secs(21), Duration::from_secs(5), GUEST_QUIET, 40) must contain kernel panic.
  • With the patch, tests/CLAUDE.md:18 ("a wedge verdict needs both the budget spent and the guest gone quiet … only a far backstop") is true again.

NOTE

• hda_tone (question 1): not this branch.

  • The testcases boot is the same on both runs: 8 jobs in the same order, and no shared or LAN work rides it.
  • The tone's own window is clean here too: underruns=0 deferred=0 starve_max=0 at 4.456 s.
  • The 104 are 39 + 65 from hda_client_stall's staged session. Its client 1 connected at 5.657 s, before soundd removed the tone's client 0 at 5.661 s. So the tone never flushed a clients=0 line, and job_window (tests/common/audio.rs:13) ran on into the next job.
  • On 630r4 the removal (6.234 s) came before the next connect (6.241 s). This is the same 39 + 65 the issue file records at f5c80264f: a race in the judge's window, not starvation. hda_client_stall's red here is the same race.

• Four waits still bounded by GUEST_WEDGED (300 s) (question 2). Until they are cut, the rule stated at tests/common/qemu.rs:303 and the PR title are false. Every other unchanged literal I checked is within three times its test.

  site	test	slowest of the six logs	cut to about
  tests/common/lan.rs:408	lan_no_lease	33 s	99 s
  tests/common/volumes.rs:563	kernel_log_file	18 s	54 s
  tests/common/usb.rs:821	usb_flush_optional	33 s	99 s
  tests/common/orphan.rs:42	guest_dies_with_its_harness	8 s	24 s

• tests/toyos.rs:11132 — readdir_bound's 545 s is three times the old shape. At this head the test took 49 s (638-whole.log), so re-derive it now.

• tests/toyos.rs:7800 — Liveness::new(40 s quiet, 35 s total): the quiet guard can never end the wait first, so the 40 s has no effect.

• toyos-i219/src/stub.rs:1165, :1044 and toyos-i219/src/regs.rs:35 — nothing writes ICS since provoke_message went. Delete the model's write arm, its read arm and the constant, so the --provoke-message deletion is whole (question 3).

• src/build.rs:3350 — INTEL_ACTUATORS: [&str; 1] keeps array-and-find machinery for one flag. One const and one contains would do.

• src/metal.rs:874 — STAGED_BOUND_MS is a bound a boot arms. toyos-tco's description says it holds "every bound a boot arms", so it belongs there beside WEDGE_BOUND_MS.

• tests/common/metal.rs:1038 — unbooted is true for any failed exit of the flashing invocation. That includes a boot that ran and was judged red, although the comment says "refused". In that case SIGKILL can cut a --swap that is still writing its readback.

• src/metal.rs:1450 — no host test reaches wait or port_accepts. The mutation let answered = listening; (no ssh true on come-back) survives every host test. The 23 T14 returns are its only evidence.

• Deletions (question 3).

  • The /home arm of readdir_bound is deleted whole.
  • netd's --exit-with-lease still ships for one live QEMU check, lan_lease_report's flap. It is tracked in issues/diagnostics/netds-lease-probe-answers-a-question-its-lines-already-answer.md with an owner and an exit condition, so it is not kept just in case.

• Prompts (question 4): no CLAUDE.md and no .claude/agents/*.md names --provoke-message, lancase, lanicscase, lanleasecase, set_width, round_trip or POLL_SECS. Nothing is owed.

• Machine record (question 5): commit the three boot.shared.* rows (1154 / 3608 / 21429) in this PR.

  • They are this head's first passing reading of shared, a boot whose list this branch moves from 38 to 62 members.
  • metaltimings::judge wrote them only because shared passed and had no row, and the harness says "commit it".
  • Landing without them leaves a record that this head's own run says is behind.

• CI's guest shards were not run under the cut ceilings (--jobs 1, KVM; question 2). nightly.yml has workflow_dispatch, so one dispatch on wt/toyos-tight measures it before landing.

REMOVE

• tests/common/qemu.rs:562-563 — "at most six times the slowest this test was ever measured": false for readdir_bound (1090 s against 49 s), and it restates the rule.
• tests/common/qemu.rs:4668 — "the slowest of which took 10 s": where a measurement came from goes in the commit, not the source.
• tests/common/power.rs:1071 — "the slowest of which took 37 s": same.
• src/metal.rs:870 — "the T14 stages its wedge 1.5 s into the kernel": a measured number that will rot.
• tests/toyos-rust-tests/src/bin/lan_hold.rs:1-4 and :9-10 — the cable, netd's records and the lease bound are all false for its one remaining user, testcases-deaf.
• tests/toyos.rs:261 — the rewritten comment restates LEASE_BOUND_MS as "twenty seconds".
• issues/hardware/the-t14-stopped-answering-ssh-between-two-lan-boots.md:9-10 — "are the two arms the metal suite flashes after lanicscase" is false.
• issues/hardware/the-cable-judge-spends-two-premises-nothing-has-measured.md:26-27 — "Closed by a lancase run" names a boot that no longer exists.
• PR body: "Runs owed", "Not run: any guest test, and the T14" and "No number is added; the next full T14 run records every boot the record lacks" are stale now that both runs happened.
• PR body: "Corrections to this branch's own record" is not main's record.
• PR body: the 20-slowest table's "kept / Not cut blind" rows carry nothing.

SEND BACK

[Japabu]

Orchestrator run at edefd81: cargo test --test toyos-build -- EXIT=0 — test result: ok. 502 passed, 502 total (309.7s) (log orch/logs/638r2-638r2-whole.log). Nightly on the branch: https://github.com/ToyOSOrg/ToyOS/actions/runs/36753172688 (running).

[Japabu]

Round 2, head edefd8193.

Round 1 BLOCKER

• tests/common/qemu.rs:564 (the backstop) is OPEN, but narrower.
  • The patch I prescribed is at :563. Case (e)1 fails at b2ccd1c84's ceiling * 2 (EXIT=101) and passes with the patch (EXIT=0), so a kernel death before the ceiling is now named.
  • A death after the ceiling is still not named. My round-1 patch did not cover the whole defect it described; BLOCKER 1 below is the rest.

Readiness

• CI host passed at edefd8193 (run 36753095781, 15m14s).
• The whole guest suite passed at edefd8193: EXIT=0, 502 passed (orchestrator comment).
• The T14 reading is from b2ccd1c84. Since then the metal loop has changed only in a refactor of wait, a same-value move of STAGED_BOUND_MS, and the swap's failure path.

Growth

• Whole branch: +520 / −948.
• Shipping code (netd, toyos-i219 lib.rs/regs.rs, toyos-tco): +18 / −69.
• Since b2ccd1c84: +161 / −122 over 20 files.

BLOCKER

• tests/common/qemu.rs:563-570 — a kernel that dies after its ceiling is still reported as a slow guest, and it is counted as a ceiling red.
  • Example: a 5 s ceiling and a death at 7 s gives "timed out after 20s, with the guest still talking 13s ago … it was working and did not finish". On main, the 300 s floor named that panic at 22 s.
  • The comment at :561-562 says "so a death is named by the arms above". That is false for this case.
  • Patch: in the backstop arm, before the TIMED_OUT format, add if let Some(line) = dying { return Some(kernel_died_here(line)); }.
  • Test: the death walk below. It fails at edefd8193 on died = 7 and passes with the patch.
• tests/checks/qemu.rs:201-220 — case (e) cannot fail on the claim :563 makes.
  • To the brief's question: case (e)1 alone does not prove the fix. It only requires a backstop of at least 11 s under a 5 s ceiling. The claim is that the backstop is at least ceiling + GUEST_QUIET.
  • The implementer is right about (e)2: it tests the order of the arms, not the backstop.
  • Mutation that passes today: let backstop = (ceiling * 2).max(ceiling + Duration::from_secs(6));. It passes:
    • (e)1, because 11 s is not past 11 s;
    • (e)2, because the panic arm is checked first;
    • every other ceiling_self_check case, whose ceilings are 153 s and 380 s;
    • a_stall_stays_red, where 30 s gives a 60 s backstop.
  • Under that mutation, a guest silent from 0 s under a 5 s ceiling reads "timed out after 11s" instead of STALLED at 15 s. That is round 1's defect back.
  • Replace (e) with two walks:

    const SHORT: Duration = Duration::from_secs(5);
    let first = |dying: Option<&str>, from: u64| {
        (from..).map(Duration::from_secs).find_map(|elapsed| {
            ceiling_verdict(dying, elapsed, SHORT, elapsed - Duration::from_secs(from), 40)
        })
    };
    for since in 0..=SHORT.as_secs() {
        let got = first(None, since);
        if !got.as_deref().is_some_and(|v| v.starts_with(STALLED)) {
            return Err(format!("silent from {since}s under a {SHORT:?} ceiling: {got:?}"));
        }
    }
    for died in 0..=4 * SHORT.as_secs() {
        let got = first(Some(KERNEL), died);
        if !got.as_deref().is_some_and(|v| v.contains("kernel panic")) {
            return Err(format!("a kernel death at {died}s under a {SHORT:?} ceiling: {got:?}"));
        }
    }

  • What each walk catches:
    • The silence walk passes at edefd8193. It fails under the mutation above and under b2ccd1c84's ceiling * 2, both on since = 0.
    • The death walk fails at edefd8193 on died = 7, and passes with BLOCKER 1's patch.
    • Together they also catch the backstop moved ahead of the panic arm, which is what (e)2 guarded.

NOTE

• Nightly 36753172688 at edefd8193: guest (1)–(12), tcg and host must be green before landing.
  • The guest shards are the only run of the cut ceilings on CI's runners, paid out through host_scale.
  • tcg runs process_stats's new 11 s ceiling under TCG, and its own bound drops to 54 min.
  • host is the new 42 min bound, and it runs the new loopback test on macos-latest.
  • build is already green. portability-linux and portability-macos are build-only, and nothing in this branch moves them.
• tests/common/metal.rs:1040 — the implementer's named gap is not a defect.
  • run returns an exit-2 refusal after the reboot only after its own come-back wait (src/metal.rs:2090-2096), so the ToyOS boot the swap dials is already over.
  • The label's refused entry is the loop's either way: the booted insert at :1062 replaces the swap's at :1052.
  • Child::kill on a swap that has already exited keeps its real status.
• tests/toyos-rust-tests/src/bin/lan_hold.rs:4 — its only user now is dump_nmi_probe's testcases-deaf (tests/toyos.rs:1726), which runs no netd.
  • Its name and its LEASE_BOUND_MS are netd's.
  • The hold should be named for what dump-deaf-cpu needs, and say so in its constant.
• PR body "Runs" carries only b2ccd1c84's readings. The head that lands has its own: CI host run 36753095781, and the whole suite at EXIT=0 with 502 passed. They belong in main's record.

REMOVE

• issues/diagnostics/the-cable-judge-reads-three-netd-records-that-cannot-arrive-on-the-t14.md — delete the whole file.
  • Its premise is false: netd's lease record reached the T14's stick at b2ccd1c84 ("this boot leased 192.168.1.49").
  • The boot and the keys it names, tests/lancase and lan.lancase.*, are deleted by this branch.

SEND BACK

[Japabu]

Round 3, head 1c81e3e66.

Round 2 BLOCKERs

• tests/common/qemu.rs:563-570, a kernel death after the ceiling reported as a slow guest: CLOSED.
  • The backstop arm now names dying (:565-567).
  • The PR body's control: edefd81's verdict code fails the death walk on died = 7 ("timed out after 20s, with the guest still talking 14s ago", EXIT=101), and 1c81e3e passes it (EXIT=0).
  • Checked by hand under a 5 s ceiling, where the backstop is 20 s: 7 s is the first death that fails at edefd81. A death at 6 s reaches the panic arm at 21 s, before the backstop fires.
• tests/checks/qemu.rs:201-221, case (e) unable to fail on the backstop claim: CLOSED.
  • The silence walk fails at since = 0 under ceiling + 6 s ("timed out after 11s") and under ceiling * 2 ("timed out after 10s"), each EXIT=101.
  • It passes at the head (EXIT=0).

Readiness

• CI host at 1c81e3e66: run 36756594838 succeeded. It printed "Host: 55 step(s), all green", checks::serial_vocabulary ... ok and checks::a_stall_stays_red ... ok.
• T14 at 1c81e3e66, --metal dump_nmi_probe: "[metal] 1 passed, 0 failed, 1 boot(s)".
  • test_rs_deaf_cpu_hold was spawned at 1.200 s. The dump ran at 11.251 s, cpu7 rejoined at 11.651 s, and the hold exited 0 at 21.201 s.
  • Source: the orchestrator's 638r3-T14-dump_nmi_probe.log. It is not on the PR.
• The whole guest suite last ran at edefd8193: 502 passed. The only change since then that a guest sees is the hold's new name, and rg lan_hold finds no reference left.

Nightly 36753172688 at edefd8193: neither red is this branch's.

• guest (10), screen_loader_lines: "the panel carried 26 rows at the GOP query and 41 at the loader's last line, a growth of 15 …". Main's scheduled nightly 36696295750, guest (8) (job 109873180506), failed with the same words, and so did wt/toyos-castore's 36736950844, guest (10).
• guest (7), blockd_survives_its_death: "role crash exited Some(1)". The guest's own blockd_io: FAIL /AFTER.BIN after the restart: Io came at 23.909 s, 2.7 s into the role.
  • No bound this branch cut shortened anything it waits on. The crash role's host ceiling is 251 s × host_scale (was 600) and was never approached. The boot passes no kernel parameter (boot(…, "blockd-death", &[])), so boot-deadline=8000/12000 and STAGED_BOUND_MS never reach it. blockd_io's crash path waits with no deadline.
  • The branch changes no line of blockd, fsd, the kernel or the crash role.
  • Its one change to that boot is BENCH_BLOCKS 8192 → 2048. That shrinks the bench partition, which lies after the FS partition. In the failing log the crash role's B2D4F6A8… at block 35072, 16384 blocks sits where it does on main.
  • The test passed elsewhere: at b2ccd1c84 (68 s) and at edefd8193 (61 s) in the orchestrator's whole runs, and on castore's nightly (58 s).

Growth

• Round 3: +36 / −66. Tests are +36 / −28 and issues −38.
• Branch: +540 / −998.
  • Shipping code (netd, toyos-i219, toyos-tco): +20 / −126.
  • src/: +145 / −76.
  • Tests: +270 / −665.
  • Issues: +103 / −129.

BLOCKER

None.

NOTE

• tests/checks/qemu.rs:205-209 — the walks step whole seconds, so a backstop one second short passes both.
  • Mutation: let backstop = (ceiling * 2).max(ceiling + GUEST_QUIET - Duration::from_secs(1));. At since = 5 the walk reaches 19 s at quiet 14, and 19 is not past 19. At quiet 15 the STALLED arm comes first.
  • The read loop polls every 100 ms (qemu.rs:3357). Under that mutation, a guest silent from any point in (4 s, 5 s] would read "timed out after 19s".
  • Patch: step since, died and quiet by Duration::from_millis(100). The silence walk then fails under the mutation from since = 4.2 s.
• tests/toyos-rust-tests/src/bin/deaf_cpu_hold.rs:7 — the boot is held 20 s for a dump that finished at 11.651 s on the T14. The judge reads the dump's own record, so this is not a flat wait on a verdict, but every full run pays about 9.5 s past the event.
• PR — the T14 reading at 1c81e3e66 and CI host run 36756594838 belong to the head that lands. Neither is in the body or in a comment.
• issues/filesystem/blockd-survives-its-death-reds-on-a-replacement-that-reorders-acknowledged-writes.md — it records the trace-order verdict, not "/AFTER.BIN after the restart: Io". That signature is recorded nowhere. Recording it falls to main, not to this branch.
• origin/main has moved past the merge base 6e1f1661a (issues: child processes — end events, a parent takes its children down, sessions and quit, with the owner's fourteen questions #617, Stage 6 step 1: a device's handler posts its watch; irq_ring's UserDev and Audio go #634, hda_tone: the judge refuses multi-stream sessions, and a dropped cpal stream waits a bounded time on soundd and then lets go #641). merge-tree is clean.
  • main's tests/toyos.rs:12182 (drain_until(Duration::from_secs(30), handler_post_said)) and the new shared member test_rs_cpal_drop_unreleased were written against the width-scaled budget.
  • Neither has a measurement under this branch's rule. No run has measured the merged head.

REMOVE

• PR body, "Unsure": "The nightly shards have run the whole suite since Delete the test tiers: a run is the whole suite or its filter #625, but none has run under these ceilings yet." — false: 36753172688 ran all twelve guest shards at edefd8193.
• PR body, "Unsure": "No T14 run has booted deaf_cpu_hold under its new name yet." — false: the orchestrator's run at 1c81e3e66 booted it.
• tests/checks/qemu.rs:201-203 — "from any point in its budget" and "at any point" claim more than a whole-second walk checks. The rest restates the loops below it.

LAND AFTER NAMED CHANGES

[Japabu]

Orchestrator runs at 1c81e3e:

• T14 cargo test --test toyos-build -- --metal dump_nmi_probe: EXIT=0 — [metal] 1 passed, 0 failed, 1 boot(s) (log orch/logs/638r3-T14-dump_nmi_probe.log).
• CI host run 36756594838: success.

[Japabu]

Mutation patches for the negative controls at 4859e8d71, each applied with git apply, built, run and reversed (run.sh, wedge.sh). The wedge's third arm reverts the whole change with git diff --binary HEAD origin/main (a93067fc2) before applying wedge-irq-storm.patch.

wedge-irq-storm.patch

--- a/kernel/src/arch/aarch64/trap.rs
+++ b/kernel/src/arch/aarch64/trap.rs
@@ -525,7 +525,7 @@
 
     pub(super) fn tick() {
         if RUNNING.load(Relaxed) {
-            TICKS.fetch_add(1, Relaxed);
+            TICKS.fetch_add(0, Relaxed);
         }
     }
 

hand-back-no-wait.patch

--- a/src/metaltalk.rs
+++ b/src/metaltalk.rs
@@ -977,7 +977,7 @@
 /// first line, so the backlog crosses at the network's pace and not the
 /// conversation's.
 fn hand_back<T>(stream: &Stream, began: Instant, by: Duration, fire: impl FnOnce() -> T) -> T {
-    match stream.wait_for_boot(by) {
+    match stream.wait_for_boot(Duration::ZERO.min(by)) {
         Some(ms) => println!(
             "  talk: the stream carried `Boot: complete` ({ms} ms), {} ms after it opened",
             began.elapsed().as_millis()

wait-for-boot-at-once.patch

--- a/src/metaltalk.rs
+++ b/src/metaltalk.rs
@@ -286,7 +286,7 @@
     /// This boot's `Boot: complete` in milliseconds, read as [`judge`] reads
     /// it, once a line carries it, or `None` after `by`.
     pub fn wait_for_boot(&self, by: Duration) -> Option<u64> {
-        self.wait_until(by, |lines| lines.iter().find_map(|line| crate::bootlog::boot_millis(line)))
+        self.wait_until(by.min(Duration::ZERO), |lines| lines.iter().find_map(|line| crate::bootlog::boot_millis(line)))
     }
 
     /// The peer, once a connection has carried a line, or `None` after `by`

said-refusal-first-line.patch

--- a/src/metal.rs
+++ b/src/metal.rs
@@ -110,7 +110,7 @@
 pub fn said_refusal(stderr: &str) -> Option<String> {
     let lines: Vec<&str> = stderr.lines().collect();
     let at = lines.iter().rposition(|line| line.starts_with(REFUSAL_HEAD))?;
-    Some(lines[at..].join("\n")[REFUSAL_HEAD.len()..].trim_end().to_string())
+    Some(lines[at..=at].join("\n")[REFUSAL_HEAD.len()..].trim_end().to_string())
 }
 
 /// Every way this loop refuses, by name.

wait-on-port-alone.patch

--- a/src/metal.rs
+++ b/src/metal.rs
@@ -1509,7 +1509,7 @@
     while began.elapsed().as_secs() < secs {
         let next = std::time::Instant::now() + POLL;
         let listening = accepts();
-        let answered = listening && (!answering || answers());
+        let answered = listening;
         if answered == answering {
             return Ok(began.elapsed().as_secs());
         }

bound-for-always-wedge.patch

--- a/src/metal.rs
+++ b/src/metal.rs
@@ -852,7 +852,7 @@
 
 /// The `boot-deadline=` bound an image armed with `armed` carries.
 pub fn bound_for(armed: &[impl AsRef<str>]) -> u64 {
-    if stages_a_wedge(armed) {
+    if stages_a_wedge(armed) && false {
         toyos_tco::STAGED_BOUND_MS
     } else {
         toyos_tco::WEDGE_BOUND_MS

backstop-one-second-short.patch

--- a/tests/common/qemu.rs
+++ b/tests/common/qemu.rs
@@ -517,7 +517,7 @@
     // For a guest that is stuck *and* chatty and so never trips the silence
     // guard; never before a guest silent since `ceiling` has been silent for
     // [`GUEST_QUIET`].
-    let backstop = (ceiling * 2).max(ceiling + GUEST_QUIET);
+    let backstop = (ceiling * 2).max(ceiling + GUEST_QUIET - Duration::from_secs(1));
     if elapsed > backstop {
         if let Some(line) = dying {
             return Some(kernel_died_here(line));

backstop-plus-six.patch

--- a/tests/common/qemu.rs
+++ b/tests/common/qemu.rs
@@ -517,7 +517,7 @@
     // For a guest that is stuck *and* chatty and so never trips the silence
     // guard; never before a guest silent since `ceiling` has been silent for
     // [`GUEST_QUIET`].
-    let backstop = (ceiling * 2).max(ceiling + GUEST_QUIET);
+    let backstop = (ceiling * 2).max(ceiling + Duration::from_secs(6));
     if elapsed > backstop {
         if let Some(line) = dying {
             return Some(kernel_died_here(line));

backstop-twice-only.patch

--- a/tests/common/qemu.rs
+++ b/tests/common/qemu.rs
@@ -517,7 +517,7 @@
     // For a guest that is stuck *and* chatty and so never trips the silence
     // guard; never before a guest silent since `ceiling` has been silent for
     // [`GUEST_QUIET`].
-    let backstop = (ceiling * 2).max(ceiling + GUEST_QUIET);
+    let backstop = ceiling * 2;
     if elapsed > backstop {
         if let Some(line) = dying {
             return Some(kernel_died_here(line));

backstop-mains.patch

--- a/tests/common/qemu.rs
+++ b/tests/common/qemu.rs
@@ -517,7 +517,7 @@
     // For a guest that is stuck *and* chatty and so never trips the silence
     // guard; never before a guest silent since `ceiling` has been silent for
     // [`GUEST_QUIET`].
-    let backstop = (ceiling * 2).max(ceiling + GUEST_QUIET);
+    let backstop = ceiling.max(Duration::from_secs(300));
     if elapsed > backstop {
         if let Some(line) = dying {
             return Some(kernel_died_here(line));

backstop-no-death.patch

--- a/tests/common/qemu.rs
+++ b/tests/common/qemu.rs
@@ -519,7 +519,7 @@
     // [`GUEST_QUIET`].
     let backstop = (ceiling * 2).max(ceiling + GUEST_QUIET);
     if elapsed > backstop {
-        if let Some(line) = dying {
+        if let Some(line) = dying.filter(|_| false) {
             return Some(kernel_died_here(line));
         }
         return Some(format!(

[Japabu]

T14 at 4859e8d71, run by the orchestrator from 638-recut/metal/request.txt: three boots, each image's sha256 checked against the request before it was flashed, each toyos-metal exit 0; then the two judges from the clean worktree at the head. No row armed the TCO watchdog, a staged wedge or the hard-lockup probe.

boot	image sha256	toyos-metal
lantalkcase (--nic 0000:00:1f.6 --talk …)	c01bdbdda51f554281b3729a9519689e020d57094720699a25a5634c67682814	exit 0; the loop printed talk: the stream carried Boot: complete (1153 ms), 822 ms after it opened
lanleasecase (--nic 0000:00:1f.6)	5efcceb3e0455de666fe68617cf8f6b4b72fbb4a88069b926ea57acf101970a7	exit 0
testcases-readdir	b51844becf2b8f8704067c1afde7dc4969082dc0f1f00092ba8e9cc7a61e62aa	exit 0

• lan_: exit 1, by main's known red alone, as the request predicted. PASS lan_talk (415 lines arrived over the cable, Boot: complete among them, in the stick's order; echo answered byte for byte with status 0, 73 ms after the stream opened; reboot taken), PASS lan_message_delivery (slot 0 handed over on vector 0x28 at 1.193 s, its first message at 8.485 s), PASS lan_lease_report (netd exit 83, leased 192.168.1.48/24, 5 sent and 14 received). FAIL lan_dhcp_lease with two findings, both the one issues/hardware/the-benchs-router-leases-toyos-another-address-than-ubuntu.md records: this boot leased 192.168.1.48 and the host pinged 192.168.1.46, and that ping was answered on the far side of the reset. 3 passed, 1 failed, 2 boot(s).
• readdir_bound: exit 0. PASS readdir_bound; 1 passed, 0 failed, 1 boot(s).

Not run, and owed: the five rows the request left out for the owner's OK (boot_deadline_ends_a_wedge, hard_lockup_ends_a_deaf_cpu, usb_reset_records_the_phase_it_cut, loader_watchdog_arms, watchdog_fed), and with them the unfiltered run that judges the shared boots at their new sizes.

Readbacks and judge logs: /Users/jan/.claude/jobs/2280e09e/tmp/scratchpad/orch/638-recut/ (metal/…, judge-lan.log, judge-readdir.log).

[Japabu]

Round 4, head 4859e8d71 (the re-cut), reviewed as a first review of this head.

Earlier BLOCKERs: none open. Round 3 closed round 2's two at 1c81e3e66. Round 3's NOTE is closed too: the walks at tests/checks/qemu.rs:199-217 now step 100 ms, and a backstop one second short reds serial_vocabulary (EXIT=101, mutations/run.log).

Evidence at 4859e8d71

• cargo run -- --ci host: EXIT=0, "Host: 67 step(s), all green" (host2.log, headed 4859e8d71).
• cargo test --test toyos-build: EXIT=0, 26 passed in 146.9 s (guest-head2.log, headed 4859e8d71).
• cargo test --test toyos-checks: EXIT=0, 31 passed.
• Ten negative-control patches: each EXIT=101 on the test the body names.
• The wedge: 37 s on the head alone, 47.6 s in the whole suite, 181 s on main's tree.
• T14 (comment 5962753432): lan_talk, lan_message_delivery, lan_lease_report and readdir_bound PASS.
• lan_dhcp_lease fails only on the router-lease red. Its first finding is the issue's words. Its second follows from it on a boot the host ends: .46 answered only after the reset.
• CI: the PR is a draft, so every job was skipped.

Growth: +648 / −677 over 36 files, net −29.

• Shipping (netd, toyos-i219 lib/regs, toyos-tco): +18 / −63.
• toyos-i219 model and tests: +2 / −57.
• src/ production: +136 / −20. src/ tests: +107 / −58.
• tests/: +160 / −351.
• issues/: +225 / −128.
• I accept the production growth for hand_back: the T14 carried Boot: complete 822 ms after the stream opened, and lan_talk passed. I accept it for bound_for and for the port poll. I do not accept it for the refusal-wait (NOTE).

Rulings on the brief

• Ceilings: the claim holds.
  • Each number checks against m2–m6: 21→63, 47→141, 12→36, 10→30, 2→6, 29→87 and 32→96.
  • The kept waits against their test's slowest run: 30 over 19, 10 over 7, 30 over 29, 40 over 32.
  • None became a duration verdict. An expiry only ends a wait whose red is a thing the guest never said (never reported, never paged, never took the screen).
  • Every kept or raised wait sits at or above its test's slowest whole run.
  • The measuring commit bf28c1e38 already ran without the width.
• lanleasecase stays: accepted. Folding it deletes --exit-with-lease, toyos_i219::lease (262 lines) and phy::Outcome with their tests. That is a high-risk driver change, and the renamed issue carries an owner and an exit.
• Tracks: the T14 runs its tests in sessions of a booted ToyOS, and every machine ships a watchdog and a hard-lockup detector #631's three citations: two hold.
  • toyos_tco::STAGED_BOUND_MS (toyos-tco/src/lib.rs:149) and the hard-lockup issue exist.
  • The allowances do not. The tree holds only their quotients, 62/67/207, as literals; 860/800/260 ms exist in this body alone.
• Off-task filings.
  • Both are filed in the present tense, with their evidence.
  • virt_mask_windows is a defect with an owner by role.
  • The fork race is tooling, which README's "the build system" allows. It names no owner.

The T14 readings this head owes

None of these ran at 4859e8d71.

1. Changed by this diff, and reset rows, so the owner's call:
   • The rows: boot_deadline_ends_a_wedge (deadlinewedge), hard_lockup_ends_a_deaf_cpu (hardlockup), usb_reset_records_the_phase_it_cut (usbload).
   • Their boot-deadline= goes from 120000 to 10000, and the lockup bound from 60 s to 5 s.
   • They were last read at b2ccd1c84, before main's kernel merges.
   • Each is one --metal <name> run of one boot; no shared member matches those names.
2. Changed by this diff, with no TCO and no staged wedge:
   • The boots: shared (62 of today's 81 shipping members), shared-2 (19), and ccorpus (every C case, up to 160, in one boot where main boots two).
   • shared at 62 has never booted on today's list. ccorpus in one chunk was read only at b2ccd1c84, under that day's kernel and libc.
   • The harness judges a member only in a run whose filter keeps it. So only an unfiltered --metal run judges these, and that run also boots everything in 1, plus loader_watchdog_arms and watchdog_fed.
3. Not owed by this diff:
   • loader_watchdog_arms and watchdog_fed: their image keeps 120000, as src/metal.rs:2851 asserts.
   • shared-debug: its 5 members make one boot at 18 or at 67.
   • Every other row: its image is unchanged, and the three boots at this head crossed the new wait_on both ways.
4. The owner's other route:
   • Reverting 62/67/207 to 38/18/90 removes 2.
   • Dropping STAGED_BOUND_MS and bound_for removes 1.
5. Whichever run closes 1 or 2 runs in Drive mode (--metal without --metal-readback). The three boots at this head were flashed by hand from request.txt, so drive (tests/common/metal.rs:805) has met no machine.

BLOCKER

• toyos-tco/src/lib.rs:149, src/metal.rs:854, tests/common/metal.rs:743 — the three staged-wedge images now end at 10 s with lockup at 5 s, and no T14 reading at this head judges them (T14 list, 1). This is a hardware claim QEMU cannot reach: hard_lockup_ends_a_deaf_cpu holds only while staging, plus 5 s, plus one sample still comes before the 10 s deadline on today's kernel. It closes with those three rows green on the T14 at 4859e8d71, or with the bound dropped.
• tests/toyos.rs:763, :853 — the T14's shared and ccorpus now run 62 and up to 160 members under the runner's 60 s JOB_BOUND_MS. The counts are priced from per-member means of lists The guest suite keeps the 21 tests only a booted machine answers; the rest are metal, host or tracked #660 has since replaced, and shared at 62 has never booted on today's list (T14 list, 2). It closes with shared, shared-2 and ccorpus green on the T14 at 4859e8d71, or with main's counts kept.

NOTE

• src/metal.rs:1512, :1523-1529 — "go down" is now one failed 1 s connect. A single lost SYN while Ubuntu is still up reads as down, then "come back" answers at once, and the loop reads a stick ToyOS never booted. That is exactly what ride_the_reboot's doc (:1362-1364) exists to prevent. Main's go-down was an ssh with ConnectTimeout=10; dial with CONNECT_SECS when going down.
• src/metal.rs:2025-2034 — the refusal-wait has no measured case. The body cites issues/hardware/the-t14-stopped-answering-ssh-between-two-lan-boots.md, but that file's recorded timeout came after lanicscase passed, a path this wait never takes. Name the refusal it is for (the reboot job's ssh dying with the machine) with a reading, or delete it along with after_the_reboot's split.
• tests/toyos.rs:763, :782, :853 — Tracks: the T14 runs its tests in sessions of a booted ToyOS, and every machine ships a watchdog and a hard-lockup detector #631 prices sessions from allowances that exist only in this body. Declare them beside JOB_BOUND_MS and derive members from them, so the counts follow the bound and Tracks: the T14 runs its tests in sessions of a booted ToyOS, and every machine ships a watchdog and a hard-lockup detector #631 has a name to cite. Separately, shared-debug's 18→67 moves nothing on a 5-member list.
• issues/build/a-suites-first-run-after-the-fork-pin-moves-races-its-own-checkout.md:29 — "Whoever next changes sysroot::fork_checkout" names nobody. The race reds every agent's first run after a pin moves (18 of 26 here), so it needs a holder now.
• issues/build/hard-lockup-bound-ms-is-read-by-nothing-but-its-own-assertion.md:22 — "Whoever next changes toyos-tco" is this branch (toyos-tco/src/lib.rs:144-149), which leaves the issue open. Name Tracks: the T14 runs its tests in sessions of a booted ToyOS, and every machine ships a watchdog and a hard-lockup detector #631's track, which plans the reader.
• issues/hardware/netds-i219-drops-a-transmit-burst-past-its-ring.md — no owner.
• issues/build/virt-mask-windows-read-nine-censuses-and-eight-windows-lines-for-one-cpu.md — its subject is the kernel's windows instrument, and its owner's track is in issues/kernel/; move it there.
• tests/checks/qemu.rs:201 — STEP restates the read loop's 100 ms poll (tests/common/qemu.rs:1571). Make it one const read by both, or the walk goes blind once the poll moves.
• tests/common/qemu.rs:493-532, :1474-1528 — on today's suite, ceiling_verdict, run_test_paced and run_test_hooked have one caller: --debug's run (tests/toyos.rs:3418). The branch reworks a backstop and its walks for that one path. File their deletion.
• tests/common/qemu.rs:172-198, :276-278 — the run's fastest boot is a virt_early_* boot whose ready marker EARLY PANIC: comes before userland (head2: 502 ms; virt_early_fault's whole test took 731 ms). So host_scale pays 1× on nearly any host, and with the width gone it is the only host correction budget_smp's doc promises. File it.
• PR body — "No T14 run has measured this head: not the fold, not hand_back, not the staged bound" is false. The body also carries none of the readings at this head from comment 5962753432. Keep the record true.

REMOVE

• src/metaltalk.rs:55 — "the stream stalls only while netd retransmits what its NIC dropped".
• src/metaltalk.rs:1475-1477 — "The lines are the T14's: … Boot: complete was 72 lines further on."
• PR body — the opening paragraph, "Re-cut onto main at a93067fc2. …".
• PR body — the section "Dropped in the re-cut".
• PR body — ", and the branch had deleted it".
• PR body — "metal: the LAN lease rides the talking boot, and two T14 flashes go #609's Driver and its reader thread existed for the swap beside it; main deleted the swap, so it is one function."

SEND BACK

[Japabu]

Negative controls at 4b009e712. Each patch is checked with git apply --check, applied, built (cargo test <target> --no-run, EXIT=0 each), run, and reversed, with the tree clean after each (mutations/run.sh and run.log under /Users/jan/.claude/jobs/2280e09e/tmp/scratchpad/orch/638-r5/). The wedge patch is run by mutations/wedge.sh.

patch	command	test it reds	exit
hand-back-no-wait	cargo test --lib -- the_hand_back_waits_for_the_boot_record_the_judge_reads	that test	101
wait-for-boot-at-once	the same	that test	101
said-refusal-first-line	cargo test --lib -- a_refusal_is_read_back_off_the_drivers_stderr	that test	101
wait-on-port-alone	cargo test --lib -- coming_back_is_ssh_answering_and_not_its_port	that test	101
go-down-dial-one-poll (new)	the same	that test: "going down dialled within [1s], where ssh's own connect is given 10s"	101
bound-for-always-wedge	cargo test --lib -- every_arm_that_stops_this_machine_is_cleared_and_judged_as_one	that test	101
backstop-one-second-short	cargo test --test toyos-checks	serial_vocabulary: "silent from 4.2s under a 5s ceiling"	101
backstop-plus-six	the same	serial_vocabulary: "silent from 0ns"	101
backstop-twice-only	the same	serial_vocabulary: "silent from 0ns"	101
backstop-mains	the same	a_stall_stays_red	101
backstop-no-death	the same	serial_vocabulary: "a kernel death at 5.2s"	101
wedge-irq-storm	cargo test --test toyos-build -- virt_irq_storm; then cargo test --test toyos-build	virt_irq_storm: "irq-storm never reported", at 37 s alone and 38 s in a 45.7 s suite	1; 1

hand-back-no-wait.patch

--- a/src/metaltalk.rs
+++ b/src/metaltalk.rs
@@ -977,7 +977,7 @@
 /// first line, so the backlog crosses at the network's pace and not the
 /// conversation's.
 fn hand_back<T>(stream: &Stream, began: Instant, by: Duration, fire: impl FnOnce() -> T) -> T {
-    match stream.wait_for_boot(by) {
+    match stream.wait_for_boot(Duration::ZERO.min(by)) {
         Some(ms) => println!(
             "  talk: the stream carried `Boot: complete` ({ms} ms), {} ms after it opened",
             began.elapsed().as_millis()

wait-for-boot-at-once.patch

--- a/src/metaltalk.rs
+++ b/src/metaltalk.rs
@@ -286,7 +286,7 @@
     /// This boot's `Boot: complete` in milliseconds, read as [`judge`] reads
     /// it, once a line carries it, or `None` after `by`.
     pub fn wait_for_boot(&self, by: Duration) -> Option<u64> {
-        self.wait_until(by, |lines| lines.iter().find_map(|line| crate::bootlog::boot_millis(line)))
+        self.wait_until(by.min(Duration::ZERO), |lines| lines.iter().find_map(|line| crate::bootlog::boot_millis(line)))
     }
 
     /// The peer, once a connection has carried a line, or `None` after `by`

said-refusal-first-line.patch

--- a/src/metal.rs
+++ b/src/metal.rs
@@ -110,7 +110,7 @@
 pub fn said_refusal(stderr: &str) -> Option<String> {
     let lines: Vec<&str> = stderr.lines().collect();
     let at = lines.iter().rposition(|line| line.starts_with(REFUSAL_HEAD))?;
-    Some(lines[at..].join("\n")[REFUSAL_HEAD.len()..].trim_end().to_string())
+    Some(lines[at..=at].join("\n")[REFUSAL_HEAD.len()..].trim_end().to_string())
 }
 
 /// Every way this loop refuses, by name.

wait-on-port-alone.patch

--- a/src/metal.rs
+++ b/src/metal.rs
@@ -1513,7 +1513,7 @@
     while began.elapsed().as_secs() < secs {
         let next = std::time::Instant::now() + POLL;
         let listening = accepts(dial);
-        let answered = listening && (!answering || answers());
+        let answered = listening;
         if answered == answering {
             return Ok(began.elapsed().as_secs());
         }

go-down-dial-one-poll.patch

--- a/src/metal.rs
+++ b/src/metal.rs
@@ -1508,7 +1508,7 @@
     mut accepts: impl FnMut(std::time::Duration) -> bool,
     mut answers: impl FnMut() -> bool,
 ) -> Result<u64, Refusal> {
-    let dial = if answering { POLL } else { std::time::Duration::from_secs(CONNECT_SECS) };
+    let dial = POLL;
     let began = std::time::Instant::now();
     while began.elapsed().as_secs() < secs {
         let next = std::time::Instant::now() + POLL;

bound-for-always-wedge.patch

--- a/src/metal.rs
+++ b/src/metal.rs
@@ -852,7 +852,7 @@
 
 /// The `boot-deadline=` bound an image armed with `armed` carries.
 pub fn bound_for(armed: &[impl AsRef<str>]) -> u64 {
-    if stages_a_wedge(armed) {
+    if stages_a_wedge(armed) && false {
         toyos_tco::STAGED_BOUND_MS
     } else {
         toyos_tco::WEDGE_BOUND_MS

backstop-one-second-short.patch

--- a/tests/common/qemu.rs
+++ b/tests/common/qemu.rs
@@ -517,7 +517,7 @@
     // For a guest that is stuck *and* chatty and so never trips the silence
     // guard; never before a guest silent since `ceiling` has been silent for
     // [`GUEST_QUIET`].
-    let backstop = (ceiling * 2).max(ceiling + GUEST_QUIET);
+    let backstop = (ceiling * 2).max(ceiling + GUEST_QUIET - Duration::from_secs(1));
     if elapsed > backstop {
         if let Some(line) = dying {
             return Some(kernel_died_here(line));

backstop-plus-six.patch

--- a/tests/common/qemu.rs
+++ b/tests/common/qemu.rs
@@ -517,7 +517,7 @@
     // For a guest that is stuck *and* chatty and so never trips the silence
     // guard; never before a guest silent since `ceiling` has been silent for
     // [`GUEST_QUIET`].
-    let backstop = (ceiling * 2).max(ceiling + GUEST_QUIET);
+    let backstop = (ceiling * 2).max(ceiling + Duration::from_secs(6));
     if elapsed > backstop {
         if let Some(line) = dying {
             return Some(kernel_died_here(line));

backstop-twice-only.patch

--- a/tests/common/qemu.rs
+++ b/tests/common/qemu.rs
@@ -517,7 +517,7 @@
     // For a guest that is stuck *and* chatty and so never trips the silence
     // guard; never before a guest silent since `ceiling` has been silent for
     // [`GUEST_QUIET`].
-    let backstop = (ceiling * 2).max(ceiling + GUEST_QUIET);
+    let backstop = ceiling * 2;
     if elapsed > backstop {
         if let Some(line) = dying {
             return Some(kernel_died_here(line));

backstop-mains.patch

--- a/tests/common/qemu.rs
+++ b/tests/common/qemu.rs
@@ -517,7 +517,7 @@
     // For a guest that is stuck *and* chatty and so never trips the silence
     // guard; never before a guest silent since `ceiling` has been silent for
     // [`GUEST_QUIET`].
-    let backstop = (ceiling * 2).max(ceiling + GUEST_QUIET);
+    let backstop = ceiling.max(Duration::from_secs(300));
     if elapsed > backstop {
         if let Some(line) = dying {
             return Some(kernel_died_here(line));

backstop-no-death.patch

--- a/tests/common/qemu.rs
+++ b/tests/common/qemu.rs
@@ -519,7 +519,7 @@
     // [`GUEST_QUIET`].
     let backstop = (ceiling * 2).max(ceiling + GUEST_QUIET);
     if elapsed > backstop {
-        if let Some(line) = dying {
+        if let Some(line) = dying.filter(|_| false) {
             return Some(kernel_died_here(line));
         }
         return Some(format!(

wedge-irq-storm.patch

--- a/kernel/src/arch/aarch64/trap.rs
+++ b/kernel/src/arch/aarch64/trap.rs
@@ -525,7 +525,7 @@
 
     pub(super) fn tick() {
         if RUNNING.load(Relaxed) {
-            TICKS.fetch_add(1, Relaxed);
+            TICKS.fetch_add(0, Relaxed);
         }
     }
 

[Japabu]

Negative controls at e5daa61ff (4b009e712 with origin/main at 322085f23, #648, merged in). The patches are the ones in comment 5963363857, unchanged; each is checked with git apply --check, applied, built (cargo test <target> --no-run, EXIT=0 each), run, and reversed, with the tree clean after each (at-merge/mutations/run.sh and run.log under /Users/jan/.claude/jobs/2280e09e/tmp/scratchpad/orch/638-r5/).

patch	command	test it reds	exit
hand-back-no-wait	cargo test --lib -- the_hand_back_waits_for_the_boot_record_the_judge_reads	that test	101
wait-for-boot-at-once	the same	that test	101
said-refusal-first-line	cargo test --lib -- a_refusal_is_read_back_off_the_drivers_stderr	that test	101
wait-on-port-alone	cargo test --lib -- coming_back_is_ssh_answering_and_not_its_port	that test	101
go-down-dial-one-poll	the same	that test: "going down dialled within [1s], where ssh's own connect is given 10s"	101
bound-for-always-wedge	cargo test --lib -- every_arm_that_stops_this_machine_is_cleared_and_judged_as_one	that test	101
backstop-one-second-short	cargo test --test toyos-checks	serial_vocabulary: "silent from 4.2s under a 5s ceiling"	101
backstop-plus-six	the same	serial_vocabulary: "silent from 0ns"	101
backstop-twice-only	the same	serial_vocabulary: "silent from 0ns"	101
backstop-mains	the same	a_stall_stays_red	101
backstop-no-death	the same	serial_vocabulary: "a kernel death at 5.2s"	101
wedge-irq-storm	cargo test --test toyos-build -- virt_irq_storm; then cargo test --test toyos-build	virt_irq_storm: "irq-storm never reported", at 37 s alone and 37 s in a 44.3 s suite	1; 1

The unmutated head: cargo test --lib over the four tests EXIT=0, cargo test --test toyos-checks EXIT=0 (31 passed).

[Japabu]

The wedge's third arm at e5daa61ff: the whole change reverted onto origin/main at 322085f23 with git diff --binary HEAD origin/main, then wedge-irq-storm.patch (comment 5963363857) applied, built (EXIT=0), cargo test --test toyos-build -- virt_irq_storm run, and both reversed with the tree clean after (at-merge/mutations/wedge-main.sh, wedge-main.log): EXIT=1, FAIL virt_irq_storm: irq-storm never reported at 181 s, against 37 s at the head.

[Japabu]

Negative controls at 75dc056f1 (1ba6b43e3 with origin/main at 73282fa93, #680, merged in; #648 came in at e5daa61ff). These patches replace the ones in comments 5963363857 and 5963458288: said-refusal-first-line is cut against the reader the #680 merge changed, and said-refusal-stamped is new. Each is checked with git apply --check, applied, built (cargo test <target> --no-run, EXIT=0 each), run, and reversed, with the tree clean after each (at-head/mutations/run.sh, wedge.sh, wedge-main.sh and their logs under /Users/jan/.claude/jobs/2280e09e/tmp/scratchpad/orch/638-r5/).

patch	command	test it reds	exit
hand-back-no-wait	cargo test --lib -- the_hand_back_waits_for_the_boot_record_the_judge_reads	that test	101
wait-for-boot-at-once	the same	that test	101
said-refusal-first-line	cargo test --lib -- a_refusal_is_read_back_off_the_drivers_stderr	that test	101
said-refusal-stamped (new)	the same	that test: left: None	101
wait-on-port-alone	cargo test --lib -- coming_back_is_ssh_answering_and_not_its_port	that test	101
go-down-dial-one-poll (new)	the same	that test: "going down dialled within [1s], where ssh's own connect is given 10s"	101
bound-for-always-wedge	cargo test --lib -- every_arm_that_stops_this_machine_is_cleared_and_judged_as_one	that test	101
backstop-one-second-short	cargo test --test toyos-checks	serial_vocabulary: "silent from 4.2s under a 5s ceiling"	101
backstop-plus-six	the same	serial_vocabulary: "silent from 0ns"	101
backstop-twice-only	the same	serial_vocabulary: "silent from 0ns"	101
backstop-mains	the same	a_stall_stays_red	101
backstop-no-death	the same	serial_vocabulary: "a kernel death at 5.2s"	101
wedge-irq-storm, at the head	cargo test --test toyos-build -- virt_irq_storm; then cargo test --test toyos-build	virt_irq_storm: "irq-storm never reported", at 37 s alone and 38 s in a 45.4 s suite	1; 1
wedge-irq-storm on the whole change reverted onto origin/main at 73282fa93 (git diff --binary HEAD origin/main)	cargo test --test toyos-build -- virt_irq_storm	the same red at 181 s	1

The unmutated head: cargo test --lib over the four tests EXIT=0, cargo test --test toyos-checks EXIT=0 (31 passed).

hand-back-no-wait.patch

--- a/src/metaltalk.rs
+++ b/src/metaltalk.rs
@@ -977,7 +977,7 @@
 /// first line, so the backlog crosses at the network's pace and not the
 /// conversation's.
 fn hand_back<T>(stream: &Stream, began: Instant, by: Duration, fire: impl FnOnce() -> T) -> T {
-    match stream.wait_for_boot(by) {
+    match stream.wait_for_boot(Duration::ZERO.min(by)) {
         Some(ms) => println!(
             "  talk: the stream carried `Boot: complete` ({ms} ms), {} ms after it opened",
             began.elapsed().as_millis()

wait-for-boot-at-once.patch

--- a/src/metaltalk.rs
+++ b/src/metaltalk.rs
@@ -286,7 +286,7 @@
     /// This boot's `Boot: complete` in milliseconds, read as [`judge`] reads
     /// it, once a line carries it, or `None` after `by`.
     pub fn wait_for_boot(&self, by: Duration) -> Option<u64> {
-        self.wait_until(by, |lines| lines.iter().find_map(|line| crate::bootlog::boot_millis(line)))
+        self.wait_until(by.min(Duration::ZERO), |lines| lines.iter().find_map(|line| crate::bootlog::boot_millis(line)))
     }
 
     /// The peer, once a connection has carried a line, or `None` after `by`

said-refusal-first-line.patch

--- a/src/metal.rs
+++ b/src/metal.rs
@@ -115,6 +115,6 @@
         .iter()
         .rposition(|line| crate::printer::unstamped(line).starts_with(REFUSAL_HEAD))?;
     let mut said = vec![&crate::printer::unstamped(lines[at])[REFUSAL_HEAD.len()..]];
-    said.extend(&lines[at + 1..]);
+    said.extend(&lines[at + 1..at + 1]);
     Some(said.join("\n").trim_end().to_string())
 }

said-refusal-stamped.patch

--- a/src/metal.rs
+++ b/src/metal.rs
@@ -113,7 +113,7 @@
     let lines: Vec<&str> = stderr.lines().collect();
     let at = lines
         .iter()
-        .rposition(|line| crate::printer::unstamped(line).starts_with(REFUSAL_HEAD))?;
+        .rposition(|line| line.starts_with(REFUSAL_HEAD))?;
     let mut said = vec![&crate::printer::unstamped(lines[at])[REFUSAL_HEAD.len()..]];
     said.extend(&lines[at + 1..]);
     Some(said.join("\n").trim_end().to_string())

wait-on-port-alone.patch

--- a/src/metal.rs
+++ b/src/metal.rs
@@ -1513,7 +1513,7 @@
     while began.elapsed().as_secs() < secs {
         let next = std::time::Instant::now() + POLL;
         let listening = accepts(dial);
-        let answered = listening && (!answering || answers());
+        let answered = listening;
         if answered == answering {
             return Ok(began.elapsed().as_secs());
         }

go-down-dial-one-poll.patch

--- a/src/metal.rs
+++ b/src/metal.rs
@@ -1508,7 +1508,7 @@
     mut accepts: impl FnMut(std::time::Duration) -> bool,
     mut answers: impl FnMut() -> bool,
 ) -> Result<u64, Refusal> {
-    let dial = if answering { POLL } else { std::time::Duration::from_secs(CONNECT_SECS) };
+    let dial = POLL;
     let began = std::time::Instant::now();
     while began.elapsed().as_secs() < secs {
         let next = std::time::Instant::now() + POLL;

bound-for-always-wedge.patch

--- a/src/metal.rs
+++ b/src/metal.rs
@@ -852,7 +852,7 @@
 
 /// The `boot-deadline=` bound an image armed with `armed` carries.
 pub fn bound_for(armed: &[impl AsRef<str>]) -> u64 {
-    if stages_a_wedge(armed) {
+    if stages_a_wedge(armed) && false {
         toyos_tco::STAGED_BOUND_MS
     } else {
         toyos_tco::WEDGE_BOUND_MS

backstop-one-second-short.patch

--- a/tests/common/qemu.rs
+++ b/tests/common/qemu.rs
@@ -517,7 +517,7 @@
     // For a guest that is stuck *and* chatty and so never trips the silence
     // guard; never before a guest silent since `ceiling` has been silent for
     // [`GUEST_QUIET`].
-    let backstop = (ceiling * 2).max(ceiling + GUEST_QUIET);
+    let backstop = (ceiling * 2).max(ceiling + GUEST_QUIET - Duration::from_secs(1));
     if elapsed > backstop {
         if let Some(line) = dying {
             return Some(kernel_died_here(line));

backstop-plus-six.patch

--- a/tests/common/qemu.rs
+++ b/tests/common/qemu.rs
@@ -517,7 +517,7 @@
     // For a guest that is stuck *and* chatty and so never trips the silence
     // guard; never before a guest silent since `ceiling` has been silent for
     // [`GUEST_QUIET`].
-    let backstop = (ceiling * 2).max(ceiling + GUEST_QUIET);
+    let backstop = (ceiling * 2).max(ceiling + Duration::from_secs(6));
     if elapsed > backstop {
         if let Some(line) = dying {
             return Some(kernel_died_here(line));

backstop-twice-only.patch

--- a/tests/common/qemu.rs
+++ b/tests/common/qemu.rs
@@ -517,7 +517,7 @@
     // For a guest that is stuck *and* chatty and so never trips the silence
     // guard; never before a guest silent since `ceiling` has been silent for
     // [`GUEST_QUIET`].
-    let backstop = (ceiling * 2).max(ceiling + GUEST_QUIET);
+    let backstop = ceiling * 2;
     if elapsed > backstop {
         if let Some(line) = dying {
             return Some(kernel_died_here(line));

backstop-mains.patch

--- a/tests/common/qemu.rs
+++ b/tests/common/qemu.rs
@@ -517,7 +517,7 @@
     // For a guest that is stuck *and* chatty and so never trips the silence
     // guard; never before a guest silent since `ceiling` has been silent for
     // [`GUEST_QUIET`].
-    let backstop = (ceiling * 2).max(ceiling + GUEST_QUIET);
+    let backstop = ceiling.max(Duration::from_secs(300));
     if elapsed > backstop {
         if let Some(line) = dying {
             return Some(kernel_died_here(line));

backstop-no-death.patch

--- a/tests/common/qemu.rs
+++ b/tests/common/qemu.rs
@@ -519,7 +519,7 @@
     // [`GUEST_QUIET`].
     let backstop = (ceiling * 2).max(ceiling + GUEST_QUIET);
     if elapsed > backstop {
-        if let Some(line) = dying {
+        if let Some(line) = dying.filter(|_| false) {
             return Some(kernel_died_here(line));
         }
         return Some(format!(

wedge-irq-storm.patch

--- a/kernel/src/arch/aarch64/trap.rs
+++ b/kernel/src/arch/aarch64/trap.rs
@@ -525,7 +525,7 @@
 
     pub(super) fn tick() {
         if RUNNING.load(Relaxed) {
-            TICKS.fetch_add(1, Relaxed);
+            TICKS.fetch_add(0, Relaxed);
         }
     }
 

[Japabu]

T14 at 75dc056f1, Drive mode, run by the orchestrator from 638-r5/metal/request.txt, with the owner's approval for the watchdog and reset rows. Each run was cargo test --test toyos-build -- --metal [filter] from the clean worktree at the head, without --metal-readback; the stick's log partition was saved before each run's first flash; no judging left a row in tests/metal (each saved patch is empty) and the worktree was clean after each.

run	exit	what it read
boot_deadline_ends_a_wedge	0 (118 s)	armed ["wedge-before-reset", "boot-deadline=10000"]; the boot deadline expired: a bound of 10000 ms, reached at 10062 ms; back in 51 s; PASS, 1 passed, 0 failed, 1 boot(s)
hard_lockup_ends_a_deaf_cpu	0 (114 s)	armed ["hard-lockup-probe", "boot-deadline=10000"]; a cpu locked up with interrupts off: cpu7 has taken no interrupt for 5000 ms, with IF clear at every sample in that span. Its bound is 5000 ms.; back in 59 s; PASS
usb_reset_records_the_phase_it_cut	0 (113 s)	armed ["usb-reset-under-load", "boot-deadline=10000"]; deadline at 10066 ms; a Bulk-Only command was open in its data phase on slot 1 after 2000 ms; the controller had that device's data endpoint Running with 252 TRB(s) it had not reached on the ring; back in 54 s; PASS
unfiltered	1 (2273 s)	50 registration(s) and 222 shared member(s) over 25 boot(s); 270 passed, 2 failed, 25 boot(s)

The unfiltered run in detail:

• The shared boots at this head's sizes, each in one boot and every member run: shared: 62 member(s) (61 ms per member over the 62 that ran), shared-2: 19 member(s) (26 ms per member), shared-debug: 5 member(s) (31 ms per member), ccorpus: 136 member(s) (25 ms per member over the 136 that ran), no ccorpus-2.
• PASS among the rest: boot_deadline_ends_a_wedge, hard_lockup_ends_a_deaf_cpu, usb_reset_records_the_phase_it_cut, loader_watchdog_arms, watchdog_fed, lan_talk, lan_message_delivery, lan_lease_report, readdir_bound, hda_tone.
• The two reds, both main's and recorded, and no other: lan_dhcp_lease (issues/hardware/the-benchs-router-leases-toyos-another-address-than-ubuntu.md) and hda_client_stall (soundd resumed 1 time(s) — the second stream did not find a suspended daemon; issues/audio/hda-client-stall-reads-one-resume-where-its-judge-wants-two.md). The third allowed red, hda_tone, passed in this run.

Logs and readbacks: /Users/jan/.claude/jobs/2280e09e/tmp/scratchpad/orch/638-r5/metal/ (<run>.log, <run>-readback/, <run>.record-rows.patch).

[Japabu]

Round 5, head 75dc056f1. This round reads round 4's BLOCKERs and what changed since 4859e8d71: 4b009e712, 1ba6b43e3, and the merges of #648 and #680.

Round 4's BLOCKERs

• toyos-tco/src/lib.rs:154, the three staged-wedge rows at 10 s: CLOSED. Comment 5966698095 ran Drive mode at 75dc056f1. Each row exited 0 alone and passed again in the unfiltered run.
  • boot_deadline_ends_a_wedge: armed ["wedge-before-reset", "boot-deadline=10000"]. The deadline was reached at 10062 ms (10063 ms in the unfiltered run). PASS.
  • hard_lockup_ends_a_deaf_cpu: armed ["hard-lockup-probe", "boot-deadline=10000"]. The detector sealed the record, not the deadline: cpu7 has taken no interrupt for 5000 ms … Its bound is 5000 ms. The wedge was staged at 1.210 s and the ring's last line is cpu7's LOCK CONTENTION at 7.082 s, so the deadline had about 2.9 s to spare. PASS both times.
  • usb_reset_records_the_phase_it_cut: the deadline was reached at 10066 ms, with 252 TRBs (119 in the unfiltered run). PASS.
• tests/toyos.rs:766, :856, the shared boots at their new sizes: CLOSED. The unfiltered --metal run exited 1 with 270 passed, 2 failed, 25 boot(s) in 2273 s.
  • shared: 62 member(s), at 61 ms per member over the 62 that ran.
  • shared-2: 19 and shared-debug: 5. ccorpus: 136 ran in one boot, with no ccorpus-2.
  • 50 registrations and 222 members make 272, which is 270 passed and 2 failed. Both reds are registrations, so every member passed.

Rulings on the brief

• Both reds belong to main alone.
  • lan_dhcp_lease: finding 1 repeats issues/hardware/the-benchs-router-leases-toyos-another-address-than-ubuntu.md and main's reading of it at 7e151819: this boot leased .48 and the host pinged .46. Finding 2 puts .46's answer 39 s after this boot's init: power: line, so Ubuntu gave it. Round 4 accepted the same reading at 4859e8d71. Finding 2 follows from finding 1, because the boot answered at its own lease ([talk] 192.168.1.48 answered a ping). The judge's timing logic is main's. The branch removed only the check on the hold job's exit, and the host's reboot ends that job.
  • hda_client_stall: soundd resumed 1 time(s) matches the issue's reading at hda_tone: the judge refuses multi-stream sessions, and a dropped cpal stream waits a bounded time on soundd and then lets go #641's 1621eb5f0 word for word, and the order of events is the same.
    • The tone's stream was removed at 5.681, and the stall was spawned at 5.683.
    • The stall opened its stream at 5.684, before soundd's 20 to 25 ms suspend (7.842→7.865 in this boot).
    • The branch touches no part of soundd, the audio judges or the testcases boot.
• The loop in Drive mode. The runs made 28 cycles through drive (25 unfiltered, 3 alone).
  • Each cycle came back in 42 to 74 s, with the stick at 0 s and a readback of that cycle's own boot (PASS: the machine booted ToyOS in …). No cycle refused.
  • So the go-down dial read no stick early. Neither the deleted refusal-wait's case nor a stamped refusal through said_refusal reached the machine.
  • The loop prints no go-down time. The dial's 10 s shows only in the absence of an early read.
• The hard-lockup owner holds at this head.
  • issues/hardware/the-t14-boots-toyos-unattended.md exists and holds the detector (:80-88).
  • Tracks: the T14 runs its tests in sessions of a booted ToyOS, and every machine ships a watchdog and a hard-lockup detector #631 cites toyos_tco::STAGED_BOUND_MS and this branch's issue, and main has neither, so Tracks: the T14 runs its tests in sessions of a booted ToyOS, and every machine ships a watchdog and a hard-lockup detector #631 lands second.
  • Tracks: the T14 runs its tests in sessions of a booted ToyOS, and every machine ships a watchdog and a hard-lockup detector #631's merge deletes that track, so root CLAUDE.md's merge rule and issues/README.md give that merge the job of moving the owner to issues/hardware/a-frozen-toyos-waits-for-a-hand-on-the-power-button.md.
  • The same merge has to reconcile the exit. Tracks: the T14 runs its tests in sessions of a booted ToyOS, and every machine ships a watchdog and a hard-lockup detector #631 closes this file by giving the constant a reader, while this file's exit deletes the constant.
  • That work belongs to Tracks: the T14 runs its tests in sessions of a booted ToyOS, and every machine ships a watchdog and a hard-lockup detector #631's merge and is not a finding here.

Evidence at 75dc056f1

• cargo run -- --ci host: EXIT=0, "Host: 67 step(s), all green".
• cargo test --test toyos-build: EXIT=0, 26 passed in 44.5 s.
• toyos-checks: EXIT=0, 31 passed.
• The twelve negative controls: EXIT=101 each.
• The wedge: 37 s alone, 38 s inside a 45.4 s suite, and 181 s on main's tree.
• CI: the PR is a draft, so every job was skipped.

Growth: +740 −671 over 38 files, net +69.

• Code alone: +425 −543, net −118.
  • Shipping (netd, toyos-i219 lib/regs, toyos-tco): +23 −63.
  • src/ production: +112 −14. src/ tests: +119 −58.
  • toyos-i219 model and tests: +2 −57.
  • tests/: +169 −351.
• issues/: +315 −128.
• Production has shrunk since round 4, because the refusal-wait and after_the_reboot are gone. Accepted.

BLOCKER

• src/metal.rs:2608-2612, :2615-2619 — the dialled wrapper and its assertion check only that wait_on hands accepts CONNECT_SECS when going down, which is the one line at :1517. The go-down-dial-one-poll control is a defect any reader of the diff catches, and the body's Unsure admits no test stages a lost SYN. A test that guards what a reader can check goes. Delete both spans and pass accepts at :2613. Cut :2590 back to "a machine going down is its port refusing". Drop the body's control row and its line "No test stages a lost SYN…".

NOTE

• issues/build/the-talking-boots-reboot-outruns-its-log-stream.md:45 — its exit, "Two consecutive T14 runs pass lan_talk", is met: PASS lan_talk at 4859e8d71 (comment 5962753432) and at 75dc056f1 (comment 5966698095), both runs with hand_back. Delete the file, its citation at issues/hardware/netds-i219-drops-a-transmit-burst-past-its-ring.md:27, and the body's "which stays until two T14 runs pass lan_talk".

REMOVE

• PR body, the "The T14." bullet — "no run has read them since main's kernel merges. They are owed at this head (The T14, below)." This has been false since comment 5966698095.

SEND BACK

[Japabu]

Round 6, head e3094a94a. This round reads round 5's findings and what changed since 75dc056f1, which is the one commit e3094a94a. origin/main is still 73282fa93, the tip that 75dc056f1 merged.

Round 5's BLOCKER

• src/metal.rs:2608-2612, :2615-2619, the test-only dialled wrapper: CLOSED.
  • Both spans are deleted, and accepts is passed directly at :2608. The doc at :2590 now reads "a machine going down is its port refusing". The body's control row and its "No test stages a lost SYN" line are both gone.
  • At this head, 638-r6/host.log shows metal::tests::coming_back_is_ssh_answering_and_not_its_port ... ok, and the run ends Host: 67 step(s), all green (host.exit EXIT=0). The run started at 07:33:27 UTC, 16 s after the commit.
  • The remaining control was run again at this head (638-r6/mutations/run.log). The patch was let answered = listening;. Unmutated EXIT=0, build EXIT=0, mutated EXIT=101 at :2605 with left: Ok(0), right: Err(Silent { what: "come back", secs: 1 }). The tree was [] before and after.

Round 5's NOTE and REMOVE

• The NOTE on issues/build/the-talking-boots-reboot-outruns-its-log-stream.md: CLOSED.
  • The file is deleted, and so is its citation in issues/hardware/netds-i219-drops-a-transmit-burst-past-its-ring.md. The body's "which stays until…" is replaced.
  • git grep for the slug at e3094a94a exits 1.
  • Its durable rule is already the doc of hand_back (src/metaltalk.rs:974-978), which is what issues/README.md's closing procedure asks. The commit message carries the evidence.
• The REMOVE of the T14 bullet's "no run has read them…": CLOSED, the sentence is gone from the body.

Whether the readings at 75dc056f1 stand for this head

• They do. git diff -U0 75dc056f1 e3094a94a -- src/metal.rs has three hunks, at :2590, :2608 and :2615. All three lie inside #[cfg(test)] mod tests, which opens at :2578.
  • Only cargo test --lib compiles that module.
  • The toyos-metal loop, the image and the toyos-build integration harness link the library without it.
• The other two files are under issues/. No include_str! or include_bytes! in the tree reads issues/, and the host steps that read it ran green at this head.
• So the T14 Drive-mode readings, build-only.log and guest.log at 75dc056f1 cover this head. toyos-checks was run again at this head: EXIT=0, 31 passed.

Evidence at e3094a94a

• cargo run -- --ci host: EXIT=0. The FAILED lines in that log belong to the control steps, which are required to fail, for example === [ci] control \wake-fence-off``.
• cargo test --test toyos-checks: EXIT=0, 31 passed.
• CI: the PR is a draft, so host, toolchain and guest were all SKIPPED. The merge queue runs them.

Growth. git diff --shortstat origin/main...e3094a94a: 38 files, +717 −704, net +13.

• Production: src/ +112 −14, and netd, toyos-i219 lib and regs and toyos-tco +23 −63.
• Tests: src/ test modules +109 −58, tests/ +169 −351, and toyos-i219's model and tests +2 −57.
• Issues: +302 −161.
• Since round 5 the branch is −56 lines, all in test code and issues. Production has not moved since round 5, when it was accepted.

BLOCKER

None.

NOTE

None.

REMOVE

None.

LAND