The boot CPU waits for an AP's echo as long as a dead CPU is: 5 s, not 100 ms

[Japabu]

time::AP_START is how long the boot CPU waits for an AP it started to echo its token. It is now DEAF_CPU's 5 s, up from 100 ms. The issue it closes is deleted with it. The harness is main's, byte for byte.

This supersedes #658 (wt/toyos-loadfree, f1ad5d1ed).

What changed, by decision

AP_START is DEAF_CPU's span (kernel/src/time.rs: one declaration, read by both architectures' bring-up loops).

• When it expires, the machine boots without that CPU and without every CPU after it, for good.
• A vCPU its host has not run yet is slow, not dead. This kernel already gives a live CPU DEAF_CPU before it calls that CPU deaf.

issues/kernel/an-ap-its-host-has-not-scheduled-for-100-ms-is-booted-without.md is deleted. Its exit has two halves:

• this bound;
• virt_smp green in a whole-suite run at a load above the dev host's 14 cores, which is the guest run owed below.

What it costs. A CPU that never echoes costs 5 s of boot, once, because the loop stops at the first one.

• Among the guest tests, only virt_failed_ap_leaves_no_hole (smp-skip-ap) meets it.
  • Its ready marker is the BSP's control-register line. after_console prints it (kernel/src/main.rs:281) before start_other_cpus runs (:438).
  • So the 5 s does not fall in the boot wait. It falls in judge_virt_job's await_marker, as 5 s of silence against GUEST_QUIET's 15 s.
• Metal, from the round-1 review: the TCO watchdog (9.6 s) is fed only from a scheduler pass (sched/driver.rs:512). The T14 logs put Boot: complete at 1195–1276 ms, so a dead first AP reaches the first pass near 6.2 s.

Dropped, and why

The whole harness half of round 1 and of #658 is dropped:

• The guest's clock: tests/common/steal.rs, its host checks, and every wait moved onto it. Nothing measured needs it.
  • Round 1's negative-control run had main's wall-clock waits, WIDTH and the 100 ms AP_START. Under 28 spinners, at load 86 → 98, it went 21/21.
  • Per the round-1 review, main's last nightly (run 36763711317) ran 20 of these 21 tests on main's waits on GitHub's runners, and all were green.
  • Its Linux reading had never run, and its unimplemented! would have panicked every wait on any host but macOS and Linux.
• WIDTH's deletion, and with it host_scale on the clock and the 20 s boot wait.
  • The deletion's only reason was the clock. Without the clock it is unmeasured.
  • Every QEMU ceiling is at most three times what its test takes; wedges end in seconds; two LAN boots ride the talking boot #638 deletes WIDTH under its own ceiling design.
• The host's load decides no verdict: every wait on a guest reads the guest's own clock, and a wait on another CPU is a bound on a dead one #658's other sites. The guest suite keeps the 21 tests only a booted machine answers; the rest are metal, host or tracked #660 cut the tests they touched.
• a-harness-wait-reads-a-starved-guest-as-a-stopped-one.md stays unfiled, because the 21 tests give it no sighting.

Gates

• cargo run -- --ci host at 8b94f8438: EXIT=0, "Host: 56 step(s), all green".
• git diff origin/main at this head names only kernel/src/time.rs and the deleted issue.
  • The kernel delta is byte-identical to 3f7081381's, where cargo run -- --build-only exited 0.

High risk: SMP bring-up

• Independent oracle. Linux waits 5000 ms for the same echo on arm64: arch/arm64/kernel/smp.c, __cpu_up, wait_for_completion_timeout(&cpu_running, msecs_to_jiffies(5000)), read at torvalds/linux 551c722f4080.

• Recorded real failure at 100 ms. virt_smp went red at 8f142ba9e on a loaded host, with SMP: cpu7 mpidr=0x7 did not echo within 100ms … and SMP: 7 of 8 MADT CPUs online.

• Negative control. No host test can reach the decision:

  • time.rs and both arch/*/smp.rs compile only for the kernel's targets.
  • kernel-loom carries smp_roster.rs to the host, but its await_echo takes the expiry as a predicate and never sees the bound.
  • A host assertion on the constant would only restate it.

  The control is therefore the guest arm: virt_smp under load with AP_START at 100 ms. It goes red only when the host leaves a vCPU unrun for more than 100 ms after its CPU_ON. 636r2-virt.log recorded that once. Round 1's negative-control run, at load 86 → 98, did not: it went 21/21.

Unsure

• 5 s is Linux's number, not one measured here. A CI runner whose hypervisor withholds a vCPU for longer still boots one CPU short, and virt_smp names that red.
• time::DEAF_CPU itself can still panic a TCG guest whose host starves it for 5 s. issues/kernel/a-shootdown-panicked-on-a-cpu-the-host-starved.md owns that.

Owed

• CI at this head.
• One whole-suite guest run under the 28-spinner load, with virt_smp and virt_failed_ap_leaves_no_hole green on main's wall-clock waits.

🤖 Generated with Claude Code

https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L

[Japabu]

Orchestrator guest runs at 3f7081381 (-LOAD = 28 CPU spinners on the 14-core host for the whole job; logs orch/logs/loadfree2-*.log):

job	load avg at start → end	expected	exit	result
whole, LOAD	28.0 → 86.3	0	0	21/21 in 176.9 s; "the guests had 35% of the 176s their waits read across"
negative control (change reverted), LOAD	86.3 → 98.3	1	0	21/21 in 42.9 s
whole, quiet	—	0	0	21/21 in 26.3 s; "the guests had 83%"

The negative control is green under a heavier load than the arm it controls: on the 21-test suite, main's wall-clock waits, WIDTH and the 100 ms AP_START did not red at load ~98. Under the same load the branch's suite took 4× longer (176.9 s vs 42.9 s).

[Japabu]

Review round 1 at 3f7081381.

Gate. CI: host passed at this head (run 36847219923, macos-latest). Guest: the orchestrator's three runs at this head all exited 0.

Size. The branch adds 486 lines and removes 134, net +352:

• kernel: +4 −2;
• issues: −30;
• harness and host checks: +482 −102.

After the cut below, it is kernel +4 −2 and issues −30, net −28.

BLOCKER

• tests/common/steal.rs:1 (with tests/checks/steal.rs, tests/checks.rs:107-121, and the steal/Moment/clock hunks of tests/common/qemu.rs and tests/toyos.rs). Delete the guest clock from this PR. It fixes no measured failure:

  • The negative control is 21/21 green at load ~98 (loadfree2-negctl-whole-LOAD.log). That control has main's wall-clock waits, WIDTH and the 100 ms AP_START reverted in.
  • Main's last completed nightly (run 36763711317, 6e0d7da82) ran main's wall-clock waits on GitHub's ubuntu runners at --jobs 1. All twelve guest shards and tcg were green. That suite held 20 of these 21 tests by name.
  • The PR body itself says every sighting of the starved-guest defect is in a cut test. It also says the clock cannot see a runner's hypervisor steal, which is the one contention its stated target has.
  • No wait among the 21 has a recorded red. The hard ceilings a starved guest could still trip are the boot wait and the screendump_* ceilings, and on CI the clock reads those as the wall.

  It is also unready on its own terms:

  • demand for Linux (steal.rs:173) is the reading for that target, and it has never run. Both host gates run on macos-latest, and it was only type-checked.
  • steal.rs:209's unimplemented! makes every guest wait panic on any host but macOS and Linux. Main's Instant ran on all of them.

• tests/common/qemu.rs:177 (with :2275 and the set_width call in tests/toyos.rs run_tasks). Keep WIDTH, set_width and main's boot wait. The deletion's only stated reason is that the clock already takes contention out. Without the clock, the deletion is a guess:

  • No run of main with only WIDTH deleted exists.
  • On CI it changes nothing: --jobs 1 means width 1, and the boot wait is 20 s either way.
  • Locally it shrinks hard ceilings that no silence gate backs: the boot wait from 120 s to 20 s, and screendump_until/screendump_while from ×12 to ×1.
  • Every QEMU ceiling is at most three times what its test takes; wedges end in seconds; two LAN boots ride the talking boot #638 (open) deletes WIDTH under its own ceiling design. That deletion, and the 28-spinner run of main without WIDTH, belong to Every QEMU ceiling is at most three times what its test takes; wedges end in seconds; two LAN boots ride the talking boot #638.

NOTE

• kernel/src/time.rs:248: AP_START at DEAF_CPU's 5 s lands on its own.
  • It is one declaration, and it matches Linux arm64's wait_for_completion_timeout(..., msecs_to_jiffies(5000)).
  • Its red arm is 636r2-virt.log: 100 ms at 8f142ba9e, where cpu7 did not echo and 7 of 8 CPUs came up under load.
  • Guest run 2 is not its negative control: it was green.
  • Metal cost, checked. The TCO is fed only from a scheduler pass (sched/driver.rs:512), and APs reach one only after smp::set_ready. The orchestrator's T14 logs put kernel to Boot: complete at 1195–1276 ms, so a dead first AP reaches the first pass near 6.2 s, inside the 9.6 s bound. The boot deadline (120 s) and the hard-lockup bound (60 s) are wider.
• The re-cut head (AP_START and the issue deletion, with the harness as main has it) is untested. It owes:
  • CI;
  • one whole-suite guest run under the 28-spinner load, with virt_smp and virt_failed_ap_leaves_no_hole green on main's wall-clock waits. The second test's boot now spends 5 s of CI's 20 s boot wait.
• The run table's "4× slower" is not the clock's cost.
  • The branch arm opened with external deps changed: cleaning …/tests/toyos-rust-tests, and its artifact-staging waits ran up to 102.9 s before its first PASS. The control's longest staging wait was 11.8 s.
  • Per-test times add up to 214 s on the branch and 169 s on the control. The gaps are virt_timer_preempts (31 s vs 10 s), iommu_virtio_platform (34 s vs 26 s) and virt_failed_ap_leaves_no_hole (12 s vs 6 s; that one is AP_START's 5 s). No other test differs by more than 2 s.
  • Guest-clock minimum waits do stretch. drain_for (qemu.rs:1427) runs until dur of guest time has passed, so every drain_serial pace and await_guest's 200 ms poll costs dur/share of wall time. At the run's 35% share that is about 570 ms a poll. But that is one poll per await, under a second per test.
  • The two arms ran on unequal builds. The control's green result stands either way.
• issues/kernel/an-ap-its-host-has-not-scheduled-for-100-ms-is-booted-without.md: deleting it is right, and the deletion goes with the AP_START half. Its exit is that bound plus virt_smp green in a whole-suite run at a load above 14 (run 1, 28→86).
  • Not filing a-harness-wait-reads-a-starved-guest-as-a-stopped-one.md is also right, but the reason is that the 21 tests give it no sighting. This branch does not fix it.
  • The two The host's load decides no verdict: every wait on a guest reads the guest's own clock, and a wait on another CPU is a bound on a dead one #658 files about cut tests rightly stay out, and the four files the body names as on main are there.
• The host's load decides no verdict: every wait on a guest reads the guest's own clock, and a wait on another CPU is a bound on a dead one #658 is still open, and this PR supersedes it.

REMOVE

• PR body: "Negative control: guest run 2 below … A did not echo within 100ms red is this half." This is false: run 2 was green.
• PR body: "On main its defect was still true of every wait above, and this branch is its exit, so it is not filed." The control shows this is false.
• PR body: the "## Guest runs owed" section. It is a scratchpad path and will rot.
• PR body: every section on the clock, WIDTH, host_scale, the share line, The host's load decides no verdict: every wait on a guest reads the guest's own clock, and a wait on another CPU is a bound on a dead one #658's carried and dropped lists, and the "Unsure" bullets. They go with the harness half.

SEND BACK

[Japabu]

Orchestrator guest run at 8b94f8438 under the 28-spinner load (load average 14.4 → 47.1): the whole suite, exit 0.

test result: ok. 21 passed, 21 total (31.0s)