Repository navigation
The kernel's idle report is gone, and the counters row measures a quiet idle second - #728
Conversation
test_rs_counters_metal printed its idle0 read, 81 lines, at the start of the very second it then measured. logkeeper writes and syncs those lines to the stick through the /log fileserver, which every counters boot so far placed on cpu7, so cpu7's idle busy fraction was the row's own output. The cpu7_scout row (wt/toyos-cpu7 at 22d2411, one T14 boot, judge EXIT=0) measured back-to-back idle seconds between counters rounds, four of each arm after a discarded warm-up. Mean idle busy: quiet (prints nothing): cpu0-6 0.00-0.03%, cpu7 0.12% loud (the row's 81 lines): cpu0-6 0.34-0.45%, cpu7 0.84% dose (four times the lines): cpu0-6 0.23-0.55%, cpu7 1.27% The row now takes idle0, idle1 and spin first and prints the three after the spin's read; the lines and their parsing are unchanged. The judge also says how many SMIs landed in the idle second: in the scout's seconds the cpu0-6 floor of about 0.45% appears exactly in the seconds whose SMI count moved, so a row's idle reading is read beside it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WcU2Dsw6mDYtwYfzVHPzM8
…PERF t14-cpu7-idles-busier-under-toyos-than-any-cpu-under-linux.md is closed: its exit asked for a reading naming what ran on cpu7 across the counters row's idle second, and for the row's doc if it was the row's own work. The cpu7_scout row (wt/toyos-cpu7 at 22d2411, image sha256 a5dcc2a77f5d35f016a663e634d1003ad30e9de027fdd53868cd8dd20fafa035, one T14 boot run by the orchestrator, judge EXIT=0) names it: the /log fileserver, pid 4, spawned onto cpu7 (dst=7), writing the row's own 81-line idle0 print to the stick. With nothing printed cpu7 reads 22 to 28 ppm in a second no earlier print spills into, under a tenth of the lowest machine-wide 10 s interval Linux's idle turbostat reads on that machine (0.11%). The fold is test_rs_counters_metal's header: nothing is printed until the three reads are taken. The SMI issue gains the MPERF reading of the same boot. The scout's seconds, as its judge printed them (busy in ppm of each CPU's stamp, cpu0..cpu7): second 0 warmup Loud: from 1172790890 ns to 2172974091 ns, smi +1, busy ppm [4979 4793 4839 5343 7107 4558 4573 19060], kicks [12 4 2 5 1 1 1 4] second 1 round Quiet: from 2172974091 ns to 3173035738 ns, smi +0, busy ppm [26 63 25 230 26 285 27 28], kicks [1 1 1 1 1 1 1 1] second 2 round Loud: from 3173035738 ns to 4173341548 ns, smi +1, busy ppm [4884 4897 4556 5831 4554 4540 4555 9875], kicks [6 2 1 3 1 1 1 3] second 3 round Dose: from 4173341548 ns to 5174342442 ns, smi +0, busy ppm [389 1470 24 3723 22 7 24 7388], kicks [6 2 1 3 1 1 1 3] second 4 round Loud: from 5174342442 ns to 6174633115 ns, smi +0, busy ppm [336 341 22 1089 22 7 22 4578], kicks [6 1 1 3 1 1 1 3] second 5 round Dose: from 6174633115 ns to 7175633698 ns, smi +1, busy ppm [4904 5746 4551 7810 4546 4535 4550 12208], kicks [6 1 1 3 1 1 1 3] second 6 round Quiet: from 7175633698 ns to 8175690541 ns, smi +0, busy ppm [21 58 25 217 24 7 25 22], kicks [1 0 1 1 1 1 1 1] second 7 round Dose: from 8175690541 ns to 9176698816 ns, smi +1, busy ppm [4819 5635 4570 7501 4568 4553 4571 13604], kicks [6 2 1 3 1 1 1 3] second 8 round Quiet: from 9176698816 ns to 10176757787 ns, smi +0, busy ppm [22 59 21 218 22 6 23 25], kicks [1 0 1 1 1 1 1 1] second 9 round Loud: from 10176757787 ns to 11177135943 ns, smi +1, busy ppm [4867 4937 4557 5511 4552 4541 4557 9970], kicks [6 2 1 4 1 1 1 3] second 10 round Quiet: from 11177135943 ns to 12177194852 ns, smi +0, busy ppm [387 73 40 356 35 16 36 4875], kicks [6 1 1 3 1 1 1 3] second 11 round Loud: from 12177194852 ns to 13177496481 ns, smi +1, busy ppm [4879 4879 4561 5643 4555 4545 4559 9318], kicks [6 1 1 3 1 1 1 3] second 12 round Dose: from 13177496481 ns to 14178489116 ns, smi +0, busy ppm [247 1076 25 3026 24 6 21 17694], kicks [6 3 1 3 1 1 1 3] Dose: mean idle busy cpu0=0.26% cpu1=0.35% cpu2=0.23% cpu3=0.55% cpu4=0.23% cpu5=0.23% cpu6=0.23% cpu7=1.27% Loud: mean idle busy cpu0=0.37% cpu1=0.38% cpu2=0.34% cpu3=0.45% cpu4=0.34% cpu5=0.34% cpu6=0.34% cpu7=0.84% Quiet: mean idle busy cpu0=0.01% cpu1=0.01% cpu2=0.00% cpu3=0.03% cpu4=0.00% cpu5=0.01% cpu6=0.00% cpu7=0.12% Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WcU2Dsw6mDYtwYfzVHPzM8
|
T14 results (orchestrator's run; worktree clean at
The change does not move cpu7: head and revert read the same. The body's "Unsure" first suspect (the round logkeeper writes for the job's launch lines — |
At head 35cd631 the T14 read cpu7 at 1.40% idle busy, and 1.43% with the row's print-after change reverted: moving the row's print did not move cpu7. The readbacks show why. The job starts about 8 ms after logkeeper does, and the stick's first sync (`usb-storage: disk 0 does not implement SYNCHRONIZE CACHE`, a record the /log fileserver's path stamps on cpu7) is at 1.182 s on head against idle0 at 1.173184 s, and at 1.183 s on the revert against idle0 at 1.173472 s: logkeeper is still writing the boot so far and the job's launch lines to the stick inside the measured second, on both arms. The binary now reads the log through logkeeper's `log` port before idle0: it prints a line and reads the served log until the line comes back, which logkeeper hands a reader only after the round is written and synced. Twice, because the round that writes the first line may itself stamp a record (the stick's first sync does), which the second round writes. Each wait is bounded by two of logkeeper's 5 s write budgets and panics past it. The testcases estate gives logkeeper its `log` port and test-runner the connector to it, which the job inherits. The cpu7 issue is restored: the head's T14 reading does not meet its exit, and no reading of this change exists yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WcU2Dsw6mDYtwYfzVHPzM8
|
T14 results at
At head cpu7 reads the same as every other CPU. All eight at ~0.46% matches the SMI floor this PR records (an SMI in the measured second costs every CPU ~4.5 ms), so this head's second likely held one; the judge log names the count. Judge logs: |
|
Review round 1, head What the readings and the code show:
BLOCKER
NOTE
REMOVE
SEND BACK |
…quiet `userland/console` (`Log::subscribe`) and `counters_metal` (`settle`) each decoded logkeeper's answer to a `READ` on its `log` port by hand: the `SERVED` frame, the boot-so-far length, the one pipe handle. Two decoders of one answer meant a change to `serve.rs`'s answer could miss one, and `counters_metal`'s copy runs only on a T14 boot. `logkeeper_api::read` is now the one decoder, and both call it. It lives in a crate of its own beside `filepicker-api`, the client side of a userland server's port: `toyos-logstream` is pure (no I/O, no `unsafe`), and the SDK is a path dependency of std and cannot take `toyos-logstream`. `counters_on_metal` now reds a kernel record stamped in a whole millisecond between `idle0` and `idle1`. `settle` waits two of logkeeper's rounds on the one record kind seen so far that a round's own write stamps (the stick's first sync); nothing held that the second was then quiet. Rejudging the round-2 readbacks of 49e49a9 with this judge: head passes (no record in 1230220488..2230283529 ns), and the whole change reverted (0c27e4f) fails on `[1.183 cpu7] usb-storage: disk 0 does not implement SYNCHRONIZE CACHE` inside 1173282164..2173482583 ns. `counters_metal` also panics by name on a served line that is not UTF-8: logkeeper writes every control byte as text, so such a line is its defect, not an empty line. Its header drops the job's start offset after logkeeper's (about 7 to 12 ms on the readbacks), a measurement. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WcU2Dsw6mDYtwYfzVHPzM8
…othing t14-cpu7-idles-busier-under-toyos-than-any-cpu-under-linux.md's exit was "a T14 reading of the row with that wait in which cpu7's idle busy fraction is within cpu0 to cpu6's". The orchestrator's T14 run of 49e49a9 (pull request #728, comment 5981716271), one `testcases` boot per arm, judged by `cargo test --test toyos-build -- --metal --metal-readback <dir> counters`: - head 49e49a9, which waits for the log before `idle0`: EXIT=0, cpu7 0.46%, cpu0 to cpu6 0.46 to 0.49%. Both settle lines (1.175 s, 1.188 s) and the stick's first sync (1.183 s) precede `idle0` (1230220488 ns); no kernel record is stamped before `idle1` (2230283529 ns). One SMI fell in the second, which is why all eight sit near 0.45%. - the whole change reverted, 0c27e4f: EXIT=0, cpu7 2.89%, cpu0 to cpu6 0.46 to 0.72%, the sync stamped 9.7 ms inside the second. So cpu7's excess was logkeeper's first rounds, written through the `/log` fileserver on cpu7 inside the measured second. The fold is `counters_metal.rs`'s header ("idle0 waits for the log to be quiet"), and `counters_on_metal` now reds a kernel record inside the second. No citation of the slug exists (`git grep`). The SMI issue quoted the row's "two idle floors", a phrase nothing in the tree says; it now states the two floors itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WcU2Dsw6mDYtwYfzVHPzM8
|
T14 result at
Linux's band printed by the row: 0.11–0.36%. With no SMI in the measured second, every CPU reads at or below 0.03%. Judge log: |
|
Review round 2, head Round 1 BLOCKERs:
Round 1 NOTEs and REMOVE: the UTF-8 panic, the "two idle floors" quote, the measurement in the header, and "Twice" being unheld (now held for kernel records) are all addressed. BLOCKER
NOTE
REMOVE SEND BACK |
…nd too The quiet check read only kernel records ([... cpuN]), so a program line stamped inside the second -- the row's own idle0 print moved back before idle1, the regression "measure first, print after" exists to prevent -- passed it. toyos-logstream gains program_ms beside record_ms: a program line's milliseconds, found from the tag back past the optional pid=, tid= and severity words, since the wall clock before it is two words or none. The judge reads the whole log with both. The window now includes the millisecond each edge falls in: a line stamped in idle0's millisecond may follow the read, and lines carry no finer time. With the edges excluded, the review's mutation (print idle0 right after reading it) can land its 81 lines in idle0's own millisecond and pass. The SMI issue cites 35cd631, which lands and carries the scout's image hash and per-second lines, rather than 22d2411, reachable only from a branch that goes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WcU2Dsw6mDYtwYfzVHPzM8
|
Round-2 answer, head The review's named mutation ( --- a/tests/toyos-rust-tests/src/bin/counters_metal.rs 2026-10-04 18:26:40
+++ b/tests/toyos-rust-tests/src/bin/counters_metal.rs 2026-10-04 18:27:00
@@ -199,6 +199,7 @@
let cap: SysCap = Endowments::get().take(SYSCAP_LABEL).expect("test-runner endows a capability");
settle();
let idle0 = read(&cap);
+ print("idle0", &idle0);
std::thread::sleep(IDLE);
let idle1 = read(&cap);
std::thread::scope(|s| {
@@ -212,7 +213,7 @@
}
});
let spin = read(&cap);
- for (phase, read) in [("idle0", &idle0), ("idle1", &idle1), ("spin", &spin)] {
+ for (phase, read) in [("idle1", &idle1), ("spin", &spin)] {
print(phase, read);
}
loaded(&cap);The control ( control.patchdiff --git a/issues/the-t14s-firmware-interrupts-every-cpu-every-2-2-s-under-toyos.md b/issues/the-t14s-firmware-interrupts-every-cpu-every-2-2-s-under-toyos.md
index aabb1e83a..de69458b0 100644
--- a/issues/the-t14s-firmware-interrupts-every-cpu-every-2-2-s-under-toyos.md
+++ b/issues/the-t14s-firmware-interrupts-every-cpu-every-2-2-s-under-toyos.md
@@ -26,9 +26,9 @@ lines are quoted on #681 (comment 5962619169; the boot itself in comment
CPUs waiting on a lock across a step, which went 4,503,694 and 4,554,675 ns
between two turns of their own spin.
- **MPERF reads the stop as about 4.55 ms of C0 on every CPU.** One T14 boot
- of a scout image without ACPI mode (`22d241174`) read twelve back-to-back
- idle seconds between counters rounds; the commit that added this bullet
- carries its per-second lines. In the five whose SMI count moved by one,
+ of a scout image without ACPI mode read twelve back-to-back idle seconds
+ between counters rounds; `35cd63142`, which added this bullet, carries its
+ image hash and per-second lines. In the five whose SMI count moved by one,
cpu2, cpu4, cpu5 and cpu6, which ran none of the log's work, read 4535 to
4571 ppm busy; in the seven where it did not, 6 to 285 ppm. So the
`counters` row's idle second reads one of two floors, about 0.45% or 0.03%
diff --git a/tests/toyos-rust-tests/src/bin/counters_metal.rs b/tests/toyos-rust-tests/src/bin/counters_metal.rs
index f8e46512c..4d023c898 100644
--- a/tests/toyos-rust-tests/src/bin/counters_metal.rs
+++ b/tests/toyos-rust-tests/src/bin/counters_metal.rs
@@ -160,7 +160,7 @@ fn print(phase: &str, read: &Read) {
/// stick, so this prints a line and reads the log until that line comes back.
/// **Twice**: the round that writes the first may itself put a record in the
/// log — the stick's first sync is one — and the second writes it. The
-/// `counters` row reds a kernel record stamped inside the idle second.
+/// `counters` row reds any line stamped inside the idle second.
fn settle() {
let pipe = logkeeper_api::read().unwrap_or_else(|why| panic!("test-runner's `log` port: {why}")).pipe;
let poller = Poller::new(1);
diff --git a/tests/toyos.rs b/tests/toyos.rs
index d7b903dab..d42f80f52 100644
--- a/tests/toyos.rs
+++ b/tests/toyos.rs
@@ -3438,9 +3438,10 @@ const SMI_SPAN_NS: u64 = 4_444_000_000;
/// reads` line names, and none stale; every CPU's performance request
/// declared at boot, `pm_enable=1`, the request Linux makes on this machine
/// (`tests/t14-linux/hwp-request.txt`), and its power envelope in every read
-/// the one its `control_regs:` line holds. No kernel record is stamped in a
-/// whole millisecond between `idle0` and `idle1`: the second is the idle
-/// machine's. From `idle0` to `spin`, at least
+/// the one its `control_regs:` line holds. No line, the kernel's or a
+/// program's, is stamped in a millisecond from `idle0`'s to `idle1`'s, either
+/// edge's included because a line stamped in it may follow the read: the
+/// second is the idle machine's. From `idle0` to `spin`, at least
/// [`SMI_SPAN_NS`] apart, every CPU's SMI count rose alike and by two or more:
/// the firmware's legacy mode, the positive control ACPI stage 1's flatness
/// is read against, and the row that stage changes. Across the spin every
@@ -3490,13 +3491,17 @@ fn counters_on_metal(back: &metal::Readback) -> Result<(), String> {
let (idle1, at1) = phase("idle1")?;
let (spin, at2) = phase("spin")?;
let (from_ms, to_ms) = (at0 / 1_000_000, at1 / 1_000_000);
- let inside: Vec<&str> = kernel
+ let inside: Vec<&str> = log
.text()
.lines()
- .filter(|line| bootlog::record_millis(line).is_some_and(|ms| from_ms < ms && ms < to_ms))
+ .filter(|line| {
+ toyos_logstream::record_ms(line)
+ .or_else(|| toyos_logstream::program_ms(line))
+ .is_some_and(|ms| (from_ms..=to_ms).contains(&ms))
+ })
.collect();
if !inside.is_empty() {
- return Err(format!("the idle second {at0}..{at1} ns holds kernel records: {inside:?}"));
+ return Err(format!("the idle second {at0}..{at1} ns holds lines: {inside:?}"));
}
let linux_request = u64::from_str_radix(include_str!("t14-linux/hwp-request.txt").trim().trim_start_matches("0x"), 16)
.map_err(|e| format!("t14-linux/hwp-request.txt: {e}"))?;
diff --git a/toyos-logstream/src/lib.rs b/toyos-logstream/src/lib.rs
index 7abd58aee..844a99adf 100644
--- a/toyos-logstream/src/lib.rs
+++ b/toyos-logstream/src/lib.rs
@@ -289,6 +289,26 @@ pub fn record_ms(line: &str) -> Option<u64> {
secs.checked_mul(1_000)?.checked_add(millis)
}
+/// The milliseconds since boot a program's line carries ([`ProgramLine`]), or
+/// `None` for any other line.
+///
+/// **Found from the tag back rather than by position**: the wall clock before it
+/// is two words or none, and the words after it are each there or not.
+pub fn program_ms(line: &str) -> Option<u64> {
+ let (head, _) = line.strip_prefix(OPEN)?.split_once(CLOSE)?;
+ let mut words = head.rsplit(' ');
+ Tag::new(words.next()?)?;
+ let field = words.find(|word| {
+ !(word.starts_with("pid=")
+ || word.starts_with("tid=")
+ || [Severity::Warn, Severity::Error, Severity::Alert].iter().any(|s| s.word() == Some(word)))
+ })?;
+ let (secs, millis) = field.split_once('.')?;
+ let secs: u64 = secs.parse().ok()?;
+ let millis: u64 = millis.parse().ok()?;
+ secs.checked_mul(1_000)?.checked_add(millis)
+}
+
/// Whether `line` opens as a program's line: what a judge of the kernel's
/// records leaves out.
pub fn is_program_line(line: &str) -> bool {
@@ -554,6 +574,27 @@ mod tests {
assert_eq!(record_ms(""), None);
}
+ /// Every head [`ProgramLine`] writes reads back to the time it carries, and
+ /// no text after the head answers for it.
+ #[test]
+ fn a_program_lines_time_is_read_inside_its_head_and_nowhere_else() {
+ let tag = Tag::new("test-runner").expect("a tag");
+ for stamp in ["", "2026-09-24 10:00:00", "---------- --------"] {
+ for severity in [Severity::Info, Severity::Warn, Severity::Error, Severity::Alert] {
+ for (tid, pid) in [(0, None), (3, None), (0, Some(9)), (3, Some(9))] {
+ let line =
+ format!("{}", ProgramLine { stamp, at_ns: 1_500_999_999, severity, tid, pid, tag, text: b"9.000" });
+ assert_eq!(program_ms(&line), Some(1_500), "{line:?}");
+ }
+ }
+ }
+ assert_eq!(program_ms("[2026-09-07 22:57:46 3.109 cpu1] exit: a pid=7"), None);
+ assert_eq!(program_ms("{2026-09-24 10:00:00 netstack} 1.000"), None);
+ assert_eq!(program_ms("{1.000 a b} x"), None);
+ assert_eq!(program_ms(" its second line"), None);
+ assert_eq!(program_ms(""), None);
+ }
+
#[test]
fn a_kernel_record_and_its_continuation_are_no_programs() {
assert_eq!(program_line("[2026-09-24 10:00:00 1.216 cpu0] Boot: complete (1216ms)"), None);The two lines inserted into copies of |
|
T14 results at
The new check reds the named mutation on the T14. The full Drive-mode suite at this head (the shared-boot NOTE) runs next. Judge logs: |
|
Full Drive-mode T14 suite at The one failure is this branch's own row, in the full profile: In the full profile the row's idle second starts at ~10.47 s (other jobs ran first), and the kernel's periodic |
The one conflict, tests/toyos-rust-tests/Cargo.toml, keeps both sides: this branch's toyos-logstream and logkeeper-api beside main's toyos-trace. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WcU2Dsw6mDYtwYfzVHPzM8
The full Drive-mode T14 suite at 19ac555 red the `counters` row in its full profile: the job's idle second ran 10.474..11.474 s, and the kernel's idle report (`scheduler::log_health`: `sched:` per CPU and the machine's `PMM:`) printed at 11.254 s inside it. In the isolated `counters` boots the second ran at ~1.2 s, just after every CPU's first report at boot, so the row's verdict depended on where in the boot the job ran. The report is legitimate kernel work with a declared cadence (`SNAPSHOT_INTERVAL`, 10 s): each CPU prints on its first idle trip at least that long after its last, and the machine's `PMM:` on the same rule from the first idle trip of all. It never wakes a CPU, so one overdue prints at whatever next wakes it. In that boot every CPU first printed at 1.135 s, so all were due from ~11.135 s, and the first wake after (11.254 s) set off the whole round. So `counters_metal` reads the report's schedule off the log it already reads to settle: each CPU's last `sched:` record, the last `PMM:` (or, before the first, the earliest `trips=1`), each due `SNAPSHOT_INTERVAL` later. When one falls due before the second could end, it sleeps until every report is due, puts a thread on every CPU for 20 ms so each passes its idle loop and prints what it owes, settles again and rechecks. A report the log does not hold counts as due now. The whole wait keeps the 20 s ceiling the two settle waits had, so the job's budget is unchanged. The judge is unchanged and stays the oracle: any line in the second reds the row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WcU2Dsw6mDYtwYfzVHPzM8
|
Full Drive-mode T14 suite at
Log: |
|
Review round 3, head Round 2 BLOCKERs:
The design question: predicting the report versus changing it. Changing the report is simpler and right. Predicting it is a workaround, built inside a test, for a kernel behaviour.
BLOCKER
NOTE
REMOVE SEND BACK |
Two conflicts against #720's coloured screen lines. toyos-logstream: #720 reads a line's head once, in `shown`, and finds a program line's time as the first head word `millis` parses. `program_ms` now reads its time through that reader instead of walking back from the tag past `pid=`/`tid=`/severity, so one function finds the field. Its host test keeps every head `ProgramLine` writes; the `{1.000 a b} x` case is dropped, being a refusal of the walk this replaces. console: keeps `logkeeper_api::read` for the handshake and #720's `Showing` for drawing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WcU2Dsw6mDYtwYfzVHPzM8
…econd it settles to The owner ruled on 2026-10-04, asked whether to remove the ~10 s idle status report (`sched:`/`PMM:`) from the idle loop, choosing "Remove it entirely": "Delete the periodic report and its counters; hang triage uses the trace diary and panic records instead." Kernel: `scheduler::log_health` and its call on every idle trip go, with `SNAPSHOT_INTERVAL`, `NEXT_HEALTH`, `IDLE_TRIPS` and the `PMM:` deadline. `pmm::dump_stats` goes with the counters only it read: `CATEGORY_STATS`, `LAST_DUMP_NANOS`, `LAST_ALLOC`, and `Category`, which existed only to index them, so `alloc_page`, `claim`, `alloc_contiguous` and `PageAlloc::new` take no category and `PhysPage` carries none. The driver's `parked_len`, `dying_len` and `stopped_len` had no other caller, and the scheduler core's `dying_len` and `stopped_len` none but its own tests, which now count `dying()` and `stopped()`. The idle loop no longer reads the clock or counts a trip for a report. Row: `counters_metal.rs` returns to `19ac555c0`'s, the settle and the judge with no prediction of the report: `Reports`, `quiet`, `REPORT`, `CLEAR`, `WAKE` and the sleep are deleted. Review round 3's design BLOCKER is closed by the ruling rather than by a second reader of the kernel's cadence. Issues: the three that read hang triage off `sched:` lines say the report is gone and quote the ruling; `toyos-explains-itself` no longer lists `PMM:`/`sched:` among the log's numbers; the 2 MiB track says where stage 3's floor is read now; the pipe-lock issue's quoted call loses its category. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WcU2Dsw6mDYtwYfzVHPzM8
|
Full Drive-mode T14 suite at
Log: |
|
Review round 4, head Round 3 BLOCKERs:
The kernel deletion is complete and safe.
BLOCKER
NOTE REMOVE SEND BACK |
…os-acpi1 tests/toyos.rs conflicted in `counters_on_metal`, in its doc and in its SMI summary line. Both sides are kept: #728's refusal of any line stamped in the idle second, and this branch's ACPI mode with the SMI count flat from `idle0` to `spin`, which replaced main's legacy-mode rise. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WcU2Dsw6mDYtwYfzVHPzM8
…once #728's `counters` row refuses any line stamped in its idle second, and this branch runs `acpiserver` on that boot. The server logs a query number the first time the embedded controller raises it, and the kernel logs the claim's first interrupt on the server's read after it. On the T14 the first query came 0.04 to 1.91 s after the server armed across every readback of this branch, so it lands in the second that follows #728's settle about half the time; the counts line, every 30 s from arming, lies past both takes. `counters_metal` now reads the log back after its three reads, through the same settle, and keeps every line stamped in the second. Where every one is the server's or the kernel's first-interrupt line, it takes the three reads again, once. Any other line, or the server's in the second take, is printed over and left to the judge, which is unchanged. The press issue gains `ff4945d6d`'s attended press: query 0x28 at 13.089 s, the press at 13.105 s, one press by the owner's account. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WcU2Dsw6mDYtwYfzVHPzM8
Conflicts against #728: - toyos-logstream: #728's `program_ms` read the old `{…}` program head through `shown()`, both gone here. Folded into `parse`: the counters judge reads `parse(l).and_then(|p| p.ms)`, one reader for every line; its readback test now asserts `parse` on every head `ProgramLine` writes. - console: #728's `logkeeper_api::read()` kept, with this branch's `asked_ms` on `stamp_ns()`. Semantic conflict: #728's idle-second check compared `counters_metal`'s `at`, read off `nanos_since_boot()` (the kernel clock's zero), with line stamps that count from the counter's zero here. `counters_metal` now reads `stamp_ns()` for `at`; the SMI span `at2 - at0` is a difference and unaffected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WcU2Dsw6mDYtwYfzVHPzM8
The kernel's ~10 s idle report (
sched:per CPU,PMM:for the machine) is deleted, with the counters only it read, and thecountersmetal row takes its idle second once the log is quiet.test_rs_counters_metalreads the log back through logkeeper'slogport beforeidle0, and waits until its own line comes back. It prints its three reads only after it has taken all three. The row's judge fails on any line, the kernel's or a program's, stamped in a millisecond the idle second touches. logkeeper's reader handshake has one decoder,logkeeper_api::read, and both the console and the row call it.Why
cpu7 read 1.4 to 1.9% idle on every
countersboot, while cpu0–6 and Linux read about 0.5% or less. The readbacks showed the stick's first sync (usb-storage: disk 0 does not implement SYNCHRONIZE CACHE) stamped 9–10 ms inside the measured second. A job starts while logkeeper is still writing the boot so far to the stick, and that write goes through the/logfileserver, which ran on cpu7.The full Drive-mode suite at
19ac555c0then showed the kernel's idle report printing inside the second when the job ran late in a boot (10.474–11.474 s, the report at 11.254 s). Round 3 predicted the report from the log; review round 3 sent that back as a second reader of the kernel's private cadence. The owner ruled on 2026-10-04, asked whether to remove the ~10 s idle status report from the idle loop, choosing "Remove it entirely": "Delete the periodic report and its counters; hang triage uses the trace diary and panic records instead."Decisions
The idle report and its counters are deleted (the ruling).
scheduler::log_healthand its call on every idle trip go, withSNAPSHOT_INTERVAL,NEXT_HEALTH,IDLE_TRIPSand thePMM:deadline.pmm::dump_statsgoes with the counters only it read:CATEGORY_STATS,LAST_DUMP_NANOS,LAST_ALLOC, andpmm::Category, which existed only to index them. Soalloc_page,claim,alloc_contiguousandPageAlloc::newtake no category, andPhysPagecarries none. The driver'sparked_len,dying_lenandstopped_lenhad no other caller. The scheduler core'sdying_lenandstopped_lenhad none but its own host tests, which now countdying()andstopped().pmm::statsstays:SYS_SYSINFOand the block layer read it.The row is the settle plus the judge.
counters_metal.rsis19ac555c0's again:Reports,quiet,REPORT,CLEAR,WAKEand the sleep are deleted.settleis the free function round 2 had rather than round 3'sLog::settle, because withquietgoneLogwould be a struct with one caller. logkeeper hands a reader of itslogport each round only after the round is written and synced (write:to_volume, thenhub.append). Sosettleprints a line and reads the served log until the line comes back. It does this twice, because the round that writes the first line can itself stamp a record (the stick's first sync), and the second round writes that record. The pipe is dropped beforeidle0. Each wait is bounded bySETTLE_BOUND(10 s, two of logkeeper's rounds at its write budget), a hang ceiling that panics by name, as does EOF.Hang triage moves to the diary and panic records. Three issues read a guest's state off
sched:lines:a-counters-read-under-host-load-can-go-silent-for-15-s,a-shared-boot-stopped-answering-and-no-capture-says-why(whose exit named "the periodic reporter") andqemu-drops-console-output-the-harness-is-slow-to-read. Each now quotes the ruling and points at the diary (kernel/src/trace.rs,/system/bin/trace) and the panic records (kernel/src/panic.rs,kernel/src/blackbox.rs). Their past sightings stay as recorded, being true of the kernel that printed them.toyos-explains-itselfno longer listsPMM:andsched:among the log's numbers. The 2 MiB track's T14 floor cited aPMM:record and now says stage 3's floor is read fromSYS_SYSINFO's used memory. The pipe-lock issue's quotedalloc_pagecall loses its category.program_msreads through Logs on screen are coloured by severity and source, with a compact stamp #720's head reader. On the merge withorigin/main(4d46c8e55),program_msbecameshown(line)?.head?matched toSource::Program, withmillisof the stamp's first word. Logs on screen are coloured by severity and source, with a compact stamp #720'sshownfinds a program line's time as the first head wordmillisparses, so one function finds that field. Its host test still writes every headProgramLinewrites (three stamps × four severities × four tid/pid shapes) and reads each back, and still checks that no text after the head answers for the time. The{1.000 a b} xcase is dropped, being a refusal of the backwards walk this replaces.consolekeepslogkeeper_api::readfor the handshake and Logs on screen are coloured by severity and source, with a compact stamp #720'sShowingfor drawing.The judge is unchanged and is the oracle.
counters_on_metalreads the whole log, programs' lines included, and fails on a line stamped in any millisecond fromidle0's toidle1's, edges included. Lines carry milliseconds only, so a line stamped inidle0's millisecond may come after the read. Rejudged at this head through the newprogram_ms, below.One handshake:
userland/logkeeper-api(round 1 BLOCKER).logkeeper_api::read()sends theREAD, decodes theSERVEDframe, the boot-so-faru64and the one pipe, and returnsServed { pipe, boot_so_far }. Why this home:toyos-logstreamis declared pure (no I/O, nounsafe).toyosSDK is a path dependency ofrust/library/stdand cannot depend ontoyos-logstream.userland/inspect's library is the genericMSG_INSPECTasker.filepicker-apiis the precedent: a userland lib that is the client side of one server's port.The crate adds no external dependency.
A served line that is not UTF-8 panics by name. logkeeper writes every control byte as text, so such a line is logkeeper's defect.
The testcases estate gains the
logport. logkeeperserves = ["log"], as it does in the shippedsystem.toml, and test-runnerreceivesit. Jobs inherit test-runner's namespace whole, so every testcases job now holdslog. The full Drive-mode suites at19ac555c0and78c21c575ran with the grant. The first one's only red is this row's report, and the second's is the census read described below.Measure first, print after. Without this, the row's 81 lines are written inside the second.
issues/t14-cpu7-idles-busier-under-toyos-than-any-cpu-under-linux.mdis closed. Its exit, "cpu7 within cpu0–6's range", is met by the readings below. The fold iscounters_metal.rs's header, and no citation of the slug exists.The SMI issue keeps its MPERF bullet and states the two idle floors itself. It cites
35cd63142, an ancestor of this head.The T14 readings (orchestrator's runs)
Logs are in the orchestrator's job directory,
.claude/jobs/2280e09e/tmp/scratchpad/orch/.49e49a963, with a negative control (comment 5981716271). The control iswt/toyos-countersquiet-revertat0c27e4f56:49e49a963withtests/exactly as the basec4ab2b1e1has it, which is the whole change reverted at the time. It never lands. Each image's sha256 was checked against its request before it was flashed, and head and revert boots were interleaved. Each arm was judged withcargo test --test toyos-build -- --metal --metal-readback <dir> counters(countersquiet-r2/metal/).49e49a963judge-head.log0c27e4f56judge-revert.logIn head's
testcases/kernel.log, both settle lines (1.175 s and 1.188 s) and the stick's first sync (1.183 s) come beforeidle0(1230220488 ns). In the revert's log, the sync is stamped 9.7 ms inside its second. All eight CPUs sit near 0.45% on head because one SMI fell in that second (issues/the-t14s-firmware-interrupts-every-cpu-every-2-2-s-under-toyos.md).09a27943e(comment 5981973507):--metal-readback countersquiet-r3/metal/head counters, EXIT=0,[metal] 3 passed, 0 failed, 3 boot(s).SMI +5 each over 12164 ms, +0 in the idle second. Every CPU reads 0.00% idle busy except cpu3 at 0.03%, against Linux's printed band of 0.11–0.36%.19ac555c0, full Drive-mode suite (countersquiet-r4/metal/drive/728r4-full.log): EXIT=1,[metal] 290 passed, 1 failed, 28 boot(s). The failure is this row: its job ran eighth of 9 in thetestcasesboot, andFAIL counters: the idle second 10474047319..11474110523 ns holds lines: ["[… 11.254 cpu3] sched: cpu=3 … trips=492", "[… 11.254 cpu3] PMM: 320/16024MB used (7852 pages free)", …]. That program is this head's. What differs is the kernel, which then printed the report.78c21c575, full Drive-mode suite (countersquiet-r5/metal/drive/728r5-full.log,cargo test --test toyos-build -- --metal): EXIT=1 (exit status: 1),[metal] 292 passed, 1 failed, 28 boot(s).counterspassed. That head'squiettook the wake path. Every CPU printedsched:andPMM:at 11.185–11.188 s, then settle lines 3–4 at 11.213 s and 11.217 s, thenidle0at 11222163868 ns andidle1at 12222223305 ns, with nothing stamped between.SMI +5 each over 11887 ms, +0 in the idle second, and every CPU read 0.00% idle busy except cpu3 at 0.03% (readback728r5-full-readback/testcases/kernel.log).irq_census_conservation, in the sametestcasesboot:FAIL irq_census_conservation: cpu1 counted 24211 interrupt(s) and attributed 24210 to sources — a source is not being counted: Census { cpu: 1, total: 24211, by_source: [1141, 22906, 0, 0, 0, 0, 0, 0, 142, 21, 0, 0] }.kernel/src/irq_census.rs.irq_took!addsTOTALfirst and the source second, andreadloadsTOTALfirst and the sources after, allRelaxed, from another CPU.counters'loadedphase, with cpu1 atkick=22906. One kick landing between the two loads gives exactlytotal = sum + 1.loadedis changed by this branch. The red is under separate investigation as main's defect, and this branch does not fix it.78c21c575'squietadded is deleted here.T14 at
e2ed813d1(orchestrator's run of the full Drive-mode suite,t14-drive.sh, worktree clean at head, 2592 s; logorch/countersquiet-r6/metal/drive/728r6-full.log): EXIT=1,[metal] 292 passed, 1 failed, 28 boot(s).counterspasses:SMI +5 each over 11878 ms, +0 in the idle second, idle busy cpu0–7 0.00–0.03% (Linux band 0.11–0.36%). The one failure isirq_census_conservation's monotonic check (cpu1's census went backwards, 56975 then 56974): main's code, the stamp-ordering exposure #734's review named and #734 files asissues/the-irq-census-judge-reds-on-two-exits-stamped-in-the-other-order.md.The judge at this head: rejudged readbacks
rejudge.shcopies each readback withoutimage.imgand runscargo test --test toyos-build -- --metal --metal-readback <copy> counters.rejudge.revise2ed813d1andrejudge.statusis empty. Logs are under.claude/jobs/2280e09e/tmp/scratchpad/orch/countersquiet-r6/.plain,insertedandedgeare copies of the09a27943eboot.insertedadds a program line stamped1.500, inside the second.edgeadds one stamped1.187,idle0's own millisecond.full-r4is the19ac555c0full readback, which holds the report inside the second.full-r5is the78c21c575full readback.plain3 passedrejudge-plain.loginsertedFAIL counters: the idle second 1187629842..2187688477 ns holds lines: ["{… 1.500 pid=9 test-runner} counters_metal idle0: printed inside the second"]rejudge-inserted.logedge1.187linerejudge-edge.logfull-r411.254sched:andPMM:linesrejudge-full-r4.logfull-r53 passedrejudge-full-r5.logRound 2's control for the judge (
09a27943e's judge, EXIT=0 oninsertedandedge, comment 5982166956) stands, sincecounters_on_metalis unchanged.Gates (head
e2ed813d1; scriptgates.sh, logs under.claude/jobs/2280e09e/tmp/scratchpad/orch/countersquiet-r6/)gates.revise2ed813d1, andgates.pre-statusandgates.statusare empty. The head contains the merge oforigin/main4d46c8e55(6d1aea97b).cargo run -- --ci host: EXIT=0,ci-host.log(Host: 75 step(s), all green). The first attempt at an earlier amend of the same commit was EXIT=1 on clippy's unusedDurationimport inscheduler.rs(ci-host-attempt1.log), fixed before this run.cargo run -- --build-only --console-boot, the only image that shipsconsole: EXIT=0,build-console.logcargo run -- --build-only: EXIT=0,build-only.logcargo test(the guest suite): EXIT=0,guest.log(28 passed, 28 total)Staged for the T14 (each staging command exits 2, the staged verdict; the full suite was then run, above). The sha256s are in
image-hashes.txt.countersat head,stage-head.log,metal/head/request.txt:testcases:23f05c9dbcc526bae9422b7add898b2f9598e97b806ff6b53ee206db48171eb5shared:db911f0bf34b5d1d6da761ee6454f2f641d568016050a7f206a5900ef701fa3dshared-debug:a9e71a82374da8218455d80fe057b774d6d674ed90e747cdab9bb4218caac2c0stage-full.log,metal/full/request.txt: 65 registrations and 228 shared members over 28 boots.testcases:7ec01fef0d1ce6983c4c809e3a807a767278cbbf5799c8fce36753c98fb717f8shared:874bcb970eeac24ae23e30c0975927614828240e10b1fdb8e77f511575cd4ec2shared-2:9bf4a1cd634d3a3a005ef71b8a5402cd468ae500bbffa8ad49f5274c2b9422c2shared-debug:63a83c7addeff11cb1933b3f19f15d3593cd5f112afb9c6fccf63508f015d36eNo QEMU tier runs
test_rs_counters_metal: it is onRUST_SKIPbecause its own row runs it, and its verdict is a reading of the T14's counters. The kernel deletion is reached by every guest boot of the guest suite and by the host suite's clippy and kernel-library tests. No guest test bootsconsole.logkeeper_api::readhas run on the T14 at09a27943e,19ac555c0and78c21c575.Checks of the high-risk parts
pmm): the change only removes accounting. No bitmap, pin or free path changes. Every allocation call site drops one argument, and the host suite's clippy builds every kernel feature set on both architectures. Nothing reads what went:git grep -woutsideissues/forCategory,dump_stats,CATEGORY_STATS,log_health,NEXT_HEALTH,IDLE_TRIPSandSNAPSHOT_INTERVALfinds no code. The one fixture string left, intoyos-blackbox's tests, is someone else's record text and not a reader.19ac555c0full readback, rejudged at this head (full-r4), is red on the report. That run is this row's program on a kernel that still printed the report.What I am unsure of
idle0's millisecond before the read is refused with the ones after it. Every T14 readback so far has 2 ms or more of clearance.waiton the job. The readings say that costs nothing measurable, but no reading separates the two.toyos-blackbox's fixtureA_BREAK_AND_ITS_RECOVERYstill holds asched:and aPMM:string as "somebody else's records". They are arbitrary non-USB text, so they are left as they are.Net (
git diff --shortstat origin/main...e2ed813d1): +301 −308.pmm.rs+8 −107,scheduler.rs+1 −67,driver.rs−17,pure/sched/cpu.rs+20 −29, and one line per allocation call site)toyos-logstreamprogram_ms+8, its test +21)userland/logkeeper-apiconsolecounters_metal.rs+75 −9,tests/toyos.rs(judge) +22 −4,testcases/system.toml+8 −4🤖 Generated with Claude Code
https://claude.ai/code/session_01WcU2Dsw6mDYtwYfzVHPzM8