Skip to content

The kernel's idle report is gone, and the counters row measures a quiet idle second - #728

Merged
Japabu merged 12 commits into
mainfrom
wt/toyos-countersquiet
Oct 4, 2026
Merged

Japabu merged 12 commits into
mainfrom
wt/toyos-countersquiet

Conversation

@Japabu

@Japabu Japabu commented Oct 4, 2026 •

Copy link
Copy Markdown
Collaborator

The kernel's ~10 s idle report (sched: per CPU, PMM: for the machine) is deleted, with the counters only it read, and the counters metal row takes its idle second once the log is quiet. test_rs_counters_metal reads the log back through logkeeper's log port before idle0, and waits until its own line comes back. It prints its three reads only after it has taken all three. The row's judge fails on any line, the kernel's or a program's, stamped in a millisecond the idle second touches. logkeeper's reader handshake has one decoder, logkeeper_api::read, and both the console and the row call it.

Why

cpu7 read 1.4 to 1.9% idle on every counters boot, while cpu0–6 and Linux read about 0.5% or less. The readbacks showed the stick's first sync (usb-storage: disk 0 does not implement SYNCHRONIZE CACHE) stamped 9–10 ms inside the measured second. A job starts while logkeeper is still writing the boot so far to the stick, and that write goes through the /log fileserver, which ran on cpu7.

The full Drive-mode suite at 19ac555c0 then showed the kernel's idle report printing inside the second when the job ran late in a boot (10.474–11.474 s, the report at 11.254 s). Round 3 predicted the report from the log; review round 3 sent that back as a second reader of the kernel's private cadence. The owner ruled on 2026-10-04, asked whether to remove the ~10 s idle status report from the idle loop, choosing "Remove it entirely": "Delete the periodic report and its counters; hang triage uses the trace diary and panic records instead."

Decisions

  • The idle report and its counters are deleted (the ruling). scheduler::log_health and its call on every idle trip go, with SNAPSHOT_INTERVAL, NEXT_HEALTH, IDLE_TRIPS and the PMM: deadline. pmm::dump_stats goes with the counters only it read: CATEGORY_STATS, LAST_DUMP_NANOS, LAST_ALLOC, and pmm::Category, which existed only to index them. So alloc_page, claim, alloc_contiguous and PageAlloc::new take no category, and PhysPage carries none. The driver's parked_len, dying_len and stopped_len had no other caller. The scheduler core's dying_len and stopped_len had none but its own host tests, which now count dying() and stopped(). pmm::stats stays: SYS_SYSINFO and the block layer read it.

  • The row is the settle plus the judge. counters_metal.rs is 19ac555c0's again: Reports, quiet, REPORT, CLEAR, WAKE and the sleep are deleted. settle is the free function round 2 had rather than round 3's Log::settle, because with quiet gone Log would be a struct with one caller. logkeeper hands a reader of its log port each round only after the round is written and synced (write: to_volume, then hub.append). So settle prints a line and reads the served log until the line comes back. It does this twice, because the round that writes the first line can itself stamp a record (the stick's first sync), and the second round writes that record. The pipe is dropped before idle0. Each wait is bounded by SETTLE_BOUND (10 s, two of logkeeper's rounds at its write budget), a hang ceiling that panics by name, as does EOF.

  • Hang triage moves to the diary and panic records. Three issues read a guest's state off sched: lines: a-counters-read-under-host-load-can-go-silent-for-15-s, a-shared-boot-stopped-answering-and-no-capture-says-why (whose exit named "the periodic reporter") and qemu-drops-console-output-the-harness-is-slow-to-read. Each now quotes the ruling and points at the diary (kernel/src/trace.rs, /system/bin/trace) and the panic records (kernel/src/panic.rs, kernel/src/blackbox.rs). Their past sightings stay as recorded, being true of the kernel that printed them. toyos-explains-itself no longer lists PMM: and sched: among the log's numbers. The 2 MiB track's T14 floor cited a PMM: record and now says stage 3's floor is read from SYS_SYSINFO's used memory. The pipe-lock issue's quoted alloc_page call loses its category.

  • program_ms reads through Logs on screen are coloured by severity and source, with a compact stamp #720's head reader. On the merge with origin/main (4d46c8e55), program_ms became shown(line)?.head? matched to Source::Program, with millis of the stamp's first word. Logs on screen are coloured by severity and source, with a compact stamp #720's shown finds a program line's time as the first head word millis parses, so one function finds that field. Its host test still writes every head ProgramLine writes (three stamps × four severities × four tid/pid shapes) and reads each back, and still checks that no text after the head answers for the time. The {1.000 a b} x case is dropped, being a refusal of the backwards walk this replaces. console keeps logkeeper_api::read for the handshake and Logs on screen are coloured by severity and source, with a compact stamp #720's Showing for drawing.

  • The judge is unchanged and is the oracle. counters_on_metal reads the whole log, programs' lines included, and fails on a line stamped in any millisecond from idle0's to idle1's, edges included. Lines carry milliseconds only, so a line stamped in idle0's millisecond may come after the read. Rejudged at this head through the new program_ms, below.

  • One handshake: userland/logkeeper-api (round 1 BLOCKER). logkeeper_api::read() sends the READ, decodes the SERVED frame, the boot-so-far u64 and the one pipe, and returns Served { pipe, boot_so_far }. Why this home:

    • toyos-logstream is declared pure (no I/O, no unsafe).
    • The toyos SDK is a path dependency of rust/library/std and cannot depend on toyos-logstream.
    • userland/inspect's library is the generic MSG_INSPECT asker.
    • filepicker-api is the precedent: a userland lib that is the client side of one server's port.

    The crate adds no external dependency.

  • A served line that is not UTF-8 panics by name. logkeeper writes every control byte as text, so such a line is logkeeper's defect.

  • The testcases estate gains the log port. logkeeper serves = ["log"], as it does in the shipped system.toml, and test-runner receives it. Jobs inherit test-runner's namespace whole, so every testcases job now holds log. The full Drive-mode suites at 19ac555c0 and 78c21c575 ran with the grant. The first one's only red is this row's report, and the second's is the census read described below.

  • Measure first, print after. Without this, the row's 81 lines are written inside the second.

  • issues/t14-cpu7-idles-busier-under-toyos-than-any-cpu-under-linux.md is closed. Its exit, "cpu7 within cpu0–6's range", is met by the readings below. The fold is counters_metal.rs's header, and no citation of the slug exists.

  • The SMI issue keeps its MPERF bullet and states the two idle floors itself. It cites 35cd63142, an ancestor of this head.

The T14 readings (orchestrator's runs)

Logs are in the orchestrator's job directory, .claude/jobs/2280e09e/tmp/scratchpad/orch/.

49e49a963, with a negative control (comment 5981716271). The control is wt/toyos-countersquiet-revert at 0c27e4f56: 49e49a963 with tests/ exactly as the base c4ab2b1e1 has it, which is the whole change reverted at the time. It never lands. Each image's sha256 was checked against its request before it was flashed, and head and revert boots were interleaved. Each arm was judged with cargo test --test toyos-build -- --metal --metal-readback <dir> counters (countersquiet-r2/metal/).

arm judge exit log cpu7 idle busy cpu0–6 idle busy SMIs in the idle second
head 49e49a963 EXIT=0 judge-head.log 0.46% 0.46–0.49% +1
revert 0c27e4f56 EXIT=0 judge-revert.log 2.89% 0.46–0.72% +1

In head's testcases/kernel.log, both settle lines (1.175 s and 1.188 s) and the stick's first sync (1.183 s) come before idle0 (1230220488 ns). In the revert's log, the sync is stamped 9.7 ms inside its second. All eight CPUs sit near 0.45% on head because one SMI fell in that second (issues/the-t14s-firmware-interrupts-every-cpu-every-2-2-s-under-toyos.md).

09a27943e (comment 5981973507): --metal-readback countersquiet-r3/metal/head counters, EXIT=0, [metal] 3 passed, 0 failed, 3 boot(s). SMI +5 each over 12164 ms, +0 in the idle second. Every CPU reads 0.00% idle busy except cpu3 at 0.03%, against Linux's printed band of 0.11–0.36%.

19ac555c0, full Drive-mode suite (countersquiet-r4/metal/drive/728r4-full.log): EXIT=1, [metal] 290 passed, 1 failed, 28 boot(s). The failure is this row: its job ran eighth of 9 in the testcases boot, and FAIL counters: the idle second 10474047319..11474110523 ns holds lines: ["[… 11.254 cpu3] sched: cpu=3 … trips=492", "[… 11.254 cpu3] PMM: 320/16024MB used (7852 pages free)", …]. That program is this head's. What differs is the kernel, which then printed the report.

78c21c575, full Drive-mode suite (countersquiet-r5/metal/drive/728r5-full.log, cargo test --test toyos-build -- --metal): EXIT=1 (exit status: 1), [metal] 292 passed, 1 failed, 28 boot(s).

  • counters passed. That head's quiet took the wake path. Every CPU printed sched: and PMM: at 11.185–11.188 s, then settle lines 3–4 at 11.213 s and 11.217 s, then idle0 at 11222163868 ns and idle1 at 12222223305 ns, with nothing stamped between. SMI +5 each over 11887 ms, +0 in the idle second, and every CPU read 0.00% idle busy except cpu3 at 0.03% (readback 728r5-full-readback/testcases/kernel.log).
  • The red is irq_census_conservation, in the same testcases boot: FAIL irq_census_conservation: cpu1 counted 24211 interrupt(s) and attributed 24210 to sources — a source is not being counted: Census { cpu: 1, total: 24211, by_source: [1141, 22906, 0, 0, 0, 0, 0, 0, 142, 21, 0, 0] }.
    • Review round 3 reads it as a torn cross-CPU read in main's kernel/src/irq_census.rs. irq_took! adds TOTAL first and the source second, and read loads TOTAL first and the sources after, all Relaxed, from another CPU.
    • The failing line was printed by cpu7 about cpu1 at a process exit, at 36.226 s. That is inside counters' loaded phase, with cpu1 at kick=22906. One kick landing between the two loads gives exactly total = sum + 1.
    • Neither the census nor loaded is changed by this branch. The red is under separate investigation as main's defect, and this branch does not fix it.
    • The 20 ms wake of every CPU that 78c21c575's quiet added is deleted here.

T14 at e2ed813d1 (orchestrator's run of the full Drive-mode suite, t14-drive.sh, worktree clean at head, 2592 s; log orch/countersquiet-r6/metal/drive/728r6-full.log): EXIT=1, [metal] 292 passed, 1 failed, 28 boot(s). counters passes: SMI +5 each over 11878 ms, +0 in the idle second, idle busy cpu0–7 0.00–0.03% (Linux band 0.11–0.36%). The one failure is irq_census_conservation's monotonic check (cpu1's census went backwards, 56975 then 56974): main's code, the stamp-ordering exposure #734's review named and #734 files as issues/the-irq-census-judge-reds-on-two-exits-stamped-in-the-other-order.md.

The judge at this head: rejudged readbacks

rejudge.sh copies each readback without image.img and runs cargo test --test toyos-build -- --metal --metal-readback <copy> counters. rejudge.rev is e2ed813d1 and rejudge.status is empty. Logs are under .claude/jobs/2280e09e/tmp/scratchpad/orch/countersquiet-r6/.

  • plain, inserted and edge are copies of the 09a27943e boot. inserted adds a program line stamped 1.500, inside the second. edge adds one stamped 1.187, idle0's own millisecond.
  • full-r4 is the 19ac555c0 full readback, which holds the report inside the second.
  • full-r5 is the 78c21c575 full readback.
copy exit verdict log
plain EXIT=0 PASS, 3 passed rejudge-plain.log
inserted EXIT=1 FAIL counters: the idle second 1187629842..2187688477 ns holds lines: ["{… 1.500 pid=9 test-runner} counters_metal idle0: printed inside the second"] rejudge-inserted.log
edge EXIT=1 names the 1.187 line rejudge-edge.log
full-r4 EXIT=1 names the 11.254 sched: and PMM: lines rejudge-full-r4.log
full-r5 EXIT=0 PASS, 3 passed rejudge-full-r5.log

Round 2's control for the judge (09a27943e's judge, EXIT=0 on inserted and edge, comment 5982166956) stands, since counters_on_metal is unchanged.

Gates (head e2ed813d1; script gates.sh, logs under .claude/jobs/2280e09e/tmp/scratchpad/orch/countersquiet-r6/)

gates.rev is e2ed813d1, and gates.pre-status and gates.status are empty. The head contains the merge of origin/main 4d46c8e55 (6d1aea97b).

  • cargo run -- --ci host: EXIT=0, ci-host.log (Host: 75 step(s), all green). The first attempt at an earlier amend of the same commit was EXIT=1 on clippy's unused Duration import in scheduler.rs (ci-host-attempt1.log), fixed before this run.
  • cargo run -- --build-only --console-boot, the only image that ships console: EXIT=0, build-console.log
  • cargo run -- --build-only: EXIT=0, build-only.log
  • cargo test (the guest suite): EXIT=0, guest.log (28 passed, 28 total)

Staged for the T14 (each staging command exits 2, the staged verdict; the full suite was then run, above). The sha256s are in image-hashes.txt.

  • counters at head, stage-head.log, metal/head/request.txt:
    • testcases: 23f05c9dbcc526bae9422b7add898b2f9598e97b806ff6b53ee206db48171eb5
    • shared: db911f0bf34b5d1d6da761ee6454f2f641d568016050a7f206a5900ef701fa3d
    • shared-debug: a9e71a82374da8218455d80fe057b774d6d674ed90e747cdab9bb4218caac2c0
  • the whole metal suite at head, no filter, stage-full.log, metal/full/request.txt: 65 registrations and 228 shared members over 28 boots.
    • testcases: 7ec01fef0d1ce6983c4c809e3a807a767278cbbf5799c8fce36753c98fb717f8
    • shared: 874bcb970eeac24ae23e30c0975927614828240e10b1fdb8e77f511575cd4ec2
    • shared-2: 9bf4a1cd634d3a3a005ef71b8a5402cd468ae500bbffa8ad49f5274c2b9422c2
    • shared-debug: 63a83c7addeff11cb1933b3f19f15d3593cd5f112afb9c6fccf63508f015d36e

No QEMU tier runs test_rs_counters_metal: it is on RUST_SKIP because its own row runs it, and its verdict is a reading of the T14's counters. The kernel deletion is reached by every guest boot of the guest suite and by the host suite's clippy and kernel-library tests. No guest test boots console. logkeeper_api::read has run on the T14 at 09a27943e, 19ac555c0 and 78c21c575.

Checks of the high-risk parts

  • Memory management (pmm): the change only removes accounting. No bitmap, pin or free path changes. Every allocation call site drops one argument, and the host suite's clippy builds every kernel feature set on both architectures. Nothing reads what went: git grep -w outside issues/ for Category, dump_stats, CATEGORY_STATS, log_health, NEXT_HEALTH, IDLE_TRIPS and SNAPSHOT_INTERVAL finds no code. The one fixture string left, in toyos-blackbox's tests, is someone else's record text and not a reader.
  • Scheduler idle loop: one call is removed and nothing is added.
  • Negative control for the row: the 19ac555c0 full readback, rejudged at this head (full-r4), is red on the report. That run is this row's program on a kernel that still printed the report.

What I am unsure of

  • That nothing else on a late-boot idle machine stamps a record inside the second. The report was the one seen, and the judge reds on any other. The full suite at this head showed it: no record in the second.
  • That the stick's first sync is the only record a round's own write stamps. The judge fails if another record lands inside the second, but two rounds might still be one short on some other boot.
  • That counting the edge milliseconds never refuses a quiet boot. A line stamped in idle0's millisecond before the read is refused with the ones after it. Every T14 readback so far has 2 ms or more of clearance.
  • cpu7 also carries test-runner, parked in wait on the job. The readings say that costs nothing measurable, but no reading separates the two.
  • toyos-blackbox's fixture A_BREAK_AND_ITS_RECOVERY still holds a sched: and a PMM: string as "somebody else's records". They are arbitrary non-USB text, so they are left as they are.

Net (git diff --shortstat origin/main...e2ed813d1): +301 −308.

part change
kernel +51 −242 (pmm.rs +8 −107, scheduler.rs +1 −67, driver.rs −17, pure/sched/cpu.rs +20 −29, and one line per allocation call site)
toyos-logstream +29 (program_ms +8, its test +21)
userland/logkeeper-api +45
console +2 −13
manifests and locks +11 userland, +21 tests
tests counters_metal.rs +75 −9, tests/toyos.rs (judge) +22 −4, testcases/system.toml +8 −4
issues +37 −36

🤖 Generated with Claude Code

https://claude.ai/code/session_01WcU2Dsw6mDYtwYfzVHPzM8

Japabu and others added 2 commits October 4, 2026 16:44
test_rs_counters_metal printed its idle0 read, 81 lines, at the start of
the very second it then measured. logkeeper writes and syncs those lines to
the stick through the /log fileserver, which every counters boot so far
placed on cpu7, so cpu7's idle busy fraction was the row's own output.

The cpu7_scout row (wt/toyos-cpu7 at 22d2411, one T14 boot, judge
EXIT=0) measured back-to-back idle seconds between counters rounds, four of
each arm after a discarded warm-up. Mean idle busy:

  quiet (prints nothing):        cpu0-6 0.00-0.03%, cpu7 0.12%
  loud  (the row's 81 lines):    cpu0-6 0.34-0.45%, cpu7 0.84%
  dose  (four times the lines):  cpu0-6 0.23-0.55%, cpu7 1.27%

The row now takes idle0, idle1 and spin first and prints the three after
the spin's read; the lines and their parsing are unchanged. The judge also
says how many SMIs landed in the idle second: in the scout's seconds the
cpu0-6 floor of about 0.45% appears exactly in the seconds whose SMI count
moved, so a row's idle reading is read beside it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WcU2Dsw6mDYtwYfzVHPzM8
…PERF

t14-cpu7-idles-busier-under-toyos-than-any-cpu-under-linux.md is closed:
its exit asked for a reading naming what ran on cpu7 across the counters
row's idle second, and for the row's doc if it was the row's own work. The
cpu7_scout row (wt/toyos-cpu7 at 22d2411, image sha256
a5dcc2a77f5d35f016a663e634d1003ad30e9de027fdd53868cd8dd20fafa035, one T14
boot run by the orchestrator, judge EXIT=0) names it: the /log fileserver,
pid 4, spawned onto cpu7 (dst=7), writing the row's own 81-line idle0 print
to the stick. With nothing printed cpu7 reads 22 to 28 ppm in a second no
earlier print spills into, under a tenth of the lowest machine-wide 10 s interval
Linux's idle turbostat reads on that machine (0.11%).
The fold is test_rs_counters_metal's header: nothing is printed until the
three reads are taken.

The SMI issue gains the MPERF reading of the same boot. The scout's seconds,
as its judge printed them (busy in ppm of each CPU's stamp, cpu0..cpu7):

  second 0 warmup Loud: from 1172790890 ns to 2172974091 ns, smi +1, busy ppm [4979 4793 4839 5343 7107 4558 4573 19060], kicks [12 4 2 5 1 1 1 4]
  second 1 round Quiet: from 2172974091 ns to 3173035738 ns, smi +0, busy ppm [26 63 25 230 26 285 27 28], kicks [1 1 1 1 1 1 1 1]
  second 2 round Loud: from 3173035738 ns to 4173341548 ns, smi +1, busy ppm [4884 4897 4556 5831 4554 4540 4555 9875], kicks [6 2 1 3 1 1 1 3]
  second 3 round Dose: from 4173341548 ns to 5174342442 ns, smi +0, busy ppm [389 1470 24 3723 22 7 24 7388], kicks [6 2 1 3 1 1 1 3]
  second 4 round Loud: from 5174342442 ns to 6174633115 ns, smi +0, busy ppm [336 341 22 1089 22 7 22 4578], kicks [6 1 1 3 1 1 1 3]
  second 5 round Dose: from 6174633115 ns to 7175633698 ns, smi +1, busy ppm [4904 5746 4551 7810 4546 4535 4550 12208], kicks [6 1 1 3 1 1 1 3]
  second 6 round Quiet: from 7175633698 ns to 8175690541 ns, smi +0, busy ppm [21 58 25 217 24 7 25 22], kicks [1 0 1 1 1 1 1 1]
  second 7 round Dose: from 8175690541 ns to 9176698816 ns, smi +1, busy ppm [4819 5635 4570 7501 4568 4553 4571 13604], kicks [6 2 1 3 1 1 1 3]
  second 8 round Quiet: from 9176698816 ns to 10176757787 ns, smi +0, busy ppm [22 59 21 218 22 6 23 25], kicks [1 0 1 1 1 1 1 1]
  second 9 round Loud: from 10176757787 ns to 11177135943 ns, smi +1, busy ppm [4867 4937 4557 5511 4552 4541 4557 9970], kicks [6 2 1 4 1 1 1 3]
  second 10 round Quiet: from 11177135943 ns to 12177194852 ns, smi +0, busy ppm [387 73 40 356 35 16 36 4875], kicks [6 1 1 3 1 1 1 3]
  second 11 round Loud: from 12177194852 ns to 13177496481 ns, smi +1, busy ppm [4879 4879 4561 5643 4555 4545 4559 9318], kicks [6 1 1 3 1 1 1 3]
  second 12 round Dose: from 13177496481 ns to 14178489116 ns, smi +0, busy ppm [247 1076 25 3026 24 6 21 17694], kicks [6 3 1 3 1 1 1 3]
  Dose: mean idle busy cpu0=0.26% cpu1=0.35% cpu2=0.23% cpu3=0.55% cpu4=0.23% cpu5=0.23% cpu6=0.23% cpu7=1.27%
  Loud: mean idle busy cpu0=0.37% cpu1=0.38% cpu2=0.34% cpu3=0.45% cpu4=0.34% cpu5=0.34% cpu6=0.34% cpu7=0.84%
  Quiet: mean idle busy cpu0=0.01% cpu1=0.01% cpu2=0.00% cpu3=0.03% cpu4=0.00% cpu5=0.01% cpu6=0.00% cpu7=0.12%

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WcU2Dsw6mDYtwYfzVHPzM8
@Japabu

Japabu commented Oct 4, 2026

Copy link
Copy Markdown
Collaborator Author

T14 results (orchestrator's run; worktree clean at 35cd63142, each image's sha256 checked against the request before its flash; boots interleaved head/revert per case).

arm counters cpu7 idle busy cpu0–6 idle busy
head 35cd63142 (measure first, print after) EXIT=0, 3 passed 1.40% 0.00–0.23%
revert 5e5bac515 EXIT=0, 3 passed 1.43% 0.00–0.25%

The change does not move cpu7: head and revert read the same. The body's "Unsure" first suspect (the round logkeeper writes for the job's launch lines — ===TEST_START=== and the spawn: records, stamped ~2 ms before idle0) fits. Judge logs: orch/countersquiet/metal/judge-head.log, judge-revert.log in the orchestrator's job directory.

At head 35cd631 the T14 read cpu7 at 1.40% idle busy, and 1.43% with the
row's print-after change reverted: moving the row's print did not move cpu7.
The readbacks show why. The job starts about 8 ms after logkeeper does, and
the stick's first sync (`usb-storage: disk 0 does not implement SYNCHRONIZE
CACHE`, a record the /log fileserver's path stamps on cpu7) is at 1.182 s on
head against idle0 at 1.173184 s, and at 1.183 s on the revert against idle0
at 1.173472 s: logkeeper is still writing the boot so far and the job's
launch lines to the stick inside the measured second, on both arms.

The binary now reads the log through logkeeper's `log` port before idle0: it
prints a line and reads the served log until the line comes back, which
logkeeper hands a reader only after the round is written and synced. Twice,
because the round that writes the first line may itself stamp a record (the
stick's first sync does), which the second round writes. Each wait is bounded
by two of logkeeper's 5 s write budgets and panics past it.

The testcases estate gives logkeeper its `log` port and test-runner the
connector to it, which the job inherits.

The cpu7 issue is restored: the head's T14 reading does not meet its exit,
and no reading of this change exists yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WcU2Dsw6mDYtwYfzVHPzM8
@Japabu Japabu changed the title The counters row measures its idle second before printing it; cpu7's excess was the row's own log The counters row takes its idle second once the log holds its own line Oct 4, 2026
@Japabu

Japabu commented Oct 4, 2026

Copy link
Copy Markdown
Collaborator Author

T14 results at 49e49a963 (orchestrator's run; worktree clean at head, each image's sha256 checked against the request before its flash; boots interleaved head/revert per case).

arm counters cpu7 idle busy cpu0–6 idle busy
head 49e49a963 (waits for the log to be quiet) EXIT=0, 3 passed 0.46% 0.46–0.49%
revert 0c27e4f56 EXIT=0, 3 passed 2.89% 0.46–0.72%

At head cpu7 reads the same as every other CPU. All eight at ~0.46% matches the SMI floor this PR records (an SMI in the measured second costs every CPU ~4.5 ms), so this head's second likely held one; the judge log names the count. Judge logs: orch/countersquiet-r2/metal/judge-head.log, judge-revert.log in the orchestrator's job directory.

@Japabu

Japabu commented Oct 4, 2026

Copy link
Copy Markdown
Collaborator Author

Review round 1, head 49e49a963, against origin/main aceb3ef3a. A trial merge (git merge-tree --write-tree origin/main 49e49a963) is clean (tree 93c27586f). The only file both sides touch is tests/testcases/system.toml, where #711 adds [programs.symbolize] beside this branch's grants. Net change is +127 −21: production +0, tests +106 −16, issues +21 −5.

What the readings and the code show:

  • The wait is bounded and fails loudly. It waits on the pipe with a Poller, bounded at 10 s per line, and panics by name when the time runs out or when logkeeper closes the pipe. The watch fires once and is armed again on every pass (kernel/src/inbox/mod.rs), and a timeout of 0 means non-blocking, so no pass blocks forever. LOG_WRITE_BUDGET does not cap a round, it only names one too slow. The 10 s bound is therefore a hang ceiling, not a guarantee. The worst case (2 × 10 s) still fits inside JOB_BOUND_MS (60 s) before LOADED runs.
  • The grants are the least that does the job. logread gives the record ring, not whether a round is durable. Only the log port answers after to_volume (userland/logkeeper/src/main.rs write). Jobs inherit test-runner's namespace whole, and no per-job grant exists, so every testcases job now holds log. Shipped system.toml already serves it.
  • Head readback, countersquiet-r2/metal/head/testcases/kernel.log. Both settle lines (1.175 s and 1.188 s) and the stick's first sync (1.183 s) come before idle0 (1230220488 ns). No record is stamped between idle0 and idle1 (2230283529 ns). One SMI fell in the idle second, and all eight CPUs read 0.46–0.49%. The revert put the sync 9.7 ms inside its second, and cpu7 read 2.89% against 0.46–0.72% on cpu0–6. The negative control is the whole change reverted (tests/ as c4ab2b1e1 has it) on the same base.
  • The SMI bullet's numbers match cpu7/metal/judge-cpu7_scout.log. Five SMI seconds (2, 5, 7, 9, 11) read 4535–4571 ppm on cpu2/4/5/6, and seven without an SMI read 6–285 ppm. The /log fileserver ran on cpu7, logkeeper and diskserver on cpu3, the scout on cpu1, and soundserver on cpu4.

BLOCKER

  • issues/t14-cpu7-idles-busier-under-toyos-than-any-cpu-under-linux.md:24-36 — the issue's exit is met but the file stays. It says "that it is cpu7's excess is not yet shown", and the exit is "a T14 reading of the row with that wait in which cpu7's idle busy fraction is within cpu0 to cpu6's". The round-2 readback at this head reads cpu7 0.46% inside 0.46–0.49%, and its control reads 2.89%. Landing it open puts a false record on main. Fix: delete the file by issues/README.md's closing procedure (the fold is already counters_metal.rs's header, and no citation of the slug exists), with the readings in that commit's message.
  • PR body — false at this head, and it becomes main's merge commit. It says "Negative control and the T14 (staged, not run)" and "No T14 reading of this head exists yet, and this branch claims no number for it". It predicts the revert's cpu7 at "about 1.4%", which read 2.89%. Fix: carry the round-2 T14 table, with each judge's command, exit code and log path, and drop the staged-not-run section.
  • PR body "Gates" — the evidence is incomplete. cargo run -- --ci host and the guest suite are each given as a command and EXIT=0 with no log, and "27 of 27 guest tests" is a count read off output, not a log. Fix: name each run's log at 49e49a963.
  • tests/toyos-rust-tests/src/bin/counters_metal.rs:165-174 — settle repeats the log port's reader handshake that userland/console/src/main.rs:79 Log::subscribe already has: endow::service(SERVICE), signal(READ), SERVED, the u64 length, one pipe handle, Pipe::from_raw. Two programs now decode logkeeper's answer separately, and this copy runs only on a T14 boot, so a change to serve.rs's answer that misses it lands unseen. Fix: one function returning the pipe and the boot's length, called by both. Its home must be a crate allowed to do I/O, since toyos-logstream is declared pure; the implementer names that home in the PR.

NOTE

  • tests/toyos-rust-tests/src/bin/counters_metal.rs:187 — from_utf8(line).unwrap_or_default() silently turns a line that is not UTF-8 into "". logkeeper writes every control byte as text, so such a line is logkeeper's defect, and it should panic by name.
  • issues/the-t14s-firmware-interrupts-every-cpu-every-2-2-s-under-toyos.md:34 — "the counters row's "two idle floors"" quotes a phrase that appears nowhere in the tree (git grep -i "idle floor" 49e49a963 finds only this line). The row does not say it, so the quote cites nothing.
  • tests/toyos-rust-tests/src/bin/counters_metal.rs:178-180 — "Twice" fits the one record kind seen (the stick's first sync). Nothing in the row holds that the second is quiet: the judge reads idle busy and holds nothing about it. It rests on the readback alone, as the body's "What I am unsure of" says.

REMOVE

  • tests/toyos-rust-tests/src/bin/counters_metal.rs:14 — "a dozen milliseconds after logkeeper does" is a measurement in a source comment; it belongs in the commit message.

SEND BACK

Japabu and others added 3 commits October 4, 2026 17:47
…quiet

`userland/console` (`Log::subscribe`) and `counters_metal` (`settle`) each
decoded logkeeper's answer to a `READ` on its `log` port by hand: the
`SERVED` frame, the boot-so-far length, the one pipe handle. Two decoders
of one answer meant a change to `serve.rs`'s answer could miss one, and
`counters_metal`'s copy runs only on a T14 boot.

`logkeeper_api::read` is now the one decoder, and both call it. It lives in
a crate of its own beside `filepicker-api`, the client side of a userland
server's port: `toyos-logstream` is pure (no I/O, no `unsafe`), and the SDK
is a path dependency of std and cannot take `toyos-logstream`.

`counters_on_metal` now reds a kernel record stamped in a whole millisecond
between `idle0` and `idle1`. `settle` waits two of logkeeper's rounds on the
one record kind seen so far that a round's own write stamps (the stick's
first sync); nothing held that the second was then quiet. Rejudging the
round-2 readbacks of 49e49a9 with this judge: head passes (no record in
1230220488..2230283529 ns), and the whole change reverted (0c27e4f) fails
on `[1.183 cpu7] usb-storage: disk 0 does not implement SYNCHRONIZE CACHE`
inside 1173282164..2173482583 ns.

`counters_metal` also panics by name on a served line that is not UTF-8:
logkeeper writes every control byte as text, so such a line is its defect,
not an empty line. Its header drops the job's start offset after
logkeeper's (about 7 to 12 ms on the readbacks), a measurement.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WcU2Dsw6mDYtwYfzVHPzM8
…othing

t14-cpu7-idles-busier-under-toyos-than-any-cpu-under-linux.md's exit was
"a T14 reading of the row with that wait in which cpu7's idle busy fraction
is within cpu0 to cpu6's". The orchestrator's T14 run of 49e49a9 (pull
request #728, comment 5981716271), one `testcases` boot per arm, judged by
`cargo test --test toyos-build -- --metal --metal-readback <dir> counters`:

- head 49e49a9, which waits for the log before `idle0`: EXIT=0, cpu7
  0.46%, cpu0 to cpu6 0.46 to 0.49%. Both settle lines (1.175 s, 1.188 s)
  and the stick's first sync (1.183 s) precede `idle0` (1230220488 ns); no
  kernel record is stamped before `idle1` (2230283529 ns). One SMI fell in
  the second, which is why all eight sit near 0.45%.
- the whole change reverted, 0c27e4f: EXIT=0, cpu7 2.89%, cpu0 to cpu6
  0.46 to 0.72%, the sync stamped 9.7 ms inside the second.

So cpu7's excess was logkeeper's first rounds, written through the `/log`
fileserver on cpu7 inside the measured second. The fold is
`counters_metal.rs`'s header ("idle0 waits for the log to be quiet"), and
`counters_on_metal` now reds a kernel record inside the second. No
citation of the slug exists (`git grep`).

The SMI issue quoted the row's "two idle floors", a phrase nothing in the
tree says; it now states the two floors itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WcU2Dsw6mDYtwYfzVHPzM8
@Japabu Japabu changed the title The counters row takes its idle second once the log holds its own line The counters row takes its idle second once the log is quiet, and its judge holds that second free of kernel records Oct 4, 2026
@Japabu

Japabu commented Oct 4, 2026

Copy link
Copy Markdown
Collaborator Author

T14 result at 09a27943e (orchestrator's run; worktree clean at head; the request listed no hashes, so the orchestrator hashed each staged image and checked it again before its flash: shared 5b59a721…ff75, shared-debug e7cc714e…33b6, testcases 9c01ad3f…ef08; all boots rc=0).

counters EXIT=0, [metal] 3 passed, 0 failed, 3 boot(s). 8 cpus, SMI +5 each over 12164 ms, +0 in the idle second.

cpu idle busy
cpu0–2, cpu4–7 0.00%
cpu3 0.03%

Linux's band printed by the row: 0.11–0.36%. With no SMI in the measured second, every CPU reads at or below 0.03%. Judge log: orch/countersquiet-r3/metal/head/judge-counters.log in the orchestrator's job directory.

@Japabu

Japabu commented Oct 4, 2026

Copy link
Copy Markdown
Collaborator Author

Review round 2, head 09a27943e, against origin/main 010a283a9. A trial merge (git merge-tree --write-tree origin/main 09a27943e) is clean, giving tree 9582207cc. Both sides touch tests/toyos.rs: main adds screen_loader_clears, which is not next to counters_on_metal. bootlog::record_millis is unchanged on main. Net change is +187 −58: production +58 −13, tests +121 −17, issues +8 −28.

Round 1 BLOCKERs:

  • CLOSED. issues/t14-cpu7-idles-busier-under-toyos-than-any-cpu-under-linux.md is deleted in 09a27943e, and the commit message carries the readings. git grep of the slug in the merged tree 9582207cc finds nothing. The exit is met twice:
    • r2 head: cpu7 0.46%, cpu0–6 0.46–0.49%, with a revert control at 2.89%.
    • r3 head: countersquiet-r3/metal/head/judge-counters.log, EXIT=0, every CPU 0.00–0.03% with +0 SMI in the second.
  • CLOSED, but see the new body BLOCKER below. The PR body's "staged, not run" claim for the r2 T14, and its 1.4% prediction, are gone.
  • CLOSED. Gate logs are named at head. ci-host.log reads "Host: 75 step(s), all green EXIT=0" and guest.log reads EXIT=0. Both ran from gates.sh with gates.rev 09a27943e and an empty gates.status.
  • CLOSED. There is now one handshake. git grep SERVED 09a27943e -- userland tests toyos-logstream finds one encoder (logkeeper/src/serve.rs:261) and one decoder (logkeeper-api/src/lib.rs:22). console and counters_metal both call logkeeper_api::read, and the T14 counters boot at head ran it (judge-counters.log, PASS). The crate is placed correctly:
    • the I/O lives here, while the pure constants stay in toyos-logstream;
    • it is shared by two programs;
    • filepicker-api is the precedent;
    • it carries description and doctest = false, and takes no external dependency.

Round 1 NOTEs and REMOVE: the UTF-8 panic, the "two idle floors" quote, the measurement in the header, and "Twice" being unheld (now held for kernel records) are all addressed.

BLOCKER

  • tests/toyos.rs:3426-3434 — the new quiet check reads only [... cpuN] records (toyos_logstream::record_ms needs [ and cpu). A {... pid=9 test-runner} program line stamped inside the second passes it. Program lines are exactly the write this branch names as the cause ("Measure first, print after. Without this, the row's 81 lines are written inside the second"), and the judge's doc claims "the second is the idle machine's". Idle busy is read, not held, so the regression lands unseen. Mutation: in counters_metal.rs main, move print("idle0", &idle0); to directly after let idle0 = read(&cap);. cargo test --test toyos-build -- --metal --metal-readback <dir> counters must go red. It stays green at this head, because those lines are program lines and printing them stamps no kernel record. Fix: the judge also refuses every program line stamped strictly inside the second. The stamp parser goes in toyos-logstream beside record_ms, since the format is {<wall> <secs.ms> [pid=N] <tag>}. Measurement, which needs no T14: rejudge a copy of countersquiet-r3/metal/head, once as it stands (EXIT=0) and once with one {… 1.500 pid=9 test-runner} counters_metal idle0: … line inserted into testcases/kernel.log (EXIT=1, naming that line). Its control is the same copy judged at 09a27943e, which stays EXIT=0.

  • PR body — false at this head, and it becomes main's merge commit. Three things are wrong:

    1. It says "At this head that boot has been staged, not run", and its first "What I am unsure of" doubts that the 49e49a963 reading carries over. The orchestrator ran it at 09a27943e: countersquiet-r3/metal/head/judge-counters.log, EXIT=0, "+0 in the idle second", cpu0–7 0.00–0.03%.
    2. It files build-only.log (17:53), build-console.log (17:54) and both rejudge-*.log (17:55) under "Gates (head 09a27943e)". All of them predate 7ae16ca06 (committed 17:56:47). rejudge.status lists the whole change as uncommitted. --console-boot is the only build that compiles console, so the console's compile at head stands on no measurement.

    Fix: carry the head T14 reading (command, exit, log). Rerun cargo run -- --build-only --console-boot and the two rejudges at the committed head, or at the head that answers the BLOCKER above, and name their logs. Drop the staged-not-run sentences and the answered doubt.

NOTE

  • issues/the-t14s-firmware-interrupts-every-cpu-every-2-2-s-under-toyos.md:29 — 22d241174 is reachable only from origin/wt/toyos-cpu7, not from main or this branch (git merge-base --is-ancestor exits 1 against both). Once that branch is deleted, the citation points at nothing. The per-second lines it backs are already in 35cd63142's message, which lands, so either cite that commit or put 22d241174 on a ref that lasts.
  • tests/testcases/system.toml:46 — every testcases job now holds log, and every discovered Rust binary rides the T14 shared boots. At head only the counters member ran (shared: 1 member(s)). No other member reads the namespace whole for log (git grep '"log"' tests/toyos-rust-tests/src hits only hierarchy_paths's path list). A full shared-boot run before landing would still show it.

REMOVE
(none)

SEND BACK

Japabu and others added 2 commits October 4, 2026 18:22
…nd too

The quiet check read only kernel records ([... cpuN]), so a program line
stamped inside the second -- the row's own idle0 print moved back before
idle1, the regression "measure first, print after" exists to prevent --
passed it. toyos-logstream gains program_ms beside record_ms: a program
line's milliseconds, found from the tag back past the optional pid=, tid=
and severity words, since the wall clock before it is two words or none.
The judge reads the whole log with both.

The window now includes the millisecond each edge falls in: a line stamped
in idle0's millisecond may follow the read, and lines carry no finer time.
With the edges excluded, the review's mutation (print idle0 right after
reading it) can land its 81 lines in idle0's own millisecond and pass.

The SMI issue cites 35cd631, which lands and carries the scout's image
hash and per-second lines, rather than 22d2411, reachable only from a
branch that goes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WcU2Dsw6mDYtwYfzVHPzM8
@Japabu

Japabu commented Oct 4, 2026

Copy link
Copy Markdown
Collaborator Author

Round-2 answer, head 19ac555c0: the patches the body's measurements rest on.

The review's named mutation (mutation.patch, committed as 4173c00c4 on wt/toyos-countersquiet-mutant, which never lands; staged at orch/countersquiet-r4/metal/mutant/):

--- a/tests/toyos-rust-tests/src/bin/counters_metal.rs	2026-10-04 18:26:40
+++ b/tests/toyos-rust-tests/src/bin/counters_metal.rs	2026-10-04 18:27:00
@@ -199,6 +199,7 @@
     let cap: SysCap = Endowments::get().take(SYSCAP_LABEL).expect("test-runner endows a capability");
     settle();
     let idle0 = read(&cap);
+    print("idle0", &idle0);
     std::thread::sleep(IDLE);
     let idle1 = read(&cap);
     std::thread::scope(|s| {
@@ -212,7 +213,7 @@
         }
     });
     let spin = read(&cap);
-    for (phase, read) in [("idle0", &idle0), ("idle1", &idle1), ("spin", &spin)] {
+    for (phase, read) in [("idle1", &idle1), ("spin", &spin)] {
         print(phase, read);
     }
     loaded(&cap);

The control (control.patch, git diff HEAD~1 HEAD at 19ac555c0, applied with git apply -R after --check and restored with git apply in rejudge.sh; with it reverted the tree is a1a14eac7, whose counters_on_metal is 09a27943e's):

control.patch
diff --git a/issues/the-t14s-firmware-interrupts-every-cpu-every-2-2-s-under-toyos.md b/issues/the-t14s-firmware-interrupts-every-cpu-every-2-2-s-under-toyos.md
index aabb1e83a..de69458b0 100644
--- a/issues/the-t14s-firmware-interrupts-every-cpu-every-2-2-s-under-toyos.md
+++ b/issues/the-t14s-firmware-interrupts-every-cpu-every-2-2-s-under-toyos.md
@@ -26,9 +26,9 @@ lines are quoted on #681 (comment 5962619169; the boot itself in comment
   CPUs waiting on a lock across a step, which went 4,503,694 and 4,554,675 ns
   between two turns of their own spin.
 - **MPERF reads the stop as about 4.55 ms of C0 on every CPU.** One T14 boot
-  of a scout image without ACPI mode (`22d241174`) read twelve back-to-back
-  idle seconds between counters rounds; the commit that added this bullet
-  carries its per-second lines. In the five whose SMI count moved by one,
+  of a scout image without ACPI mode read twelve back-to-back idle seconds
+  between counters rounds; `35cd63142`, which added this bullet, carries its
+  image hash and per-second lines. In the five whose SMI count moved by one,
   cpu2, cpu4, cpu5 and cpu6, which ran none of the log's work, read 4535 to
   4571 ppm busy; in the seven where it did not, 6 to 285 ppm. So the
   `counters` row's idle second reads one of two floors, about 0.45% or 0.03%
diff --git a/tests/toyos-rust-tests/src/bin/counters_metal.rs b/tests/toyos-rust-tests/src/bin/counters_metal.rs
index f8e46512c..4d023c898 100644
--- a/tests/toyos-rust-tests/src/bin/counters_metal.rs
+++ b/tests/toyos-rust-tests/src/bin/counters_metal.rs
@@ -160,7 +160,7 @@ fn print(phase: &str, read: &Read) {
 /// stick, so this prints a line and reads the log until that line comes back.
 /// **Twice**: the round that writes the first may itself put a record in the
 /// log — the stick's first sync is one — and the second writes it. The
-/// `counters` row reds a kernel record stamped inside the idle second.
+/// `counters` row reds any line stamped inside the idle second.
 fn settle() {
     let pipe = logkeeper_api::read().unwrap_or_else(|why| panic!("test-runner's `log` port: {why}")).pipe;
     let poller = Poller::new(1);
diff --git a/tests/toyos.rs b/tests/toyos.rs
index d7b903dab..d42f80f52 100644
--- a/tests/toyos.rs
+++ b/tests/toyos.rs
@@ -3438,9 +3438,10 @@ const SMI_SPAN_NS: u64 = 4_444_000_000;
 /// reads` line names, and none stale; every CPU's performance request
 /// declared at boot, `pm_enable=1`, the request Linux makes on this machine
 /// (`tests/t14-linux/hwp-request.txt`), and its power envelope in every read
-/// the one its `control_regs:` line holds. No kernel record is stamped in a
-/// whole millisecond between `idle0` and `idle1`: the second is the idle
-/// machine's. From `idle0` to `spin`, at least
+/// the one its `control_regs:` line holds. No line, the kernel's or a
+/// program's, is stamped in a millisecond from `idle0`'s to `idle1`'s, either
+/// edge's included because a line stamped in it may follow the read: the
+/// second is the idle machine's. From `idle0` to `spin`, at least
 /// [`SMI_SPAN_NS`] apart, every CPU's SMI count rose alike and by two or more:
 /// the firmware's legacy mode, the positive control ACPI stage 1's flatness
 /// is read against, and the row that stage changes. Across the spin every
@@ -3490,13 +3491,17 @@ fn counters_on_metal(back: &metal::Readback) -> Result<(), String> {
     let (idle1, at1) = phase("idle1")?;
     let (spin, at2) = phase("spin")?;
     let (from_ms, to_ms) = (at0 / 1_000_000, at1 / 1_000_000);
-    let inside: Vec<&str> = kernel
+    let inside: Vec<&str> = log
         .text()
         .lines()
-        .filter(|line| bootlog::record_millis(line).is_some_and(|ms| from_ms < ms && ms < to_ms))
+        .filter(|line| {
+            toyos_logstream::record_ms(line)
+                .or_else(|| toyos_logstream::program_ms(line))
+                .is_some_and(|ms| (from_ms..=to_ms).contains(&ms))
+        })
         .collect();
     if !inside.is_empty() {
-        return Err(format!("the idle second {at0}..{at1} ns holds kernel records: {inside:?}"));
+        return Err(format!("the idle second {at0}..{at1} ns holds lines: {inside:?}"));
     }
     let linux_request = u64::from_str_radix(include_str!("t14-linux/hwp-request.txt").trim().trim_start_matches("0x"), 16)
         .map_err(|e| format!("t14-linux/hwp-request.txt: {e}"))?;
diff --git a/toyos-logstream/src/lib.rs b/toyos-logstream/src/lib.rs
index 7abd58aee..844a99adf 100644
--- a/toyos-logstream/src/lib.rs
+++ b/toyos-logstream/src/lib.rs
@@ -289,6 +289,26 @@ pub fn record_ms(line: &str) -> Option<u64> {
     secs.checked_mul(1_000)?.checked_add(millis)
 }
 
+/// The milliseconds since boot a program's line carries ([`ProgramLine`]), or
+/// `None` for any other line.
+///
+/// **Found from the tag back rather than by position**: the wall clock before it
+/// is two words or none, and the words after it are each there or not.
+pub fn program_ms(line: &str) -> Option<u64> {
+    let (head, _) = line.strip_prefix(OPEN)?.split_once(CLOSE)?;
+    let mut words = head.rsplit(' ');
+    Tag::new(words.next()?)?;
+    let field = words.find(|word| {
+        !(word.starts_with("pid=")
+            || word.starts_with("tid=")
+            || [Severity::Warn, Severity::Error, Severity::Alert].iter().any(|s| s.word() == Some(word)))
+    })?;
+    let (secs, millis) = field.split_once('.')?;
+    let secs: u64 = secs.parse().ok()?;
+    let millis: u64 = millis.parse().ok()?;
+    secs.checked_mul(1_000)?.checked_add(millis)
+}
+
 /// Whether `line` opens as a program's line: what a judge of the kernel's
 /// records leaves out.
 pub fn is_program_line(line: &str) -> bool {
@@ -554,6 +574,27 @@ mod tests {
         assert_eq!(record_ms(""), None);
     }
 
+    /// Every head [`ProgramLine`] writes reads back to the time it carries, and
+    /// no text after the head answers for it.
+    #[test]
+    fn a_program_lines_time_is_read_inside_its_head_and_nowhere_else() {
+        let tag = Tag::new("test-runner").expect("a tag");
+        for stamp in ["", "2026-09-24 10:00:00", "---------- --------"] {
+            for severity in [Severity::Info, Severity::Warn, Severity::Error, Severity::Alert] {
+                for (tid, pid) in [(0, None), (3, None), (0, Some(9)), (3, Some(9))] {
+                    let line =
+                        format!("{}", ProgramLine { stamp, at_ns: 1_500_999_999, severity, tid, pid, tag, text: b"9.000" });
+                    assert_eq!(program_ms(&line), Some(1_500), "{line:?}");
+                }
+            }
+        }
+        assert_eq!(program_ms("[2026-09-07 22:57:46 3.109 cpu1] exit: a pid=7"), None);
+        assert_eq!(program_ms("{2026-09-24 10:00:00 netstack} 1.000"), None);
+        assert_eq!(program_ms("{1.000 a b} x"), None);
+        assert_eq!(program_ms("  its second line"), None);
+        assert_eq!(program_ms(""), None);
+    }
+
     #[test]
     fn a_kernel_record_and_its_continuation_are_no_programs() {
         assert_eq!(program_line("[2026-09-24 10:00:00 1.216 cpu0] Boot: complete (1216ms)"), None);

The two lines inserted into copies of countersquiet-r3/metal/head/testcases/kernel.log (after its line 363, the second settle line; one copy each):

{2026-10-04 16:12:18 1.500 pid=9 test-runner} counters_metal idle0: printed inside the second
{2026-10-04 16:12:18 1.187 pid=9 test-runner} counters_metal idle0: printed in idle0's own millisecond

@Japabu Japabu changed the title The counters row takes its idle second once the log is quiet, and its judge holds that second free of kernel records The counters row takes its idle second once the log is quiet, and its judge holds that second free of every line Oct 4, 2026
@Japabu

Japabu commented Oct 4, 2026

Copy link
Copy Markdown
Collaborator Author

T14 results at 19ac555c0 (orchestrator's run; worktree clean at head; each image's sha256 checked against image-hashes.txt before its flash; boots interleaved head/mutant; both judged from the head worktree).

arm counters detail
head 19ac555c0 EXIT=0, 3 passed SMI +6 each over 12205 ms, +1 in the idle second; cpu7 0.46% (all CPUs at the SMI floor)
mutant 4173c00c4 (prints idle0 inside the second) EXIT=1, 2 passed, 1 failed FAIL counters: the idle second 1258944500..2259315764 ns holds lines: ["{… 1.258 pid=9 test-runner} counters_metal idle0: …"

The new check reds the named mutation on the T14. The full Drive-mode suite at this head (the shared-boot NOTE) runs next. Judge logs: orch/countersquiet-r4/metal/judge-head.log, judge-mutant.log in the orchestrator's job directory.

@Japabu

Japabu commented Oct 4, 2026

Copy link
Copy Markdown
Collaborator Author

Full Drive-mode T14 suite at 19ac555c0 (orchestrator's run, t14-drive.sh, worktree clean at head; the harness built, flashed, booted, read back and judged every boot; 2538 s): EXIT=1, [metal] 290 passed, 1 failed, 28 boot(s).

The one failure is this branch's own row, in the full profile:

FAIL counters: the idle second 10474047319..11474110523 ns holds lines: ["[… 11.254 cpu3] sched: cpu=3 ready=0 dying=0 stopped=0 parked=3 current=None trips=492", "[… 11.254 cpu3] PMM: 320/16024MB used (7852 pages free)", "[… 11.254 cpu3]   kernel-heap  alloc=…", … "[… 11.254 cpu1] sched: cpu=1 …", …]

In the full profile the row's idle second starts at ~10.47 s (other jobs ran first), and the kernel's periodic sched:/PMM: report lands inside it at 11.254 s. In the isolated runs the second started at ~1.2 s, before that report's first round. The new check is doing its job: the kernel itself writes inside the measured second. No other row failed. Log: orch/countersquiet-r4/metal/drive/728r4-full.log in the orchestrator's job directory.

Japabu and others added 2 commits October 4, 2026 19:53
The one conflict, tests/toyos-rust-tests/Cargo.toml, keeps both sides:
this branch's toyos-logstream and logkeeper-api beside main's toyos-trace.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WcU2Dsw6mDYtwYfzVHPzM8
The full Drive-mode T14 suite at 19ac555 red the `counters` row in its
full profile: the job's idle second ran 10.474..11.474 s, and the kernel's
idle report (`scheduler::log_health`: `sched:` per CPU and the machine's
`PMM:`) printed at 11.254 s inside it. In the isolated `counters` boots the
second ran at ~1.2 s, just after every CPU's first report at boot, so the
row's verdict depended on where in the boot the job ran.

The report is legitimate kernel work with a declared cadence
(`SNAPSHOT_INTERVAL`, 10 s): each CPU prints on its first idle trip at least
that long after its last, and the machine's `PMM:` on the same rule from the
first idle trip of all. It never wakes a CPU, so one overdue prints at
whatever next wakes it. In that boot every CPU first printed at 1.135 s, so
all were due from ~11.135 s, and the first wake after (11.254 s) set off the
whole round.

So `counters_metal` reads the report's schedule off the log it already reads
to settle: each CPU's last `sched:` record, the last `PMM:` (or, before
the first, the earliest `trips=1`), each due `SNAPSHOT_INTERVAL` later. When
one falls due before the second could end, it sleeps until every report is
due, puts a thread on every CPU for 20 ms so each passes its idle loop and
prints what it owes, settles again and rechecks. A report the log does not
hold counts as due now. The whole wait keeps the 20 s ceiling the two settle
waits had, so the job's budget is unchanged. The judge is unchanged and stays
the oracle: any line in the second reds the row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WcU2Dsw6mDYtwYfzVHPzM8
@Japabu Japabu changed the title The counters row takes its idle second once the log is quiet, and its judge holds that second free of every line counters row measures a quiet idle second, wherever in the boot it runs Oct 4, 2026
@Japabu

Japabu commented Oct 4, 2026

Copy link
Copy Markdown
Collaborator Author

Full Drive-mode T14 suite at 78c21c575 (orchestrator's run, t14-drive.sh; worktree clean at head; 2537 s): EXIT=1, [metal] 292 passed, 1 failed, 28 boot(s).

  • counters now passes in the full profile: 8 cpus, SMI +5 each over 11887 ms, +0 in the idle second; cpu0–2, cpu4–7 idle busy 0.00%, cpu3 0.03% (Linux band 0.11–0.36%).
  • The one failure is another row:
    FAIL irq_census_conservation: cpu1 counted 24211 interrupt(s) and attributed 24210 to sources — a source is not being counted: Census { cpu: 1, total: 24211, by_source: [1141, 22906, 0, 0, 0, 0, 0, 0, 142, 21, 0, 0] }
    
    It passed in the two previous full runs today (The userland lock takes the per-range maxima the five main locks already carry, the second alignment of the one-workspace track #732 at 7015d8b52, this branch at 19ac555c0). A red is a defect, not a flake: the orchestrator is sending it to its owner separately. Whether this branch's 20 ms wake of every CPU makes it more likely is for that investigation to say.

Log: orch/countersquiet-r5/metal/drive/728r5-full.log in the orchestrator's job directory.

@Japabu

Japabu commented Oct 4, 2026

Copy link
Copy Markdown
Collaborator Author

Review round 3, head 78c21c575, against origin/main 4d46c8e55. Net change (git diff --shortstat origin/main...78c21c575): +351 −58. Production is +99 −13 and tests +244 −17. Of the tests, counters_metal.rs is +193 −9, and about 100 of those lines are this round's report prediction. Issues are +8 −28.

Round 2 BLOCKERs:

  • CLOSED. The judge now refuses a program's line inside the second.
    • The rejudged copies at 19ac555c0 go red with the line and stay green without it: inserted and edge each EXIT=1 and name the line, the control (09a27943e's judge) EXIT=0 on both, and plain EXIT=0 (orch/countersquiet-r4/rejudge-*.log).
    • The named mutation went red on the T14: mutant 4173c00c4 EXIT=1, FAIL counters: … holds lines: ["{… 1.258 pid=9 test-runner} counters_metal idle0: …", with head EXIT=0 (orch/countersquiet-r4/metal/judge-mutant.log, judge-head.log).
    • 78c21c575 touches only counters_metal.rs, so that judge is the head's.
  • CLOSED. The round-2 falsehoods in the PR body are gone.
    • The 09a27943e T14 reading is carried.
    • build-console.log was built at the committed head: build-console.rev is 78c21c575 and build-console.status is empty.
    • The rejudges ran at a committed head: rejudge.rev with an empty rejudge.status.
    • The body has gone false again since; see below.
  • Round-2 NOTEs are closed:
    • The SMI issue now cites 35cd63142, which is an ancestor of head (merge-base --is-ancestor exits 0).
    • Two full Drive-mode suites ran with the log grant. Neither failed a row the grant reaches.

The design question: predicting the report versus changing it. Changing the report is simpler and right. Predicting it is a workaround, built inside a test, for a kernel behaviour.

  • What the head does. It works out the kernel's private schedule from the kernel's debug text:

    • it parses sched: cpu=, trips=1 and PMM: ;
    • it copies SNAPSHOT_INTERVAL as REPORT;
    • it copies the rule for the first PMM: deadline;
    • it adds a 50 ms fudge (CLEAR);
    • it sleeps to a predicted time, spins every CPU for 20 ms to make the kernel print, and repeats.
  • Who reads those lines. Nothing in the tree parses them except this row. git grep 'sched: cpu=\|PMM: ' at head finds only issue quotes and a toyos-blackbox fixture string.

  • What the report costs. On a shipping idle machine it is periodic idle work:

    • a clock read on every idle trip;
    • about a dozen records per CPU round every 10 s or so;
    • logkeeper writing them to the stick.

    That is the cost the counters row exists to set against Linux. The row now picks a second the shipping machine does not have.

  • What changing it would take. Taking the occupancy and page-pool snapshot out of the idle loop deletes code on both sides:

    • log_health's prints, NEXT_HEALTH, IDLE_TRIPS and pmm::dump_stats;
    • Reports, quiet, REPORT, CLEAR, WAKE and the sleep.

    If the snapshot is still wanted, it becomes an on-demand read rather than a timer. The row is then round 2's settle plus the judge.

  • Why the implementer could not do it. The report has a real diagnostic use: three issues use sched: lines in hang triage. It is also kernel work outside this brief's fence. So it is the owner's ruling, and the right move was to stop and report rather than build around it.

  • Measured. The head does hold up on the one boot that ran it. The full readback at head shows quiet took the wake path:

    • settle lines 1–2 at 10.430 s and 10.435 s;
    • all eight sched: lines and PMM: at 11.185–11.188 s;
    • settle lines 3–4 at 11.213 s and 11.217 s;
    • idle0 at 11222163868 ns and idle1 at 12222223305 ns, with nothing stamped between;
    • [counters] … +0 in the idle second, every CPU at 0.00% except cpu3 at 0.03%.

    (orch/countersquiet-r5/metal/drive/728r5-full-readback/testcases/kernel.log, lines 596–653, and 728r5-full.log.)

BLOCKER

  • tests/toyos-rust-tests/src/bin/counters_metal.rs:67,181-216,282-311 — This is the design BLOCKER. REPORT is a private copy of kernel/src/scheduler.rs's SNAPSHOT_INTERVAL, and Reports::see/due are a second reader of the kernel's report text and of its PMM: deadline rule. No declaration ties them to the kernel, so if the kernel changes its cadence or format, the prediction goes quietly wrong. The isolated counters boot runs its second at about 1.2 s, where nothing is due, so only a full-profile T14 suite would ever see it. Both rules forbid this: "a sibling of something the tree already has", and "never work around it". Fix: take the report question to the owner.

    • If he rules that the idle report goes or becomes on-demand, it lands with that change, in this branch or one that lands first. Reports, quiet, REPORT, CLEAR, WAKE and the sleep are deleted, and the row keeps Log::settle and the judge. A full-profile T14 suite with counters green is its measurement.
    • If he rules that the report stays as it is, the duplicated cadence is a compromise. It is recorded in issues/ with an owner, this head's full-run evidence and an exit, and the prediction stays.
  • Merge with origin/main 4d46c8e55 — This is not a clean merge. git merge-tree --write-tree origin/main 78c21c575 exits 1, with conflicts in toyos-logstream/src/lib.rs and userland/console/src/main.rs against Logs on screen are coloured by severity and source, with a compact stamp #720.

  • PR body — The body is false at this head, and it becomes main's merge commit.

    1. "Staged, not run … the whole metal suite at head" and "This head's program has not booted yet". The full suite has run at 78c21c575: EXIT=1, [metal] 292 passed, 1 failed, 28 boot(s), with counters passing (orch/countersquiet-r5/metal/drive/728r5-full.log).
    2. "What I am unsure of", first bullet: "No boot of this head has run that loop yet". That boot ran the loop once, and every CPU printed at 11.185–11.188 s (readback lines 598–615).
    3. The irq_census_conservation red in that run, in the same testcases boot, is nowhere in the body.

    Fix: carry that run's command, exit and log. Drop the staged sentences and the answered doubt. Record the red along with what the NOTE below shows of its cause.

NOTE

  • kernel/src/irq_census.rs:80-91 (main's code, off this branch's fence; to be filed, not fixed here) — The irq_census_conservation red at head is a torn read across CPUs, not a source left uncounted.
    • irq_took! (arch/x86_64/percpu.rs:730) adds TOTAL first and the source second.
    • read loads TOTAL first and the sources after, all Relaxed, from another CPU.
    • The failing line was printed by cpu7 about cpu1, at a process exit (process.rs:1067) at 36.226 s (readback line 12102). That falls inside counters' loaded phase, with cpu1 at kick=22906. One kick landing between the two loads gives exactly total = sum + 1.
    • Main's loaded and the census print are unchanged on this branch, so the race is main's. Its judge's message ("a source is not being counted") misnames it. This is for the investigation the orchestrator opened.
  • tests/toyos-rust-tests/src/bin/counters_metal.rs:76 — WAKE is a flat 20 ms. On the head's boot every CPU printed within 3 ms of its thread starting, and the threads then spun on to 11.206–11.212 s. This is moot if the design BLOCKER deletes it. If the owner keeps the report, the wake waits on each CPU's sched: record appearing in the log rather than on a duration.

REMOVE
(none)

SEND BACK

Two conflicts against #720's coloured screen lines.

toyos-logstream: #720 reads a line's head once, in `shown`, and finds a
program line's time as the first head word `millis` parses. `program_ms`
now reads its time through that reader instead of walking back from the
tag past `pid=`/`tid=`/severity, so one function finds the field. Its
host test keeps every head `ProgramLine` writes; the `{1.000 a b} x`
case is dropped, being a refusal of the walk this replaces.

console: keeps `logkeeper_api::read` for the handshake and #720's
`Showing` for drawing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WcU2Dsw6mDYtwYfzVHPzM8
…econd it settles to

The owner ruled on 2026-10-04, asked whether to remove the ~10 s idle
status report (`sched:`/`PMM:`) from the idle loop, choosing "Remove it
entirely": "Delete the periodic report and its counters; hang triage
uses the trace diary and panic records instead."

Kernel: `scheduler::log_health` and its call on every idle trip go, with
`SNAPSHOT_INTERVAL`, `NEXT_HEALTH`, `IDLE_TRIPS` and the `PMM:` deadline.
`pmm::dump_stats` goes with the counters only it read: `CATEGORY_STATS`,
`LAST_DUMP_NANOS`, `LAST_ALLOC`, and `Category`, which existed only to
index them, so `alloc_page`, `claim`, `alloc_contiguous` and
`PageAlloc::new` take no category and `PhysPage` carries none. The
driver's `parked_len`, `dying_len` and `stopped_len` had no other caller,
and the scheduler core's `dying_len` and `stopped_len` none but its own
tests, which now count `dying()` and `stopped()`.
The idle loop no longer reads the clock or counts a trip for a report.

Row: `counters_metal.rs` returns to `19ac555c0`'s, the settle and the
judge with no prediction of the report: `Reports`, `quiet`, `REPORT`,
`CLEAR`, `WAKE` and the sleep are deleted. Review round 3's design
BLOCKER is closed by the ruling rather than by a second reader of the
kernel's cadence.

Issues: the three that read hang triage off `sched:` lines say the
report is gone and quote the ruling; `toyos-explains-itself` no longer
lists `PMM:`/`sched:` among the log's numbers; the 2 MiB track says
where stage 3's floor is read now; the pipe-lock issue's quoted call
loses its category.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WcU2Dsw6mDYtwYfzVHPzM8
@Japabu Japabu changed the title counters row measures a quiet idle second, wherever in the boot it runs The kernel's idle report is gone, and the counters row measures a quiet idle second Oct 4, 2026
@Japabu

Japabu commented Oct 4, 2026

Copy link
Copy Markdown
Collaborator Author

Full Drive-mode T14 suite at e2ed813d1 (orchestrator's run; worktree clean at head; 2592 s): EXIT=1, [metal] 292 passed, 1 failed, 28 boot(s).

Log: orch/countersquiet-r6/metal/drive/728r6-full.log in the orchestrator's job directory.

@Japabu

Japabu commented Oct 4, 2026

Copy link
Copy Markdown
Collaborator Author

Review round 4, head e2ed813d1, against origin/main 0613f93f7. git merge-tree --write-tree origin/main e2ed813d1 exits 0 (tree f98cf08ea). Net (git diff --shortstat origin/main...e2ed813d1): +301 −308. Kernel −191 net, logkeeper-api +45, console −11, toyos-logstream +8 production and +21 test, counters_metal.rs +75 −9, judge +22 −4.

Round 3 BLOCKERs:

  • CLOSED — design (the predicted report). The owner's ruling is now in the kernel, and the row no longer predicts it.
    • Reports, quiet, REPORT, CLEAR, WAKE and the sleep are gone from counters_metal.rs.
    • scheduler::log_health, SNAPSHOT_INTERVAL, NEXT_HEALTH, IDLE_TRIPS, pmm::dump_stats, Category and its counters are deleted.
    • Measured on the T14 by the orchestrator's full Drive-mode suite at this head (orch/countersquiet-r6/metal/drive/728r6-full.log). Line 147056 reads [counters] 8 cpus, SMI +5 each over 11878 ms, +0 in the idle second, and every CPU reads 0.00–0.03% idle busy (lines 147057–147064). The judge refused no line inside the second.
    • The control is full-r4 rejudged at head (rejudge-full-r4.log, EXIT=1, naming the 11.254 sched:/PMM: lines). That boot ran this row's program on a kernel that still printed the report.
  • CLOSED — merge with Logs on screen are coloured by severity and source, with a compact stamp #720. program_ms is shown(line)?.head? matched to Source::Program, with millis of the stamp's first word, so there is one finder of the field. The host test writes every ProgramLine head and reads each back (ci-host.log, Host: 75 step(s), all green, EXIT=0, gates.rev = head, both status files empty). console compiles at head (build-console.log EXIT=0).
  • OPEN — PR body. It has gone false again; see the BLOCKER below.

The kernel deletion is complete and safe.

  • A search of the merged tree f98cf08ea outside issues/ finds nothing left behind: no pmm::Category, Category::<variant>, dump_stats, log_health, NEXT_HEALTH, IDLE_TRIPS, SNAPSHOT_INTERVAL, dying_len, stopped_len, parked_len or CATEGORY_STATS.
  • Every alloc_page, alloc_contiguous, PageAlloc::new and claim call in the merged tree takes the new arity, including the 21 commits main has gained since.
  • Nothing is orphaned. Cadence has seven other users. CpuSched::parked, dying and stopped are still read by driver.rs, invariants.rs, sim and loom. hw::now_ns is still read by payload.rs.
  • PhysPage is built in three places (pmm.rs:192, pmm.rs:227 and from_raw), and its drop path only frees.
  • Only one string outside issues/ still holds a sched: or PMM: line: the toyos-blackbox fixture text. Nothing reads it as a record.
  • The idle loop loses one call and gains nothing.
  • File grants: one port per file-server role, a connection is the grant it was minted, and each instance has a quarter of each bound #729 makes log a file-server role name as well. The role's port is fs:-prefixed (toyos/src/fs.rs:44), so it does not collide with logkeeper's log port in tests/testcases/system.toml.

BLOCKER

  • PR body, "This head's kernel has not booted on the T14", "Staged for the T14, not run", and the first "What I am unsure of" bullet ("The staged full suite is the run that shows it") — false at this head, and on evidence.
    • The full suite has run at e2ed813d1: cargo test --test toyos-build -- --metal, exit status 1, [metal] 292 passed, 1 failed, 28 boot(s), log orch/countersquiet-r6/metal/drive/728r6-full.log. That run is the measurement round 3 asked for, and it is not in the body.
    • Its red is not recorded in the body or in any issue on main. The red is FAIL irq_census_conservation: cpu1's census went backwards, 56975 then 56974 (line 147021). The issue it is attributed to, issues/the-irq-census-judge-reds-on-two-exits-stamped-in-the-other-order.md, exists only on open The interrupt census keeps one word per delivery, so it always adds up #734. git ls-tree origin/main issues/ has no such file.
    • The body still carries the 78c21c575 conservation red as the open question, but The interrupt census keeps one word per delivery, so it always adds up #734 has since answered it.
    • Fix:
      1. Carry the head run in the body: its command, exit, log and the [counters] lines.
      2. Record the census red as main's, citing The interrupt census keeps one word per delivery, so it always adds up #734 and its issue.
      3. Drop the staged-not-run sentences and the answered doubt. The isolated counters staging may stay only as what it is.

NOTE
(none)

REMOVE
(none)

SEND BACK

@Japabu
Japabu marked this pull request as ready for review October 4, 2026 20:52
@Japabu
Japabu enabled auto-merge October 4, 2026 20:52
@Japabu
Japabu added this pull request to the merge queue Oct 4, 2026
Merged via the queue into main with commit 9de5d7d Oct 4, 2026
6 checks passed
@Japabu
Japabu deleted the wt/toyos-countersquiet branch October 4, 2026 21:34
Japabu added a commit that referenced this pull request Oct 4, 2026
…os-acpi1

tests/toyos.rs conflicted in `counters_on_metal`, in its doc and in its SMI
summary line. Both sides are kept: #728's refusal of any line stamped in the
idle second, and this branch's ACPI mode with the SMI count flat from `idle0`
to `spin`, which replaced main's legacy-mode rise.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WcU2Dsw6mDYtwYfzVHPzM8
Japabu added a commit that referenced this pull request Oct 4, 2026
…once

#728's `counters` row refuses any line stamped in its idle second, and this
branch runs `acpiserver` on that boot. The server logs a query number the
first time the embedded controller raises it, and the kernel logs the claim's
first interrupt on the server's read after it. On the T14 the first query
came 0.04 to 1.91 s after the server armed across every readback of this
branch, so it lands in the second that follows #728's settle about half the
time; the counts line, every 30 s from arming, lies past both takes.

`counters_metal` now reads the log back after its three reads, through the
same settle, and keeps every line stamped in the second. Where every one is
the server's or the kernel's first-interrupt line, it takes the three reads
again, once. Any other line, or the server's in the second take, is printed
over and left to the judge, which is unchanged.

The press issue gains `ff4945d6d`'s attended press: query 0x28 at 13.089 s,
the press at 13.105 s, one press by the owner's account.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WcU2Dsw6mDYtwYfzVHPzM8
Japabu added a commit that referenced this pull request Oct 4, 2026
Conflicts against #728:
- toyos-logstream: #728's `program_ms` read the old `{…}` program head
  through `shown()`, both gone here. Folded into `parse`: the counters
  judge reads `parse(l).and_then(|p| p.ms)`, one reader for every line;
  its readback test now asserts `parse` on every head `ProgramLine`
  writes.
- console: #728's `logkeeper_api::read()` kept, with this branch's
  `asked_ms` on `stamp_ns()`.

Semantic conflict: #728's idle-second check compared `counters_metal`'s
`at`, read off `nanos_since_boot()` (the kernel clock's zero), with line
stamps that count from the counter's zero here. `counters_metal` now
reads `stamp_ns()` for `at`; the SMI span `at2 - at0` is a difference and
unaffected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WcU2Dsw6mDYtwYfzVHPzM8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant