Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .claude/agents/orchestrator.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ A mutation loop, guest or metal, starts on a clean worktree at the head under re
clean: `git apply --check`, `git apply`, the tests by name, `git apply -R`. None runs while an
agent edits that worktree. A queue script passes only flags `src/testargs.rs` declares: any other
word becomes the run's filter, and a one-test run reports as a pass.
A run is the whole suite, or a positional filter that reaches any enabled test.

## The bench

Expand Down
6 changes: 2 additions & 4 deletions .github/workflows/nightly.yml
Original file line number Diff line number Diff line change
@@ -1,15 +1,13 @@
name: nightly

# Everything that boots a guest, the host gate again to write the cache the
# merge queue restores, and portability: on main every night, the weekly tier
# on Sundays, and on any branch by `gh workflow run nightly.yml --ref <branch>`.
# merge queue restores, and portability.
# Every job's logic is `cargo run -- --ci <job>` (src/ci.rs); this file says
# where each one runs.

on:
schedule:
- cron: '0 3 * * 1-6'
- cron: '0 3 * * 0'
- cron: '0 3 * * *'
workflow_dispatch:

# Never cancelled: `build` may be an hour into a bootstrap.
Expand Down
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ The bar is not yet the tree. The standing failures are declared rather than remo

## Build & test

The testing rules live where they are enforced: known reds in `src/redlist.rs`, tiers in `src/tiers.rs`, the PR gate and the nightly in `.github/workflows/`. Operationally:
The testing rules live where they are enforced: known reds in `src/redlist.rs`, the PR gate and the nightly in `.github/workflows/`. Operationally:

- `cargo run` builds everything (toolchain, kernel, bootloader, userland, image) and launches QEMU; `--build-only` skips the launch. `cargo test` runs the QEMU harness; `cargo run -- --ci host` runs every host suite, as the PR gate's required `host` check does.
- **Agents never run QEMU.** An agent verifies with host tests and builds the image at most; the orchestrator runs every guest test, one suite at a time.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ opened: 2026-09-27

# A disk whose port went away panics the boot at ROOT's hold

`usb_transport_break` (nightly tier) fails its boot with:
`usb_transport_break` fails its boot with:

PANIC: panicked at src/rootfs.rs:177:19:
boot: the partition ROOT was read from, <guid>, cannot be held: Unusable
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -40,9 +40,7 @@ session against an unchanged tree, not a re-classification of its `Sched`.

Not the diff it was seen from: that branch's kernel change to this path is the
deletion of an unreachable `if rflags & TF != 0` branch in
`arch::LogCommitGuard::close`, which removes instructions and adds none, and
`kernel_log_file` is Nightly for `Why::Cost` rather than for flakiness
(`src/tiers.rs`).
`arch::LogCommitGuard::close`, which removes instructions and adds none.

## Promoted 2026-08-25

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ opened: 2026-09-27

# `screen_diag_boot` finds no `i8042:` line on the panel

`screen_diag_boot` (nightly tier) fails with `"i8042:" is not on screen five
`screen_diag_boot` fails with `"i8042:" is not on screen five
seconds after the boot finished`. The decoded panel ends at `Boot: complete`,
and no i8042 line is painted in the five seconds after it. The i8042 health
line (`report_health`, `to_screen`) never reaches the panel on the diag image.
Expand Down
6 changes: 3 additions & 3 deletions issues/build/a-metal-only-row-cannot-be-disabled.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ opened: 2026-09-29
# A metal-only row cannot be disabled, so its red has no legal home

`tests/toyos.rs`'s `check_redlist` accepts a `src/redlist.rs` row only for a
name `schedule` holds: the shared boot's discovered binaries, `SCREEN_TESTS`
name `registered` holds: the shared boot's discovered binaries, `SCREEN_TESTS`
and `MACHINE_TESTS`. A `METAL` row listed in `METAL_ONLY` has no QEMU
registration and is in none of them, yet the metal branch filters `METAL`
rows through the same `keep` that reads the redlist. A metal-only red can
Expand All @@ -17,13 +17,13 @@ row before any boot.
## Measured

`hda_tone` and `hda_client_stall` red on the T14 run of `main` at `7e151819`.
Only the code reading is verified: `check_redlist` tests `schedule.contains`, and
Only the code reading is verified: `check_redlist` tests `registered.contains`, and
the metal branch filters `METAL` rows through the same `keep`. Running
`--known-red hda_tone` and `--list` against a tree with such a row is owed and
not in this tree's record.

## Exit condition

`check_redlist` accepts a name `METAL` registers as well as one `schedule`
`check_redlist` accepts a name `METAL` registers as well as one `registered`
does, and `cargo run -- --known-red hda_tone` can answer YES from a row the
suite accepts.
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,11 @@ Whether `collect` is where the kernel takes `PROCESS_TABLE` is
A settled join then takes the lock on every wake of its wait.

**Exit**: taking `PROCESS_TABLE` outside `ask_join`'s `collect` reds a host
test or a fast-tier test. Owner: orchestrator.
test. Owner: orchestrator.

The syscall's answer-keeping is the same gap: `sys_thread_join`
(`kernel/src/syscall/proc.rs:154`) holds one `Join` across every ask, but
`a_join_asked_after_it_collected_keeps_its_answer` only replicates that shape
rather than calling the syscall, so a `Join::ask` built fresh per ask
(`join-per-ask`) stays host-green and reds only on the nightly guest
(`join-per-ask`) stays host-green and reds only on the guest
`fpu_isolation` (`thread_join failed`, `left: 18446744073709551614`).
5 changes: 2 additions & 3 deletions issues/build/a-shards-boot-width-does-not-price-its-tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,7 @@ reference and multiplies every liveness ceiling by the result; every shard print
it (`host: fastest boot N ms against the reference 1320 ms — liveness ceilings
paid at Wx width`). The proposal was to spend the same factor on the *duration
profile*: divide each shard's measured prices by its width in
`35383398^:src/durations.rs`'s merge, so `src/tiers.rs`'s ceiling compares like with like
across shards of different speed, with timer-anchored names exempt because a
`35383398^:src/durations.rs`'s merge, with timer-anchored names exempt because a
fixed wait does not shrink on a fast host.

**Measured over six hosted twelve-shard runs and refused.** Dividing by the
Expand Down Expand Up @@ -60,7 +59,7 @@ reverse regression is 1.248 with `var(ln width)` 7.86x `var(factor)`: even read
charitably, ln(width) is the shared cause buried under eight times its own
variance of noise, and dividing by it adds that noise to every price.

**Within-name slope of ln(price) on ln(width), by `src/tiers.rs` relegation
**Within-name slope of ln(price) on ln(width), by
class** — the prediction was ~1 for ordinary and `Why::Cost` names and ~0 for
`Why::TimerAnchored`:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,4 @@ p10-p90 shard spread that explains ordinary variance. `dump_nmi_probe` grew
+29% over the same window (6,284 -> 8,098 ms). Either the boots these ride got
genuinely slower — a real kernel-cost regression nobody measured on purpose —
or the shard fleet's pricing shifted; the two hypotheses separate on a bisect
of the boot's own timestamps across the window's landings. Noticed while
relegating the straddler batch; the relegation hides the symptom from the
per-PR gate, which is exactly why the growth is recorded here.
of the boot's own timestamps across the window's landings.
Original file line number Diff line number Diff line change
Expand Up @@ -22,4 +22,4 @@ the listener's first accept, bounded by a ceiling that fails loudly, and only
then pace the offers into the closed connection.

**Exit condition**: the test waits for the accept event instead of for a fixed
span, and stays green under a loaded fast tier.
span, and stays green under a loaded suite.
Original file line number Diff line number Diff line change
Expand Up @@ -24,4 +24,4 @@ hypothesis, and one red at 18x its price is not a rate.
Exit: a rate — the same suite run repeatedly with and without a second
worktree's build on the host — that says whether this is contention the harness
should schedule around or a defect in the guest's own boot, and the name is
either re-tiered or fixed at the cause.
fixed at the cause.
6 changes: 2 additions & 4 deletions issues/build/parallel-tests-red-under-other-suites.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,9 +64,7 @@ changes.
`qemu::budget(20 s)`, the phase's. Still `Sched::Parallel`. **The duration
profile's share of this is closed**: nothing is retyped against a clock any
more — `/system/bin/terminal` prints `terminal: ready` and `shell_echoes` waits on
that (`tests/toyos.rs`'s `SURFACE_UP`) — and the four-minute lane holder that
the profile used to seat a second desktop beside, `desktop_window_child`, is
`Tier::Nightly` and so never in a pull request's parallel phase.
that (`tests/toyos.rs`'s `SURFACE_UP`).
- **`desktop_locale_detect`** — retired 2026-09-04, green 5 of 5 beside a full
fast tier.
- **`netd_connection_caps`** — retired 2026-09-04, green 5 of 5 beside a full
Expand Down Expand Up @@ -500,6 +498,6 @@ mechanism for it.
driver nor this test, and the test was green in its previous fast-tier run.

Owner: the i8042 tally. **Exit condition**: the verdict revising itself in a
parallel run — a loaded full fast tier in which `i8042_undecoded_bytes`'
parallel run — a loaded full suite in which `i8042_undecoded_bytes`'
first mute line names nothing and its second names the sequence, or the
retirement's clause narrowed to the conditions under which it holds.
4 changes: 2 additions & 2 deletions issues/build/process-stats-exits-101-beside-other-guests.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,5 +21,5 @@ harness naming a hypothesis, and one red is not a rate.

Exit: a rate — the same suite run repeatedly with and without a second
worktree's build on the host — that says whether this is contention the harness
should schedule around or a defect the guest has, and the name is either
re-tiered or fixed at the cause.
should schedule around or a defect the guest has, and the name is
fixed at the cause.
Original file line number Diff line number Diff line change
Expand Up @@ -51,4 +51,4 @@ stdio console.
The console chardev the harness reads cannot refuse a write — for example a
file chardev the harness follows, or a `virtserialport`, which QEMU throttles
instead of dropping. Shown by `log_stream_stalled_reader` green in 20 of 20
runs beside a full nightly.
runs beside a full suite.
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,7 @@ opened: 2026-09-16
Both `Sched::Parallel`, both first sightings, both in one `cargo test` fast-tier
run on `wt/toyos-usbhang` sharing the host's twelve guest slots throughout with
two other worktrees' suites (`[host-slots]` names `pid 49668` and `pid 2726`
holding slots across the whole run). The branch's own diff at the time —
`tests/toyos.rs`'s tier table, `src/tiers.rs`'s `RELEGATED` and
`tests/test-durations`, all for `usb_reset_records_the_phase_it_cut` — touches
holding slots across the whole run). The branch's own diff at the time touches
neither SMP bring-up, `spawn_init`, nor `rootfs.rs`, so it is filed rather than
chased. Same shape as
`issues/build/parallel-tests-red-under-other-suites.md`, filed separately
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,6 @@ of 6 green off the one red.
No layer (QEMU input, i8042/USB, translator, console) has been measured yet.

**Exit condition.** The layer that drops the characters is identified and
fixed, and the test then passes 10 of 10 consecutive runs of the Fast tier
fixed, and the test then passes 10 of 10 consecutive runs
under the host load of the original failure (other suites running in
parallel); then the redlist row goes.
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,9 @@ opened: 2026-08-15
# The shard split prices a task's boot and not the image it builds first

`Shard::keep` partitions the twelve shards by `tests/test-durations`, and that
profile records what a *test* took — the number the ten-second Fast ceiling is
also read against. A machine test that boots a config the shared image does not
cover builds that image first, and nothing prices that build at all.
profile records what a *test* took. A machine test that boots a config the
shared image does not cover builds that image first, and nothing prices that
build at all.

**Measured on run `31896922288`** (`main` at `e064a96`, twelve KVM shards, the
Fast tier), reading each shard's job log for the wall clock between one `PASS`
Expand Down Expand Up @@ -68,11 +68,8 @@ where it landed.

Two directions, and they are not exclusive:

- **Price a task by build + boot.** The profile cannot simply absorb it: the
same file is what the `durations` verdict reads against the 10,000 ms Fast
ceiling, so a task priced at 203 s would red the tier gate it has nothing to
do with. It wants a second profile — per *config*, not per test — that only
`Shard::keep` reads.
- **Price a task by build + boot.** It wants a second profile — per *config*,
not per test — that only `Shard::keep` reads.
- **Make the second image cheap.** 145 s to add `netd` and `sshd` to an image
is a full recompile of those programs, not a relink.

Expand All @@ -93,9 +90,7 @@ the number this entry says nothing prices is measured on every run, deliberately
removed from the price, and then dropped on the floor. What is missing is two
smaller things: nothing attributes that duration to the *config* that caused it
rather than to the worker thread that happened to ask first, and nothing writes
it anywhere. `tests/test-durations` cannot be where it goes — the same file is
read against `FAST_CEILING_MS` (`src/tiers.rs:84`, 10,000 ms), which is this
entry's own reason for wanting a second profile.
it anywhere.

**What no work on this host can supply is the seed and the verdict.** The cost
being priced is a *cold-checkout* rebuild on a hosted shard: `actions/checkout`
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ The rules a prompt can read off a branch move into `.claude/agents/reviewer.md`,
and the gates that held them go.

- A test is green and fast or it is deleted in the same pull request and filed;
then `src/redlist.rs`, `src/tiers.rs`, `35383398^:src/durations.rs` and
then `src/redlist.rs`, `35383398^:src/durations.rs` and
`tests/test-durations` have no subject and go.
- The toolchain is content-addressed by the four trees that produce it, one
directory per hash, never mutated; then the sysroot claim, `src/buildlock.rs`
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,5 +27,5 @@ re-run. The logd branch, its tree as committed at `ebf2f76a`, measured the same

The case's premise is arranged rather than raced: a write after a verified
flush is made to go out before the job ends the boot (or the case waits on the
break before it starts the job), and ten consecutive `--nightly
usb_transport_break` runs are green.
break before it starts the job), and ten consecutive
`usb_transport_break` runs are green.
Original file line number Diff line number Diff line change
Expand Up @@ -14,5 +14,5 @@ compiles. PR #564's round-2 review handed `pass_block`'s drain
`Entered::Pass { depth: 0 }`: every host test and the fast tier stayed green,
and only the nightly `blocked_dump` went red.

**Exit**: a mutation of either call site's `Entered` reds a host test or a
fast-tier test. Owner: orchestrator.
**Exit**: a mutation of either call site's `Entered` reds a host test.
Owner: orchestrator.
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ opened: 2026-09-27

# `blockd_survives_its_death` reds on a replacement that rewrites acknowledged writes out of order

The nightly-tier test kills blockd with writes in flight and judges QEMU's
The test kills blockd with writes in flight and judges QEMU's
block trace. It has gone red on one signature, the same sectors each time:

```
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ opened: 2026-09-27

# `home_budget_refusal_retried` is red on every nightly

`home_budget_refusal_retried` (nightly tier) is red, and red alone, on the
`home_budget_refusal_retried` is red, and red alone, on the
nightlies of three trees, each time with the same two shapes:
- main at e8d7c9c0 (run 36228604597);
- PR #524's branch at 8c5be843 (run 36273557690);
Expand Down
2 changes: 1 addition & 1 deletion issues/hardware/tearing-is-what-gop-cannot-give-back.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ pixels — which is the "reference capture" half of a pixel-identity check, with
no new mechanism at all.

**A per-client damage oracle exists and is green.** `desktop_typing_damage`
(`tests/toyos.rs:6670`, registered at `:823`, Nightly) reads the compositor's
(`tests/toyos.rs:6670`, registered at `:823`) reads the compositor's
own `damage_px_max` while a client is driven and refuses a frame over 2% of the
screen. Measured on the dev host 2026-08-31: `eight lines typed, 16
appearances; biggest frame 9472 of 2073600 px over 2 intervals`, PASS. It bounds
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,8 @@ Held by the orchestrator. Every stage waits on PR #536 (`wt/toyos-fsd`).

## Stages

Every guest test this track names is registered at `Tier::Fast` or
`Tier::Nightly` with no `src/redlist.rs` row. A deleted or disabled test covers
nothing.
Every guest test this track names is registered with no `src/redlist.rs` row.
A deleted or disabled test covers nothing.

1. **The rename**, one mechanical PR, first after #536. Before stage 1 is
briefed, the exit's search below runs once over the `rust/` fork's delta as
Expand Down
6 changes: 3 additions & 3 deletions issues/kernel/a-claims-own-refusals-are-read-by-nothing.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ The refusals `kernel/src/pcidev/mod.rs` raises that are read back are
`iommu_virtio_platform`'s no-unit arm, `Refusal::NoInterrupt` by
`virtio_net_no_msix`, `Refusal::CapsTruncated` by `pci_claim_caps_truncated`,
the domain by `userdev_dma_fault`, and `SYS_DEVICE_REG_READ`'s bound by netd's
own `config_space_is_bounded`. These are reached by no test in any tier:
own `config_space_is_bounded`. These are reached by no test:

- `ClaimError::Ambiguous`, a config naming a device this machine has two of;
- `ClaimError::KernelDriven`, a claim on a function one of this kernel's own
Expand All @@ -35,8 +35,8 @@ own `config_space_is_bounded`. These are reached by no test in any tier:
no BAR at all, so what is unread is the hand-over that succeeds.

A one-field mutation of any of them — an index bound compared `<=` rather than
`<`, a window overlap accepted, a grant total never summed — leaves every arm in
every tier green.
`<`, a window overlap accepted, a grant total never summed — leaves every arm
green.

Owned by whoever next adds a boot config whose own test binary holds a claimable
function: netd holds this machine's only claim, and a test that makes it fail
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ Until `logd` reads the frame the owner word is 0, and `push` then keeps
nothing for anyone. A child that starts flooding in that window can take
every slot, the owner's included.

`log_ring_keeps_the_owners_slots` (fast tier) is red this way beside the
`log_ring_keeps_the_owners_slots` is red this way beside the
other `log_` guests and green alone. Its `/log` holds `===READY===`,
`===TEST_START test_rs_log_flood===` and 1917 flood lines: exactly the ring's
1919 shared slots, with no slot left for test-runner's `===TEST_END`.
Expand Down
5 changes: 1 addition & 4 deletions issues/kernel/desktop-window-child-freeze.md
Original file line number Diff line number Diff line change
Expand Up @@ -134,10 +134,7 @@ machine that is still running passes somewhere. Judge the next occurrence by the
signature exactly as before; a green run of this test proves what it always
proved, which is nothing.

Two things a reader looking for the next sighting needs. The test is
`Tier::Nightly` (`src/tiers.rs`), so a plain `cargo test` does not run it at all
— `cargo test --test toyos-build -- --nightly desktop_window_child` does. And
the one instrument that could name what a stopped CPU is doing has still never
The one instrument that could name what a stopped CPU is doing has still never
been fired at one: `sched::dump`'s NMI probe separates a CPU spinning with `IF`
clear from one halted with its kick undelivered from one wedged below the
interrupt layer. Take `info registers -a` over QMP before pressing Ctrl+Alt+D,
Expand Down
2 changes: 1 addition & 1 deletion issues/kernel/no-kernel-address-is-drawn-per-boot.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ with `RANDOMIZE_BASE` and `RANDOMIZE_MEMORY`
(`debian.master/config/annotations:10674,10677`): at most 9 bits of text slot,
and a direct map placed at PUD granularity.

**Exit**: on every proving machine, over 64 boots in a `Tier::Weekly` test,
**Exit**: on every proving machine, over 64 boots in a test,
each claimed bit is set 12 to 52 times, the guard of
`issues/kernel/the-kernel-has-no-stack-protector.md` and the first spawn's
bases included. **Mutation**: a fixed base. **Oracle**: Linux's bits.
Loading
Loading