From 64f58547cf6056ad15abe44f621f6d54e4db055d Mon Sep 17 00:00:00 2001 From: japabu Date: Wed, 30 Sep 2026 22:57:31 +0200 Subject: [PATCH 1/5] blockd sets its ring cursors before it answers an open MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `blockd_survives_its_death` has two red signatures. Neither is a write blockd lost or reordered; the second is a defect in blockd's open, fixed here. 1. "the blockd after it wrote [281592, 282600] first" (the deleted issue). Every red it cites predates the judge that reads it now: 8c5be843 and a55d62c6 do not contain 17bb26416 (#534, "a reissue keeps the order of the writes that overlap"), and the issue was filed at 59bd29ff2, which does not either; its branch merged #534 thirteen minutes later (672c4991d). The two writes are one block each, 1008 sectors apart, so the judge since #534 takes [281592, 282600] as a reissue of [282600, 281592]. What reorders them is QEMU, not blockd: - the client reissues in first-acknowledged order and puts the two on the wire together, since they do not overlap (`Client::next_request` keeps an overlapping pair apart: `overlapping_writes_go_out_again_in_order`); - blockd's `roomiest` takes the last of equally empty queues, so the first goes down SQ 4 (the nightly log's "WITHHELD ... queue 4 identifier 0" is that choice) and the second down SQ 3; - QEMU v11.1.1's `nvme_process_db` schedules each SQ's bottom half, and `aio_bh_enqueue` inserts at the list's head while `aio_bh_poll` dequeues from it: two doorbells rung before the main loop polls run their SQs last-first, and `pci_nvme_write` is traced as each SQ is processed. So the trace reads [281592, 282600] exactly when both doorbells land inside one poll. NVMe promises no order between commands outstanding at once (NVM Express 1.4, §6.3 "Command Ordering Requirements": each command "is processed as an independent entity without reference to other commands submitted to the same I/O Submission Queue or to commands submitted to other I/O Submission Queues", and a host that needs an order enforces it above the controller), and two writes to disjoint blocks leave the same medium in either order. Across the orchestrator's logs on this host, 32 passes, all TCG, every one tracing [282600, 281592]; CI's reorder is KVM's timing. The version of Debian sid's QEMU on CI was not measured. 2. "blockd_io: FAIL /AFTER.BIN after the restart: Io" (nightly 36753172688, guest 7). blockd answered MSG_OPENED and only then made its ends of the session page (`admit` called `layout::server`). A fresh region is zero, so a first open cannot tell; a reconnect sends the page its last server wrote, whose SQ head and CQ tail are still that server's. `Session::reconnect` pumps the moment it hears, and its first wait drains: a client that gets there before blockd's `admit` finds no room to ask (the stale head is past its zeroed tail) and a completion ring holding the dead session's completions. The first one names a tag the client no longer has in flight, `Client::complete` answers `Violation::Tag`, `Session::wait` ends the session, and the FAT32 call on it is `Io`. In the log, the relay of the replacement blockd's (pid 14) output had reached its partition table at 23.909 and not its "session 0 opened" line, which blockd prints after `admit`, when the client failed at 23.909: consistent with the race, not proof of it. No acknowledged write is lost to this: the session's end requeues every acknowledged, unflushed write for the next reconnect. fsd reconnects on `Ended` and would meet the same race. The fix: `Service::open` makes the server's rings and the `Opening` carries them, so no answer is sent before the cursors are zero, and `admit` takes them from it. `layout::server` states the contract. The model (`toyos-blockring/src/model.rs`) used to zero all four cursors at a session's end, which no end does: the page now keeps each cursor where its end left it until the next session's two ends set theirs, and a ring is held to its queue only while an end is looking at it. New host test `the_new_server_sets_its_cursors_before_the_client_looks`: over the page a crashed model run leaves, a client looking before the new server's ends are made gets `HeadPastTail` asking and the last session's completion hearing, which `Client::complete` refuses with `Tag`; after, the page is a fresh one. With the model's session end zeroing the page again (a checked patch, restored), that test is FAILED (EXIT=101). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L --- ...ement-that-reorders-acknowledged-writes.md | 29 -------- toyos-blockring/src/layout.rs | 3 + toyos-blockring/src/model.rs | 66 ++++++++++++++----- userland/blockd/src/main.rs | 9 ++- 4 files changed, 60 insertions(+), 47 deletions(-) delete mode 100644 issues/filesystem/blockd-survives-its-death-reds-on-a-replacement-that-reorders-acknowledged-writes.md diff --git a/issues/filesystem/blockd-survives-its-death-reds-on-a-replacement-that-reorders-acknowledged-writes.md b/issues/filesystem/blockd-survives-its-death-reds-on-a-replacement-that-reorders-acknowledged-writes.md deleted file mode 100644 index e7e39d5c174..00000000000 --- a/issues/filesystem/blockd-survives-its-death-reds-on-a-replacement-that-reorders-acknowledged-writes.md +++ /dev/null @@ -1,29 +0,0 @@ ---- -status: open -kind: defect -opened: 2026-09-27 ---- - -# `blockd_survives_its_death` reds on a replacement that rewrites acknowledged writes out of order - -The test kills blockd with writes in flight and judges QEMU's -block trace. It has gone red on one signature, the same sectors each time: - -``` -FAIL blockd_survives_its_death: QEMU's trace, after the reset: Ok([411648]); after the kill: Err("blockd died with the writes at sectors [282600, 281592] acknowledged and no flush after them (and 280584 withheld); the blockd after it wrote [281592, 282600] first") -``` - -- PR #524's branch nightly at `8c5be843` (run 36273557690), wide and alone, - with x86-64 linked by toyos-ld; -- PR #532's branch nightly at `a55d62c6` (run 36287592139), wide and alone; -- PR #532's branch on the dev host, `cargo test --test toyos-build -- --nightly - blockd_survives_its_death`: 1 red in 5. - -It passed in main's nightly at `fd62f567` (run 36278449733) and in #532's -nightly at `e4317d3f` (run 36281465192), whose guest binaries are the ones -`a55d62c6` booted. So it is a rate, it predates the rust-lld switch, and -`cargo run -- --known-red` answers NO. Whether the defect is blockd's replay -order or the test's reading of the trace is not measured. - -Exit: the ordering the verdict names cannot happen, or the verdict is shown -wrong about it, with the test green across a run of repeats. diff --git a/toyos-blockring/src/layout.rs b/toyos-blockring/src/layout.rs index 05f60c90d85..0029ce7cdb5 100644 --- a/toyos-blockring/src/layout.rs +++ b/toyos-blockring/src/layout.rs @@ -62,6 +62,9 @@ pub fn client(page: &[W; RING_WORDS]) -> ClientRings { /// The server's ends of a session page it was sent, every word it owns set to /// 0. Whatever the client left in its own is bounded when first looked at. +/// +/// Made before the open is answered: a client that reconnects sends the page +/// its last server wrote, and reads these two words the moment it hears. pub fn server(page: &[W; RING_WORDS]) -> ServerRings { (Consumer::new(page, REQUESTS), Producer::new(page, COMPLETIONS)) } diff --git a/toyos-blockring/src/model.rs b/toyos-blockring/src/model.rs index 0edc7115fba..d91b435c5cd 100644 --- a/toyos-blockring/src/model.rs +++ b/toyos-blockring/src/model.rs @@ -35,7 +35,7 @@ use core::sync::atomic::Ordering; use std::collections::HashSet; use toyos_blockhold::Holds; -use toyos_transport::{Consumer, Place, Producer, Untrusted, Word}; +use toyos_transport::{Consumer, Place, Producer, Untrusted, Violation, Word}; use crate::client::{Client, Outcome, Ticket, MAX_ATTEMPTS}; use crate::entry::{Completion, Op, Request}; @@ -129,21 +129,32 @@ struct Queues { impl Queues { fn new() -> Self { let page = core::array::from_fn(|_| Shared(Cell::new(0))); - let (client, server) = Self::ends(&page); + let (client, server) = (Self::client_ends(&page), Self::server_ends(&page)); Self { sq: VecDeque::new(), cq: VecDeque::new(), page, client, server } } - /// Both ends over `page`, every cursor stored 0. - fn ends(page: &[Shared; WORDS]) -> (ClientEnds, ServerEnds) { - ((Producer::new(page, SQ), Consumer::new(page, CQ)), (Consumer::new(page, SQ), Producer::new(page, CQ))) + /// The client's ends over `page`, its two cursors stored 0. + fn client_ends(page: &[Shared; WORDS]) -> ClientEnds { + (Producer::new(page, SQ), Consumer::new(page, CQ)) } - /// The session is over: what either queue held is gone, and the next - /// session's two ends start over the same page. - fn reset(&mut self) { + /// A server's ends over `page`, its two cursors stored 0. + fn server_ends(page: &[Shared; WORDS]) -> ServerEnds { + (Consumer::new(page, SQ), Producer::new(page, CQ)) + } + + /// The session is over: what either queue held is gone, and the page keeps + /// every cursor where its end left it. + fn ended(&mut self) { self.sq.clear(); self.cq.clear(); - (self.client, self.server) = Self::ends(&self.page); + } + + /// The next session over the same page: both ends set their cursors + /// before the client looks at it ([`crate::layout::server`]). + fn reopen(&mut self) { + self.client = Self::client_ends(&self.page); + self.server = Self::server_ends(&self.page); } fn send(&mut self, request: Request) { @@ -178,12 +189,13 @@ impl Queues { Some(want) } - /// A ring whose queue is empty gives nothing. - fn hold_empty(&mut self) { - if self.sq.is_empty() { + /// A ring whose queue is empty gives nothing to an end that is looking: + /// the server while it lives, the client while its session is up. + fn hold_empty(&mut self, server: bool, client: bool) { + if server && self.sq.is_empty() { assert_eq!(self.server.0.pop(&self.page), Ok(None), "the request ring gave what the queue does not hold"); } - if self.cq.is_empty() { + if client && self.cq.is_empty() { assert_eq!(self.client.1.pop(&self.page), Ok(None), "the completion ring gave what the queue does not hold"); } } @@ -271,6 +283,7 @@ fn start(failures: Failures) -> World { } fn connect(world: &mut World) { + world.queues.reopen(); let mut holds = Holds::new(); holds.hold(0, BLOCKS as u64, 1).expect("a fresh server holds nothing"); world.server = Some(Server { session: ServerSession::new(0, BLOCKS as u64), holds }); @@ -462,7 +475,7 @@ fn key(world: &World) -> String { } fn dfs(run: &mut Run, mut world: World) { - world.queues.hold_empty(); + world.queues.hold_empty(world.alive, world.client.up()); if run.broken.is_some() || !run.seen.insert(key(&world)) { return; } @@ -633,7 +646,7 @@ fn next(script: &[Step], world: &World) -> Vec<(String, After)> { if !world.alive && world.client.up() { let mut w = world.clone(); w.client.session_ended(); - w.queues.reset(); + w.queues.ended(); after("notice".into(), Ok(w)); } @@ -770,6 +783,29 @@ mod tests { world } + /// The page a client reconnects over holds the dead server's cursors until + /// the new server sets its own. A client looking before then has no room + /// to ask and meets the last session's completion, which it refuses as the + /// server breaking the protocol; once the server's ends are made, as + /// blockd makes them before it answers the open, the page is a fresh one. + #[test] + fn the_new_server_sets_its_cursors_before_the_client_looks() { + let world = walk(start(at_most(0, 1, 0)), &["ask", "take", "done", "read", "crash", "notice"]); + let page = &world.queues.page; + let mut client = world.client.clone(); + client.session_started(); + let (mut asks, mut hears) = Queues::client_ends(page); + assert_eq!(asks.space(page), Err(Violation::HeadPastTail)); + let stale = hears.pop(page).expect("a tail inside the ring").expect("the last session's completion"); + let stale = Completion::decode(stale).expect("a completion the last server wrote"); + assert_eq!(client.complete(stale), Err(Violation::Tag)); + + let (mut asks, mut hears) = Queues::client_ends(page); + let _server = Queues::server_ends(page); + assert_eq!(asks.space(page), Ok(DEPTH)); + assert_eq!(hears.pop(page), Ok(None)); + } + /// Two states a key that orders tags by value merges, though a reset parts /// them: F2 on the device and slot 1 free, reached with W1 asked after W0 /// was answered, and with the two in flight together. Named alike, they act diff --git a/userland/blockd/src/main.rs b/userland/blockd/src/main.rs index dc53ba86c29..29cc50afe3f 100644 --- a/userland/blockd/src/main.rs +++ b/userland/blockd/src/main.rs @@ -253,6 +253,9 @@ struct Service { /// A session decided on and not yet told to its client. struct Opening { region: Region, + /// Made before the client is told: it reads them the moment it hears, and + /// a region it opens again holds the last server's until they are. + rings: ServerRings, device_addr: u64, first: u64, blocks: u64, @@ -309,7 +312,8 @@ impl Service { let (first, blocks) = self.place(guid)?; match self.ctrl.up().claim().dma_map(region.handle()) { Ok(mapping) if mapping.bytes == SESSION_BYTES as u64 => { - Ok(Opening { region, device_addr: mapping.device_addr, first, blocks, unique: guid }) + let rings = layout::server(region.words()); + Ok(Opening { region, rings, device_addr: mapping.device_addr, first, blocks, unique: guid }) } // A region longer than a session would spend the claim's bound on // the kernel's side for every other client: refused whole. @@ -334,14 +338,13 @@ impl Service { fn admit(&mut self, opening: Opening, conn: Connection) -> u64 { let id = self.next_id; self.next_id += 1; - let rings = layout::server(opening.region.words()); self.sessions.insert( id, Served { conn, region: opening.region, device_addr: opening.device_addr, - rings, + rings: opening.rings, state: ServerSession::new(opening.first, opening.blocks), unique: opening.unique, closing: false, From 7a8c3b21e8de65f589b62bfad9efe99088d958cc Mon Sep 17 00:00:00 2001 From: japabu Date: Fri, 2 Oct 2026 10:19:13 +0200 Subject: [PATCH 2/5] blockd's answer to an open is made with its ends of the page; the issue stays open The first round's fix moved `layout::server` from `admit` into `Service::open`, where nothing but reading holds it: with that move reverted at 0878aa112 and the round's tests kept, `cargo test -p toyos-blockring` and blockd's host test both exit 0. The only arm that could red was a guest run of `blockd_survives_its_death`, which #660 cut. The order is now a type. `wire::Opened`'s fields are private and `Opened::over(page, blocks, unique)` returns a server's ends and its answer together; `layout::server` is the crate's own. blockd can encode no `MSG_OPENED` before its two cursors are 0, and its `Opening` carries both. A compile-fail case on `Opened` holds the refusal (E0451), beside a block that compiles. The issue is not closed. Its exit asks for the test green across a run of repeats, and the test and its trace judge left the suite in 520c0d129, so nothing that runs can meet it. Its body now says what the first round found (every cited red predates #534's judge, whose comparison passes the cited trace: run on the two orders, it passes where the comparison before it fails) and where the test went. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_013UDZQ6fSKw14e4w2TKTRfm --- ...ement-that-reorders-acknowledged-writes.md | 23 ++++++++-- toyos-blockring/src/layout.rs | 6 +-- toyos-blockring/src/model.rs | 2 +- toyos-blockring/src/wire.rs | 44 ++++++++++++++++++- userland/blockd/src/main.rs | 19 ++++---- userland/blockd/src/session.rs | 2 +- 6 files changed, 73 insertions(+), 23 deletions(-) diff --git a/issues/filesystem/blockd-survives-its-death-reds-on-a-replacement-that-reorders-acknowledged-writes.md b/issues/filesystem/blockd-survives-its-death-reds-on-a-replacement-that-reorders-acknowledged-writes.md index 05e5f8a0bf9..3b12bc49eb7 100644 --- a/issues/filesystem/blockd-survives-its-death-reds-on-a-replacement-that-reorders-acknowledged-writes.md +++ b/issues/filesystem/blockd-survives-its-death-reds-on-a-replacement-that-reorders-acknowledged-writes.md @@ -6,8 +6,8 @@ opened: 2026-09-27 # `blockd_survives_its_death` reds on a replacement that rewrites acknowledged writes out of order -The test kills blockd with writes in flight and judges QEMU's -block trace. It has gone red on one signature, the same sectors each time: +The test killed blockd with writes in flight and judged QEMU's +block trace. It went red on one signature, the same sectors each time: ``` FAIL blockd_survives_its_death: QEMU's trace, after the reset: Ok([411648]); after the kill: Err("blockd died with the writes at sectors [282600, 281592] acknowledged and no flush after them (and 280584 withheld); the blockd after it wrote [281592, 282600] first") @@ -21,8 +21,23 @@ FAIL blockd_survives_its_death: QEMU's trace, after the reset: Ok([411648]); aft It passed in main's nightly at `fd62f567` (run 36278449733) and in #532's nightly at `e4317d3f` (run 36281465192), whose guest binaries are the ones -`a55d62c6` booted. So it is a rate, and it predates the rust-lld switch. Whether the defect is blockd's replay -order or the test's reading of the trace is not measured. +`a55d62c6` booted. So it is a rate, and it predates the rust-lld switch. + +Each of those reds is the verdict of a judge that held the replacement to the +order of every write. `17bb26416` (#534) holds it to the order of the writes +that overlap, and none of `8c5be843`, `a55d62c6` and `59bd29ff2`, where this +was filed, contains it. The two writes are 1008 sectors apart and a request is +at most 256 long: #534's comparison passes the trace above, and the one before +it does not. The client puts two such writes on the wire together, and NVM +Express 1.4 §6.3 orders no two commands outstanding at once; `64f58547c`'s +message has how QEMU reorders them. A pass prints the order the first blockd +wrote in and not the replacement's, so no recorded run shows #534's judge a +reordered trace. + +`520c0d129` (#660) cut the test and its judge from the guest suite, and nothing +that runs reads a device's trace. Stage D of +`issues/build/the-guest-suite-runs-only-what-no-cheaper-tier-reaches.md` owes +blockd's restart to a host test: that is the test this exit waits on. Exit: the ordering the verdict names cannot happen, or the verdict is shown wrong about it, with the test green across a run of repeats. diff --git a/toyos-blockring/src/layout.rs b/toyos-blockring/src/layout.rs index 0029ce7cdb5..56f65d70163 100644 --- a/toyos-blockring/src/layout.rs +++ b/toyos-blockring/src/layout.rs @@ -62,10 +62,8 @@ pub fn client(page: &[W; RING_WORDS]) -> ClientRings { /// The server's ends of a session page it was sent, every word it owns set to /// 0. Whatever the client left in its own is bounded when first looked at. -/// -/// Made before the open is answered: a client that reconnects sends the page -/// its last server wrote, and reads these two words the moment it hears. -pub fn server(page: &[W; RING_WORDS]) -> ServerRings { +/// A server's way to them is [`crate::wire::Opened::over`]. +pub(crate) fn server(page: &[W; RING_WORDS]) -> ServerRings { (Consumer::new(page, REQUESTS), Producer::new(page, COMPLETIONS)) } diff --git a/toyos-blockring/src/model.rs b/toyos-blockring/src/model.rs index d91b435c5cd..67444d3117d 100644 --- a/toyos-blockring/src/model.rs +++ b/toyos-blockring/src/model.rs @@ -151,7 +151,7 @@ impl Queues { } /// The next session over the same page: both ends set their cursors - /// before the client looks at it ([`crate::layout::server`]). + /// before the client looks at it ([`crate::wire::Opened::over`]). fn reopen(&mut self) { self.client = Self::client_ends(&self.page); self.server = Self::server_ends(&self.page); diff --git a/toyos-blockring/src/wire.rs b/toyos-blockring/src/wire.rs index 489444b9c06..fadee718f7a 100644 --- a/toyos-blockring/src/wire.rs +++ b/toyos-blockring/src/wire.rs @@ -5,6 +5,16 @@ //! with it, and is answered [`MSG_OPENED`] or [`MSG_REFUSED`] with a //! [`Refusal`]. After `MSG_OPENED` the connection carries nothing but doorbell //! bytes, each way. +//! +//! **The answer comes with the server's ends of the page.** A client looks at +//! the page the moment it hears, and one that opens the same region again +//! sends the cursors its last server left on it. So [`Opened::over`] makes a +//! server's ends and its answer together: nothing else makes the ends, and an +//! answer is otherwise only read off the wire ([`Opened::decode`]). + +use toyos_transport::Word; + +use crate::layout::{self, ServerRings, RING_WORDS}; /// Open the partition whose unique GUID is the payload, over the region sent /// with it. Handles: the region. @@ -24,15 +34,45 @@ pub const GUID_BYTES: usize = 16; /// What an open was answered with: the partition's length in blocks, and its /// unique GUID as the table stores it. +/// +/// A server has one from [`Opened::over`], with its ends of the page: +/// +/// ``` +/// use core::sync::atomic::AtomicU32; +/// use toyos_blockring::{layout::RING_WORDS, wire::Opened}; +/// let page: [AtomicU32; RING_WORDS] = core::array::from_fn(|_| AtomicU32::new(0)); +/// let (_ends, _answer) = Opened::over(&page, 1, [0; 16]); +/// ``` +/// +/// and never without them: +/// +/// ```compile_fail,E0451 +/// let _answer = toyos_blockring::wire::Opened { blocks: 1, unique: [0; 16] }; +/// ``` #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub struct Opened { - pub blocks: u64, - pub unique: [u8; GUID_BYTES], + blocks: u64, + unique: [u8; GUID_BYTES], } impl Opened { pub const BYTES: usize = 8 + GUID_BYTES; + /// A server's ends of the session `page` it was sent, every word it owns + /// set to 0, and the answer to send after: `blocks` of the partition + /// `unique`. + pub fn over(page: &[W; RING_WORDS], blocks: u64, unique: [u8; GUID_BYTES]) -> (ServerRings, Self) { + (layout::server(page), Self { blocks, unique }) + } + + pub fn blocks(&self) -> u64 { + self.blocks + } + + pub fn unique(&self) -> [u8; GUID_BYTES] { + self.unique + } + pub fn encode(&self) -> [u8; Self::BYTES] { let mut out = [0u8; Self::BYTES]; out[..8].copy_from_slice(&self.blocks.to_le_bytes()); diff --git a/userland/blockd/src/main.rs b/userland/blockd/src/main.rs index 731467cd5b2..b98b9a6cbe3 100644 --- a/userland/blockd/src/main.rs +++ b/userland/blockd/src/main.rs @@ -55,7 +55,7 @@ use toyos_abi::part::{PartGuid, GUID_TEXT_LEN}; use toyos_abi::syscall::{DEV_PREFIX, SyscallError}; use toyos_blockhold::Holds; use toyos_blockring::entry::{Completion, Op}; -use toyos_blockring::layout::{self, ServerRings, DEPTH}; +use toyos_blockring::layout::{ServerRings, DEPTH}; use toyos_blockring::server::{ServerSession, Taken}; use toyos_blockring::wire::{self, Opened, Refusal}; use toyos_blockring::{BLOCK_BYTES, PORT, SESSION_BYTES}; @@ -247,13 +247,10 @@ struct Service { /// A session decided on and not yet told to its client. struct Opening { region: Region, - /// Made before the client is told: it reads them the moment it hears, and - /// a region it opens again holds the last server's until they are. rings: ServerRings, + opened: Opened, device_addr: u64, first: u64, - blocks: u64, - unique: [u8; 16], } impl Service { @@ -306,8 +303,8 @@ impl Service { let (first, blocks) = self.place(guid)?; match self.ctrl.up().claim().dma_map(region.handle()) { Ok(mapping) if mapping.bytes == SESSION_BYTES as u64 => { - let rings = layout::server(region.words()); - Ok(Opening { region, rings, device_addr: mapping.device_addr, first, blocks, unique: guid }) + let (rings, opened) = Opened::over(region.words(), blocks, guid); + Ok(Opening { region, rings, opened, device_addr: mapping.device_addr, first }) } // A region longer than a session would spend the claim's bound on // the kernel's side for every other client: refused whole. @@ -339,8 +336,8 @@ impl Service { region: opening.region, device_addr: opening.device_addr, rings: opening.rings, - state: ServerSession::new(opening.first, opening.blocks), - unique: opening.unique, + state: ServerSession::new(opening.first, opening.opened.blocks()), + unique: opening.opened.unique(), closing: false, requests: 0, posted: false, @@ -717,13 +714,13 @@ fn handshake(service: &mut Service, p: Pending, msg_type: u32, payload_len: usiz refuse(&p.conn, why); } Ok(opening) => { - let opened = Opened { blocks: opening.blocks, unique: guid }; + let opened = opening.opened; if p.conn.try_send_bytes(wire::MSG_OPENED, &opened.encode()).is_err() { service.abandon(opening); return; } let id = service.admit(opening, p.conn); - println!("blockd: session {id} opened {} ({} blocks)", guid_text(guid), opened.blocks); + println!("blockd: session {id} opened {} ({} blocks)", guid_text(guid), opened.blocks()); } } } diff --git a/userland/blockd/src/session.rs b/userland/blockd/src/session.rs index e00613a55c5..8de1de510bd 100644 --- a/userland/blockd/src/session.rs +++ b/userland/blockd/src/session.rs @@ -159,7 +159,7 @@ impl Session { /// The partition's length in blocks. pub fn blocks(&self) -> u64 { - self.opened.blocks + self.opened.blocks() } /// The most requests this session has had on the wire at once. From 9a2ea85e9a54fbf67882e05c66da74c7b37518f3 Mon Sep 17 00:00:00 2001 From: japabu Date: Fri, 2 Oct 2026 11:00:49 +0200 Subject: [PATCH 3/5] The reorder issue closes on #534; the model is main's again, and one test reads a used page Answers the review of 7a8c3b21e on #643. The issue `blockd-survives-its-death-reds-on-a-replacement-that-reorders- acknowledged-writes` is deleted, and not on this branch's type. Its subject was closed before it was filed: 17bb26416 (#534) is the fix for its signature. That commit's message names run 36273557690 and the pair [282600, 281592] / [281592, 282600], and its judge holds the replacement to the same writes as a multiset, in order only among writes that overlap. `git merge-base --is-ancestor 17bb26416 ` exits 1 for 8c5be843, a55d62c6 and 59bd29ff2, the three heads the issue cites and was filed at, and 0 for origin/main: every red it records is the verdict of the judge before #534, which is the exit's second arm, "the verdict is shown wrong about it". `Opened::over` orders the handshake and bears on neither arm. The exit's last clause named a test 520c0d129 (#660) deleted, and Stage D of `the-guest-suite-runs-only-what-no-cheaper-tier-reaches` already owes its replacement. The slug was cited nowhere else in the tree. Its durable rule is at its site already: requests in flight at once are unordered (`toyos-blockring/src/lib.rs`, and `Client::next_request`, which keeps an overlapping pair apart). What reorders the two writes, QEMU's bottom halves and NVM Express 1.4 section 6.3, is in 64f58547c's message. What the branch does leave owed is filed: `nothing-reopens-a-blockd-session- over-a-used-page`. The nightly's `blockd_io: FAIL /AFTER.BIN after the restart: Io` (run 36753172688, guest 7) was fixed by reading and is reproduced by nothing, and no tier reopens a session over a used page. The model is origin/main's again. The last round split its session end into `ended` and `reopen` and gave `hold_empty` two parameters so that the page kept a dead session's cursors. The search does not visit the difference: a state is keyed by its queues, and nothing looks at the page between `notice` and `reconnect`. With and without the split, `cargo test -p toyos-blockring --lib -- --nocapture --test-threads 1` prints the same eighteen verdict lines (end states and flushes given up, per bound). The one test that wanted a used page walks to `crash`, where main's model already holds one, and ends the clone's session itself. That test is named for what it can fail on: `a_used_page_reads_fresh_once_a_servers_ends_are_made_over_it`. Its old name stated blockd's order, which it stays green without. The compile-fail case on `Opened` names no error code. rustdoc reads one only on a nightly build and the host suites run a stable one, so `E0451` was read by nothing: `E0308` in its place left the doc-tests green. The block beside it that compiles is what holds the case, as in `toyos-net-wire`. Filed as `a-compile-fail-case-names-an-error-code-rustdoc-never-reads`, with the same mutation run on `toyos_bootmap::DirectMapEnd` (E0603) and `toyos_transport::Place` (E0080): both doc-test runs exit 0 under E0308. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_013UDZQ6fSKw14e4w2TKTRfm --- ...names-an-error-code-rustdoc-never-reads.md | 35 +++++++++++ ...ement-that-reorders-acknowledged-writes.md | 43 ------------- ...opens-a-blockd-session-over-a-used-page.md | 36 +++++++++++ toyos-blockring/src/model.rs | 60 +++++++------------ toyos-blockring/src/wire.rs | 2 +- 5 files changed, 95 insertions(+), 81 deletions(-) create mode 100644 issues/build/a-compile-fail-case-names-an-error-code-rustdoc-never-reads.md delete mode 100644 issues/filesystem/blockd-survives-its-death-reds-on-a-replacement-that-reorders-acknowledged-writes.md create mode 100644 issues/filesystem/nothing-reopens-a-blockd-session-over-a-used-page.md diff --git a/issues/build/a-compile-fail-case-names-an-error-code-rustdoc-never-reads.md b/issues/build/a-compile-fail-case-names-an-error-code-rustdoc-never-reads.md new file mode 100644 index 00000000000..9d35eb5caf5 --- /dev/null +++ b/issues/build/a-compile-fail-case-names-an-error-code-rustdoc-never-reads.md @@ -0,0 +1,35 @@ +--- +status: open +kind: tooling +opened: 2026-10-02 +--- + +# A compile-fail case names an error code rustdoc never reads + +rustdoc reads the error code of a `compile_fail` block only on a nightly build +(`ErrorCodes::from(…is_nightly_build())`, `rust/src/librustdoc/doctest.rs`), +and the host suites run the host's stable one, 1.98.1 where this was measured: +the `toyos` toolchain builds no rustdoc. There the code is a word of the fence, +and the block passes on any compile error, a renamed item or a typo included. + +Each code changed to `E0308`, a checked patch run and restored, leaves its +crate's doc-tests green: + +| case | names | `cargo test -p --doc` under `E0308` | +|---|---|---| +| `toyos_bootmap::DirectMapEnd` | `E0603` | EXIT=0 | +| `toyos_transport::Place`, three blocks | `E0080` | EXIT=0 | + +What does hold a case to its reason in this tree is the block beside it that +compiles (`toyos-net-wire/src/lib.rs`, `mod compile_fail`). `Place`'s three +have one; `DirectMapEnd`'s has none. The "compile-fail case" that Stages C and +H of `issues/build/the-guest-suite-runs-only-what-no-cheaper-tier-reaches.md` +exit on is held the same way and no other. + +**Exit**: `git grep 'compile_fail,E'` finds nothing outside `rust/` and every +compile-fail case sits beside a block that compiles; or the host suites' +rustdoc reads the code, and a case under a wrong one reds its crate's +doc-tests. + +Owner: the orchestrator, which dispatches the stages whose exits name a +compile-fail case. diff --git a/issues/filesystem/blockd-survives-its-death-reds-on-a-replacement-that-reorders-acknowledged-writes.md b/issues/filesystem/blockd-survives-its-death-reds-on-a-replacement-that-reorders-acknowledged-writes.md deleted file mode 100644 index 3b12bc49eb7..00000000000 --- a/issues/filesystem/blockd-survives-its-death-reds-on-a-replacement-that-reorders-acknowledged-writes.md +++ /dev/null @@ -1,43 +0,0 @@ ---- -status: open -kind: defect -opened: 2026-09-27 ---- - -# `blockd_survives_its_death` reds on a replacement that rewrites acknowledged writes out of order - -The test killed blockd with writes in flight and judged QEMU's -block trace. It went red on one signature, the same sectors each time: - -``` -FAIL blockd_survives_its_death: QEMU's trace, after the reset: Ok([411648]); after the kill: Err("blockd died with the writes at sectors [282600, 281592] acknowledged and no flush after them (and 280584 withheld); the blockd after it wrote [281592, 282600] first") -``` - -- PR #524's branch nightly at `8c5be843` (run 36273557690), wide and alone, - with x86-64 linked by toyos-ld; -- PR #532's branch nightly at `a55d62c6` (run 36287592139), wide and alone; -- PR #532's branch on the dev host, `cargo test --test toyos-build -- --nightly - blockd_survives_its_death`: 1 red in 5. - -It passed in main's nightly at `fd62f567` (run 36278449733) and in #532's -nightly at `e4317d3f` (run 36281465192), whose guest binaries are the ones -`a55d62c6` booted. So it is a rate, and it predates the rust-lld switch. - -Each of those reds is the verdict of a judge that held the replacement to the -order of every write. `17bb26416` (#534) holds it to the order of the writes -that overlap, and none of `8c5be843`, `a55d62c6` and `59bd29ff2`, where this -was filed, contains it. The two writes are 1008 sectors apart and a request is -at most 256 long: #534's comparison passes the trace above, and the one before -it does not. The client puts two such writes on the wire together, and NVM -Express 1.4 §6.3 orders no two commands outstanding at once; `64f58547c`'s -message has how QEMU reorders them. A pass prints the order the first blockd -wrote in and not the replacement's, so no recorded run shows #534's judge a -reordered trace. - -`520c0d129` (#660) cut the test and its judge from the guest suite, and nothing -that runs reads a device's trace. Stage D of -`issues/build/the-guest-suite-runs-only-what-no-cheaper-tier-reaches.md` owes -blockd's restart to a host test: that is the test this exit waits on. - -Exit: the ordering the verdict names cannot happen, or the verdict is shown -wrong about it, with the test green across a run of repeats. diff --git a/issues/filesystem/nothing-reopens-a-blockd-session-over-a-used-page.md b/issues/filesystem/nothing-reopens-a-blockd-session-over-a-used-page.md new file mode 100644 index 00000000000..26322aa9057 --- /dev/null +++ b/issues/filesystem/nothing-reopens-a-blockd-session-over-a-used-page.md @@ -0,0 +1,36 @@ +--- +status: open +kind: tooling +opened: 2026-10-02 +--- + +# Nothing reopens a blockd session over a used page + +A nightly went red on `blockd_io: FAIL /AFTER.BIN after the restart: Io` (run +36753172688, guest 7). blockd answered an open and only then made its ends of +the session page. A client that reconnects sends the page its last server +wrote, and one that looks before the new server's ends are made has no room to +ask (`Violation::HeadPastTail`) and hears the dead session's completion +(`Violation::Tag`). The order is a type now: `wire::Opened::over` makes a +server's ends and its answer together. + +That was fixed by reading. Nothing reproduced the `Io` before it, and nothing +that runs reopens a session over a used page: + +- blockd's host test reaches `Service::place` and `Service::listing`, never + `Service::open`; +- `toyos-blockring`'s model test holds what the transport answers over a page + a crashed session left, and is green with blockd's order reverted; +- no host runs the client's glue (`Session::pump`, `drain` and `wait` in + `userland/blockd/src/session.rs`), so that `HeadPastTail` and `Tag` end the + session and the caller's call is `Io` is read off it, not run; +- `520c0d129` cut `blockd_survives_its_death`, the guest test that killed + blockd under its client, and no guest test or metal row ends blockd. + +**Exit**: the restart test that Stage D of +`issues/build/the-guest-suite-runs-only-what-no-cheaper-tier-reaches.md` owes +for `blockd_survives_its_death`, at the tier that stage builds it on: red with +that `Io` under a mutation that moves blockd's two stores after its answer, +and green without it. + +Owner: the orchestrator, which dispatches Stage D. diff --git a/toyos-blockring/src/model.rs b/toyos-blockring/src/model.rs index 67444d3117d..9a103bc0e73 100644 --- a/toyos-blockring/src/model.rs +++ b/toyos-blockring/src/model.rs @@ -129,32 +129,21 @@ struct Queues { impl Queues { fn new() -> Self { let page = core::array::from_fn(|_| Shared(Cell::new(0))); - let (client, server) = (Self::client_ends(&page), Self::server_ends(&page)); + let (client, server) = Self::ends(&page); Self { sq: VecDeque::new(), cq: VecDeque::new(), page, client, server } } - /// The client's ends over `page`, its two cursors stored 0. - fn client_ends(page: &[Shared; WORDS]) -> ClientEnds { - (Producer::new(page, SQ), Consumer::new(page, CQ)) + /// Both ends over `page`, every cursor stored 0. + fn ends(page: &[Shared; WORDS]) -> (ClientEnds, ServerEnds) { + ((Producer::new(page, SQ), Consumer::new(page, CQ)), (Consumer::new(page, SQ), Producer::new(page, CQ))) } - /// A server's ends over `page`, its two cursors stored 0. - fn server_ends(page: &[Shared; WORDS]) -> ServerEnds { - (Consumer::new(page, SQ), Producer::new(page, CQ)) - } - - /// The session is over: what either queue held is gone, and the page keeps - /// every cursor where its end left it. - fn ended(&mut self) { + /// The session is over: what either queue held is gone, and the next + /// session's two ends start over the same page. + fn reset(&mut self) { self.sq.clear(); self.cq.clear(); - } - - /// The next session over the same page: both ends set their cursors - /// before the client looks at it ([`crate::wire::Opened::over`]). - fn reopen(&mut self) { - self.client = Self::client_ends(&self.page); - self.server = Self::server_ends(&self.page); + (self.client, self.server) = Self::ends(&self.page); } fn send(&mut self, request: Request) { @@ -189,13 +178,12 @@ impl Queues { Some(want) } - /// A ring whose queue is empty gives nothing to an end that is looking: - /// the server while it lives, the client while its session is up. - fn hold_empty(&mut self, server: bool, client: bool) { - if server && self.sq.is_empty() { + /// A ring whose queue is empty gives nothing. + fn hold_empty(&mut self) { + if self.sq.is_empty() { assert_eq!(self.server.0.pop(&self.page), Ok(None), "the request ring gave what the queue does not hold"); } - if client && self.cq.is_empty() { + if self.cq.is_empty() { assert_eq!(self.client.1.pop(&self.page), Ok(None), "the completion ring gave what the queue does not hold"); } } @@ -283,7 +271,6 @@ fn start(failures: Failures) -> World { } fn connect(world: &mut World) { - world.queues.reopen(); let mut holds = Holds::new(); holds.hold(0, BLOCKS as u64, 1).expect("a fresh server holds nothing"); world.server = Some(Server { session: ServerSession::new(0, BLOCKS as u64), holds }); @@ -475,7 +462,7 @@ fn key(world: &World) -> String { } fn dfs(run: &mut Run, mut world: World) { - world.queues.hold_empty(world.alive, world.client.up()); + world.queues.hold_empty(); if run.broken.is_some() || !run.seen.insert(key(&world)) { return; } @@ -646,7 +633,7 @@ fn next(script: &[Step], world: &World) -> Vec<(String, After)> { if !world.alive && world.client.up() { let mut w = world.clone(); w.client.session_ended(); - w.queues.ended(); + w.queues.reset(); after("notice".into(), Ok(w)); } @@ -783,25 +770,24 @@ mod tests { world } - /// The page a client reconnects over holds the dead server's cursors until - /// the new server sets its own. A client looking before then has no room - /// to ask and meets the last session's completion, which it refuses as the - /// server breaking the protocol; once the server's ends are made, as - /// blockd makes them before it answers the open, the page is a fresh one. + /// A used page reads fresh once a server's ends are made over it. A + /// client's ends alone leave the dead server's two cursors on it: no room + /// to ask, and the last session's completion to hear, which the client + /// refuses as the server breaking the protocol. #[test] - fn the_new_server_sets_its_cursors_before_the_client_looks() { - let world = walk(start(at_most(0, 1, 0)), &["ask", "take", "done", "read", "crash", "notice"]); + fn a_used_page_reads_fresh_once_a_servers_ends_are_made_over_it() { + let world = walk(start(at_most(0, 1, 0)), &["ask", "take", "done", "read", "crash"]); let page = &world.queues.page; let mut client = world.client.clone(); + client.session_ended(); client.session_started(); - let (mut asks, mut hears) = Queues::client_ends(page); + let (mut asks, mut hears): ClientEnds = (Producer::new(page, SQ), Consumer::new(page, CQ)); assert_eq!(asks.space(page), Err(Violation::HeadPastTail)); let stale = hears.pop(page).expect("a tail inside the ring").expect("the last session's completion"); let stale = Completion::decode(stale).expect("a completion the last server wrote"); assert_eq!(client.complete(stale), Err(Violation::Tag)); - let (mut asks, mut hears) = Queues::client_ends(page); - let _server = Queues::server_ends(page); + let ((mut asks, mut hears), _server) = Queues::ends(page); assert_eq!(asks.space(page), Ok(DEPTH)); assert_eq!(hears.pop(page), Ok(None)); } diff --git a/toyos-blockring/src/wire.rs b/toyos-blockring/src/wire.rs index fadee718f7a..71236c917bf 100644 --- a/toyos-blockring/src/wire.rs +++ b/toyos-blockring/src/wire.rs @@ -46,7 +46,7 @@ pub const GUID_BYTES: usize = 16; /// /// and never without them: /// -/// ```compile_fail,E0451 +/// ```compile_fail /// let _answer = toyos_blockring::wire::Opened { blocks: 1, unique: [0; 16] }; /// ``` #[derive(Clone, Copy, Debug, PartialEq, Eq)] From 0d2642da252b6853a1c48f38fbb0ce2900537c3a Mon Sep 17 00:00:00 2001 From: japabu Date: Fri, 2 Oct 2026 11:20:36 +0200 Subject: [PATCH 4/5] issues: the guest test crate still depends on blockd_io's three crates Found while checking who else names toyos-blockring. 520c0d129 deleted `tests/toyos-rust-tests/src/bin/blockd_io.rs` and left `blockd`, `toyos-blockring` and `toyos-fat32` in that crate's manifest: `git grep` for the three under `tests/toyos-rust-tests` finds `Cargo.toml` and `Cargo.lock` alone. Filed, not fixed: it is off this branch's task. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_013UDZQ6fSKw14e4w2TKTRfm --- ...-crate-depends-on-three-crates-no-test-uses.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 issues/build/the-guest-test-crate-depends-on-three-crates-no-test-uses.md diff --git a/issues/build/the-guest-test-crate-depends-on-three-crates-no-test-uses.md b/issues/build/the-guest-test-crate-depends-on-three-crates-no-test-uses.md new file mode 100644 index 00000000000..85bd5f49308 --- /dev/null +++ b/issues/build/the-guest-test-crate-depends-on-three-crates-no-test-uses.md @@ -0,0 +1,15 @@ +--- +status: open +kind: tooling +opened: 2026-10-02 +--- + +# The guest test crate depends on three crates no test uses + +`tests/toyos-rust-tests/Cargo.toml` names `blockd`, `toyos-blockring` and +`toyos-fat32`, "for `blockd_io`". `520c0d129` deleted +`tests/toyos-rust-tests/src/bin/blockd_io.rs`, and `git grep` for the three +under `tests/toyos-rust-tests` finds the manifest and its lockfile alone. + +**Exit**: the three lines and their comment are gone, the lockfile follows, and +`cargo test --test toyos-build` is green. From a5654353ff5ea602dc34ae1b1a42cbb8785e8f8a Mon Sep 17 00:00:00 2001 From: japabu Date: Fri, 2 Oct 2026 11:38:04 +0200 Subject: [PATCH 5/5] issues: the guest crate's three dead dependencies name their owner The review of 0d2642da2 on #643 found the file with evidence and an exit and no owner; the branch's other two issue files name theirs, and a compromise is recorded with all three. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_013UDZQ6fSKw14e4w2TKTRfm --- ...the-guest-test-crate-depends-on-three-crates-no-test-uses.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/issues/build/the-guest-test-crate-depends-on-three-crates-no-test-uses.md b/issues/build/the-guest-test-crate-depends-on-three-crates-no-test-uses.md index 85bd5f49308..dacd305d119 100644 --- a/issues/build/the-guest-test-crate-depends-on-three-crates-no-test-uses.md +++ b/issues/build/the-guest-test-crate-depends-on-three-crates-no-test-uses.md @@ -13,3 +13,5 @@ under `tests/toyos-rust-tests` finds the manifest and its lockfile alone. **Exit**: the three lines and their comment are gone, the lockfile follows, and `cargo test --test toyos-build` is green. + +Owner: the orchestrator.