diff --git a/issues/a-netstack-client-cannot-tell-a-reset-from-the-peers-fin.md b/issues/a-netstack-client-cannot-tell-a-reset-from-the-peers-fin.md index 42c6605898..b1513cae8e 100644 --- a/issues/a-netstack-client-cannot-tell-a-reset-from-the-peers-fin.md +++ b/issues/a-netstack-client-cannot-tell-a-reset-from-the-peers-fin.md @@ -4,25 +4,55 @@ kind: defect opened: 2026-09-26 --- -# A netd client cannot tell a reset connection from the peer's FIN - -A piped TCP connection reaches its client as two pipes, and the only thing -the receive pipe can say about the end of the stream is EOF: its write end -closed. `bridge_piped` (`userland/netd/src/main.rs`) closes that end for -every ending alike — the peer's FIN, the peer's RST, and netd resetting the -connection itself through `PipedConnection::refuse` (a pipe whose ring page -could not be allocated, or a handle netd cannot write or read). The client -reads the same zero-byte `read` in all of them, so a stream cut short by a -reset is indistinguishable from one the peer finished, and a protocol with no -length framing of its own takes a truncated stream as a whole one. - -The `ResourceExhausted` case is reachable without a hostile client: the ring -page is allocated on a pipe's first write, and a peer that speaks first (an -ssh banner) makes netd's first write the allocation. The only record is -netd's own log line. - -Read from the code, not measured. - -Exit condition: the pipe protocol carries a reset as something other than -EOF — for instance an error the client's next `read` returns — and a test -whose connection netd resets sees that error rather than EOF. +# A netstack client cannot tell a reset connection from the peer's FIN + +A stream reaches its client as two pipes, and the receive pipe's end says only +that it ended. std (`ended`, `sdk/std/sys/net/connection.rs`) reads the kind +off the send pipe: one with no reader behind a receive pipe at its end is a +failure. libc reads no kind: its `recv` answers every end 0 and every refused +read `EIO` (`userland/libc/src/socket.rs`), so a C client reads a reset as the +peer's FIN. + +On the node that order holds: a failure lets the send pipe go before the +receive pipe, and an orderly end keeps a send pipe whose last byte and FIN are +queued until its writer leaves (`userland/netstack/node/src/streams.rs`'s +header; its host tests in `userland/netstack/node/tests/streams.rs`, each red +with the order broken). + +What `main` ships is netstack on smoltcp, whose `bridge_piped` +(`userland/netstack/src/main.rs`) closes the receive pipe and then, in the same +pass, the send pipe of a connection that is no longer open, whether it failed +or ended in order. A client that reads the end between the two reads a +failure as a FIN, and one that reads it after both reads an orderly end as a +failure. A client that shut its sending half down before the peer's FIN is +the second: its connection is done in both directions at that FIN. Measured +in a guest on `main`, std, against the host kernel's TCP behind QEMU's user +network, the same program on the harness host's TCP (macOS) as the oracle: + +- a client that shut its sending half down with nothing pending, then read the + peer's four bytes and its FIN, read the four and then `ConnectionReset`, in + each of two runs, where the host read `Ok(0)`; a C client doing the same + read `recv` 0 with `main`'s libc, which reads every end so, and + `ECONNRESET` with a libc that reads the kind as std does; +- a client that read the peer's FIN and then shut its sending half down read + its next read as `Ok(0)` in two runs and as `ConnectionReset` in two others, + where the host read `Ok(0)`: the race between the two closes; +- a reset mid-stream, a reset after the client's half-close and a FIN before + the client's write read as the host read them, in each of four runs. + +**Exit condition**: netstack runs on the node, and on `tests/netcase`: + +- a guest std test whose client shuts its sending half down reads its peer's + bytes and then the end as an end, one whose peer resets a stream reads the + reset as a reset, and each line it prints is the host kernel's for the same + program against the same peer; +- libc reads an end's kind as std does, and a guest C case reads `recv` 0 at + the peer's FIN after `shutdown(SHUT_WR)`, `ECONNRESET` on a reset mid-stream + and on one after `SHUT_WR`, `send` `EPIPE` after `SHUT_WR` and `recv` 0 + after `SHUT_RD`; and it is red with `recv`'s probe of the send pipe replaced + by a plain 0, with `shutdown` not marking the sending half shut, and with + `recv` not answering 0 after `SHUT_RD`. The libc change and its host test + were written for the node and are posted on pull request #803 for the move + to carry with that case. + +**Owner**: whoever holds `issues/toyos-has-its-own-network-stack.md`. diff --git a/issues/a-shutdown-of-the-sending-half-drops-what-the-send-pipe-still-holds.md b/issues/a-shutdown-of-the-sending-half-drops-what-the-send-pipe-still-holds.md index e9f641083a..e592ad2b78 100644 --- a/issues/a-shutdown-of-the-sending-half-drops-what-the-send-pipe-still-holds.md +++ b/issues/a-shutdown-of-the-sending-half-drops-what-the-send-pipe-still-holds.md @@ -17,7 +17,15 @@ stream that ends short with a clean FIN. Dropping the write end instead is the path that works: the bridge reads the pipe to its end and closes the socket after the last byte. -Read from the code, not measured. `netstack_socket_churn` +Measured in a guest on `main`, against the host kernel's TCP behind QEMU's +user network, with a peer that echoes what it read once the client's FIN +arrives: a client that wrote 1,048,576 bytes and shut its sending half down +at once read 65,536 bytes back as sent in two runs and 65,535 in two others, +where the same program on a host's TCP reads all 1,048,576 and then the end. +What it read after them, `ConnectionReset`, is not this defect but +`issues/a-netstack-client-cannot-tell-a-reset-from-the-peers-fin.md`'s: a +client that shut its sending half down with nothing pending reads it too. +`netstack_socket_churn` (`tests/toyos-rust-tests/src/bin/netstack_socket_churn.rs`) writes into the pipe after the shutdown, which is the client's own error and not this. diff --git a/issues/a-streams-failure-reaches-its-client-as-a-reset-whatever-it-was.md b/issues/a-streams-failure-reaches-its-client-as-a-reset-whatever-it-was.md new file mode 100644 index 0000000000..0e177c786a --- /dev/null +++ b/issues/a-streams-failure-reaches-its-client-as-a-reset-whatever-it-was.md @@ -0,0 +1,37 @@ +--- +status: open +kind: defect +opened: 2026-10-09 +--- + +# A stream's failure reaches its client as a reset, whatever it was + +The pipe ABI tells a stream's client how the stream ended by the order its two +pipes end in (`userland/netstack/node/src/streams.rs`'s header), which carries +one bit: the peer's FIN, or a failure. The node knows which failure +(`toyos_net_tcp::Failure`: a reset, R2's timeout, an ICMP error), and std +answers every one `ConnectionReset`, where Linux answers a connection [tcp] +gave up on `ETIMEDOUT` and one an ICMP error ended `EHOSTUNREACH` or +`ENETUNREACH`. libc reads no failure yet +(`issues/a-netstack-client-cannot-tell-a-reset-from-the-peers-fin.md`), and +the change the move carries for it answers every one `ECONNRESET`. + +A pipe's end carries no word, and every other carrier costs what this one does +not: a third pipe or a kept request connection is a 2 MiB page each +(`kernel/src/pipe.rs`, `PIPE_SIZE`), and a reason the node keeps for a client to +ask about after both pipes are gone is kept for ever for a client that died, +since nothing then tells the node it left. The clean carrier is a word the +writer of a pipe sets as it lets its end go and the reader reads at the end, +which is the kernel's to add. + +The same word set would answer a connect that an ICMP error ended, which the +move's netstack answers `ERR_OTHER` for want of a word. + +Read from the code, not measured. + +**Exit condition**: a pipe's end carries its reason, the node writes +`Failure`'s through it, and std and libc answer each as Linux does; a node +host test reads each failure's word, and a guest test whose peer is +unreachable reads it as unreachable. + +**Owner**: whoever holds `issues/toyos-has-its-own-network-stack.md`. diff --git a/issues/libc-answers-epipe-and-raises-no-sigpipe.md b/issues/libc-answers-epipe-and-raises-no-sigpipe.md new file mode 100644 index 0000000000..139b02096d --- /dev/null +++ b/issues/libc-answers-epipe-and-raises-no-sigpipe.md @@ -0,0 +1,36 @@ +--- +status: open +kind: defect +opened: 2026-10-09 +--- + +# libc answers EPIPE and raises no SIGPIPE + +POSIX's `write` to a pipe or socket whose reader is gone, and its `send` on a +stream that can send no more, fail `EPIPE` and also send the calling thread +`SIGPIPE`, whose default action ends the process; `send` raises none only with +`MSG_NOSIGNAL`. ToyOS's libc fails `EPIPE` and raises nothing +(`userland/libc/src/posix_io.rs`, `SyscallError::Gone`): it has no signals +(`signal`, `sigaction` and `raise` in `userland/libc/src/misc.rs` do nothing), +and none of those +`issues/a-childs-end-is-an-event-and-a-parent-takes-its-children-down.md` +has it imitate is `SIGPIPE`. So every write behaves as if `MSG_NOSIGNAL` were +given, and a C program that relies on the default action to stop writing into a +closed pipe runs on, and stops only if it reads the error. `send` on a stream answers `EIO` for every refusal today; the libc +change the move carries for +`issues/a-netstack-client-cannot-tell-a-reset-from-the-peers-fin.md` answers +`EPIPE` after `shutdown(SHUT_WR)`, and raises nothing either. + +Whether libc raises `SIGPIPE` on these, or states the departure as it states +others, is a decision of libc's design that nobody has made. + +Read from the code, not measured. + +**Exit condition**: the decision is made and recorded at `posix_io.rs`, and a +guest C case asserts it on both paths, a `write` into a pipe whose reader is +gone and a `send` after `shutdown(SHUT_WR)`: with `SIGPIPE` raised, the default +action ends the process, an ignored one leaves `EPIPE` and `MSG_NOSIGNAL` +raises none; with the departure kept, each answers `EPIPE` and the process +lives on. + +**Owner**: `userland/libc`. diff --git a/issues/toyos-has-its-own-network-stack.md b/issues/toyos-has-its-own-network-stack.md index e05edcd90b..724dcc3228 100644 --- a/issues/toyos-has-its-own-network-stack.md +++ b/issues/toyos-has-its-own-network-stack.md @@ -32,6 +32,7 @@ What the node does not yet meet: - The node counts a wake unspent until an accept arrives, and cannot tell an accept on its way from one that will never be sent: an owner that reads a wake and sends no accept is owed one wake fewer from then on, and the last connection waiting is not announced. That is the node's half of `issues/an-accept-that-never-reaches-netstack-strands-its-listener.md`. Exit: that issue's, by a client that cannot spend a wake without its accept arriving, or by an accept that waits in netd and needs no wake. - A stream its client can see no more, its reader gone or the peer's FIN read through and its client writing no more, is reset once its pipe has given up no byte for 100 s, and a peer address restarts that clock for at most 16 such streams at once (`OWNERLESS_PER_PEER`, `userland/netstack/node/src/streams.rs`); one past them has 100 s from the pass that found it so, until one of the 16 is done and it takes its place. What holds in this stage, from the code: per address, 16, an estimate no measurement set. There is no floor on the rate: each of the 16 is kept for as long as its pipe gives up one byte per 100 s, which for a full 2 MiB pipe is 2,097,152 x 100 s, over six years, and then for its tail in [tcp] (the line below on a client gone with nothing left in its pipe). The 16 is no bound across addresses: addresses are not counted, and where a connect's address is its client's choice, an accepted stream's is the peer's, which on the link answers for as many addresses as it likes. The bound across them is the node's places (`userland/netstack/node/src/places.rs`), of which every stream holds one until the node lets it go, and there is no second one: in `peers_at_more_addresses_than_there_are_places_hold_no_stream_past_them` (`userland/netstack/node/tests/listeners.rs`) peers at four addresses bring sixteen connections each to a node with three places, which holds two such streams at a time, each kept alive past its 100 s, and refuses every accept and connect past them. `issues/netstack-cuts-a-departed-clients-unsent-tail-at-the-ceiling.md` stays open until netd runs on the node: on a host its exit's second clause is met (`a_departed_clients_tail_arrives_whole_while_its_peer_takes_it`) and its first for 16 streams an address. Exit: the T14 or a guest shows a peer holding one at a byte per 100 s, and the cut takes a floor on the rate, or the owner rules it needs none; and the move closes that issue with its bench row. - A stream its peer reset is gone from the node at once, so for a client that still holds it `Node::set_nodelay` answers `false` and `Node::nodelay` `None`: `issues/a-stream-its-peer-reset-refuses-the-option-requests-a-host-answers.md`, carried by the node as built. std's `nodelay()` asks netd nothing: it answers from the value its `TcpStream` holds (`sdk/std/sys/net/connection.rs`), and libc's `getsockopt` of `TCP_NODELAY` from its socket's entry (`userland/libc/src/socket.rs`). What the node carries is `set_nodelay` on a reset stream, refused where macOS refuses it under another kind and Linux answers. The node cannot hold the stream for the client: its two pipe ends are all it has of the client, letting both go is how the client learns of the reset, and with both gone nothing tells it when the client leaves, so a kept stream would be kept for ever. Exit, outside the node: that issue's guest test is green, and a `getsockopt` of `TCP_NODELAY` on a stream its peer reset is answered, by libc from a value its socket holds as std's does or by a pipe ABI that gives netd a handle whose other end's leaving the kernel reports. +- How a stream ended reaches its client as the order its two pipes end in, which is one bit: the peer's FIN, or a failure. The node keeps a send pipe whose FIN is queued until its writer leaves and lets a failed stream's send pipe go first (`userland/netstack/node/src/streams.rs`), so std reads a FIN as a FIN and a failure as `ConnectionReset`, whichever failure it was: `issues/a-streams-failure-reaches-its-client-as-a-reset-whatever-it-was.md`. libc reads no end's kind until the move carries its change for `issues/a-netstack-client-cannot-tell-a-reset-from-the-peers-fin.md` (posted on pull request #803) with that issue's guest C case on `tests/netcase`. Exit: those two issues'. - Every frame and every deadline ends in a pass over every stream, every transmit opportunity walks every stream twice (once for the count each peer address keeps alive, once to pass those whose connect is not answered), and `Node::next_deadline` reads every stream: linear in streams per frame, as netd's `bridge_piped` is today. For an idle stream a pass is one `recv_with`, two `status` and one read of its client's pipe, which in netd is a system call: 1,000 system calls a frame with 1,000 idle streams. A pass's `recv_with` also re-files its connection's deadline and offers it to the transmit round, so one frame makes every idle stream eligible and the next opportunity serves each to learn it has nothing. [tcp] cannot say which connections a segment, a timer, an ICMP error or a failed next hop moved: it reports `drain_eligible` and `drain_gone`, which are the round's. The change: [tcp] marks a held connection in `for_conn`, `tick`, `icmp` and `end` and hands the marks out once (`drain_moved`); the node maps a connection to its stream, keeps its deadlines ordered, and passes only the streams [tcp] moved, the one a pipe's wake names, and those due. Exit: at the move, netd's CPU time per frame on the T14 with 1, 100 and 1,000 idle streams beside one bulk stream is measured, and the change lands if 1,000 idle streams cost more than 12.3 µs a frame over what one costs. The 12.3 µs is the time a full frame takes on a wire of 1 Gbit/s, 1,538 bytes of 8 ns with its preamble and gap: past it the passes alone take longer than the bulk stream's frames take to arrive. It is a threshold by arithmetic and no measurement; by estimate, not measured, 1,000 system calls exceed it. - A client that is gone with nothing left in its pipe is finished by [tcp]'s rules for a user who let go: a reset once it has been idle 60 s, where netd on smoltcp reset it 100 s after the client left whatever the peer said. A peer that acknowledges a byte a minute holds such a connection for as long as it has bytes to acknowledge. Exit: the T14 or a guest shows such a peer holding one, and [tcp]'s rule takes a bound from the client's leaving; or the owner rules the idle bound is the one. - `Tcp::ready`, `Tcp::orphans` and `Tcp::options` are additions to [tcp] the specifications do not hold, made for the node's wakes, its places and the copy a stream keeps of its connection's options; their tests (`toyos-net-shard/tcp/tests/held.rs`) name no scenario id. `Shard::listen` refuses an address [ip] does not hold usable, by the rule [udp]'s bind refuses by (`udp.bind-address-not-local`), and counts nothing for it; the node's test of it (`a_listener_is_at_the_address_it_named_and_only_one_the_machine_holds`) names no scenario either. A listener whose address the machine loses stands as a datagram socket bound to it does, with its port and its place and its owner told nothing, and answers again once the address is back (`a_listener_stands_while_its_address_is_lost_and_answers_when_it_is_back`, `userland/netstack/node/tests/lease.rs`); no host was read for it. Exit: the specifications hold the three calls and the refusal, and the tests name their scenarios. diff --git a/userland/netstack/node/src/streams.rs b/userland/netstack/node/src/streams.rs index 41e86527a6..6e5bc08f44 100644 --- a/userland/netstack/node/src/streams.rs +++ b/userland/netstack/node/src/streams.rs @@ -4,10 +4,16 @@ //! **The node holds a stream for as long as it holds one of its pipes.** A pass //! ([`Node::bridge`]) moves what each side takes and lets go of an end that is finished: the //! to-client end once the peer's FIN or a failure has been read through it or its reader is gone, -//! the from-client end once its last byte is queued and the FIN after it, or the connection is -//! over. With neither end left the node closes the connection, which [tcp] then finishes alone -//! by its own rules for a user who let go (a reset if text is unread, RFC 9293 §3.6.1), and the -//! stream's id names nothing. +//! the from-client end once the connection failed, or its writer is gone and its last byte is +//! queued with the FIN after it. With neither end left the node closes the connection, which +//! [tcp] then finishes alone by its own rules for a user who let go (a reset if text is unread, +//! RFC 9293 §3.6.1), and the stream's id names nothing. +//! +//! **How a stream ended is the order its ends go in**, which is all a pipe can say: a client that +//! reads the end of the to-client pipe and finds the from-client pipe still read had the peer's +//! FIN, and one that finds it gone had a failure. So a failure lets the from-client end go before +//! the to-client end, and an orderly end keeps a from-client end whose last byte and FIN are +//! queued, read no more, until its writer leaves or the client lets go of the stream. //! //! **Nothing leaves the stack that the client's pipe did not take**, and **nothing leaves the //! pipe that the stack will not take**: a byte moved is a byte acknowledged to whoever sent it, @@ -149,8 +155,10 @@ struct Stream { conn: ConnId, /// The peer's address: what [`OWNERLESS_PER_PEER`] counts by. remote: Ipv4Addr, - to_client: Option>, + /// Before `to_client`, so a stream dropped whole lets its send pipe go first: the order a + /// failure owes its client. from_client: Option>, + to_client: Option>, /// The connect is not answered yet, and no byte moves. connecting: bool, /// The connect's, while connecting; then the cut's, once the client can see the stream no @@ -159,6 +167,11 @@ struct Stream { /// The client writes no more: its writer is gone, it shut its sending half down or it /// closed. An empty pipe is then the end. done_writing: bool, + /// The FIN is queued after the client's last byte. A from-client end still held is kept for + /// its client to find there, until its writer leaves. + fin_queued: bool, + /// The client let go of the stream, or nobody reads it: nobody asks how it ended. + left: bool, /// One of its peer address's [`OWNERLESS_PER_PEER`]: a byte given up restarts the cut's clock. extended: bool, /// The to-client pipe refused bytes [tcp] holds. @@ -181,6 +194,8 @@ impl Stream { connecting: false, deadline: None, done_writing: false, + fin_queued: false, + left: false, extended: false, held: false, room: false, @@ -227,14 +242,23 @@ impl Stream { break; } // Nobody reads what the peer sends. - (Ok(Received::Data(_)), Some(WriteRefusal::Gone)) => self.to_client = None, + (Ok(Received::Data(_)), Some(WriteRefusal::Gone)) => { + self.to_client = None; + self.left = true; + } (Ok(Received::Data(_)), Some(WriteRefusal::Broken)) => { stack.tcp_abort(now, conn); return false; } - // The peer's FIN after its last byte, or the connection's failure: the client - // reads the end. - (Ok(Received::End) | Err(Error::Failed(_)), _) => self.to_client = None, + // The peer's FIN after its last byte: the client reads the end, and its send pipe + // stays. + (Ok(Received::End), _) => self.to_client = None, + // The connection's failure: the client's send pipe goes first, so that the end it + // reads finds no reader behind it. + (Err(Error::Failed(_)), _) => { + self.from_client = None; + self.to_client = None; + } (Err(Error::WouldBlock), _) => break, (Err(refusal), _) => unreachable!("[tcp] refused the holder of a connection a read: {refusal:?}"), } @@ -243,16 +267,17 @@ impl Stream { let mut gave_up = false; while let Some(pipe) = self.from_client.as_mut() { // Never more than [tcp] has room for, and so never a read of no bytes, whose answer - // would be the end's. + // would be the end's; [tcp] has none once the FIN is queued. let room = stack.tcp_status(conn).writable.min(CHUNK); if room == 0 { break; } let mut chunk = [0u8; CHUNK]; let Some(space) = chunk.get_mut(..room) else { unreachable!("room is at most a chunk") }; - let read = match pipe.read(space) { - Ok(read) => read, - Err(ReadRefusal::Empty) if self.done_writing => 0, + // The read, and whether the pipe's writer is gone, which a read of 0 is. + let (read, writer_gone) = match pipe.read(space) { + Ok(read) => (read, read == 0), + Err(ReadRefusal::Empty) if self.done_writing => (0, false), Err(ReadRefusal::Empty) => break, Err(ReadRefusal::Broken) => { stack.tcp_abort(now, conn); @@ -261,7 +286,10 @@ impl Stream { }; if read == 0 { stack.tcp_shutdown_write(now, conn); - self.from_client = None; + self.fin_queued = true; + if writer_gone { + self.from_client = None; + } break; } let Some(bytes) = space.get(..read) else { unreachable!("a pipe read {read} bytes into room for {room}") }; @@ -270,9 +298,15 @@ impl Stream { } let status = stack.tcp_status(conn); - // A connection that is over takes no more of the client's bytes: its writes fail rather - // than fill a pipe nobody drains. - if matches!(status.state, State::Closed | State::TimeWait) { + // A connection that failed takes no more of the client's bytes: its writes fail rather + // than fill a pipe nobody drains, and its end reads as a failure. One that ended in order + // ended after the client's FIN, so its writing was over already. + if status.failure.is_some() { + self.from_client = None; + } + // A finished send pipe is kept only for a client that can still read the end and look + // behind it. + if self.fin_queued && self.left { self.from_client = None; } self.room = status.writable > 0; @@ -280,8 +314,9 @@ impl Stream { stack.tcp_close(now, conn); return false; } - // Nothing of the stream reaches its client again, alive or not. - if self.to_client.is_none() && self.done_writing { + // Nothing of the stream reaches its client again, alive or not, and its pipe still holds + // what may be bytes to send. + if self.to_client.is_none() && self.done_writing && !self.fin_queued { if !self.extended { let kept = extended.entry(self.remote).or_insert(0); if *kept < OWNERLESS_PER_PEER { @@ -373,6 +408,7 @@ impl Node { } stream.to_client = None; stream.done_writing = true; + stream.left = true; self.bridge(now); } @@ -402,7 +438,12 @@ impl Node { pub fn pipe_gone(&mut self, now: Instant, id: StreamId, end: PipeEnd) { let Some(stream) = self.streams.live.get_mut(&id).filter(|stream| !stream.connecting) else { return }; match end { - PipeEnd::ToClient => stream.to_client = None, + PipeEnd::ToClient => { + stream.to_client = None; + stream.left = true; + } + // A finished send pipe was kept for its writer alone. + PipeEnd::FromClient if stream.fin_queued => stream.from_client = None, PipeEnd::FromClient => stream.done_writing = true, } self.bridge(now); @@ -427,7 +468,12 @@ impl Node { pub fn watches(&self) -> impl Iterator + '_ { self.streams.live.iter().filter(|(_, stream)| !stream.connecting).map(|(id, stream)| { let (from, to) = (stream.from_client.is_some(), stream.to_client.is_some()); - (*id, Watch { readable: from && stream.room, writer: from && !stream.done_writing, writable: to && stream.held, reader: to }) + (*id, Watch { + readable: from && stream.room, + writer: from && (!stream.done_writing || stream.fin_queued), + writable: to && stream.held, + reader: to, + }) }) } diff --git a/userland/netstack/node/tests/streams.rs b/userland/netstack/node/tests/streams.rs index bbccc29658..93ea1799a0 100644 --- a/userland/netstack/node/tests/streams.rs +++ b/userland/netstack/node/tests/streams.rs @@ -60,6 +60,8 @@ struct Ends { read_broken: bool, to_dropped: bool, from_dropped: bool, + /// The ends the node let go of, in the order it did. + let_go: Vec, } type Client = Rc>; @@ -87,7 +89,9 @@ impl ToClient for WriteEnd { impl Drop for WriteEnd { fn drop(&mut self) { - self.0.borrow_mut().to_dropped = true; + let mut ends = self.0.borrow_mut(); + ends.to_dropped = true; + ends.let_go.push(PipeEnd::ToClient); } } @@ -110,7 +114,9 @@ impl FromClient for ReadEnd { impl Drop for ReadEnd { fn drop(&mut self) { - self.0.borrow_mut().from_dropped = true; + let mut ends = self.0.borrow_mut(); + ends.from_dropped = true; + ends.let_go.push(PipeEnd::FromClient); } } @@ -126,6 +132,11 @@ fn dropped(client: &Client) -> (bool, bool) { (ends.to_dropped, ends.from_dropped) } +/// The ends the node let go of, in the order it did. +fn let_go(client: &Client) -> Vec { + client.borrow().let_go.clone() +} + // ---- the far end ---- #[derive(Clone, Copy, Debug, PartialEq, Eq)] @@ -837,7 +848,125 @@ fn a_shutdown_sends_what_the_pipe_held_and_then_the_fin() { assert!(net.node.shutdown_write(net.now, id)); net.pump(); assert_eq!((net.far.received.as_slice(), net.far.fin), (&b"written before the shutdown"[..], true)); - assert_eq!((dropped(&client), net.node.streams()), ((false, true), 1)); + assert_eq!((dropped(&client), net.node.streams()), ((false, false), 1), "the send pipe is kept for its writer"); + assert_eq!(net.watch(id), Some(Watch { readable: false, writer: true, writable: false, reader: true })); +} + +// RFC 9293 §3.6, case 1, as a client that shut its sending half down reads it: the peer's FIN +// after its last byte ends the to-client pipe while the from-client pipe is still there, which is +// what an orderly end is to std's `ended`. The send pipe goes when its +// writer does, and not before, however long that is. +#[test] +fn the_peers_fin_after_the_clients_shutdown_leaves_the_send_pipe_to_its_writer() { + let (mut net, id, client) = established(); + client.borrow_mut().outbox.extend(b"request"); + assert!(net.node.shutdown_write(net.now, id)); + net.pump(); + assert_eq!((net.far.received.as_slice(), net.far.fin), (&b"request"[..], true)); + let frame = net.far.text(b"response"); + net.deliver(&frame); + let frame = net.far.fin(); + net.deliver(&frame); + assert_eq!(client.borrow().inbox, b"response"); + assert_eq!((let_go(&client), net.node.streams()), (vec![PipeEnd::ToClient], 1)); + assert_eq!(net.watch(id), Some(Watch { readable: false, writer: true, writable: false, reader: false })); + + net.fire(net.now.after(Duration::from_secs(300))); + assert_eq!((let_go(&client), net.node.streams(), net.events()), (vec![PipeEnd::ToClient], 1, vec![]), "past TIME-WAIT"); + client.borrow_mut().writer_gone = true; + net.node.pipe_gone(net.now, id, PipeEnd::FromClient); + assert_eq!((let_go(&client), net.node.streams()), (vec![PipeEnd::ToClient, PipeEnd::FromClient], 0)); + assert!(net.far.resets.is_empty()); +} + +// RFC 9293 §3.6, case 1, with the client's writer gone after its shutdown and before the peer's +// FIN: the finished send pipe goes with its writer, and the stream lives on its to-client pipe, +// which still carries the peer's text and then its end. +#[test] +fn a_writer_that_leaves_after_the_shutdown_leaves_the_stream_to_its_reader() { + let (mut net, id, client) = established(); + assert!(net.node.shutdown_write(net.now, id)); + net.pump(); + assert!(net.far.fin); + client.borrow_mut().writer_gone = true; + net.node.pipe_gone(net.now, id, PipeEnd::FromClient); + assert_eq!((let_go(&client), net.node.streams()), (vec![PipeEnd::FromClient], 1)); + assert_eq!(net.watch(id), Some(Watch { readable: false, writer: false, writable: false, reader: true })); + + let frame = net.far.text(b"late"); + net.deliver(&frame); + let frame = net.far.fin(); + net.deliver(&frame); + assert_eq!(client.borrow().inbox, b"late"); + assert_eq!((let_go(&client), net.node.streams()), (vec![PipeEnd::FromClient, PipeEnd::ToClient], 0)); + assert!(net.far.resets.is_empty() && net.events().is_empty()); +} + +// RFC 9293 §3.6, case 2, with the client's end a shutdown: the connection is over in order, and +// the send pipe is still there for the client that reads the end after it. +#[test] +fn a_shutdown_after_the_peers_fin_keeps_the_send_pipe_past_the_connections_end() { + let (mut net, id, client) = established(); + let frame = net.far.fin(); + net.deliver(&frame); + assert_eq!(let_go(&client), vec![PipeEnd::ToClient]); + client.borrow_mut().outbox.extend(b"noted"); + assert!(net.node.shutdown_write(net.now, id)); + net.pump(); + assert_eq!((net.far.received.as_slice(), net.far.fin), (&b"noted"[..], true)); + assert_eq!((let_go(&client), net.node.streams()), (vec![PipeEnd::ToClient], 1)); + client.borrow_mut().writer_gone = true; + net.node.pipe_gone(net.now, id, PipeEnd::FromClient); + assert_eq!((let_go(&client), net.node.streams()), (vec![PipeEnd::ToClient, PipeEnd::FromClient], 0)); + assert!(net.far.resets.is_empty()); +} + +// RFC 9293 §3.10.7.4 after the client's FIN: the reset lets the send pipe go before the to-client +// pipe ends, so a client that shut its sending half down reads a failure and not a FIN. +#[test] +fn a_reset_after_the_clients_shutdown_lets_the_send_pipe_go_first() { + let (mut net, id, client) = established(); + assert!(net.node.shutdown_write(net.now, id)); + net.pump(); + assert!(net.far.fin); + net.far.manner = Manner::Deaf; + let frame = net.far.rst(); + net.deliver(&frame); + assert_eq!((let_go(&client), net.node.streams()), (vec![PipeEnd::FromClient, PipeEnd::ToClient], 0)); +} + +// A client whose reader is gone reads no end, so its finished send pipe goes with its reader, +// however that was found gone. +#[test] +fn a_finished_send_pipe_goes_with_its_reader() { + for kernel in [true, false] { + let (mut net, id, client) = established(); + assert!(net.node.shutdown_write(net.now, id)); + net.pump(); + assert_eq!(dropped(&client), (false, false), "kernel: {kernel}"); + client.borrow_mut().reader_gone = true; + if kernel { + net.node.pipe_gone(net.now, id, PipeEnd::ToClient); + } else { + let frame = net.far.text(b"to nobody"); + net.deliver(&frame); + } + assert_eq!((let_go(&client), net.node.streams()), (vec![PipeEnd::ToClient, PipeEnd::FromClient], 0), "kernel: {kernel}"); + } +} + +// A reset after the peer's FIN ends the send pipe its client still writes into. +#[test] +fn a_reset_after_the_peers_fin_ends_the_send_pipe() { + let (mut net, _, client) = established(); + let frame = net.far.fin(); + net.deliver(&frame); + net.far.manner = Manner::Deaf; + client.borrow_mut().outbox.extend(b"late"); + net.bridge(); + let frame = net.far.rst(); + net.deliver(&frame); + assert_eq!((let_go(&client), net.node.streams()), (vec![PipeEnd::ToClient, PipeEnd::FromClient], 0)); } #[test] @@ -876,7 +1005,7 @@ fn a_reset_ends_both_pipes_and_the_stream() { net.far.manner = Manner::Deaf; let frame = net.far.rst(); net.deliver(&frame); - assert_eq!((dropped(&client), net.node.streams()), ((true, true), 0)); + assert_eq!((let_go(&client), net.node.streams()), (vec![PipeEnd::FromClient, PipeEnd::ToClient], 0), "the send pipe first"); assert!(net.events().is_empty() && net.far.resets.is_empty()); } @@ -1136,7 +1265,7 @@ fn a_pipe_that_is_no_pipe_resets_its_connection() { client.borrow_mut().read_broken = true; net.bridge(); } - assert_eq!((dropped(&client), net.node.streams()), ((true, true), 0), "write: {write}"); + assert_eq!((let_go(&client), net.node.streams()), (vec![PipeEnd::FromClient, PipeEnd::ToClient], 0), "write: {write}"); let [reset] = &net.far.resets[..] else { panic!("one reset, not {:?}", net.far.resets) }; assert_eq!(reset.seq, net.first(), "write: {write}"); }