A TCP stream is one region and one connection to netstack (netstack data-path track, stage 3)

[Japabu]

What this is

Stage 3 of issues/netstack-moves-every-byte-through-two-kernel-pipes.md, built as #905's design round set it out and as the orchestrator ruled on R1 to R3 (#905 (comment)). Draft. The track's stage-3 exit is met on the T14 (below): in every run of internet_download the kernel copies 0.00013 to 0.00015 bytes of pipe per byte downloaded, against 2.003 on main's pipe path under the same instrument, at the same rate and C0.

A TCP stream — a connected one and an accepted one alike (R1) — is one region the client makes and lends, which is [tcp]'s two buffers, and the connection it was opened on, which carries the request and its answer and then only doorbells and the hang-up. A received byte is copied once by netstack into the region and once by the client out of it; a sent byte once by the client into the region and twice by netstack: out of the region into [tcp]'s segment scratch, then into the frame. A frame builder that copies the region's spans straight into the frame would remove the second; the track records it. Neither passes through a kernel pipe or the PIPES lock.

The change, by layer

• toyos-transport (lib/blockring/transport): stream.rs gains the ring back; new session.rs is the duplex session — the client's read, write, poll (watch/unwatch) and register sequences written once over a Lock and Bells the adapter supplies, and the server's publish, release, finish, fail and rung. Amendments A (a bell per direction), B (no end held across a wait; a take passed on), D (a registration arms and rings) and rulings 2 and 3 (a fatal end readies the send direction; write reads it before it copies) live here.
• [tcp]: Ring's storage is its own Vec or a Lent one (copied into and out of, never borrowed). Tcp::lend moves a synchronized connection onto the region's rings, everything it held at its offsets (amendment C); consume and queue are the client's read and publish, with no copy into [tcp]; the send buffer is the whole send ring. recv_with and its take.rs tests go; lent.rs holds the lend, the window update a read owes (RFC 9293 §3.8.6.2.2) and the send ring as segment text.
• The node: a stream is a connection, its lent rings and the server end ([Session] keeps the mapping in netstack). A pass reads the head, publishes, releases, queues, turns the client's shutdown into the FIN and stores a failure as the end. A hostile word is a Violation that resets that stream alone. The client leaving is Node::left; OWNERLESS_LIFE, OWNERLESS_PER_PEER and the cut go, [tcp]'s ORPHAN_IDLE owning the orphan. An accept lends the region (AcceptRefused::NoRegion).
• netstack: lent.rs judges a region through SharedMemory::lent before mapping it; a stream is one watch, its connection, for toyos::net::stream::SERVER_WATCH (READABLE alone: the kernel refuses OTHER_END_GONE on a connection, and a connection whose client is gone reads as ended, which drained takes as the client leaving); WATCHES_PER_PLACE goes, so issues/netstack-spends-two-poller-slots-per-piped-connection.md is closed. PLACE_BYTES is the region and three pipe pages.
• toyos::net::stream: the region's one declaration, the process's ends in a fixed table (MAX_STREAMS = 128) behind futex locks, and connect, accept, read, write, shutdown, nodelay, register, watch/unwatch, adopt, duplicate, let_go, is_stream. libc's three-state futex lock moves to toyos::sync and libc uses it. TcpConnectPiped, TcpAcceptPiped, TcpShutdown, TcpSetOption, read_end, write_refused and StreamRefusal go.
• std: TcpStream is a handle entered in toyos::net; FromRawFd, IntoRawFd, From<OwnedFd> and From<TcpStream> for OwnedFd take only the process's own ends; File::from_raw_fd of a stream's handle and every spawn path (stdio slots, inherit_handle, endow, provide) refuse it by name before anything moves; take_error answers the end; netstack gone is ConnectionAborted. std::os::toyos::net::TcpStreamExt gives mio the bell and register (R2). OwnedHandle::from_raw's # Safety excludes a stream's handle.
• rust/ (73f63e2441b on the fork's main): OwnedFd's drop and BorrowedFd::try_clone_to_owned call toyos::net::stream::let_go and duplicate; library/Cargo.lock gains toyos-transport and toyos-untrusted.
• R3: both crates join build/sdkversion.rs's PUBLISHED before toyos and build/sysroot.rs's sources, manifests and std patch, with a rustc-dep-of-std feature.
• libc: sockets over toyos::net::stream; recv/send answer ECONNRESET, ETIMEDOUT, EHOSTUNREACH, ECONNABORTED; poll asks the rings first and watches the connection for POLLIN and the bell for POLLOUT.
• mio (836d14f on toyos) and socket2 (12abd8f on toyos): their ToyOS arms hold std's TcpStream/TcpListener; neither depends on toyos any more; mio registers the bell as writable and calls register.
• kernel: SYS_CONNECTION_JOIN, ConnectionEnd::joined, HandleQueue::dead and the pipe ends' reference go; abuse_pipe_owner loses its join arm.
• The track carries R1 to R3: stage 3 is every stream, with stage 4's two listener cases and the join's deletion; stage 4 keeps the listener's connection, the datagram socket and the pipe ABI's last messages.
• Because the node, a host member, now builds toyos, host clippy reaches toyos; fourteen lints there are fixed (mechanical: as_chunks, is_multiple_of, a Default, an is_empty, a range, and a reasoned allow on OwnedHandle::into_raw's forget).

Net lines against main f0a987b (git diff --numstat): production +3,128 −1,338, tests +1,605 −1,051, issues +162 −224. Production grows by the session (session.rs ~850 lines with its tests), the SDK's stream end (~700) and netstack's adapter; the pipe paths it replaces were smaller because the kernel carried their state.

Round 4 (after the round-3 review)

finished's doc comment back on finished; stage 3 marked done in issues/netstack-moves-every-byte-through-two-kernel-pipes.md (the track whose stages 1 and 2 are marked done; toyos-has-its-own-network-stack.md's stages are another track's), with its exit readings and the evidence commit; main 9842eeda5 merged and the host suite rerun at the merge. No bench rerun: the comment and records change nothing the T14 boots read, and the merge brings only main's #923 (microcode as a file per CPU), outside the stream path; the T14 readings stand on c5c69f12e and main f0a987bc6, as recorded.

The T14 reading (round 3)

One T14 boot per image, internet_download, six whole runs of the 170,439,044-byte file in each, every sha256 right, judge EXIT=0 for both. The two trees differ by this change alone: baseline f11ebeaf0 is main f0a987bc6 plus the instrument, stage 3 81afbe6c6 is this head c5c69f12e plus the same instrument (27 files, git diff 0dcdbe24c 4313cb0b0), and git diff f11ebeaf0 81afbe6c6 equals git diff f0a987bc6 c5c69f12e. The baseline is the control. The scout's rows (C0 5.310 ms, H 0.2728, P 0.0397 ms per MB, 785.95 Mb/s) were read on an older base, before #917 and the rest of main since, so they are context and not the comparison.

Medians of runs 2 to 6, per MB downloaded (MB = 10⁶ bytes), lock times in ms of TSC; computed from the judge logs by round3/analysis/per-run.py:

	baseline (pipes)	stage 3 (region)
pipe_copy_bytes per byte downloaded	2.0029 (every run 2.0029)	0.000135 (every run 0.000129 to 0.000147)
pipe lock hold H	0.587 ms	0.179 ms
pipe lock spin P	0.108 ms	0.092 ms
system C0	7.770 ms	7.735 ms
rate, Mb/s	824.1 (778 to 868)	812.7 (722 to 878)

• The exit is met: pipe_copy_bytes is under 0.01 per byte downloaded in every run, all six included. What remains on pipes is the doorbells, DNS and the listener.
• The lock's hold falls 3.3x (0.41 ms per MB less), and its spin slightly. C0 does not fall: 7.735 against 7.770 ms per MB is within run-to-run spread (7.61 to 7.97 on the baseline). The pipe copies moved to netstack's copy into the region and the client's copy out of it, so no CPU saving is claimed; netring: a session region replaces the pipes between netstack and its clients (design, and the track that stages it) #863's S rule bounded the possible saving at 0.192 ms per MB, under this spread.
• The rates are equal within run-to-run noise.
• Run 1 is slow on both images: the baseline's took 12.747 s at 107.0 Mb/s (busy 0.016), stage 3's 2.900 s at 470.1 Mb/s, against 722 to 878 Mb/s for every later run. Round 2's cut first run (23.7 Mb/s for 34 s) is, by inference only, the same stall taken further: it is far below even the baseline's slowest first run here (107 Mb/s), and no boot has measured the two together. It is on main's pipe path too, so it is not this change's; it is filed as issues/the-t14s-first-download-after-boot-runs-slower-than-the-rest.md (owner netstack; candidates toyos-i219: drive the PCH's MAC in MSI mode, and prove its message before driving it #917's MSI mode, the client's wake to read, a connection's warm-up) and not chased here.

Review round 1

The build review (#905 (comment)) sent back five blockers; the first T14 internet_download was red for the first.

1. Every stream was closed at its first watch. netstack watched a stream's connection for READABLE | OTHER_END_GONE; the kernel refuses OTHER_END_GONE on anything not a pipe end (kernel/src/inbox/mod.rs, ops::pipe_end_watch) with NotSupported, and serve.rs read the refusal as the client leaving. The T14 said netstack: closing a stream — the kernel refused the watch of its connection: NotSupported, then https_download failed. Fix: the flags are one constant, toyos::net::stream::SERVER_WATCH = READABLE, which serve.rs watches with throughout. Guard: stream_handles' netstack stand-in watches its connection with SERVER_WATCH and asserts the kernel takes the watch and answers it with the client's hang-up, which then reads Ok(0); red under the old flags (Err(NotSupported)). A QEMU virtio-net run through the real netstack (evidence only, not landing) carries a std TcpStream connect, 4 MiB both ways, shutdown and end, and an accepted stream the guest answers and drops, whose host peer reads the end; under the old flags it reproduces the T14's line and fails. That the dropped accepted stream reaches its peer as an end shows a READABLE-only watch reports the hang-up, so the poller-slot issue's closure stands.
2. A freed slot was reusable before its halves were cleared. free_unused now leaves the slot reserved (Making); free clears the halves, closes the bell and region, then unreserves. The table test asserts the freed slot is not taken before; red with the slot freed early.
3. The send copy count is corrected above, in [tcp]'s Outgoing comment and in the track.
4. Issue records made false: toyos-has-its-own-network-stack.md loses the ownerless-clock and reset-option bullets and names lend/consume/queue; a-streams-failure-reaches-its-client-as-a-reset-whatever-it-was.md is narrowed to what is owed (node tests for TIMED_OUT and UNREACHABLE and a guest unreachable case); a-stream-its-peer-reset-refuses-the-option-requests-a-host-answers.md and nothing-reads-whether-a-streams-tcp-nodelay-reached-netstack.md close (no option request is left; the node's nodelay tests read the word's effect on the wire).
5. The 8 MiB per connect and accept and the 128-stream bound are filed in issues/a-stream-commits-eight-mebibytes-and-a-process-holds-128.md, exits demand-allocated regions and a growable or measured table.

The six suggestions are not taken in this round: none is a defect, and each would reopen evidence (the loom models, the session) for a change the T14 reading does not need. Suggestions 3 and 4 (the deadline read against the clock; a live-end count before TABLE) are worth their own change.

Review round 2

The round-2 review (#905 (comment)) closed round 1's five blockers and sent back on the T14 reading (R1), with two records to correct (R2, R3).

• R1, the reading. internet_download from the instrument tree 4313cb0b0 (stage 3 at 0dcdbe24c): cut bytes=101985020 secs=34.371 mbps=23.7 rtt_ms=[17.5 15.8 16.0 16.0 16.1] busy=0.007 cpu_s=1.799 cpu_ms_per_mb=17.64; idle LAN 0.34–0.42 ms. The row reads PASS because cut exits 0, but no run was whole, so no perf run= line exists: no pipe_copy_bytes per byte downloaded, no C0, hold or spin per MB. The exit is not met and not read. It is the same measurement as the scout's rows: the same https_download binary and file (170,439,044 bytes) from the same host on the same T14, the scout's handshakes 14.8–16.5 ms against these 15.8–17.5 ms, the rate by the same formula; the scout's six whole runs took about 1.7 s each (614.6–845.4 Mb/s). So the exit compares like with like, and the gap, about 33x with the CPU idle, is a stall, not a cost.
• R1, the window hypothesis tested on the host. 23.7 Mb/s over 16 ms is about 47 KB in flight, under RECEIVE_BUFFER_INITIAL = 65,535, so a window that never grows on a lent ring would explain it. a_lent_receive_ring_grows_the_window_as_a_read_does ([tcp]'s net.rs, with the harness's new Net::lend) runs the T14-shaped download (gigabit, 15 ms, timestamps, 32 MiB) with the reader taking text out of its lent ring and giving it back through consume: the window passes 65,535 and reaches the 4 MiB buffer. Under a consume that grows nothing it is red at 65,535. [tcp]'s window on a lent ring is therefore not the cause; the node consumes on every receive batch (Node::receive ends in a pass), so the window sees the client's reads as they happen. Unverified alternatives remain: the client's wakes, or toyos-i219: drive the PCH's MAC in MSI mode, and prove its message before driving it #917's switch of the I219 to MSI, which entered with main 6caaadf8f after the first T14 boot.
• R1, the attribution, now read (round 3 above). Two internet_download images were booted once each, both main f0a987bc6 (which carries toyos-i219: drive the PCH's MAC in MSI mode, and prove its message before driving it #917) under one instrument (git diff 0dcdbe24c 4313cb0b0, 27 files): the baseline wt/toyos-netring-s3-baseline f11ebeaf0 (main's pipe path) and the retake wt/toyos-netring-s3-instrument 81afbe6c6 (this head). git diff f11ebeaf0 81afbe6c6 is exactly git diff f0a987bc6 c5c69f12e, so the pair differs by this change alone: the whole change reverted. Both were whole and equal in runs 2 to 6, and both slow in run 1: the slowdown is main's, filed, and stage 3's exit is read.
• R2. The network-stack issue's §3.10.7.4 bullet now says text published into a client's receive ring.
• R3. Slot::Making's doc names both reservations: a connect or accept not yet answered, and an end free is letting go of. serve.rs's watch comment is rewrapped.

Checks

Evidence: https://github.com/ToyOSOrg/evidence/tree/4f33146cba323f79708ae51780f395d93619b81e/ (branch pr-905), every log masked (<tree>, ~, <tmp>, <mac>, <email>). Round 1's logs are under round1/ and round1b/, round 2's (with the T14 readbacks, masked further: <ip>, <guid>, <serial>, <machine>) under round2/, round 3's T14 readbacks and analysis under round3/; the rest are the first round's.

check	command	head	exit	log
host suite	cargo run -- --ci host	dddbafe (main 9842eed merged)	0 (83 steps green)	round4/host.log
host suite	cargo run -- --ci host	3375d0b	0 (83 steps green)	round3/gates/host.log
host suite	cargo run -- --ci host	c5c69f1	0 (83 steps green)	round2/gates/host.log
image	cargo run -- --build-only	c5c69f1	0	round2/gates/build-only.log
the lent-ring window test and its control	round2/controls/lent-window.sh (cargo test -p toyos-net-tcp --test net a_lent_receive_ring)	1920ac4's files	0; 101 (window 65,535)	round2/controls/lent-window-1920ac48d.log
T14 stream_handles (orchestrator)	testcases boot	0dcdbe2	pass	round2/metal/stage3-*
T14 internet_download (orchestrator)	instrument 4313cb0b0	0dcdbe2	row PASS, run cut at 23.7 Mb/s: exit unread	round2/metal/instrument-*
T14 internet_download, baseline control (orchestrator)	baseline f11ebeaf0	main f0a987b	0; six whole runs	round3/metal/baseline-*
T14 internet_download, retake (orchestrator)	instrument 81afbe6c6	c5c69f1	0; six whole runs, exit met	round3/metal/stage3-*, round3/analysis/
host suite, image	as above	0dcdbe2	0, 0	round1b/
guest suite (55 machine and screen tests, QEMU)	cargo test --test toyos-build	0dcdbe2	0	round1b/guest-suite.log
stream_handles in QEMU	controls/guest.sh … green	0dcdbe2	Some(0)	round1b/stream_handles-qemu-0dcdbe24c.log
host suite, image	as above	14142f2	0, 0	round1/gates/
instrument tree's image	cargo run -- --build-only	327685710 (superseded by 4313cb0b0, staged)	0	round1/gates/instrument-build-only.log
std streams through the real netstack, QEMU virtio-net (scratch job, not landing), and its control	round1/netcheck/netcheck.sh	14142f2	0; under the old watch flags 1	round1/netcheck/green-and-control-14142f258.log
stream_handles and the server-watch control, QEMU	controls/guest.sh … green server-watch-other-end-gone	14142f2	Some(0); Some(101)	round1/controls/stream_handles-green-and-server-watch-14142f258.log
the slot test and its control	cargo test -p toyos --lib net::stream	c0e87be	0; 101	round1/controls/slot-freed-early-c0e87be2e.log
host suite, image (first round)	as above	432df72	0, 0	gates/
guest suite (53 machine and screen tests, QEMU)	cargo test --test toyos-build	e7daded	0	guest/suite.log
stream_handles in QEMU	the harness's --debug boot, run test_rs_stream_handles (controls/guest.sh green)	e7daded	0	guest/stream_handles-qemu.log
loom: three models, 13 cases, five preemptions, and each control	controls/controls.sh (cargo test -p toyos-transport [--features <c>] --test loom -- …)	e7daded	0 (13 green), then 101 for each of the seven controls	loom/session-models-and-controls.log
loom unbounded	loom/unbounded.sh (LOOM_MAX_PREEMPTIONS=255, 600 s cap)	e7daded	10 of 13 green; the three a_poll_beside_* gave no verdict in 600 s	loom/unbounded.log
ruling 1's case and its three controls	controls/accept.sh	e7daded	0; 101, 101, 101	controls/accept-copy.log
ruling 3's case and its control	controls/write.sh	e7daded	0; 101	controls/write-after-reset.log
stream_handles' six std controls, QEMU	controls/guest.sh <control>…	a39ff1b7d	each exit_code: Some(101)	controls/stream_handles-std-controls.log

What each control reds:

• one-doorbell → both two_sleepers_*. no-pass-on → all three a_poll_beside_*. register-no-arm, register-no-ring, register-neither → all eight a_source_registered_*. want-relaxed → both two_sleepers_*, the three a_poll_beside_* and the four racing registration cases. head-only → exactly the two reset cases. Each control is in build/ci.rs's CONTROLS, and build/clippy.rs's $CONTROLS shape lints it.
• accept-drops-in-order → a_listener_answers_a_syn_… and what_a_child_held_in_order_past_a_hole_and_its_fin_reaches_its_stream; accept-drops-out-of-order and accept-drops-fin → the second alone.
• write-copies-before-end → a_write_with_room_after_a_reset_answers_it_and_copies_nothing (copied is 3, not 0).
• into-raw-lets-go → the client never says shared; file-takes-a-stream → the as-file child is taken; each spawn control → its own path's line (stdio slot took the stream's handle, slot past stdio took …, endow took …). spawn-takes-provide reds on the launcher's refusal (provide answered only a launch carries a provided connector, and this process holds no launcher): testcases gives the test no launcher, so provide's "handle moves and entry stays" arm is shown red but not by a moved handle.

The six std controls ran at a39ff1b7d, before the squash and two merges of main; the std, SDK and test sources they mutate are unchanged since but for module headers and the clippy fixes above, which change no behaviour they read. The green run of stream_handles was repeated at e7daded. e7daded to 432df72 is two merges of main (ABI/SDK changes for HDA and ps2server, none in this branch's files). Round 1 changes serve.rs's watch, toyos::net::stream's table and stream_handles, and merges main 6caaadf, whose kernel change (#910: a process's end published after its releases) touches the hang-up path; so the host suite, the image, stream_handles and the netstack run were rerun at 14142f2. main 5b48a3d (#922's audio rows, #918's claim vectors) was then merged at 0dcdbe2, touching nothing on the stream path; the host suite, the image, the guest suite and stream_handles were rerun there, and the netstack run and its controls stand on 14142f2. Round 2 adds a [tcp] test, two comment fixes and main f0a987b (#919's citation sweep, #917 already in); host suite and image were rerun at c5c69f1, and nothing else could have changed. The loom models, the [tcp] and node controls and the std controls read none of what round 1 changed and stand on their earlier heads.

Independent oracles: RFC 9293 (§3.8.6.2.2, §3.10.4, §3.6.1, §3.10.7.4, §3.10.2, §3.10.8), POSIX.1-2024 (send, recv, select), virtio 1.2 §2.9 and, unchanged, the host kernel's TCP in the node's differential tests. The whole change reverted cannot be the control for the unit and guest tests, which name APIs that do not exist on main; each decision's own control above stands for them. On metal it is the control: the T14 baseline is main under the same instrument, and the two trees differ by this change alone.

Why stream_handles is a guest test, and where it runs

std's ToyOS arms build only for toyos and stand on the kernel's handle table, and a refused spawn needs a real spawn. It plays netstack itself (a port of its own served as netstack, the region judged and published into with toyos-transport's server end), so no NIC or peer is needed. It is a shared-boot member: on the T14 it rides the testcases boot; under QEMU it ran through the harness's debug boot. Supported: x86-64 (T14, QEMU); AArch64 untested.

T14

First boot (432df72 / instrument 0de249252): stream_handles passed; internet_download was red, exit 1, every stream closed at its first watch (round 1's blocker 1). Second boot (0dcdbe2 / instrument 4313cb0b0): stream_handles passed; internet_download crawled and was cut (Review round 2 above). Third (c5c69f1 / instrument 81afbe6c6, beside the baseline f11ebeaf0): both green, every run whole; the exit is met (the T14 reading above). The instrument is the scout's (git diff 777c96b2b 2f9e090b8) less every netstack stamp and the cadence K word, which stage 3's tree no longer has, and the baseline carries exactly the same files. stream_handles passed on the second boot's testcases, at 0dcdbe2; since then only a [tcp] test, two comments, an issue and main f0a987b changed, none of which stream_handles reads.

Removed tests, and what holds their behaviour

• node/tests/streams.rs is rewritten over the real session client end on a heap region (35 cases). Cut, with pipe meaning only: the pipe-end ordering cases (a_reset_after_the_clients_shutdown_lets_the_send_pipe_go_first, a_finished_send_pipe_goes_with_its_reader, the_peers_fin_after_the_clients_shutdown_leaves_the_send_pipe_to_its_writer, a_writer_that_leaves_after_the_shutdown_leaves_the_stream_to_its_reader, a_shutdown_after_the_peers_fin_keeps_the_send_pipe_past_the_connections_end), the ownerless clock (a_departed_clients_unsent_bytes_have_100_seconds, …tail_arrives_whole…, …100_seconds_without_progress, the three sixteen-per-peer cases), the pipe refusals (a_pipe_that_is_no_pipe_resets_its_connection, a_refused_watch_resets_only_a_stream_that_holds_the_end) and the pipe-room cases (a_full_pipe_keeps_the_rest_in_the_stack…, nothing_leaves_the_pipe_that_the_stack_will_not_take, a_send_pipe_found_empty…). Their region meanings are held by the reset-is-the-end cases, a_head_past_what_was_published_resets_that_stream_alone, a_tail_past_the_send_rings_room_resets_that_stream, text_published_after_the_shutdown_resets_that_stream, a_write_larger_than_the_window_is_queued_whole_and_arrives_whole and a_read_opens_a_shut_window_at_the_next_opportunity.
• listeners.rs: an_accept_answers_the_option_of_a_stream_its_own_pass_let_go and an_accepted_stream_is_one_of_its_peers_addresss_sixteen go with the pipe and the clock (ruling 1 named the first's replacement, now what_a_child_held_…); the departed-client place cases are rewritten over [tcp]'s ORPHAN_IDLE.
• shard/tcp/tests/take.rs goes with recv_with; lent.rs holds the window update it held.
• toyos::net's read_end and write_refused tests go with the functions.

Risks and what I am unsure of

• The data path has carried twelve whole T14 downloads (six per image on round 3's two boots, every sha256 right; stage 3's six through the region), beside the QEMU scratch run. No landing suite test carries TCP through netstack (none has since ff1a11d); the stream_handles server-watch arm is the landing guard for the boundary that broke. Run 1's stall is main's too (filed).
• MAX_STREAMS = 128 a process is an estimate, and 8 MiB of contiguous memory is committed at every connect and accept: both filed in issues/a-stream-commits-eight-mebibytes-and-a-process-holds-128.md.
• Every OwnedFd drop, try_clone_to_owned, File::from_raw_fd and spawn takes the table's lock for a lookup; not measured.
• toyos::net::stream's read and write wait with a fresh one-handle Poller when a deadline is set; a blocking call with no deadline waits in SYS_READ on the doorbell.
• A stream's handle cannot move to another process (issues/a-stream-cannot-move-between-processes.md); provide's refusal is shown only by its launcher-less red under QEMU.
• The three poll-beside-a-blocking-call models gave no verdict unbounded in 600 s each (as netring: a session region replaces the pipes between netstack and its clients (design, and the track that stages it) #863's B model did); they pass at five preemptions, where their control reds.
• ConnectionAborted for netstack gone remains the closest word, as the design said.

🤖 Generated with Claude Code

https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy

[Japabu]

Rulings on R1-R3, and acceptance.

• R1: accepted streams move onto the region in stage 3, with amendment C's copy, stage 4's two listener cases, and SYS_CONNECTION_JOIN's deletion. One kind of stream at every stage.
• R2: mio and socket2 reach streams through std, plus the small std::os::toyos::net extension. One copy of the stream state.
• R3: toyos-transport and toyos-untrusted become published SDK crates in std's graph.

The track's text moves with these in the build's diff. Accepted: build stage 3.

[Japabu]

Review of #905 at 432df72ed (main 3ba4a4a2f merged)

I read the committed head with git show/git archive, not the worktree, which the implementer is editing. I ran no tests or builds. Evidence was read at pr-905 4e5a13018, and the T14 readback the orchestrator relayed was read too.

Architecture

The shape matches R1–R3 and the track. A stream is one region plus the connection it was opened on. The duplex sequences are written once in toyos-transport::session. The client's end table is the one std links, and mio and socket2 reach streams through std plus std::os::toyos::net (R2). toyos-transport and toyos-untrusted are published into std's graph (R3). SYS_CONNECTION_JOIN and its kernel plumbing are deleted cleanly, and the pinned fork commits contain no remaining callers. Ownership is right: the kernel gains nothing, netstack judges the region through SharedMemory::lent and judges every client word, and a hostile word resets only that stream. The fork commits (rust/ 73f63e2441b, mio 836d14f, socket2 12abd8f) are small, give their reasons, depend on ToyOS crates by version, and are pinned in this change. One layering point: the pure node crate now depends on the whole toyos SDK just for the region layout (a suggestion below).

The problem is the data path. It has never carried a byte on any machine, and at this head it cannot.

Blockers

1. system/netstack/src/serve.rs:741: every stream is closed at its first watch. netstack watches a stream's connection for OTHER_END_GONE, alone until the connect is answered and with READABLE after. The kernel refuses OTHER_END_GONE on anything that is not a pipe end, whatever else the watch asks (kernel/src/inbox/mod.rs:544, via ops::pipe_end_watch). tests/guest/rust-tests/src/bin/pipe_other_end_gone.rs:76 asserts exactly that refusal for a connection, and the issue this PR deletes said it in its exit: "is refused OTHER_END_GONE … a connection has two other ends for the one bit". Sockets::answered takes the NotSupported as the client leaving, so node.left closes the stream.

   • Result: no TCP stream can work. The T14 instrument boot shows exactly this: netstack: closing a stream — the kernel refused the watch of its connection: NotSupported, then https_download.rs:282 fails with netstack error (readback lines 350–397).
   • The deleted issue no longer holds. Deleting issues/netstack-spends-two-poller-slots-per-piped-connection.md rests on this watch. That closure holds only once a READABLE-only watch of the connection is shown to answer the client's hang-up. The kernel comment in arm ("that last holder posts the end's own data watch too") suggests it does, and then drained reads Ok(0). That has to be shown, not inferred.
   • Why it landed unseen: nothing reaches the netstack–kernel boundary. The node tests use a heap region and counted doorbells. stream_handles plays netstack with its own code and never watches the connection. The loom models model the kernel. The PR's "evidence for the data path" therefore never touched serve.rs or lent.rs.
   • Required, besides the fix:
     • (a) A check that would have been red. The cheapest is to make the server's watch flags one declaration in toyos::net::stream. stream_handles' netstack stand-in then watches its connection with that constant and asserts the kernel accepts the watch, answers a client ring, and answers the client's hang-up. That rides the existing testcases boot and needs no NIC. It is red under the head's flags. A type-level alternative is also fine: the SDK poller takes OTHER_END_GONE only for pipe-end types.
     • (b) Before the T14 measurement is retaken, one run where a std TcpStream goes through the real netstack and SDK: a connect, bytes both ways and a close. One local QEMU virtio-net boot is enough, as evidence rather than a new suite row. Run the accept path too if it is cheap (a listener on the same netstack).

2. sdk/toyos/src/net/stream.rs:312–319, 555–559, 592–597: a freed slot is reusable before its halves are cleared.

   • Problem: returned/forget set the slot Free under TABLE. free() then clears HALVES[slot] after TABLE is dropped. A concurrent open() can reserve that slot and install its own halves (lines 702–703) in between. The old end's free() then sets them to None.
   • Consequence: the new stream's next call panics at expect("a live end has both halves") (line 327). The window spans a whole connect, so it is rare, but it is a real ordering bug in the table every std program uses.
   • Fix: clear the halves before the slot becomes Free, either under TABLE or by keeping the slot reserved until free() returns.

3. The send-path copy count is false, and a comment went stale with it. For a lent ring, Ring::payload (system/netstack/shard/tcp/src/ring.rs:170) copies the region into scratch. The builder then copies scratch into the frame (stack.rs:197). So netstack copies a sent byte twice. The PR body says "a sent byte once by the client into the region and once by netstack into the frame". stack.rs:52's "payload borrowed from the send buffer" is now false for every stream.

   • Required: either remove the copy (let the frame builder take the payload through Lent::take straight into the frame), or correct the claim and record the extra copy in the track. In both cases fix or delete the Outgoing comment.

4. Issue records this change made false. These describe mechanisms this branch deleted, so they are not unrelated cleanup:

   • issues/toyos-has-its-own-network-stack.md line 32 (the OWNERLESS_PER_PEER 100 s cut, deleted).
   • The same file, line 75 (Tcp::recv_with and take.rs, deleted).
   • The same file, line 35 (its recv_with, pipe read and Node::pipe_gone costs).
   • issues/a-streams-failure-reaches-its-client-as-a-reset-whatever-it-was.md. The end word now carries TIMED_OUT/UNREACHABLE to std and libc. Its text names a pipe ABI and sdk/libc/src/streamend.rs, neither of which exists. Close it, or narrow it to what is still owed (its guest exit).
   • issues/a-stream-its-peer-reset-refuses-the-option-requests-a-host-answers.md. TcpSetOption is gone, and set_nodelay is now a region word.
   • issues/nothing-reads-whether-a-streams-tcp-nodelay-reached-netstack.md. It is still true in substance, since nodelay() reads back the word the client wrote, but it names MsgType::TcpSetOption.

   Update or close each by issues/README.md's procedure.

5. Debt that is admitted but not recorded. The 8 MiB of contiguous memory committed at every connect and accept, and the 128-stream cap per process (stream.rs:89, "an estimate: no measurement sets it"), appear only in the PR's Risks. The 8 MiB is the owner-accepted cost of netring: a session region replaces the pipes between netstack and its clients (design, and the track that stages it) #863's design, so it is not a defect of this build. But root CLAUDE.md's "no silent debt" wants both in issues/ with evidence, owner and exit. The 128 is also a new limit for std programs, which had no per-process table before. One issue, or a line in the track, is enough. Name the exit for each: demand-allocated regions for the first, and a measured bound or a growable table for the second.

Settled or acceptable

• Session protocol. I traced read, write, watch/unwatch, register, shutdown and the server's publish/release/finish/fail/rung. The orders hold: the end is loaded before the tail, a fatal end makes the send direction ready to every look, write reads it before it copies, the drain comes before the arm, no half is held across Bells::wait, and the pass-on goes through leave. I found no lost wake beyond what the models check.
• Loom bound. Thirteen cases at five preemptions, each control red within the bound, is sufficient evidence for these claims. Preemption bounding is loom's documented practice, and the kernel's models use it. The three a_poll_beside_* cases giving no verdict unbounded does not make the bounded verdict owed elsewhere. A cheaper unbounded model is optional (suggestion 1).
• Hostile word and hang-up. A hostile word resets that stream alone: Violation leads to fail(RESET), tcp_abort and a log event, and the node serves on. The client hanging up as the close is sound: published bytes are queued and the connection is closed (§3.10.4/§3.6.1), with [tcp]'s ORPHAN_IDLE bounding the orphan.
• std refusals. refuse_streams is the first statement of spawn, so it covers stdio, inherit_handle, endow and provide before anything moves. The six std controls red their own lines. The provide control reds on the launcher's refusal rather than a moved handle. It still shows the check alone produces the green answer, and the ordering is settled by inspection. That is disclosed, and acceptable.
• The 14 clippy fixes are forced by the node building toyos on the host. They are mechanical, and the one allow gives its reason. In scope.
• No whole-change revert. The justification holds: the new tests name APIs main lacks, each decision has its own control, and the T14 base is the scout's rows.
• Reused evidence. e7daded to 432df72 is two merges of main, and host and image checks were rerun at the head. Acceptable, though the host and image logs carry no revision line of their own; record the head in the log next time.

What the T14 reading must show for the track's pipe_copy_bytes exit

It must come from an instrument tree rebuilt on the fixed head, not 0de249252, which carries this bug. The tree's stage-3 code must equal the landing head, so git diff <head> <instrument> shows only the instrument's hunks. In every run of round 3's internet_download:

• The download completes and its bytes match.
• pipe_copy_bytes is below 0.01 per byte downloaded. Only doorbell bytes, the listener and DNS still use pipes.
• C0, the pipe lock's hold and its spin per MB are posted beside the scout's rows (C0 5.310 ms, H 0.2728 ms, P 0.0397 ms per MB, 785.95 Mb/s).

A run whose download takes more than twice its recorded time is the existing bug guard. #863's S rule decides whether a CPU saving may be claimed. A missing saving does not fail the exit.

Suggestions (optional)

1. The a_poll_beside_* models' server loop (serve, which waits and calls rung until quit) is what makes the state space unbounded. A server that answers at most the two rings these scenarios need may finish unbounded.
2. session.rs:365's saturating_sub on waiters hides an internal invariant break. An assertion would fail loudly.
3. Doorbells::wait with a deadline (stream.rs:373–384) reads "nothing after the poller returned" as TimedOut without checking the deadline. Compare against the clock instead. It also makes a fresh Poller per wait, a cost worth noting next to the per-call TABLE lookups.
4. Every OwnedFd drop, dup and spawn in every Rust program now takes TABLE and scans 256 entries. A live-end count read before the lock would make programs that hold no streams pay nothing. This is unmeasured, and not a defect.
5. The pure node crate depends on all of toyos only for toyos::net::stream's layout and codes. A protocol module free of syscalls, or toyos-transport, would keep the node's dependency surface to what it uses.
6. netstack's drained reads 64 bytes per syscall until WouldBlock. A larger buffer costs nothing and bounds the syscalls a ring storm can force per pass.

SEND BACK

[Japabu]

Review round 2 of #905 at 0dcdbe24c (main 5b48a3d17 merged; fix commits 8b1e167fd, c0e87be2e)

I read the two fix commits and both merges. I checked the kernel's watch path (kernel/src/inbox/mod.rs arm, object/ops.rs, pipe.rs release, scheduler::wake_pipe_readers) and the evidence at pr-905 7cfb70e6f: round1/ and round1b/, including the netcheck log and patches. I also read both T14 readbacks: metal/stage3/testcases and metal/instrument/download, with each judge-*.log and kernel.log. I checked that the instrument tree 4313cb0b0 differs from 0dcdbe24c only by the instrument's 27 files, and that none of them is under system/netstack. I ran nothing.

Architecture

Unchanged from round 1, and still sound. The watch fix makes the server's interest one declaration, toyos::net::stream::SERVER_WATCH, which netstack and the guest test share. That is the right boundary. The slot fix uses the existing Making state, so it adds no new state. The remaining problem is not the structure. It is that the one measurement this stage exists for came back at about 1/33 of the rate it is compared against, and nothing yet says why.

Round-1 blockers

1. Closed. The flags are SERVER_WATCH = READABLE, and serve.rs:741 watches with it at every stage of a stream.
   • The guard. stream_handles' hang_up_answers_the_server_watch is red under the old flags (Some(101), [Err(NotSupported)], round1/controls/…-14142f258.log). It is green on QEMU at 0dcdbe24c and green on the T14 (testcases readback, kernel.log line 365).
   • The QEMU run and its control. The netcheck run carried a std TcpStream through the real netstack: 4 MiB echoed, a shutdown, the end read, and an accepted stream dropped with its host peer reading the end. It is green at 14142f258, and its control reproduces the T14's NotSupported line and exits 1. That is what round 1 asked for.
   • The poller-slot issue's deletion is now justified. By inspection, a connection's read_watch is the read watch of its rx pipe. A writer end's last release calls wake_pipe_readers, which posts that watch, and has_data is true once the writer is gone. Together these mean a READABLE-only watch armed before the hang-up is woken by it. The netcheck's dropped accepted stream shows this end to end. The guest test arms after the hang-up, so it exercises only the level path; the kernel reading above covers the armed path.
   • WATCHES_PER_PLACE is gone. The issue's exit also asked the kernel to report a connection's dead direction, but that is moot: one watch now sees the client's single hang-up.
2. Closed.
   • free_unused leaves the slot Making. free clears both halves, closes the bell and region, and only then calls unreserve.
   • All three callers (returned at 324 and 565, forget at 602) reach free.
   • The table test asserts that reserve skips the freed slot. It is red with the slot set Free early (slot-freed-early-c0e87be2e.log).
3. Closed. "Twice by netstack" now appears consistently in the body, in stack.rs's Outgoing comment and in the track's stage-3 paragraph, which also records the builder change that would remove the scratch copy.
4. Closed, with one leftover (R2 below).
   • The ownerless-clock and reset-option bullets are gone, and lend/consume/queue replace recv_with.
   • The failure issue is narrowed to its owed tests.
   • Both option and nodelay issues are deleted. No citation of their slugs remains outside issues/ (git grep).
5. Closed. issues/a-stream-commits-eight-mebibytes-and-a-process-holds-128.md has evidence, an owner and an exit for each limit, and its frontmatter is valid.

Reuse of earlier evidence

Acceptable.

• The loom models drive toyos-transport only. Neither fix touches it, and git diff e7dadeda4 HEAD over system/netstack/node, shard (except the Outgoing comment), lib/blockring/transport, sdk/std and rust is empty.
• The node controls (accept and write) read nothing that changed.
• The std controls each mutate one refusal or sharing line. The table change touches only the free path, and the green stream_handles was rerun at the head, on QEMU and on the T14.
• The merges. A process's end is published only after its own releases have run (device lifecycle stage 1a) #910 touches the hang-up path, and the netcheck and server-watch control were rerun after it, at 14142f258. pcidev: a claim takes an interrupt and gives it back; the four-slot limit goes #918 and Bench audio rows run with soundserver's master muted #922 touch nothing on the stream path.

One merge does matter, for the measurement rather than for the controls. #917 (63816ac88: the I219 driven in MSI mode) came in with 6caaadf8f. It was not in the first T14 boot's tree, and it is in this one (see R1).

The T14 reading against the track's exit

The exit is not met. The reading does not contain the quantities the exit names. The instrument's kernel log has one transfer line:

https_download: cut bytes=101985020 secs=34.371 mbps=23.7 rtt_ms=[17.5 15.8 16.0 16.0 16.1] busy=0.007 … cpu_s=1.799 cpu_ms_per_mb=17.64

• The first run was cut by the job's window. It reached 60% of the 170,439,044-byte file. cut exits 0, so the row reads PASS, but it judges nothing.
• No whole run happened, so there is no perf run= line. With no perf line there is no pipe_copy_bytes over a download, no C0, and no hold or spin per MB. The only counters printed are the 2-second idle window's (k.pipe_copy_bytes=416), which say nothing about the download.

The 23.7 Mb/s is the same measurement as the scout's rows. It comes from the same job binary and the same file from the same host, on the same T14, over the same route. The handshake RTTs match: the scout read 14.8 to 16.5 ms, this boot 15.8 to 17.5 ms. The rate is computed by the same report formula, from the first body read to the last. The scout's six whole runs read 614.6 to 845.4 Mb/s (median 785.95), about 1.7 s per 170 MB. Here 102 MB took 34 s. The differences are real but small next to the gap: one partial first run against a median of six, the cadence K absent, and main's movement since the scout. The result is a regression of roughly 33x, far past the 2x bug guard.

busy=0.007, so the machine is idle while the transfer crawls. It is stalled on something, not short of CPU.

Findings

R1. BLOCKER: the download rate on the T14 fell from about 786 to 23.7 Mb/s, and the cause is not established.

• Where: metal/instrument/download/kernel.log:372.
• Consequence: the track's stage-3 exit cannot be read, and the stage's main path is about 33x slower than the reading it replaces.
• Two candidates. Both are inferences, not measurements:
  • A receive window stuck near its initial 64 KiB. 23.7 Mb/s times 16 ms is about 47 KB in flight, under limits::RECEIVE_BUFFER_INITIAL = 65,535. The window grows only through Rx::after_read, which a lent ring reaches through Conn::consume, and only once rx.rtt is sampled. That sampling and growth have not been shown to work on the lent path.
  • toyos-i219: drive the PCH's MAC in MSI mode, and prove its message before driving it #917's MSI mode. It entered between the first T14 boot and this one, and no download reading on main since it is known to me.
• Required: attribute the regression, fix it if it is stage 3's, and take the reading in which every run is whole.
• Cheapest discriminator: one T14 boot of internet_download from main 5b48a3d17 plus the scout's instrument. That is the pipe path, after toyos-i219: drive the PCH's MAC in MSI mode, and prove its message before driving it #917, and it is the whole-change-reverted control this measurement needs. Red there means the cause is independent of this PR, and it gets recorded under issues/ without blocking. Green there means stage 3 owns it.
• Host-side check for the window hypothesis: a node test that lends a region, reads through consume across several round trips with timestamps, and asserts the offered window grows past 65,535. That needs no metal.

R2. BLOCKER (small): a bullet this branch edited is left false. issues/toyos-has-its-own-network-stack.md, the RFC 9293 §3.10.7.4 bullet, still justifies keeping pre-reset text by "text a pass has already written to a client's pipe cannot be flushed". There is no pipe now. The text is published into the region's receive ring, and the same argument holds for it. Replace "written to a client's pipe" with "published into a client's receive ring". issues/README.md makes records this change touches part of its review.

R3. BLOCKER (small): a doc comment is now false. sdk/toyos/src/net/stream.rs, Slot::Making: "Taken by a connect or an accept that has not answered." Since c0e87be2e, a slot is also Making while free clears an end's halves. Say both, for example "Reserved: by a connect or an accept that has not answered, or by an end free is letting go of."

Suggestions (optional)

• system/netstack/src/serve.rs:734: the reflowed watch doc comment now has a line of about 120 columns. Rewrap it.
• Round 1's six suggestions stay optional as declined. Number 3 (comparing a timed-out wait against the clock) is worth its own change, as the body says.

PR description (for the orchestrator)

The T14 section still says the retake is "not yet run". It should record the readback above: stream_handles green on testcases, and internet_download cut at 101,985,020 bytes and 23.7 Mb/s with no perf line. The track's exit should be stated as open, with R1's attribution owed. A PASS row must not be read as the exit.

SEND BACK

[Japabu]

Review round 3 of #905 at 3375d0b6e (base f0a987bc6), focused on the T14 exit

I read 1920ac48d, the c5c69f12e merge and 3375d0b6e, and the evidence on pr-905 at 7cc1bda0d (round2/controls/, round3/metal/*-judge.log, round3/analysis/). I ran no builds or tests; the only things I recomputed were text comparisons with git diff and a few figures from the logs. Round 2's R2 and R3 are fixed as asked (the §3.10.7.4 bullet, Slot::Making), and the watch comment is rewrapped.

The pair differs by the change alone: verified

• 81afbe6c6 is c5c69f12e with 4313cb0b0 merged in. f11ebeaf0 is a single commit on f0a987bc6.
• The +/- lines of git diff f0a987bc6 f11ebeaf0 and git diff c5c69f12e 81afbe6c6 are identical: 27 files, +1205/−25.
• git diff f11ebeaf0 81afbe6c6 equals git diff f0a987bc6 c5c69f12e apart from index hashes and two hunk offsets.
• 3375d0b6e adds only the new issue file over c5c69f12e, so the stage-3 image's code is the head's code.

The baseline is therefore the whole change reverted onto the measured base, as root CLAUDE.md asks for.

The readings follow from the published logs: verified

• Both judge logs carry six whole bytes=170439044 lines, all with the pinned sha256, plus PASS and 0 past its record.
• I recomputed run 3 by hand:
  • stage 3: k.pipe_copy_bytes=22938 / 170,439,044 = 0.000135; hold 79,038,318 cycles at 2.419 GHz = 0.1917 ms/MB; C0 7.599.
  • baseline: 341,365,514 bytes = 2.0029 per byte.
  • All three match per-run.txt.
• The medians in the body's table match per-run.txt.
• In stage 3, pipe_copy_bytes lies between 0.000129 and 0.000147 in all six runs, run 1 included. The track's exit (under 0.01 per byte in every run) is met.
• The scout's rows are rightly demoted to context. Both images read C0 near 7.7 to 7.9 ms/MB against the scout's 5.31, so the base moved, not this change.

The exit statement is honest

The body claims no CPU saving. C0 7.735 vs 7.770 lies inside the baseline's own 7.61–7.97 spread, and the body attributes the hold's 3.3x fall to the copies leaving the lock, not to CPU. It says what the control shows: equal rates (812.7 vs 824.1 Mb/s, overlapping ranges), copies gone from the pipe, hold down. The track's text needs no change for this PR; marking stage 3 done is the orchestrator's, as d0181491b did for stages 1 and 2.

The new issue: acceptable

issues/the-t14s-first-download-after-boot-runs-slower-than-the-rest.md records only measured rows: both run 1s, and round 2's cut run stated neutrally. Its candidates are labelled unmeasured. "The first boot that downloaded fast (the scout's, all six runs whole)" is true: the scout's run 1 read 845.4 Mb/s. The exit, run 1 inside the band of the same boot's runs 2–6, can be read off one boot.

Independence holds for the phenomenon "run 1 is slow". It appears on main's pipe path in the control boot, at 107 Mb/s over 12.7 s.

The window test: sound

• The claim. a_lent_receive_ring_grows_the_window_as_a_read_does lends after the handshake. The application reads the lent ring itself and gives the bytes back through consume. assert_exact checks the bytes, and the window must pass 65,535 and reach BUFFER.
• The control. It deletes after_read from Conn::consume and fails on the claim's own assertion: window: 65535, exit 101 (lent-window-1920ac48d.log).
• What it rules out. A lent ring that never grows its window cannot explain round 2's crawl.

Findings

BLOCKER: system/netstack/shard/tcp/tests/common/net.rs:495–498. A doc comment is misplaced and now false. Net::lend was inserted between finished's doc comment and finished. As a result:

• lend's doc opens with "Every application finished: both FINs in, or a failure."
• finished has no doc.

The fix is to move that line back above pub fn finished. No test needs to rerun.

For the orchestrator (PR description, not code)

• "Round 2's cut first run (23.7 Mb/s for 34 s) was the same stall taken further." This is an inference, not a measurement. Between round 2's image and this one, the code changed by comments and a test (5b48a3d17..f0a987bc6 touches [tcp] only in comments), so a common cause is plausible. But a 34 s crawl at 23.7 Mb/s is far below anything the single baseline boot showed (its run 1: 107 Mb/s over 12.7 s). Label it as an inference; the issue file already words it correctly.
• "The whole change reverted cannot be the control here" (Checks, Independent oracles) now conflicts with "The baseline is the control". Scope it to the unit and guest tests.
• "The data path has crossed netstack only under QEMU" (Risks) is stale. It has now carried twelve whole downloads on the T14.

SEND BACK

[Japabu]

Evidence branch pr-905 was rewritten (owner-authorized) to mask machine identifiers in its history: tip is now 27d5a14e7 (was 4f33146); file contents at the tip are unchanged apart from the masked strings, so commit hashes cited above no longer resolve — read the same paths at the new tip.