Repository navigation
netring: a session region replaces the pipes between netstack and its clients (design, and the track that stages it) - #863
Conversation
The session-region design that replaces the pipes between netstack and its clients, and its roast, are this pull request's body. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017cSFvbD35xJ2kGANVdm23C
…e for a mapping nothing makes The track records the copies and the lock as a present-state weakness, with round 3's T14 rows as evidence and the stages the design in PR #863's body names. The defect is the pipe-section finding of that design: SYS_PIPE_MAP has had no caller outside the guest tests since a237346, and it is why every pipe and every connection direction costs a 2 MiB page. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017cSFvbD35xJ2kGANVdm23C
|
The netring scout patch: applies with diff --git a/kernel/src/counters.rs b/kernel/src/counters.rs
index df5f87d3b..9d2144048 100644
--- a/kernel/src/counters.rs
+++ b/kernel/src/counters.rs
@@ -133,7 +133,10 @@ fn sample(me: usize) -> [u64; WORDS] {
| Counter::InboxRearms
| Counter::PipeLockSpinCycles
| Counter::PipeLockHoldCycles
- | Counter::PipeCopyBytes => Some(soft::get(me, counter)),
+ | Counter::PipeCopyBytes
+ | Counter::PipeLockTakes
+ | Counter::PipeReadCopyCycles
+ | Counter::PipeWriteCopyCycles => Some(soft::get(me, counter)),
};
if let Some(value) = value {
words[1] |= 1 << counter as usize;
diff --git a/kernel/src/pipe.rs b/kernel/src/pipe.rs
index 6bd53207e..e46d1e9af 100644
--- a/kernel/src/pipe.rs
+++ b/kernel/src/pipe.rs
@@ -184,6 +184,7 @@ fn with_pipes<R>(f: impl FnOnce(&IdMap<PipeId, Pipe>) -> R) -> R {
let guard = PIPES.lock();
let t1 = soft::now();
soft::add(Counter::PipeLockSpinCycles, t1.wrapping_sub(t0));
+ soft::add(Counter::PipeLockTakes, 1);
let r = f(guard.as_ref().expect("pipes not initialized"));
soft::since(Counter::PipeLockHoldCycles, t1);
r
@@ -196,6 +197,7 @@ fn with_pipes_mut<R>(f: impl FnOnce(&mut IdMap<PipeId, Pipe>) -> R) -> R {
let mut guard = PIPES.lock();
let t1 = soft::now();
soft::add(Counter::PipeLockSpinCycles, t1.wrapping_sub(t0));
+ soft::add(Counter::PipeLockTakes, 1);
let r = f(guard.as_mut().expect("pipes not initialized"));
soft::since(Counter::PipeLockHoldCycles, t1);
r
@@ -235,7 +237,12 @@ pub fn try_read(pipe_id: PipeId, buf: &mut UserBytesMut) -> Option<usize> {
.as_mut()
.expect("available() > 0 implies a ring")
.ring
- .read(len, |off, src| buf.write_run(off, &src));
+ .read(len, |off, src| {
+ let t = crate::counters::soft::now();
+ let r = buf.write_run(off, &src);
+ crate::counters::soft::since(toyos_abi::counters::Counter::PipeReadCopyCycles, t);
+ r
+ });
crate::counters::soft::add(toyos_abi::counters::Counter::PipeCopyBytes, n as u64);
let boost = pipe.rt_boost_pending;
pipe.rt_boost_pending = false;
@@ -270,7 +277,12 @@ pub fn try_write(pipe_id: PipeId, buf: &UserBytes) -> Option<PipeWrite> {
return Some(PipeWrite::NoMemory);
};
if backing.ring.space() > 0 {
- let n = backing.ring.write(buf.len(), |off, mut dst| buf.read_run(off, &mut dst));
+ let n = backing.ring.write(buf.len(), |off, mut dst| {
+ let t = crate::counters::soft::now();
+ let r = buf.read_run(off, &mut dst);
+ crate::counters::soft::since(toyos_abi::counters::Counter::PipeWriteCopyCycles, t);
+ r
+ });
crate::counters::soft::add(toyos_abi::counters::Counter::PipeCopyBytes, n as u64);
Some(PipeWrite::Wrote(n))
} else {
diff --git a/sdk/abi/src/counters.rs b/sdk/abi/src/counters.rs
index 090907631..686d189bf 100644
--- a/sdk/abi/src/counters.rs
+++ b/sdk/abi/src/counters.rs
@@ -73,10 +73,16 @@ pub enum Counter {
PipeLockHoldCycles,
/// Bytes pipe reads and writes copied.
PipeCopyBytes,
+ /// Takes of the pipe table's lock.
+ PipeLockTakes,
+ /// Stamp ticks of a pipe read's copy out of the ring, under the lock.
+ PipeReadCopyCycles,
+ /// Stamp ticks of a pipe write's copy into the ring, under the lock.
+ PipeWriteCopyCycles,
}
impl Counter {
- pub const COUNT: usize = 21;
+ pub const COUNT: usize = 24;
pub const ALL: [Counter; Self::COUNT] = [
Self::Stamp,
Self::Smi,
@@ -99,6 +105,9 @@ impl Counter {
Self::PipeLockSpinCycles,
Self::PipeLockHoldCycles,
Self::PipeCopyBytes,
+ Self::PipeLockTakes,
+ Self::PipeReadCopyCycles,
+ Self::PipeWriteCopyCycles,
];
pub const fn name(self) -> &'static str {
@@ -124,6 +133,9 @@ impl Counter {
Self::PipeLockSpinCycles => "pipe_lock_spin_cycles",
Self::PipeLockHoldCycles => "pipe_lock_hold_cycles",
Self::PipeCopyBytes => "pipe_copy_bytes",
+ Self::PipeLockTakes => "pipe_lock_takes",
+ Self::PipeReadCopyCycles => "pipe_read_copy_cycles",
+ Self::PipeWriteCopyCycles => "pipe_write_copy_cycles",
}
}
@@ -147,7 +159,10 @@ impl Counter {
| Self::InboxRearms
| Self::PipeLockSpinCycles
| Self::PipeLockHoldCycles
- | Self::PipeCopyBytes => Rights::COUNTERS.union(Rights::TRACE),
+ | Self::PipeCopyBytes
+ | Self::PipeLockTakes
+ | Self::PipeReadCopyCycles
+ | Self::PipeWriteCopyCycles => Rights::COUNTERS.union(Rights::TRACE),
}
}
}
diff --git a/tests/guest/rust-tests/src/bin/https_download.rs b/tests/guest/rust-tests/src/bin/https_download.rs
index b1885732a..1dd8f7d61 100644
--- a/tests/guest/rust-tests/src/bin/https_download.rs
+++ b/tests/guest/rust-tests/src/bin/https_download.rs
@@ -58,7 +58,7 @@ const PROBES: usize = 5;
/// cadence's `K` netstack is told before it: a palindrome, so drift across the
/// boot weighs every `K` alike. A netstack with no cadence refuses the word, and
/// the label is then the run's place in the order.
-const K_ORDER: [u32; 6] = [8, 16, 32, 32, 16, 8];
+const K_ORDER: [u32; 6] = [32; 6];
/// MEASUREMENT ONLY: the word that sets netstack's `K`: `K=` and the value.
fn set_k(k: u32) { |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Scout image build, |
The scout boot, judged: S = 0.192 ms per MB, so the rule says the build may claim a CPU savingThe boot. One T14 boot of The judge. The instrument is sound
The rows, per MB downloaded (MB = 10⁶ bytes), TSC time at the counter's 2,419,361,856 HzM is read from a counter. D is derived from M rows, and the derivation is named.
Arithmetic. Each M cell is counter ÷ 2,419,361,856 × 10³ ÷ 170.439044. For example, run 1's R is 48,835,466 ÷ 2,419,361,856 × 10³ ÷ 170.439044 = 0.1184 ms. D is computed per run first, and then the median is taken. From the medians:
The decision, by the rule stated before the boot
What the rule does not seeThese points do not change the rule's verdict. They bound what the build's pull request can claim.
The script every number above came from (python3 -I scout.py judge.log, EXIT=0)import re, sys, statistics as st
log = open(sys.argv[1]).read().splitlines()
TSC = None
runs = []
dl = {}
for l in log:
if '[download] perf idle' in l:
TSC = int(re.search(r'tsc_hz=(\d+)', l).group(1))
m = re.search(r'\[download\] perf run=(\d+) k=32 (.*)', l)
if m:
kv = dict(w.split('=',1) for w in m.group(2).split())
runs.append((int(m.group(1)), kv))
m = re.search(r'\[download\] whole .* mbps=([\d.]+) .* run=(\d+)', l)
if m: dl[int(m.group(2))] = float(m.group(1))
c0 = {}
cur = None
for l in log:
m = re.match(r'netperf: run (\d+) per MB', l)
if m: cur = int(m.group(1))
m = re.search(r'M system C0 \(sum mperf / tsc_hz\)\s+([\d.]+) ms', l)
if m and cur: c0[cur] = float(m.group(1))
m = re.search(r'M netstack cy_ring_write \(tsc\)\s+([\d.]+) ms', l)
if m and cur: c0.setdefault(('rw',cur), float(m.group(1)))
rows = {k: [] for k in 'R W H P S X takes copyB C0 ringw mbps hold_per_take R+W<=H'.split()}
for n, kv in runs:
B = int(kv['bytes']); MB = B/1e6
ms = lambda c: int(kv[c])/TSC*1e3/MB
R=ms('k.pipe_read_copy_cycles'); W=ms('k.pipe_write_copy_cycles'); H=ms('k.pipe_lock_hold_cycles'); P=ms('k.pipe_lock_spin_cycles')
takes=int(kv['k.pipe_lock_takes'])/MB; cb=int(kv['k.pipe_copy_bytes'])/B
ok = int(kv['k.pipe_read_copy_cycles'])+int(kv['k.pipe_write_copy_cycles']) <= int(kv['k.pipe_lock_hold_cycles'])
hpt = int(kv['k.pipe_lock_hold_cycles'])/int(kv['k.pipe_lock_takes'])
vals = dict(R=R,W=W,H=H,P=P,S=H-R+P,X=H-R-W,takes=takes,copyB=cb,C0=c0[n],ringw=c0[('rw',n)],mbps=dl[n],hold_per_take=hpt)
for k,v in vals.items(): rows[k].append(v)
rows['R+W<=H'].append(ok)
for k,v in rows.items():
if k=='R+W<=H': print(k, 'all' if all(v) else 'NOT all'); continue
print(f"median {k}: {st.median(v):.4f} sorted {[round(x,4) for x in sorted(v)]}")The worktree and the readback are left as they were. Nothing in the PR's code was changed. |
|
The scout judge, |
|
The scout judge log, masked, part 2 of 2 (lines 389-722 of 722) |
|
The scout flash, |
|
Design acceptance, recorded by the orchestrator for the owner. The owner accepted the session-region design after the scout's result (#863 (comment)). His answer to "Build it?", verbatim: "i agree but explain why an abi change shouldnt that be completely userland? isnt this why we have userland drivers?" His question sets the constraint for every stage: the data path is userland's. A kernel change lands only where the kernel is the sole authority able to answer, as stage 1's two answers are: what a received handle is, and a lent region's length and whether it is kernel-allocated RAM. A stage that adds anything else to the kernel says why userland cannot do it. Stages build in the order the body gives, each its own pull request; stage 1 first. |
main moved by issues and tests/common/qemu.rs alone since 06788fc; nothing under kernel, sdk, system, lib or userland that the design cites changed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy
…design as amended Stage 0 is done: the scout's reading on #863 (S = 0.192 ms per MB, the copies 73% of the hold) joins the evidence. The owner's acceptance and its constraint, that the kernel gains only what it alone can answer, is the design line every stage keeps (#863 (comment)). Stage 1 is #867, narrowed by the orchestrator's ruling there (#867 (comment)): the kernel's one new answer is fstat's DeviceMemory, SYS_HANDLE_RECV answers no kind and SYS_SHM_MAP no length, and a receiver judges a lent region through fstat before it maps it. It closes the lent-window issue alone; issues/a-received-handle-has-no-knowable-type.md stays open, narrowed by #867. Stage 2 is #878. The exit read pipe_copy_bytes "per MB downloaded", a unit the row does not print: the row is per byte downloaded, 2.00 today. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy
|
Review of #863 at Net lines ( Checked
BLOCKER
NOTE
SEND BACK |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy
…for send room, and the pipe page has an owner Review #863 (comment). The owner ruled zero copy out unless the design showed a useful benefit; the track said he dropped it. It now says the design dropped it under his condition, on its own reading of a 4% gain (BLOCKER 1). Stage 3 has its own exit (BLOCKER 5): a loom model of one stream's reader and writer asleep at once, and the T14 boot of round 3's download. That boot reads pipe_copy_bytes, which no landing tree has: round 3's b245592 added it, and the boot carries the scout's measurement-only commits onto stage 3's head (BLOCKER 4). The track's own exit reads stage 3's and the pipe ABI gone, `pipe_pair` in toyos::net included, so a datagram socket cannot keep its pipes under another name (NOTE 3). Stage 3 rings a writer waiting for send room on a pipe of the stream's own, the amendment PR #863's body puts to the orchestrator: one netstack-to-client doorbell for both wants lets a reader drain the writer's wake, and a TcpStream split across two threads hangs (BLOCKER 6). A scratch loom model over #878's stream ends at 7ea0ff9 deadlocks with one doorbell and passes with two. The scout's S bounds the lock's share the design removes, not the receive path's saving: the pipe syscalls' work outside the hold goes too. The pipe-page defect has an owner, the orchestrator who stages the track that pays its pages (BLOCKER 2), and an exit two searches read (BLOCKER 3): the used-memory guest test it named reads a machine-wide figure every demand fault and heap growth moves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy
… run -- --ci host, EXIT=0 Run 2026-10-12 10:08:49Z to 10:12:22Z, on wt/toyos-netring with main at 528b9a6c2 merged.
|
Review of #863 at Net lines ( Earlier BLOCKERs
Earlier NOTEs
The new citations, each read against its source (fetched to scratch with the project
Checked
BLOCKER
NOTE
SEND BACK |
|
Rulings on round 6's BLOCKERs (#863 (comment)).
NOTE: the body's sentence "ToyOS does not follow Linux and references it nowhere in the tree" goes. The body says the record cites no other implementation as authority, under #902. If the body would pass 65,536 characters, the mTCP and TAS text comes out first. |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy
…datagram is never Ok, and the track cites no implementation Round 6's review of #863 and the orchestrator's rulings on it (issuecomment-6135709402): - Stage 3: a spawn's stdio slots, CommandExt::inherit_handle into any slot, CommandExt::endow, Command::provide and toyos::ipc's send_handles each ask toyos::net before anything moves and answer its refusal, so no refused move leaves an entry that outlives its number. The bare dup, SYS_HANDLE_SEND and SYS_SPAWN stay unchecked. The exit gains a case per path; endow's and provide's red when the handle moves and its entry stays. - Stage 4: toyos::net's send_to refuses what a sender can judge (a payload past a bound toyos-abi declares and [udp] reads, port 0, an unspecified address) and writes no record; what only netstack can judge is the socket's pending error on its next call (POSIX XSH 2.10.10). The exit gains a case per refusal, red when it answers Ok. - The stream-move issue names every path and drops "unrefused". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy
… run -- --ci host, EXIT=0
|
Rulings on round 7's two open points. (1) |
|
Review of #863 at Net lines ( Earlier BLOCKERs
Earlier NOTE
Checked
BLOCKER
NOTE
SEND BACK |
…agram socket's two waits under loom, measurement only toyos-transport as at 9b860ac69, unchanged since 7ea0ff9d6. `sh run.sh <dir> <rev>` with the model's lib/ copied from that tree under src/; each case has its own exit.
It brings #874, #896 and #887; none moves a path the record cites. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy
…or wakes every sleeper, and a sender waits at a full queue Round 7's review of #863 (SEND BACK) and the orchestrator's ruling on send_handles. - send_handles takes OwnedHandle, which safe code makes only from a Pipe: a stream's handle reaches it only by breaking OwnedHandle::from_raw's contract, which stage 3 extends to name it. The check, its exit case and its lookup go from the track and the stream-move issue. - netstack stores a datagram socket's pending error, then exchanges both client wants and rings each that was set; a look after the arm reads pending beside its ring. Stage 4's exit gains a loom model that reds when the look reads pending before the arm, when the wants go before the store, and when only rx_data is rung. - A full [udp] queue holds the send ring; a send_to waits for room on the socket's own bell, as POSIX's sendto blocks, and stage 4's exit gains a model that reds at a refusal at the full queue and at room rung on the connection. - limits::MAX_PAYLOAD goes: every reader takes toyos-abi's bound. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy
… run -- --ci host, EXIT=0 The worktree masked as <tree>, the home directory as ~ and the user's temporary directory as <tmp>.
|
Rulings on round 8's Unsure items: the node crate takes |
|
Review of #863 at Net lines ( Earlier BLOCKERs
Earlier NOTE
Checked
BLOCKER
NOTE
Once BLOCKER 1 closes, I read the design as ready for stages 3 and 4 to build. Stage 3's exit, and every part of stage 4's except the room model's controls, now name a negative control for each claim the design makes. SEND BACK |
… nonblocking send at a full ring answers WouldBlock Round 8's BLOCKER on #863: the room model's datagram left netstack's queue before netstack's first look, so no interleaving had netstack hold a record at a full queue, and a netstack that waited for the client's next ring after the leave stayed green. Stage 4's exit now has the leave as its own step, interleaved with netstack's looks, and a control that waits for the client's ring after the leave without looking again; and names `want-relaxed`, which the record claimed red. Round 8's NOTE: a case for a nonblocking `send_to` over a full send ring, red when `set_nonblocking` sets nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy
… run -- --ci host, EXIT=0; the datagram room model holding a record at a full queue, measurement only host.log: the worktree masked as <tree>, the home directory as ~ and the user's temporary directory as <tmp>. datagram-waits/: toyos-transport as at 67a1d5963, unchanged since 7ea0ff9d6. `sh run.sh <dir> <rev>` with the model's lib/ copied from that tree under src/; each case has its own exit. Round 8: the card is its own thread, so netstack can hold a record at a full queue, and netstack_waits_for_a_ring_after_the_leave is the new control.
|
Review of #863 at
Net lines ( Earlier BLOCKER
Earlier NOTEs
Checked
BLOCKER
NOTE
The design is ready for stage 3 to build: nothing open touches streams. Stage 4 is ready once BLOCKER 1 closes. That takes one sentence in the track and one exit case, with no change to the design. SEND BACK |
…e queue or is dropped [udp]'s queue empties by two paths: a datagram leaves when Udp::serve offers it and the caller takes it, and one whose hop Udp::fail names is dropped. The track named only the first, and no exit case could fail if stage 4 hooked the second look to the transmit path alone. Stage 4's exit gains a node case over fail: a full queue to a silent hop, a sender asleep over a full send ring, woken when [ip] gives the hop up. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy
… run -- --ci host, EXIT=0
|
Review of #863 at
Net lines ( Earlier BLOCKER
Earlier NOTE
Checked
BLOCKER NOTE The track can land as the record its stages build from. Stages 3 and 4 each have an exit that a build or test can fail, and that exit names its controls. Nothing in the design is left open. LAND |
#893 cut acpi_power_button and acpi_press_across_a_lock_wait and replaced the latter's held arm with acpi_press_is_one_sci's press arm; this branch had cut acpi_press_across_a_lock_wait too. The conflicts take main's side in tests/common/qemu.rs (both sides dropped acpi_tables; main also drops second_slot), tests/guest/rust-tests/src/bin/acpi_mediated.rs (the press arm; this branch's RegisterCached rename auto-merges) and tests/toyos.rs's RUST_SKIP comment. In tests/common/power.rs acpi_power_button goes with main, which leaves power::acpi_tables_loaded with no caller, since this branch's acpi_tables_on_metal reads each bench's blocks itself: it goes, with ACPI_TABLE, ACPI_S5_HANDED and slp_typ_a. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy
What this lands
A design round, built on nothing: three issue files and this body, which is the design as accepted and amended, and its roast. No source, test or manifest changes.
issues/netstack-moves-every-byte-through-two-kernel-pipes.md(kind: track): the pipe copies and the lock as a present-state weakness, its evidence, the owner's constraint, the stages, 0 to 2 done, the exits of stages 3 and 4, and the track's exit.issues/a-pipe-is-a-two-mebibyte-page-for-a-mapping-nothing-makes.md(kind: defect): the one thing the pipe section below found to fix, with its owner and an exit two searches read. Not fixed here.issues/a-stream-cannot-move-between-processes.md(kind: defect): the regression amendment E brings, a stream's handle that reads and writes the stream in another process today and after stage 3 is refused loudly by every std path that would move it there, and bysend_handles's type, with its owner and E's exit.Amendments A to E, and the rulings on rounds 4 to 7 and on the second triage, were each accepted by the orchestrator, in the comments linked where each is set out. Round 7's answers, the datagram socket's two waits, were accepted (rulings 13 and 14); rounds 8's and 9's, the hold and the drop, are this round's.
The owner's rulings, as the orchestrator relayed them
He also asked whether the pipe design itself has a problem: its own section, from the code.
What was ruled since, and where
an_accept_answers_the_option_of_a_stream_its_own_pass_let_go;endmakes the send direction ready to every look;writeafter a fatalendanswers its error and copies nothing;len,portandaddr;nodelay;File::from_raw_fdand in a spawn's stdio, with a case in stage 3's exit that reds when either takes it silently;endowandprovide(send_handlesby ruling 12), refuses a stream's handle before anything moves, with a case per path in stage 3's exit, and no refused move leaves atoyos::netentry; the baredup,SYS_HANDLE_SENDandSYS_SPAWNstay unchecked;Okfor a datagram refused: what the sender can judge is refused atsend_toagainst one bound intoyos-abi, and what only netstack can judge is the socket's pending error on its next call, with a case per refusal in stage 4's exit (the orchestrator's narrowing of round 5's ruling 2);toyos-abias a dependency for the one bound;send_handlestakesOwnedHandle, so a stream's handle reaches it only by breakingOwnedHandle::from_raw's contract: the type refuses it, no exit case tests it, and its check goes, a lookup under a lock per handle moved being no nil cost;toyos-abifor the bound, beside [udp];tx_room, as POSIX'ssendtobacks.The starting facts, each read again at 06788fc
Each fact below was read at 06788fc, again at 97f07c3 after #861 (ab655a1), and holds; later merges move among the paths they cite only netstack's lease, resolver and lookup serving (b805e61).
Ring::write_atcopies into aVec<u8>that grows by doubling (system/netstack/shard/tcp/src/ring.rs). Round 3 times it ascy_ring_write.PIPESlock with preemption off. True:Node::passhandstcp_recv's bytes toPipe::write_nonblock(system/netstack/src/pipes.rs);pipe::try_writeandtry_readcopy insidewith_pipes_mut(kernel/src/pipe.rs), andLock::lockdisables preemption (kernel/src/sync.rs). Sent bytes go into a pipe, out into netstack'sCHUNKbuffer (streams.rs), into the send ring bytcp_send, and into the frame, copied again into the card's slot.issues/toyos-has-its-own-network-stack.md. The nearest isissues/the-global-pipe-lock-spans-a-user-copy.md, which records the lock and not the copies. The new track records them.lib/blockring/transporthas nothing block-specific in it. True: entry queues with clamped peer cursors, an arena of slot runs and a tag table, overWords an adapter hands in,forbid(unsafe_code), loom-modelled. Its byte ring, the one addition below, stage 2 (toyos-transport: a byte stream between two ends that do not trust each other, and its wake handshake (netring stage 2) #878) landed asstream.rs.system/netstack/i219/): one receive ring, no queue, split or offload register is written. What the part itself has is not read here; the zero-copy section does not rest on it.The session-region design
A TCP stream is one shared region and one IPC connection. The client makes both: the region with
SYS_SHM_CREATE, the connection by connecting tonetstack. The connect request carries the region's handle. The connection carries that request and its answer, then nothing but one-byte doorbells each way and its own hang-up (amendment E), and netstack rings a writer waiting for send room on a bell of the stream's own (amendment A). A stream belongs to the process that connected or accepted it (amendment E).The region
8 MiB, four 2 MiB pages, physically contiguous as every
SYS_SHM_CREATEregion is today:The header's words: the receive ring's
tail(netstack) andhead(client); the send ring'stail(client) andhead(netstack: what the peer acknowledged); four want words, one per wait, each set by exchange by the side that will sleep and cleared by exchange by the side that rings (rx_dataandtx_roomset by the client,rx_roomandtx_dataset by netstack); the receive ring'send(netstack, written once, after the last tail:0open, then FIN, reset, timed out, unreachable, prohibited); the send ring'send(client: shut down for writing); andnodelay(client).The rings are the TCP buffers, not a copy of them, for a passive child from its accept on (amendment C). [tcp]'s
Ringtoday stores into aVec; it becomes a ring over the region's words through the same adaptertoyos-transportuses, sowrite_atplaces an out-of-order segment's payload at its offset in the receive ring andcommitis the published tail. The window is the receive ring's room. On the send sidetcp_send's copy goes: the bytes the client published are what [tcp] segments from, and the head moves only as the peer acknowledges. One copy of every received byte is gone (heap ring to pipe), and two of every sent byte (pipe to chunk, chunk to send ring).A passive child's bytes wait in netstack until its accept (amendment C). Its SYN-ACK offers a window before any region exists, so what arrives first stays in [tcp]'s heap storage: at most 65,535 bytes a child, its first window, and
LISTEN_READY, 128, children a listener. The accept copies them into the region's receive ring at their offsets, with any out-of-order range and a FIN, frees that storage, and only then answers. An active open's region exists before its SYN.The addition to
toyos-transport(stage 2, #878, landed): a single-producer byte ring,stream.rs, besidequeue.rs.Release, load the peer's withAcquireonly when asked for room or what is ready, and hand the adapterSpans to copy through.u32under a mask would be right only for a power of two, and the send ring is not one. The peer's cursor is believed only as a position at most the ring away from this end's: a word at or past the lap names no position, and either isViolation::TailPastDepth/HeadPastTail, asqueue.rsnames them.Acquire), then reads the peer's cursor again, and sleeps only if that look finds nothing. The waker publishes its cursor, then exchanges the want for 0 (Release), and rings only if it was set. Either the sleeper's exchange reads the waker's clear and sees the cursor stored before it, or the clear follows and finds the want set. A sleeper drains its doorbell before it looks, never between arming and sleeping.Nothing in it is network-specific: a third byte stream between two processes would take it as it is.
Why the kernel changes at all
The owner's question. The data path is userland's: the stream's bytes, cursors, wants and end words live in a region two processes share, and the kernel neither copies nor reads one of them. The kernel's part is what it already does: make and map the region (
SYS_SHM_CREATE,SYS_SHM_MAP), and carry the doorbells and the hang-up on the connection and on the bell, both pipes.Most of what changes is not the kernel's ABI. The netstack message set below is a userland protocol between netstack and
toyos::net, as any server's is.The kernel's syscall ABI changes twice, and the first has landed:
SYS_FSTATanswersDeviceMemoryfor a region that is not memory the kernel allocated, where before it answered every regionSharedMemory(kernel/src/object/ops.rs). Only the kernel knows what memory backs a region; without the answer, a client could lend a device's BAR as its stream, and netstack would write received bytes into device memory.SYS_CONNECTION_JOIN, whose only caller outside the abuse tests is std'smake_socket_fd.The protocol, and what the kernel answers about a lent region
MsgType::TcpConnect { addr, port, timeout_ms }with one handle, the region; answeredTcpConnectResponse { local_port }with one handle, the stream's bell (amendment A), or an error, on the same connection, which is then the stream's.MsgType::TcpListen { addr, port, nodelay }on a connection that is then the listener's, on which netstack writes one wake for each child that waits, aslisteners.rs'sWakeis written today. An accept takes a wake first (ruling 5): a nonblocking one with none answersWouldBlock, and a blocking one waits on the connection for one, so no region or connection is made until a child is there. POSIX'sacceptgives the same two answers to an empty queue: it "shall block until a connection is present", or, withO_NONBLOCKset, "shall fail and set errno to [EAGAIN] or [EWOULDBLOCK]". That nothing is made first is the design's own: an accept loop that callsacceptuntilWouldBlockwould make 8 MiB of contiguous memory per event to hear it. A process makes a connection to netstack only by connecting, so the accept is thenMsgType::TcpAccept { listener }on a new connection with a fresh region, answered once the child's bytes are in it (amendment C) with the stream's bell andTcpAcceptResponse { remote_addr, remote_port, local_port, nodelay }(ruling 6): the peer, and thenodelaythe child inherited from its listener, which netstack has also written into the region's header before it answers. POSIX leaves "implementation-defined which socket options, if any, on the accepted socket will have a default value determined by a value previously customized by setsockopt() on socket"; the child takesnodelay, as it does today (TcpAcceptPipedResponse'soptions), since a program that set it on its listener set it for the streams it accepts. An accept that took a wake and failed before netstack had its request rings the listener's connection, and netstack writes the wake again, which isissues/an-accept-that-never-reaches-netstack-strands-its-listener.md's exit. The listener id stays the numberissues/netstack-socket-ids-are-ambient.mdrecords.MsgType::UdpBind { addr, port }with a 2 MiB region whose two rings carry records, a record beinglen,port,addrand the payload padded to a word, and whose header holds apendingword beside the rings' words: a send is a record and at most a doorbell, where today it is a pipe write and a request on a new connection. netstack drops and counts a received datagram its client's ring has no room for, which is UDP's answer, where today a datagram that does not fit ends the socket. netstack judges each record's words (ruling 4) as the hostile-client section says.Okfor a datagram refused (ruling 9), as every refusal issend_to's error today (udp_send_to,system/netstack/src/serve.rs).send_tofirst takes an errorpendingholds. It then refuses, writing no record, what a sender can judge: a payload pasttoyos_abi::net::UDP_MAX_PAYLOAD, as POSIX'ssendtofails "[EMSGSIZE] The message is too large to be sent all at once, as the socket requires",EMSGSIZEin libc andInvalidInputin std, which has no kind for it; port 0; an unspecified address, eachInvalidInputas today. The bound is one declaration:toyos-abigains anetmodule for it, the Ethernet MTU less the IPv4 and UDP headers, and [udp]'slimits::MAX_PAYLOADgoes (ruling 11). [udp] (system/netstack/udp/src/lib.rs, asserting at compile time that such a payload fits its MTU, and the pin in itstests/send.rs), the node'sname.rsandresolve.rs, the node takingtoyos-abitoo, andserve.rsread it. What only netstack can judge, [udp]'s refusal of a connected socket's other destination, of a route, of a broadcast without permission or of a source address, and a pending ICMP error, the socket's nextsend_to,recv_fromortake_erroranswers and clears: POSIX's pending error "is returned in response to a subsequent send(), recv(), or getsockopt() operation on the socket, and the pending error is then cleared" (XSH 2.10.10).rx_datarung on the connection andtx_roomon a bell of the socket's own that netstack makes at the bind and alone writes (amendment A). Two things are the datagram's own:pendingwakes as a fatalenddoes. netstack stores the error inpendingif it is 0 (Release), then exchanges both client wants for 0 and rings each that was set: a refusal makes both directions ready, as POSIX'sselectcounts a descriptor ready "whether or not the function would transfer data successfully". A refusal while one is pending is counted, not stored. Every look, a call's after it drains and arms and a registration's, loadspending(Acquire) beside its ring; a call exchanges it for 0 and answers it when it is set, and sleeps only if neither has anything. A look that readpendingbefore the arm would miss a refusal stored between the two, whose exchange finds no want.limits::TX_DATAGRAMS, has room, and looks at a held socket again when a datagram leaves that queue or is dropped; [udp]'sTxQueueFullstays its own bound, which netstack no longer meets. Asend_towith no room for its record waits as a stream's writer does: it drains the bell, armstx_room, reads the head andpendingagain, and only then waits on the bell and any write timeout, as POSIX'ssendto"shall block until space is available or a timeout occurs". Nonblocking, it answersWouldBlock, POSIX's "[EAGAIN] or [EWOULDBLOCK]", once std'sUdpSocket::set_nonblocking, which answersOkand sets nothing today, sets it; stage 4's exit has that case, red whenset_nonblockingsets nothing (round 8's NOTE), and a node case overUdp::fail, red when only a leave looks again (round 9). Room rung on the connection would bring back amendment A's defect: arecv_fromtakes the byte, and the sender sleeps on with room.TcpShutdown,TcpSetOption,TcpClose,UdpSendTo,UdpRecvFrom,UdpCloseand the three*Pipedmessages go: shutdown andnodelayare header words, close is the hang-up, a datagram is a record.SYS_SHM_MAPof a handle that is no region isHandleError::WrongType, which ends the caller, andSharedMemory::adoptmaps what it is handed and believes the size it is told, so it is only for a region the kernel itself handed the process (sdk/toyos/src/shm.rs).SYS_FSTATresolves a handle of any kind and needs no right: only a region of memory the kernel allocated answersSharedMemory, and it answers the region's size. netstack takes a lent region throughSharedMemory::lent, which refuses by name a handle that is no region, a region that is not memory the kernel allocated, and one that is not exactly 8 MiB (2 MiB for a datagram socket), each before anything is mapped; netstack closes the connection on the refusal.SYS_HANDLE_RECVandSYS_SHM_MAPare unchanged. fileserver and diskserver take their lent regions throughSharedMemory::lentalready.issues/a-received-handle-has-no-knowable-type.mdstays open for the kindsfstatdoes not settle, none of which this design has netstack use.std,
poll, libc and mioThe session's client end is
toyos::net::Streaminsdk/toyos/src/net.rs, one implementation that std (sdk/std/sys/net/connection.rs), libc (sdk/libc/src/socket.rs) and the mio and socket2 forks call, as they all calltoyos::netnow.as_raw_fdanswers it, and the hang-up comes once no process holds a handle of it.toyos::netkeeps the process's ends by handle. std'stry_cloneand libc'sdupmake a handle of the same connection and enter it against the end they duplicate, asTcpStream::duplicateshares itsArcs today (sdk/std/sys/net/connection.rs); so doesBorrowedFd::try_clone_to_owned, whichOwnedFd::try_clonecalls, once stage 3 routes its ToyOS arm, the bareduptoday (therust/fork'sos/fd/owned.rs:159-165), throughtoyos::net. Stage 3 adds std'sFromRawFd,IntoRawFd,From<OwnedFd>andFrom<TcpStream> for OwnedFdforTcpStream, which therust/fork'sos/fdcompiles out fortoyostoday (mod.rs:16-17,owned.rs:378-418), so a program that turns a stream into its handle and back builds unchanged. Taken back, a handle the process holds an end for shares that end. A handle the process holds no end for, one moved in from another process or duplicated pasttoyos::netby the baredupsyscall, is no stream in it:toyos::netrefuses it by name and writes nothing on its connection, so two processes never share a region (amendment E), and std'sfrom_raw_fd, which cannot fail, panics with that refusal (issues/a-stream-cannot-move-between-processes.md). A handle given out stays entered against its end, andOwnedFd's ToyOSdrop, which closes by the raw syscall today (owned.rs:241-244), lets a handle go throughtoyos::netastry_clone_to_owned's arm enters one, so no entry outlives its handle's number. The end goes with the process's last handle of the stream.File::from_raw_fdtakes any handle (sdk/std/os/io.rs) and reads and writes it with the samesyscall::readandwriteasTcpStream(sdk/std/sys/fs.rs), andStdio::from(File)hands it to a child (sdk/std/sys/process.rs), so a stream's handle serves the stream through either. Over the region it is the doorbell connection. SoFile::from_raw_fd, the one way a raw handle becomes aFileon ToyOS (From<OwnedFd> for Fileis compiled out,owned.rs:358-359), askstoyos::netwhether the handle is entered against a stream end, and panics with the refusal if it is, asTcpStream::from_raw_fdpanics on a foreign handle. Every path std offers to move a raw handle into another process asks the same of each handle before anything moves, and answers the refusal: a spawn, for each handle it would put in a child's slot, byStdio::inherit()orCommandExt::inherit_handleinto any slot, and each it would move byCommandExt::endoworCommand::provide(sdk/std/sys/process.rs), with no child started and the handle still the caller's, so no refused move leaves an entry that outlives its handle's number.toyos::ipc'ssend_handles(sdk/toyos/src/ipc.rs) is refused by its type (ruling 12): it takesOwnedHandles, which safe code makes only from aPipe(sdk/toyos/src/lib.rs), so a stream's handle reaches it only throughunsafe fn OwnedHandle::from_raw, whose# Safety, naming only the close today, stage 3 extends to exclude a handletoyos::netanswers for. The baredup,SYS_HANDLE_SENDandSYS_SPAWNsyscalls stay unchecked. NoFilethen holds a stream's handle, andFile's drop, a raw close (sys/fs.rs), never lets one go pasttoyos::net.readcopies from the receive ring into the caller's buffer, a userland copy of the bytes the kernel copies today. Empty: it drains the connection's doorbell, setsrx_databy exchange, reads the tail again, and only then lets its end go and waits on the connection.writefirst reads the receive ring'sendwithAcquire: a fatal one, any but a FIN, is its error, and it copies nothing (ruling 3). RFC 9293 has a reset in a synchronized state flush "All segment queues" and delete the TCB (§3.10.7.4), as a user timeout does (§3.10.8), and answers a SEND that finds no TCB "error: connection does not exist" (§3.10.2); POSIX'ssendfails[ECONNRESET], "A connection was forcibly closed by a peer". Otherwise it copies into the send ring and rings iftx_datawas set; full, it drains the stream's bell, setstx_roomby exchange, reads the head again, and only then lets its end go and waits on the bell (amendment A). Each direction's end is behind its own mutex, held to drain, arm, look and copy and never across a wait, so any number of threads, through any of the process's handles of the stream, may wait on one direction, each counted on its one end, and a call under a timeout waits on nothing but its doorbell and its deadline (amendment B).WouldBlock, andWouldBlockis answered only after the want is set and the cursor read again, so the wake that follows reaches the doorbell that direction's sleeper waits on. mio waits for an event after it registers (amendment D), soregisterandreregistertake each named direction's end, arm its want, look, let the end go, and ring netstack if the direction is ready, and netstack rings back by amendment B's rule; a registration drains nothing and counts nothing (amendment D). The mio fork registers the connection for itsREADABLEinterest and the bell for itsWRITABLEinterest, each watched for reading. libc'spollof a socket asks the rings first, so a level-triggeredpollon a ring that holds bytes answers at once; for a direction that is not ready it takes that direction's end as any waiter does, counts itself in, drains, arms and looks again, lets the end go, and waits on the connection forPOLLINand on the bell forPOLLOUT. A doorbell is drained only under its direction's end, before the drainer arms and looks, and a thread that took a doorbell while another waits on its direction passes the take on (amendment B). The mio fork counts nothing: a source is read toWouldBlock, which leaves the want armed, so the next publish rings its watch. A program that polls the raw handle withtoyos::pollerand does not read toWouldBlocksees edges, not levels: the module header says so.endword set, read withAcquireafter the tail: a FIN isOk(0), a resetConnectionReset, a timeoutTimedOut, an ICMP failure its own word. What rests on pipe ordering today, that the send pipe closes before the receive pipe so a client probing an empty write learns reset from FIN (TcpStream::ended,Node::pass), is deleted, and a failure says which failure it was (issues/a-streams-failure-reaches-its-client-as-a-reset-whatever-it-was.md).shutdown(Write)stores the send ring'sendafter its last tail and rings; netstack sends the FIN once it has segmented to that tail. A fatalendmakes the send direction ready to every look (ruling 2), awrite's, a registration's, apoll's and netstack's ring back. Awritethen answers the error without waiting, and POSIX'sselectcounts a descriptor "ready for writing when a call to an output function with O_NONBLOCK clear would not block, whether or not the function would transfer data successfully". A writer or aWRITABLEregistration that arms after a reset over a full send ring does not sleep. A FIN is not fatal, so half-close keeps the send direction.OTHER_END_GONE. Today the node cannot learn it once the peer's FIN has been read through (theOWNERLESS_LIFEparagraph instreams.rs). The peer-side bounds on an ownerless stream stay [tcp]'s; the half that exists only because a pipe cannot say its reader left goes.What it deletes
src/pipes.rs; in the node,ToClient,FromClient,ReadRefusal,WriteRefusal,Pipes,PipeEnd,pipe_gone,pipe_broken, the end-ordering of the two ends, theCHUNKread, thedrainedbookkeeping that therx_room/tx_datawants replace, and the listener'sWakepipe; inserve.rs,data_pipes, the datagram socket's pipes and its request per datagram, and two watches a stream (issues/netstack-spends-two-poller-slots-per-piped-connection.md,issues/netstack-watches-both-pipes-of-every-connection-on-every-pass.md). [tcp]'sVecstorage underRingstays, for a passive child before its accept alone (amendment C).toyos::net: the*Pipedmessages,DATA_HANDLES,NOTIFY_HANDLESand the pipe ends in every answer type.make_socket_fd,TcpStream::ended's empty-write probe,UdpSocket's two pipe handles.SYS_CONNECTION_JOINandConnectionEnd::joined.PLACE_BYTESfalls from 12 MiB (two 2 MiB pipes and two 4 MiB buffers) to the 8 MiB region, plus its doorbells' pipe pages, 14 MiB, untilissues/a-pipe-is-a-two-mebibyte-page-for-a-mapping-nothing-makes.mdcloses (amendment A).What a hostile client can do to a region it shares with netstack
netstack reads five kinds of word the client writes. A datagram record's
len,portandaddrare copied out of the region once and judged on the copy (ruling 4): alenpast the bytes the record holds is aViolationthat ends that socket alone, asudp_send_torefuses one today (system/netstack/src/serve.rs). That is the one bound netstack's serving code judges: the payload, port and address go to [udp], whosesubmitstill refuses a payload past the bound (Counter::ExceedsMtu), port 0 and an address it cannot send to, each by its counter (system/netstack/udp/src/lib.rs), since a client that writes records pasttoyos::netis not held by its checks. Each refusal is the socket's pending error, and the socket serves on. The send ring's tail and the receive ring's head are each believed only as a position at most the ring away, and otherwise are aViolationthat resets that stream alone. The want,endandnodelaywords are each a bit whose worst value costs netstack one doorbell or one pass (any want but 0 is set);pending's worst hides only the socket's own refusals from it. The send ring's bytes netstack copies into a frame before it checksums them, so a client that rewrites a byte after publishing it, or between a send and its retransmission, corrupts only its own stream. netstack never reads back a byte it wrote into the receive ring, so out-of-order payload the client can see and overwrite before it is committed is the client's own. A doorbell storm is bounded by the pass: doorbells coalesce, and a stream rung with nothing new published costs one look, and a ring back for each want the client set on a direction that is ready (amendment B). A client that unmaps or closes its handle takes nothing from netstack, which holds its own handle and mapping until it lets the stream go.Amendments A and B, accepted
Each is set out in full, with the alternatives it weighed and refused and its model, in #863 (comment), as this body carried it through round 4.
Amendment A: a writer waiting for send room has a doorbell of its own. Accepted by the orchestrator: #863 (comment).
rx_dataandtx_roomon the connection's one netstack→client pipe, and a pipe's byte goes to one reader: the reader could take the byte rung fortx_room, and the writer sleep on with room in its send ring.rx_datarings the connection;tx_roomrings the stream's bell, a pipe netstack makes at connect and accept and alone writes (amendment E). Writing the receive ring'send, netstack exchanges both client wants for 0 and rings each that was set, so a writer asleep wakes and reads why (ruling 2). The bell's write end gone reads as the stream gone. netstack's own wants share the connection's client→netstack pipe, its one thread their only sleeper. A reader,READABLEandPOLLINread only the connection; a writer,WRITABLEandPOLLOUTonly the bell.PIPESentry more per stream, and the bell's page, allocated on its first ring: once the stream's writer finds its send ring full, or once it is registered forWRITABLE(amendment D). Untilissues/a-pipe-is-a-two-mebibyte-page-for-a-mapping-nothing-makes.mdcloses,PLACE_BYTESis 14 MiB where it is 12 today.want-relaxed, EXIT=101 (the model, its log).Amendment B: within a direction no end is held across a wait, and a take is passed on. Accepted by the orchestrator: #863 (comment).
pollof the same direction watched it; the kernel answers that watch by the pipe's level at its look, and arms it again if the pipe is empty (kernel/src/inbox/polls.rs,lookinmod.rs), so a level-triggeredpollslept over a readable socket.pollwhose interest names the direction, counts itself in under the end before it drains, arms and looks, and out when it returns. A thread that takes the direction's doorbell, by a wait or a drain, and leaves the end with another waiter counted sets the want by exchange again and looks, and rings netstack if the direction is ready (bytes or theendin the receive ring, room or a fatalendfor the send ring). netstack, rung by a client, exchanges each client want whose direction is ready for 0 and rings the one that was set; stage 3 adds this exchange to each ofstream.rs's ends.pollof one direction, the blocking call leaving a byte, theendor room, at most five preemptions as the kernel's own models bound theirs. The rule as written, B less the pass and B underwant-relaxedare each red in all three cases, EXIT=101; B passes all three, EXIT=0 (the model, its log).Amendments C, D and E, accepted
Accepted by the orchestrator: #863 (comment). Each is set out in full, with the alternatives it weighed and refused, in the comment that proposed it: #863 (comment). They answer the triage's three BLOCKERs (#863 (comment)), each read again at e7e1179; each stands on a published specification, per #885's rule, or on its own terms (ruling 10).
Amendment C: a passive child's bytes wait in netstack until its accept.
stack.rs:418), and the ACK that ends the handshake runs the segment's text as ESTABLISHED (stack.rs:852-864). As first written, the design made the region only at the accept and deleted [tcp]'s heap storage, so that text had nowhere to go.listeners.rs'sa_listener_answers_a_syn_and_wakes_its_owner_when_the_handshake_ends(listeners.rs:455) is green at e7e1179 and red, EXIT=101, with the child's bytes dropped at its accept, as the design first written dropped them (pre-accept.log). Ruling 1's case, a child holding text in order, text past a hole and its FIN behind that before its accept, the hole filled after it, ran as a scratch test against today's node tests at 04ff95b: green, EXIT=0, and red, EXIT=101, under each control dropping one of the three at the accept. Dropping the in-order bytes reds the first case too (pre-accept.log).Amendment D: a registration arms its wants.
stream.rs:73-74), as tokio'sconnect_mioand first I/O do (tokiostream.rs:155,registration.rs:169). As first written no registration armed a want, soTcpStream::connectand sshserver's first I/O would wait forever.WRITABLE.end, room, and a reset over a full send ring (ruling 2), each made ready before the registration or racing it (registered-ready.log). The design as first written is red in all eight cases, EXIT=101; D passes all eight, EXIT=0; D less the arm and D less the ring are each red in all eight;want-relaxedreds D in the four racing cases;head-only, whose looks read only the send ring's head as the record stood at 148d265, reds D in the two reset cases alone. Each of the 30 reds is loom'sdeadlock, both threads blocked.Amendment E: the connection carries one request, then only doorbells.
MsgType::Regiontravelled on a connection where every byte after the answer is a doorbell: netstack would drain the request's 8-byteIpcHeader(ipc.rs:93-99) as rings, and the answer would land in the pipe the stream's readers drain.MsgType::Regiongoes, andtoyos::netrefuses by name a handle the process holds no end for (the handle bullet).issues/a-stream-cannot-move-between-processes.md.Regionon the connection, and of its boundary (round 5's BLOCKER 3), and the tests of rulings 7 and 8.Rounds 7 and 8: the datagram socket's two waits
pendingwas read by no look, so arecv_fromcould sleep on an error; a sender at a full ring had no doorbell.recv_fromis spawned only where room rings the connection, its one shared step with the sender.stream.rs(the model, its log), eleven cases and four underwant-relaxed. Unbounded, arecv_fromand asend_toasleep when the refusal lands answer it, EXIT=0, as does arecv_fromrung whatever its want, underwant-relaxedtoo; red, EXIT=101,deadlock:pendingread before the arm, under the handshake and the ring outside it; the wants exchanged before the store; onlyrx_datarung; both handshake cases underwant-relaxed. At five preemptions, every record leaves in order, none refused, EXIT=0, in 542 s; red, EXIT=101: room rung on the connection (deadlock); a refusal at the full queue (its assertion); round 8's control, netstack waiting after the leave for the client's next ring (deadlock, all three threads);want-relaxed(deadlock).want-relaxed.True zero copy: dropped
Received bytes straight from the card into the application's region would remove
cy_ring_write, the one copy the session design keeps, and nothing else:readstill copies from the region into the caller's buffer, becausestd::io::Readhands in the buffer. Round 3 measured that copy at 0.225 ms per MB of 5.447 ms of C0 at K = 32, 4.1%, and the scout read the same share (0.2185 of 5.310). To get it:pcidev::dma_map), which spendsMAX_GRANT_TOTAL, 32 MiB a claim (kernel/src/pcidev/mod.rs): four 8 MiB sessions.For a 4% gain the design would trade [tcp] parsing what the client cannot touch for per-flow hardware state, per-client IOMMU windows and a reassembly that copies when loss is common. Dropped under the owner's condition, on this reading, the design's and not his; the layout carries no field for it.
Does the pipe design itself have a problem?
What pipes are for once netstack leaves them. Every IPC connection, two pipes (
connect_through,kernel/src/syscall/ipc.rs) carrying frames of at most 8 KiB; stdio and pipelines between programs (sdk/std/sys/pipe.rs), the one user whose volume is unbounded; one-byte wakes (toyos::wake, the mio fork's waker, the supervisor's pipes) and logkeeper's stream to each log reader. Bulk data elsewhere already moves through regions: a file server's window (toyos::fs::WINDOW_BYTES), a block session (lib/blockring), sound and the compositor's buffers.Are two copies inherent? For a kernel-buffered byte stream whose reader and writer each hand in their own buffer, yes: the kernel holds the bytes between the two calls. The defect was not the pipe's: it was netstack keeping every byte in its own TCP ring and in the pipe, two buffers of one stream, which is the third copy and the 12 MiB a place. The session design makes the ring the buffer.
What
SYS_PIPE_MAPis for. Nothing, today. Its last caller, netd, mapped its clients' rings only to read the closed flags, and stopped in a237346 when those were found forgeable; since then only guest tests call it. It is the reason a pipe's ring is a whole 2 MiB page, the only page this kernel maps, and whypublish_endsstill writes flags into it: filed asissues/a-pipe-is-a-two-mebibyte-page-for-a-mapping-nothing-makes.md, not fixed here.Is the global
PIPESlock a flaw beyond its issue?issues/the-global-pipe-lock-spans-a-user-copy.mdalready names the whole of it: one table lock is also every ring's serialization, so a copy of a length the caller chooses, and a first write's allocation and 2 MiB zeroing, run under the lock every connection, readiness poll, clone and close in the machine takes. Its fix direction, a per-pipe lock under the table lock and the allocation out from under both, answers it, so nothing new is filed. Round 3 measured it, 0.281 ms held and 0.047 ms spun per MB, and the scout found the two copies 73% of the hold.What it buys, measured and not guessed
Round 3's rows. Round 3's B at K = 32, medians of the six runs per MB downloaded, from the judge logs b 1a b 2a b 3a and their halves posted on #861:
pipe_lock_hold_cyclespipe_lock_spin_cyclescy_ring_writecy_pipe_wpipe_copy_bytesThe lock and its copies are 0.328 ms of 5.447, 6.0%. These rows cannot say how much of the hold is the two copies and how much is everything else the lock covers, and so which part the session design removes and which part the per-pipe lock of the lock's own issue would.
The scout (stage 0). One T14 boot of
internet_download, round 3's B with three kernel counters more and every run told K = 32:pipe_lock_takes, and the stamp ticks oftry_read's andtry_write's copy alone (pipe_read_copy_cyclesR,pipe_write_copy_cyclesW), each timed around thewrite_run/read_runthe ring hands its run to. Patch: the comment; branchwt/toyos-netring-scout2f9e090, measurement only, never landing.The rule, stated before the boot. What the session design removes of the pipe lock on the receive path is S = H − R + P per MB (H the hold, P the spin): the write copy and every non-copy hold and spin, while R stays as the client's own copy out of the region. S ≥ 0.10 ms per MB (about 2% of round 3's C0): the build may claim a CPU saving, and its own T14 boot is judged against the scout's rows. S < 0.10: the design is built on the owner's ruling for what pipes cannot say and the 4 MiB a place, and claims no saving. H − R − W > W: most of the hold is not a copy, and the per-pipe lock takes most of the gain without this design, so that issue goes first. The instrument is sound only if R + W ≤ H in every run and
pipe_copy_bytesis 2.00.The reading (judged; the judge,
cargo test --test toyos-build -- --metal --metal-readback <scratchpad>/netring/scout internet_downloadon 2f9e090, EXIT=0, its log whole and masked in part 1 and part 2; the flash, toyos-metal's lines and the boot). toyos-metal exited 0, and all six runs read the whole 170,439,044-byte file with the right sha256. R + W ≤ H in all six;pipe_copy_bytes2.0028 in all six. Medians per MB: R 0.1197 ms, W 0.0806, H 0.2728, P 0.0397, 185 takes, C0 5.310 ms, 785.95 Mb/s.PIPES, and the takes, 185 per MB against about 20 data writes, are mostly not the stream's copies. So the lock's share it removes lies between W = 0.081 and S = 0.192 ms per MB. netstack's pipe-write syscalls and the client's read syscalls go too, and their entry, lookup and return run outside the hold, in neither number; the doorbells' syscalls come in. Stage 3's boot reads the whole, which may be more than S, and its pull request claims no more than that boot reads.The roast
Cost. By the files it rewrites, not a measurement: netstack's
serve.rs,client.rsand the node'sstreams.rs,listeners.rsanddatagram.rswith their tests (2,908 lines of tests innode/tests/streams.rs,listeners.rsanddatagram.rsbywc -l, most asserting pipe behaviour);toyos::net; std'sconnection.rs; libc'ssocket.rs; [tcp]'sRingover the region. The byte ring and its loom model intoyos-transportare paid: stage 2 (#878) landed them. Production should shrink:pipes.rs, the end-ordering, the per-datagram request,SYS_CONNECTION_JOINand the doubled buffers go, but for a passive child before its accept (amendment C); the client end is new.ABI churn. In the kernel:
SYS_FSTATanswersDeviceMemory, a newFileType, which every reader of a region's file type sees (stage 1, landed), andSYS_CONNECTION_JOINgoes (stage 4);SYS_HANDLE_RECVandSYS_SHM_MAPkeep their shape. In userland: the netstack message set is replaced. Three forks move in stage 3's pull request: mio (itstoyos_stream.rsandtoyos_listener.rs) and socket2 (src/sys/toyos.rs), each a commit on itstoyosbranch and a lockfile bump, andrust/, a commit on its branch and a gitlink bump: itslibrary/std/src/os/fd/owned.rsholds the ToyOS arms ofOwnedFd's drop andBorrowedFd::try_clone_to_ownedthat stage 3 routes throughtoyos::net, and itsos/fdcompiles out fortoyosthe conversions stage 3 adds. In stage 4toyos-abigains the datagram payload bound, and [udp] and the node a dependency ontoyos-abi.Risk.
toyos-transport's discipline and loom model carry over to the byte ring, and itsCONTROLSfeatures (publish-relaxed,no-clamp, and toyos-transport: a byte stream between two ends that do not trust each other, and its wake handshake (netring stage 2) #878'shead-relaxedandwant-relaxed) give the negative controls.PIPES, one byte per sleep rather than per batch, and a stream holds three pipes where it held two (amendment A).READABLEandPOLLIN, and the bell forWRITABLEandPOLLOUT. So two limits serve half the sockets watched both ways that they serve today. libc'spolltakesnfdsup toPoller::MAX_HANDLES, 256 (sdk/libc/src/posix_io.rs). A mio ring holdsMAX_PENDING_WATCHES, 1024 (kernel/src/inbox/mod.rs). A socket watched one way costs what it does today.pending.limits::TX_DATAGRAMS, 16, isTxQueueFull, answeredResourceExhaustedbytoyos::netandErrorKind::Otherby std (system/netstack/node/src/datagram.rs,sdk/std/sys/net/connection.rs), and the caller keeps its datagram. After stage 4 a blockingsend_tosleeps until netstack takes its record, which waits on [udp]'s queue: a datagram there leaves once its next hop answers, or is dropped once [ip] gives the hop up afterBROADCAST_SOLICIT, 3, unanswered requests (system/netstack/ip/src/nud.rs). While 16 wait on one silent hop, a send to any other hop waits too, where today it is refused. A program that retried onOthersleeps instead. The bell costs what a stream's does (amendment A).OwnedFddrop andtry_clone_to_ownedin every ToyOS program, eachFile::from_raw_fd, and each handle a spawn puts in a child's slot, endows or provides looks intoyos::net's table of the process's stream ends, under its lock. In std on ToyOS onlysdk/std/sys/net/connection.rs's sockets hold anOwnedFd(rg -n OwnedFd sdk/std); another crate's pays at its drop. Not measured: nothing is built.Known limit, a regression from today's ToyOS: a stream does not move between processes (amendment E, ruling 7). A program that hands a stream to another process, through a
File, a child's slot, an endowment or a provided name, orsend_handles, which its type refuses, as an inetd-style server does through its child's stdio, works today and fails loudly after stage 3; one that takes a moved stream back as aTcpStream, as socket activation does, is refused. Filed asissues/a-stream-cannot-move-between-processes.md, owned by the orchestrator, with E's exit: a stream moved by its handle reads and writes in the receiving process, and the process that moved it holds no end.What it saves. On the receive path, of the pipe lock, between the write copy, 0.081 ms per MB, and S = 0.192, 3.6% of C0, by the scout, and beside it the pipe syscalls' work outside the hold, which no row reads apart; on the send path two copies of every byte and a pipe; the request and connection per datagram; the
PIPEStraffic of every stream; and the structural debts above, three of them filed issues it closes.The alternative it is measured against is to keep the pipes and take the per-pipe lock: no ABI change, the lock's convoy gone, every copy kept. The scout's third rule is where that would have won, and it did not fire.
Staging
The track's stages, each a pull request, built in this order:
fstat'sDeviceMemory, andSharedMemory::lentjudging a lent region before it maps it. Useful alone: fileserver and diskserver judge their lent regions through it.toyos-transport.toyos::net, std's raw-handle conversions forTcpStream, and its refusal of a stream's handle as aFileor on any path that moves a handle. Its exit is the track's (issues/netstack-moves-every-byte-through-two-kernel-pipes.md): amendment A's, B's and D's loom models over the code it lands, each red under its controls,head-onlyamong D's (ruling 2); the tests of E's refusal and its boundary (round 5's BLOCKER 3), ofFile::from_raw_fd(ruling 7), of each std path that moves a raw handle (ruling 8;send_handleshas none, its type refusing the handle, ruling 12) and of awriteafter a reset (ruling 3), each red when the path takes the handle silently or the write copies; and the T14 boot, with the scout's instrument carried onto its head, readingpipe_copy_bytesunder 0.01 per byte downloaded beside the scout's rows.SYS_CONNECTION_JOINwith it. Its exit is the track's: amendment C's listener case and ruling 1's, red under its three controls; ruling 4'slenpast its record; a case per refusal, red when a call answersOk(ruling 9); a nonblockingsend_toat a full ring answeringWouldBlock; and rounds 7's and 8's loom models, red under their section's controls.Stages 3 and 4 change the ABI with the work that needs it, as root
CLAUDE.mdasks, and each says why userland cannot do what it adds to the kernel, if it adds anything.Checks
Net lines (
git diff --shortstat origin/main...5664103e4,origin/mainat b6a5565): 3 files changed, 280 insertions(+), all underissues/; production 0, tests 0. Each stage names its own negative control and oracle in its pull request.head-onlyfor ruling 2; round 7's look before the arm, wants exchanged before the store,rx_dataalone rung, room on the connection and a refusal at the full queue; round 8's wait for the client's ring after the leave; and stage 2'swant-relaxedfor A, B, D and rounds 7 and 8. E, its cost and rulings 3 to 9 rest on readings, and the exits carry their tests. The independent oracles are RFC 9293, POSIX.1-2024 (IEEE Std 1003.1-2024:accept,select,send,sendto, XSH 2.10.10) and virtio 1.2, each at the version cited.Gates
cargo run -- --ci hoston 5664103,mainat b6a5565 merged: EXIT=0, 02:50:23Z (commit 02:50:15Z) to 02:55:22Z, endingHost: 76 step(s), all green; its 49test result: FAILEDlines are each inside a[ci] controlstep. Posted whole, masked (<tree>,~,<tmp>): host.log at evidence 361e205f.streamend.rsintotoyos::net, pipe readers stage 3 deletes, and growsnode/tests/streams.rs;origin/mainis b6a5565.toyos-transportis unchanged since 7ea0ff9, which A's and B's models ran, D's and rounds 7's and 8's a copy of:git diff --stat 7ea0ff9d6 HEAD -- lib/blockring/transport lib/untrustedis empty. Each model case iscargo test --offline --release --test <model> -- --exact <case>, run by itsrun.shbeside its source and lockfile; rounds 7's and 8's log has fifteenEXIT=lines.ungated, 0, 0, 0 (log); unbounded,pass_on_bytes_leftstopped at 600 s with no verdict, EXIT=143 (log).sh run.sh <tree> <dir>applies the case, runscargo test -p toyos-net-node --test listeners, EXIT=0 (32 passed), then each control, EXIT=101, 101, 101, then restores,git status --porcelainempty (log). The first round'snowhere.patch: EXIT=0, then 101 (log).--metal-readbackon 2f9e090, EXIT=2, the staging verdict, not a pass (log and request.txt); its judge is under "The reading".What I am unsure of
toyos::nettakingtoyos-transportas a dependency moves the sysroot's identity in a waybuild/identity.rscharges every build for; read before stage 3.WouldBlockafter its event. A mio program that reads a socket from a thread mio did not wake, and stops short ofWouldBlock, can miss its next event, as mio documents.reregisterthat changes only the token should ring netstack too. As written, D's does.🤖 Generated with Claude Code
https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy