Skip to content

ToyOS's own network stack, stage 3: toyos-net-ip, toyos-net-udp and toyos-dhcp - #635

Merged
Japabu merged 25 commits into
mainfrom
wt/toyos-netstack
Oct 1, 2026
Merged

Japabu merged 25 commits into
mainfrom
wt/toyos-netstack

Conversation

@Japabu

@Japabu Japabu commented Sep 30, 2026 •

Copy link
Copy Markdown
Collaborator

Stage 3 of ToyOS's own network stack (issues/design-debt/toyos-has-its-own-network-stack.md). Three crates, built clean-room from ip.md, udp-dhcp.md and the RFCs. Like toyos-net-wire and toyos-net-tcp, they are pure: no_std with alloc, forbid(unsafe_code), and clippy forbids arithmetic side effects, indexing, unwrap, expect, panic and as outside tests. They read no clock, draw no randomness and do no I/O. Nothing in the image uses them yet: netd moves over at stage 5.

What changed

  • toyos-net-ip: per-interface addressing and host routing; ARP under one reachability machine (RFC 4861 NUD with RFC 7048's UNREACHABLE, a typestate enum whose states hold only their own fields); RFC 5227 conflict detection with defence (b); ICMP echo, errors, suppression and the two-level error limiter; the IGMPv3 any-source host (RFC 9776) with IGMPv2 compatibility mode; frame dispatch and the IPv4 input policy; pull egress.
  • toyos-net-udp: one socket per port (U-1), RFC 6056 Algorithm 1 ephemeral ports (U-2), the demultiplexer, receive and send queues of 16 datagrams or 64 KiB, broadcast permission (U-4), send-to-self and loopback refused (U-8), ICMP errors for connected sockets only, and a bounded hold for what closed sockets had accepted.
  • toyos-dhcp: the RFC 2131 client with RFC 3396 long options, RFC 4039 rapid commit, an RFC 4361 client identifier a server may echo (RFC 6842), and conflict detection through toyos-net-ip. Its draws come in a fixed order: the xid first, then any jitter.
  • toyos-net-wire now also holds what every net crate shares: the caller's Instant, SipHash-2-4 (moved from toyos-net-tcp with its bytes unchanged), the counters! macro with RefusalLog, which admits one log line per rule per 10 s, and the IPv4 address classes and prefixes (addr: is_host, is_martian, Cidr). Every net crate reads the address arithmetic from there and keeps no copy of it. The IGMPv3 report builder takes GroupRecord, whose IS_IN carries its sources for §10.5's group-and-source answer, and MulticastAddr is Ord.
  • toyos-net-tcp: connect refuses a remote that addr::is_host refuses, and its own unicast() is gone (decision 17).
  • The track issue lists what the stage 3 specifications still owe (OUT-07, US-57, dhcp.renew-unroutable, FRA, PMTU), the three places this stage departs from them, and stage 3's missing independent oracle, each with an exit.
  • No new dependency: Cargo.lock gains only the three crates.

Decisions

  1. Transmit back-pressure is designed in. Egress is pulled with credit (ip §5, udp §U4.6). A frame is built when it is handed off, and every timer it feeds starts then. A datagram toyos-net-ip holds for resolution spends no credit, so the socket's next datagram does not wait behind it. A full send queue refuses the sender (udp.tx-queue-full) and the caller keeps its datagram.
  2. ip.md §5.4, §6.2 and §6.5 as the owner amended them. REACHABLE, STALE, DELAY, PROBE and UNREACHABLE each carry a pending queue that only drains (Released: pop, no push). INCOMPLETE's Pending is the only queue with push, and resolution moves it into the resolved state. NUD-28's doctests push into each resolved queue (compile_fail) beside a twin that pushes into INCOMPLETE's and drains every resolved one. The control queue keeps only the order: resolution gives each datagram a turn there, which counts against its 64 frames. A turn that finds it full waits and enters as room appears, ahead of any frame queued later (OUT-08). When the turn comes, the oldest datagram in the entry's queue leaves, to the MAC the entry holds at that moment (NBR-08). An entry whose queue holds any is evicted after every other candidate, and then they are dropped as nb.pending-evicted with their turns, and each sender is told (NUD-29). Its idle lifetime cannot end while it holds any: an entry whose lifetime passed meanwhile goes when the last one leaves, and one whose lifetime has not passed stays (NUD-29; NUD-30, for STALE and quiescent UNREACHABLE). However an entry goes, its turns and its queued requests go with it, so no item in the control queue outlives its entry. A later entry at the same address therefore sends its request from its own place in the queue (NUD-30).
  3. IP-D1, as the owner ruled: probing ends within 200 ms. PROBE_WAIT, PROBE_MIN, PROBE_MAX and ANNOUNCE_WAIT are RFC 5227's scaled by 200/7,000; ANNOUNCE_INTERVAL, DEFEND_INTERVAL, MAX_CONFLICTS and RATE_LIMIT_INTERVAL keep the RFC's values. The owner ruled two exceptions to the 200 ms bound: §8.3's rate limit, which holds the first probe back (ACD-08 pins its 60 s), and a starved control queue, which holds a probe until it has room (ACD-18). ACD-01 asserts the scaled constants, 28,571,428 and 57,142,857 ns, and the 114,285,713 ns floor as literals.
  4. Frame dispatch lives in Ip::receive (wire.md §3.3), so the shard hands raw frames to it. The nine [shell] dispatch scenarios (ETH-10, 11, 12, 14, 19, 22–25) are therefore tested here.
  5. US-43 is read with the frame filter first: a frame to an unjoined group's MAC is eth.not-for-us before toyos-net-ip sees the datagram. The test checks both layers; the departure is in the track.
  6. IGMP-23 follows W-2: a query without Router Alert is refused (RFC 9776 §9.1), which inverts wire.md's scenario as ip.md §16.3 rules.
  7. Undrained refusals are bounded at 1,024 per crate, and one past that is counted (ip.event-overflow, udp.event-overflow, dhcp.event-overflow), as in toyos-net-tcp. MOD-01, US-20 and DH-38 leave 1,025 logged refusals undrained and find 1,024 and one overflow.
  8. nb.pending-dropped is added: each held datagram dropped at FAILED or link-down is counted, beside the "host unreachable" its sender is told.
  9. dhcp.renew-unroutable is netd's, at stage 5 (udp-dhcp.md §D2.3, [shell][5]): the client never learns of toyos-net-udp's refusal, which counts it as udp.no-route, and keeps its schedule. The track owes it.
  10. The client never asks [ip] to announce. On link-up, toyos-net-ip announces the held address itself (IP-D6, C-4). DH-64 still names an announce request; the departure is in the track.
  11. §D4.2's degradation (routers, resolvers, T1 and T2) runs on the lease an accepted ACK makes, and each refusal is counted there.
  12. Every client message is padded to RFC 1542's 300 bytes; a long host name can make one longer.
  13. Closing is bounded. What closed sockets had accepted is held to one socket's queue together (limits::CLOSED_DATAGRAMS, 16); a close past it refuses the rest udp.tx-discarded-on-close, since nobody is left to refuse them to. A closed socket's turn leaves with it, and the closed sockets' datagrams take one turn among the sockets rather than going first. udp-dhcp.md §U9 (3) lets every one leave; the departure is in the track.
  14. The acquisition exception admits a destination that names one host (C-3, which the owner ruled stands over H-1's wording). A host destination only arrives in a frame to our MAC, because §4.1 step 3 refuses one in a group frame first, so the separate link check was dead and is gone. The exception stays on while the only address is still probing, which is not usable (§2.4 (5)).
  15. ACD's own frames wait for room (the spec owner's ruling on the round-4 ACD note). A probe or an announcement that finds the control queue full is not counted as sent: it waits as a turn does and goes out at the next opportunity, so an address is never verified without its three probes on the wire (RFC 5227 §2.1). A defence waits one at a time. What waits for room is therefore bounded: turns within PENDING_TOTAL per interface, one probe per tentative address, and one owed announcement and one defence per usable address.
  16. A request is built when it leaves, from the state of that moment (§6.4): broadcast in INCOMPLETE and UNREACHABLE, to the cached MAC in PROBE, and in REACHABLE, STALE and DELAY not at all, since §6.3 sends none in them. OUT-08 pins both halves: a request queued before the reply is dropped for REACHABLE, and one queued in INCOMPLETE and still queued in PROBE leaves unicast to the cached MAC.
  17. One host predicate. toyos-net-tcp's connect refuses what toyos_net_wire::addr::is_host refuses. HS-35's three addresses stay refused, and 0/8 and 127/8 remotes are refused too (MUST-46).

For the spec owner

  • OUT-08 is adjusted. B's request, queued before the reply, is dropped at egress, because §6.3 sends nothing to maintain REACHABLE. With credit, out the 63 other queued frames, then 1, 2, 3; the scenario says "the 64 queued frames". It gains an arm: B's request, queued in INCOMPLETE and still queued when an assertion, a send and DELAY's 5,000 ms bring B to PROBE, leaves unicast to MAC B, and it is the only request.
  • ACD-18 is a proposed id, for decision 15. It fills the control queue from a second interface during probing. The address stays tentative and ip.control-queue-full stays 0 until the queue drains; then the probes leave on the schedule, and only then is the address verified. It fills the queue again when a defence is due, and asserts that one defence waits.
  • NUD-29 gains three arms: a later entry at B's address, made after B's eviction, sends its datagram on its own turn, behind a defence queued first; B's turns wait inside a full control queue, and the room its eviction frees goes to a waiting defence, which leaves ahead of the new entry's request; and RTE-004's prefixes put X on both interfaces: when X on if0 ends its FAILED hold-down, the datagrams released to X on if1 still leave to X.
  • NUD-30 gains its quiescent UNREACHABLE twin, and an arm in which a request waits behind B's datagrams past B's idle lifetime. B goes as 2 leaves and takes the request with it. A later entry at B's address then sends its own request, behind a reply queued first.
  • NBR-08 gains an arm: a datagram released to R that still waits for room leaves to R's new MAC.
  • HS-35 adds 127.0.0.1 and 0.1.2.3.

Scenario coverage

Every test names its scenario id, and every scenario has one test; a scenario with several inputs is one test that checks each.

  • ip.md: 215 scenarios. 189 carry [5], and 188 of those are tested. NUD-28 is covered by compile-fail doctests, each beside a twin that compiles. OUT-07 ([5][shard]) is deferred to stage 4: it needs a TCP segment and the shard's scheduler choosing between a flow and toyos-net-ip's own frames. The 26 that carry only [9] (FRA-01–20, PMTU-01–06: fragments and path MTU) belong to IP hardening. ACD-18, a proposed id, is tested too.
  • udp-dhcp.md: 133 scenarios (US-01–57, DH-01–76), 132 tested. US-57 ([shell], the pipe ABI's mapping of refusals) is deferred to stage 5, where netd does that mapping.
  • wire.md: all 53 scenarios tagged [ip] or [udp] are tested, as are the nine [shell] dispatch scenarios and ETH-32.

Negative control: one checked mutation per rule

Each mutation is a patch. It was checked with git apply --check and applied; the mutated tree was shown to build (cargo test -p toyos-net-ip -p toyos-net-udp -p toyos-dhcp -p toyos-net-wire -p toyos-net-tcp --no-run, exit 0 for every one) and run with the same packages and --no-fail-fast; the patch was then reversed with git apply -R, and git status --porcelain was empty after each. The baseline on 754ce5e exits 0, 1,045 passed. All 71 below were run on 754ce5e, and every one exits 101. Each goes red on the same tests as on c1474cd, and Md is new. The first run hit its time limit during R01. Its half-applied patch was reversed and the tree was checked clean. That run had also found that N23 as written no longer builds, so N23 was re-expressed. N23 and R01–R20 then ran in a second run on the same head.

The final review's mutation. Md reverts this round's request change onto the head, in source only: purge_entry is drop_turns again, and NUD-30's new arm stays.

Mutation Exit Red
Md remove leaves its entry's queued requests 101 NUD-30, in its third arm: the orphaned request leaves as the new entry's, ahead of the reply

The round-6 review's mutations, each the review's edit on the lines as they stand: drop_turns is now purge_entry, and drained's condition is now in leave. Mc-r reverts round 6's NOTE fix. The term comes back in leave's condition, and request_leaves' drop branch again deletes a drained entry.

Mutation Exit Red
M-a purge_entry ignores a turn's interface 101 NUD-29, in its two-interface arm: if1's 1 and 2 never leave
M-b the room purge_entry frees is not refilled 101 NUD-29, in its defence arm: the new entry's request leaves ahead of the defence
Mc-r a request queued behind a drained entry's datagrams keeps it until the request is taken out 101 NUD-30, in its third arm: B is still held after 2 leaves

The round-5 review's mutations, each the review's edit on the lines as they stand. N24 (request_leaves drops a request without deleting a drained entry) has no site left: request_leaves deletes nothing, and Mc-r restores that deletion. N23 also adds .. to purge_entry's request pattern, which its new field needs to build.

Mutation Exit Red
N21 remove leaves the turns; now purge_entry keeps them and still drops the requests 101 NUD-29, in its first arm: B's first turn carries the later entry's datagram out ahead of the defence
N23 a request's form is fixed when it is queued, as 02dde43's to: Option<MacAddr> fixed it 101 OUT-08, in its second arm: the request leaves broadcast

Round 6's rules. N25 is the same edit on the line as it stands.

Mutation Rule Exit Red
N22 an eviction drops the turns and keeps them counted ip §6.5 101 NUD-29, in its second arm: 62 datagrams accepted, not 64
N25 leave's idle condition ignores the released queue, so it deletes an entry whose lifetime passed while it still holds datagrams ip §6.3's idle note 101 NUD-30

The round-4 review's mutations. A1 is the same edit on the line as it stands.

Mutation Exit Red
A1 a released datagram's leaving deletes its entry before the entry's lifetime has passed 101 NUD-29
A2 a quiescent UNREACHABLE entry idles out while it holds released datagrams 101 NUD-30
B every probing interval halved: first use at 57 to 100 ms 101 ACD-01

Round 5's rules. N16 and N18 are the same edits on the lines as they stand.

Mutation Rule Exit Red
N12 Released gains push ip §6.2; NUD-28 101 NUD-28's four push doctests
N13 a probe that finds the control queue full counts as sent the spec owner's ruling; RFC 5227 §2.1 101 ACD-18
N14 an announcement that finds it full counts as sent the spec owner's ruling 101 ACD-18; NUD-29
N15 every defence waits, not one at a time decision 15 101 ACD-18
N16 a request queued before resolution leaves for a REACHABLE neighbour ip §6.3 101 OUT-08
N18 a released datagram's MAC is written at resolution decision 2; ip §5.2 101 NBR-08; NUD-30
N19 connect admits 0/8 and 127/8, as the deleted unicast() did MUST-46; decision 17 101 TCP HS-35

The earlier rounds' mutations, run again. Where the code they mutate moved, each is the same edit on the lines as they stand: M02r, N04r (a turn that finds the queue full is dropped, so its datagram never leaves), N05r, N07r, N08r (now: no released datagram's leaving deletes anything), R07r and R08r. R02p, R05p, R15p, M04p and M05p are round 3's re-expressions. N06 ("an evicted entry's released datagrams still leave"), N17 ("a turn names only its address") and N20 ("an eviction takes its turns out of the interface's count") have no site left: turns carry no id, an evicted entry's datagrams and turns go with it, and N21 and N22 mutate that.

Mutation Exit Red
R01 [ip]'s undrained-refusal bound removed 101 MOD-01
R02p the acquisition exception admits a group destination 101 US-44
R03 dhcp.bootp-reply not logged 101 DH-43
R04 RATE_LIMIT_INTERVAL 6 s 101 ACD-08
R05p close drops accepted datagrams 101 US-53
R06 the xid unchecked 101 DH-37, 42, 62, 73, 74
R07r STALE evicted before FAILED 101 NUD-22
R08r UNREACHABLE requests without traffic 101 NUD-12
R09 TCP to a group or broadcast delivered 101 IN-09
R10 IGMP query sources unbounded 101 IGM-12
R11 [udp]'s undrained-refusal bound removed 101 US-20
R12 [dhcp]'s undrained-refusal bound removed 101 DH-38
R13 a quote's source unchecked 101 ICD-04; wire ICMP-36
R14 the echo-reply bound removed 101 ECHO-05
R15p the router may be yiaddr 101 DH-59
R16 the v2 leave always sent 101 IGM-19
R17 icmp.source-quench not logged 101 MOD-02
R18 a parse refusal names 0.0.0.0 as its peer 101 DH-38–40, 43, 45, 47, 48, 51–58
R19 dhcp.no-subnet-mask not logged 101 DH-57
R20 dhcp.xid-mismatch logged 101 DH-42
Mutation Rule Exit Red
N01 close holds every accepted datagram architecture §3.3; decision 13 101 US-53
N02 close leaves its socket's turn behind "one datagram per socket in turn" 101 US-53
N03 closed sockets' datagrams go first decision 13 101 US-53
N04r a turn that finds the queue full is dropped ip §5.4 as amended 101 OUT-08; NUD-29, 30; NBR-08
N05r eviction ignores released datagrams ip §6.8, owner's ruling 101 NUD-29
N07r the idle lifetime deletes an entry holding released datagrams ip §6.3, owner's ruling 101 NUD-30
N08r the last released datagram's leaving deletes nothing owner's ruling (b) 101 NUD-30
N09 every route refusal is udp.no-source-address udp §U4.4 101 US-24
N10 an unexpected OFFER, ACK or NAK names no reason DH-73 101 DH-16, 62, 73, 74
N11 ip.acquisition-admitted not counted ip §4.2 as amended 101 US-44
M01 the first conflict loses the address RFC 5227 §2.4 (b); IP-D3 101 ACD-07, 10, 12, 17, 18; NUD-29
M02r no request spacing RFC 1122 §2.3.2.1; RFC 4861 §7.3.3; ip §6.3 101 NUD-16
M03 the per-destination bucket ignored RFC 1812 §4.3.2.8; RFC 4443 §2.4 (f); ip §9.4 101 RL-01, 02, 06; ECHO-08; CLK-09
M04p no acquisition exception RFC 2131 §2; C-3 101 US-44; DH-07, 75
M05p the exception outlives the address C-3 101 DH-75
M06 Ip::transmit ignores its credit ip §5.4; pull egress 101 OUT-06, 08; NUD-30; ACD-18; US-26
M07 queries without Router Alert accepted RFC 9776 §9.1; W-2 101 IGM-13; wire IGMP-23
M08 IGMPv1 queries accepted IP-D11 (modern only) 101 IGM-20
M09 no LOCKTIME RFC 4861 Appendix C; ip §7.3 101 NBR-09, 10
M10 errors about errors RFC 1122 §3.2.2 101 ICG-03
M11 a connected socket hears anyone U-11; udp §U3 (5) 101 US-15, 18, 37, 55
M12 ICMP errors to unconnected sockets udp §U8; U-5 101 US-47, 52
M13 the send queue unbounded udp §U4.6; U-6 101 US-25
M14 the exception reaches any socket on 68 udp §U5.3 101 US-44
M15 a held datagram spends UDP's credit udp §U4.6 (4) 101 US-26
M16 the xid unchecked RFC 2131 §4.4.1; udp §D3.1 (8) 101 DH-37, 42
M17 chaddr unchecked udp §D3.1 (4) 101 DH-41
M18 retransmissions at the 60 s floor, not halved RFC 2131 §4.4.5 101 DH-19, 20, 24, 31
M19 five transmissions before giving up udp §D6.3; RFC 2131 §3.1 (5) 101 DH-09, 67
M20 FORCERENEW taken as an ACK RFC 3203 §6; modern only 101 DH-52
M21 no NAK back-off H-7; §D11 101 DH-30
M22 the refusal log interval at 1 s ip §13.1 101 TCP MOD-08, 09
M23 one SipHash rotation changed SipHash-2-4 101 TCP ISN-001–006, 008; PORT-01–04

M22 and M23 go red only in toyos-net-tcp, which tests the shared log policy and SipHash's reference vectors for every crate that uses them.

Independent oracle

  • The RFC text of each rule, cited in the module headers and in the tables above.
  • The specifications' byte vectors, which the readers derived from the RFCs, are reproduced byte for byte: V-ARP-, V-IGMP3-, V-IGMP-, V-ICMP-, V-UDP-* and V-DHCP-*. Every DHCP message is framed through toyos-net-ip and toyos-net-udp and compared as a whole frame: DISCOVER, REQUEST, RENEW, REBIND, REBOOT and DECLINE.
  • Checksums in the tests come from the tests' own RFC 1071 sum, not from toyos-net-wire's.
  • SipHash-2-4's published reference vectors (Aumasson and Bernstein) are checked by ISN-001, and they cover the function now in toyos-net-wire.
  • The gap: none of these is independent of the readers except the SipHash vectors. No behaviour of the three crates is checked against anything a reader did not write. The track records it, with its exit: exchanges captured from slirp and the T14 replay through them at stage 5 and match.

Gates, on 754ce5e

  • cargo run -- --ci host: EXIT=0 (55 steps, all green, workspace clippy with warnings denied among them).
  • cargo test -p toyos-net-ip -p toyos-net-udp -p toyos-dhcp -p toyos-net-wire -p toyos-net-tcp --no-fail-fast: EXIT=0, 1,045 passed.
  • cargo clippy -p toyos-net-ip -p toyos-net-udp -p toyos-dhcp --all-targets -- -D warnings: EXIT=0.
  • The 71 mutations above: each builds with exit 0 and runs with exit 101.

Clean room

No other stack's source was opened while writing this branch. That includes smoltcp, lwIP, netstack3, FreeBSD, Linux and netd's glue beyond the pipe ABI.

🤖 Generated with Claude Code

https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L

Japabu and others added 10 commits September 29, 2026 21:46
…net crate shares

Stage 3 adds three crates (toyos-net-ip, toyos-net-udp, toyos-dhcp) that
need the same injected Instant, the same SipHash-2-4 keyed function and the
same counter declaration toyos-net-tcp carried privately. Each moves to
toyos-net-wire, the crate every net crate already depends on, so the shard
that composes them hands one Instant to all of them and no crate carries a
second copy of either.

toyos-net-tcp re-exports Instant, siphash24 and Key, so its API is
unchanged; the TSval of an instant becomes a free function, and its
counters are declared through toyos_net_wire::counters!. Its suite is the
same 654 tests, green, before and after.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stage 3's first crate, written clean-room from the stage-3 IP
specification and the RFCs: per-interface addresses and the host routing
table, frame dispatch and the IPv4 input policy, the reachability machine
over ARP (RFC 4861 §7.3 with RFC 7048's UNREACHABLE), RFC 5227 conflict
detection on the owner's 200 ms schedule, ICMP echo and errors under the
two-level limiter, inbound errors attributed for the transports, and the
IGMPv3 host with its IGMPv2 compatibility mode.

Pull egress throughout: [ip]'s own frames wait in one FIFO and are built
when they leave; every timer they feed starts at the hand-off; a UDP
datagram is written straight into the caller's frame or held here for its
next hop, and a full ring spends no datagram.

toyos-net-wire gains what W-1 needs (IGMPv3 IS_IN (B), RFC 9776 §5.2
Table 5), `MulticastAddr: Ord`, and the counter macro takes an
expression for a name and generates the per-rule refusal limiter, which
toyos-net-tcp now uses instead of its own copy.

Scenario tests so far: CLK, ADDR, RTE, IN, OUT, NUD, and the wire
specification's owed [ip] and [shell] rows (IPP, ETH, IP, IPO, UDP-22).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every [5] scenario of the IP specification has its test but OUT-07, which
needs the composed shard (stage 4): 185 of 186, plus the NUD-28
compile-fail doctests, and the wire specification's owed [ip] and [shell]
rows except UDP-17 and UDP-18, which toyos-net-udp owns.

Scenarios whose numbers the IP-D1 ruling moved are tested on the ruled
schedule: probing scaled so the add-to-first-announcement worst case is
200 ms, ANNOUNCE_WAIT after the third probe's hand-off, announcements
2 s apart as RFC 5227 keeps them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stage 3's second crate, written clean-room from Part U of the stage-3
UDP/DHCP specification and the RFCs: sockets bound to any address, one
address or one peer, at most one socket per port whatever the address
(U-1), RFC 6056 Algorithm 1 over 49152-65535 with one draw per bind
(U-2), bounded queues both ways (16 datagrams, 64 KiB) whose transmit
side refuses back to the caller, broadcast only by permission (U-4),
0.0.0.0 as a source only for the acquisition socket and only to the
limited broadcast, and ICMP errors for connected sockets alone (U-5).

Egress is pulled: a datagram leaves only when the transmit opportunity
offers it to toyos-net-ip, which writes it into the device's frame or
holds it for its next hop; a held one spends no credit, so a socket's
next datagram is never behind it (US-26). Accepted datagrams of a closed
socket still leave (U-9).

toyos-net-ip now counts every datagram it held and dropped
(nb.pending-dropped): at resolution failure, and at link-down, where a
datagram resolution had released but not yet sent is dropped and its
sender told too.

Every Part U scenario has its test but US-57, the pipe ABI mapping, which
is netd's (stage 5); UDP-17 and UDP-18 are here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L
Stage 3's third crate, written clean-room from Part D of the stage-3
UDP/DHCP specification and the RFCs: RFC 2131's client with RFC 3396 long
options, RFC 4039 rapid commit, an RFC 4361 client identifier (H-2) a
server may echo (RFC 6842), and RFC 5227 conflict detection through
toyos-net-ip before an acknowledged address is used (H-3 = IP-D1). Pure:
each call takes the time and a draw source and returns at most one
transmission, a configuration change and an address request.

Messages are built to one layout, options in §D2.2's order, padded to
300 bytes; a server's is read by §D3's checks in order, each refusal one
named counter, BOOTP replies, unauthenticated FORCERENEW and a lease
without a mask refused and logged. Renewals follow RFC 2131 §4.4.5's
halving with the 60 s floor; a late timer runs only the latest due event;
NAK loops back off (H-7).

The tests compare every client message the stack frames, through
toyos-net-ip and toyos-net-udp, against the specification's frame vectors
byte for byte (DISCOVER, REQUEST, RENEW, REBIND, REBOOT, DECLINE).

DH-73's property found that link_up on a lease already expired entered
INIT-REBOOT with a deadline in the past; such a lease now expires.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L
…resentable

toyos-net-ip: FAILED carries nothing (its hold-down is the entry's deadline),
so Nud::Failed is a unit variant; the accessors no test or caller read go
(Ip::mac, Ip::joined, Linked/Probing/Unreachable::mac, Unreachable::requests,
Failed::since), and Limiter::global_burst is private. NUD's fail takes the
INCOMPLETE entry's queue from the arm that matched it rather than re-matching
a state it cannot be in.

toyos-net-udp and toyos-dhcp: a refusal is a Refusal drained by
drain_refusals, as it was the only event either crate had. UDP's port-0
checks yield the typed port they refuse on, so no unreachable second check
remains; DHCP's renew takes the lease's timers from the one caller that has
them, the lease degradation is a Client method, and the two limits nothing
read (JITTER, NAK_BACKOFF_MAX) go.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L
…ue's bound

ip.md §5.4 lists datagrams released from a pending queue among the frames
its 64-frame FIFO drops when full. A released datagram's sender was already
told it is held, so dropping it there is the silent loss the owner's premise
rules out ("a full ring backpressures the sender and never drops"); it stays
bounded by PENDING_TOTAL, counted where it was held until it leaves. The
test fills the queue with 64 of [ip]'s own frames and resolves eight
neighbours holding 64 datagrams: all 128 frames leave and nothing is counted
ip.control-queue-full. A mutation making release subject to the bound turns
it red, and no other test did.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L
A checked mutation per rule found four that no test turned red:

- NUD request spacing removed: NUD-16's advice at 600 lands on PROBE,
  which ignores advice anyway, so the scenario never reached the rule. The
  new NUD-16 test sends the request at 0, confirms at 5 and advises at 100:
  the poll waits for 1,000 (ip.md §6.3, RFC 1122 §2.3.2.1).
- Transmit credit ignored by [ip]: OUT-06 now asserts that no credit moves
  nothing and one credit moves the oldest frame.
- The acquisition exception delivered to any socket on port 68: US-44 now
  also has a socket on 68 without the mark, which must not hear it
  (udp-dhcp.md §U5.3).
- A datagram [ip] holds spending UDP's credit: with one credit, the next
  socket datagram leaves while the first waits for resolution (§U4.6 (4)).

NUD-28's FAILED block had become vacuous when FAILED lost its fields; it now
states that FAILED carries nothing, and the compiling twin binds STALE's
Linked so the STALE block can fail only on the missing queue.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L
…3 owes

ip.md §17 gives every scenario exactly one test, and a scenario with several
inputs is one test that checks each; the four checks the surviving mutations
asked for become parts of NUD-16, OUT-06, US-26 and US-44 instead of second
tests under the same ids.

The track's list of wire scenarios owed by stages 3 to 5 is gone: every one
is tested here. In its place, what the stage 3 specifications still owe
(OUT-07 to the shard, US-57 to netd, the fragment and path-MTU scenarios to
IP hardening) and the three places stage 3 departs from them, each with its
exit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L
@Japabu
Japabu marked this pull request as ready for review September 30, 2026 12:21
@Japabu

Japabu commented Sep 30, 2026

Copy link
Copy Markdown
Collaborator Author

Review of #635 at ce7fcb9. This is the first posted round. The earlier reviewer was cut off and posted nothing.

PROVENANCE: CLEAN.

  • Checked: the new crates' src and tests, and toyos-net-wire's additions.
  • Compared against: smoltcp 0.12.0 and both ToyOSOrg/smoltcp checkouts (330 files); netstack3 core ip, device, udp, datagram, icmp_echo and base (124 files); ref-lwip (acd.c, dhcp.c, dhcp.h, udp.c); and the Linux and FreeBSD igmp, arp, neighbour, icmp, udp, if_ether, ip_icmp, udp_usrreq, in_pcb, dhclient and if_llatbl sources.
  • Code: 24- and 40-token shingles, identifiers normalised, with at least 3 control-flow tokens. None in common with smoltcp or netstack3, in src or in tests.
  • Comments: 6-word shingles. The only shared phrases are RFC 9776's own wording ("group-and-source-specific", "the interface timer", "the response to a general query"). Tests share none.
  • Identifiers: none distinctive to those stacks appears (DynamicNeighborState, UnreachableMode, ClientState, RetryConfig, neighbor_cache, lastconflict, NUD_, lle_, …).
  • Where the NUD typestate resembles netstack3's, its fields are ip.md §6.2's table.

Ready for review.

  • CI host is SUCCESS on ce7fcb9 (run 36714140937, step cargo run -- --ci host success).
  • The new tests and NUD-28's doctests run in its host-workspace step, and workspace clippy is clean.
  • These are pure host crates, so no hardware reading is owed.
  • The branch merges cleanly onto origin/main deb8fa3.

Growth (git diff --shortstat origin/main...HEAD): 56 files, +12671/−160.

  • src: +5258/−149.
  • tests: +7344/−10.
  • Manifests, lock and issue: +69/−1.
  • The production growth is accepted as stage 3 of the track. The deletions it owes are named below: dhcp's and udp's copies of the address arithmetic, tcp's dead re-exports, Ip::wire_refusals, and acquisition()'s link argument once it is fixed.

Measured on ce7fcb9.

  • Baseline: cargo test -j 2 -p toyos-net-ip -p toyos-net-udp -p toyos-dhcp -p toyos-net-wire -p toyos-net-tcp --no-fail-fast gave EXIT=0, 1039 passed.
  • I planted twenty mutations. Each was one literal edit, saved as a patch and built (exit 0). Each was run with the command above and restored with git checkout -- <file>. The tree was clean after each.
  • Red (exit 101):
    • R05, close drops accepted datagrams: US-53.
    • R06, xid unchecked (the PR's M16): DH-37, 42, 62.
    • R08, UNREACHABLE requests without traffic: NUD-12.
    • R10, IGMP query sources unbounded: IGM-12.
    • R13, quote source unchecked: ICMP-36, ICD-04.
    • R14, echo-reply bound removed: ECHO-05.
    • R15, router may be yiaddr: DH-59.
    • R16, v2 leave always sent: IGM-19.
  • Green (exit 0, 1039 passed): R01, R02, R03, R04, R07, R09, R11, R12, R17, R18, R19, R20. Each one is in a BLOCKER below, with its patch in the appendix.
  • Probes P1, P2 and P3 ran as temporary test files, deleted afterwards (tree clean). Each exited 101. Their bodies are in the appendix.

Scenario map.

  • ip.md: 185 of 212 ids have exactly one test each.
  • udp-dhcp.md: 132 of 133 ids have exactly one test each.
  • The 27 without a test are OUT-07, FRA-01–20, PMTU-01–06 and US-57.
  • wire.md: all 62 ids tagged [ip], [udp] or [shell] have a test, and so does ETH-32.
  • Spot-checked against the spec text:
    • ip.md: NUD-02, 04, 05, 11, 16, 22, 28; NBR-09, 10; ACD-01, 07, 08, 11, 12, 16; OUT-06; CLK-09; RL-01, 02; ECHO-08; ICG-03; IGM-13, 20; RTE-15.
    • udp-dhcp.md: US-15, 20, 25, 26, 43, 44, 53; DH-07, 19, 20, 22, 24, 26, 30, 31, 33, 35, 37, 38–43, 64, 73, 74, 75.
    • Deferred: OUT-07 ([shard]), US-57 ([shell]), FRA-01 and PMTU-01 ([9]). The track records each deferral.
  • Deviations found are listed below. Every other checked scenario matches its test.

BLOCKER

  • toyos-net-udp/src/lib.rs:540 — close() moves accepted datagrams into orphans, a queue with no bound. — P1 accepts 1,600 datagrams (1.6 MB) on one port with no credit and holds all of them. Any pipe-ABI client can grow netd this way, and because orphans drain first they starve every other socket. This breaks architecture §3.3 ("never pushes segments into an unbounded queue") and the header's "Back-pressure is a refusal". Bound them and refuse past the bound; P1 turns green.
  • toyos-net-udp/src/lib.rs:532 — close() leaves the slot's index in active. — A socket that reuses the slot pushes the index again and gets two turns: P2 gives "yy" where line 640's "one datagram per socket in turn" promises "yz". Fix it; P2 turns green.
  • toyos-net-ip/src/input.rs:50 — acquisition() admits a group IP destination. — P3 (UDP to unjoined 239.9.9.9:68, frame to A's MAC, no address) is delivered as Cast::Acquisition, but C-3 admits "a unicast IPv4 datagram" only. Require a unicast host destination. Step 3 already refuses a unicast destination in a group frame, so the link check (R02 green) becomes dead and goes. P3 becomes US-44's negative case.
  • toyos-net-ip/src/input.rs:147 — the non-unicast TCP gate has no test. — It enforces tcp.md §2.1's MUST-57, which Delivery::Tcp's contract (lib.rs:188) promises [tcp]. R09 (arm deleted) is green. Add a SYN to 255.255.255.255, one to 192.0.2.255 and one to joined 224.0.0.251: none may produce a Delivery::Tcp, and ip.tcp-not-unicast must be 3.
  • toyos-net-ip/src/counters.rs:128, toyos-net-udp/src/lib.rs:256, toyos-dhcp/src/lib.rs:427 — the undrained-refusal bound is untested in all three crates. — Decision 7 claims this bound, and a peer drives the memory it limits. R01, R11 and R12 are green. Add one test per crate: 1,025 logged refusals left undrained give 1,024 events and *.event-overflow = 1.
  • toyos-dhcp/tests/parse.rs:21 — refused() expects rule.logged() from the crate itself, so nothing pins which DHCP refusals are logged. — R03 (bootp-reply unlogged), R19 (no-subnet-mask unlogged) and R20 (xid-mismatch logged) are all green. No test checks a parse refusal's peer: R18 (Peer::From(UNSPECIFIED)) is green. Pass the expected flag from §D3.1 and §D14 at each call, and assert the Refusal's peer. icmp.source-quench's log flag is also pinned by nothing (R17 green); pin it either way.
  • toyos-net-ip/tests/acd.rs:167 — ACD-08 reads RATE_LIMIT_INTERVAL from the crate. — The scenario says "no earlier than 60 s", and IP-D1 keeps RFC 5227's rate limit, yet R04 (60 s → 6 s) is green. Assert 60,000 ms as a literal, as ACD-11 and ACD-12 do for DEFEND_INTERVAL.
  • toyos-net-ip/src/nud.rs:208 — §6.8's eviction order (FAILED, then quiescent UNREACHABLE, then STALE) is pinned only for FAILED against REACHABLE. — R07 (STALE evicted before FAILED) is green. NUD-22 needs a full table holding one entry of each evictable class, evicted in that order.
  • toyos-dhcp/src/lib.rs:303 — toyos-dhcp and toyos-net-udp copy toyos-net-ip's address arithmetic. — mask, same_subnet, broadcast and the edge check at :726 copy Cidr's mask, contains, broadcast and is_edge. toyos-net-udp/src/lib.rs:219 invalid() copies is_class_e. The PR body's "no sibling crate keeps a copy" is false. Put one Cidr and one address-class module in toyos-net-wire and delete the copies.

NOTE

  • toyos-net-tcp/src/lib.rs:53 — pub use toyos_net_wire::REFUSAL_LOG_INTERVAL has no user. — Delete it. The siphash24 re-export at :56 serves only tcp's tests (isn.rs, port.rs), which can import it from toyos-net-wire; delete it too.
  • toyos-net-ip/src/lib.rs:359 — wire_refusals has only test callers. — wire_counters answers the same; delete it.
  • toyos-net-wire/Cargo.toml:5 — the description names wire formats only. — The crate now also holds the clock, SipHash and the counters macro.
  • toyos-dhcp/tests/link.rs:29 — DH-64 specifies "an announce request", but the test asserts none (decision 10, C-4). — This departs from a scenario, like US-43, and the track does not record it.
  • toyos-dhcp/tests/props.rs:107 — DH-73 says "exactly one reason", but the test asserts at most one. — A refusal that counts nothing still passes.
  • toyos-net-ip/src/lib.rs:448 — NUD-28's compile-fail set does not cover Probing or Unreachable holding no pending queue.
  • toyos-net-ip/src/igmp.rs:384 — merging a peer's query sources (up to about 366) with list.contains is quadratic per packet, because the 64-source cap applies after the merge. — Cap before merging.
  • toyos-net-udp/src/lib.rs:224 — no test tells udp.no-route from udp.no-source-address. — Every route refusal mapped to no-source-address passes. Add a send through an empty gateway list, which must give udp.no-route.
  • Oracles — stage 3's only oracles are the reader-built vectors and the tests' own checksum. — The track records this weakness for toyos-net-wire (slirp and T14 captures) but not for these crates.

REMOVE

  • issues/design-debt/toyos-has-its-own-network-stack.md:25 — the §5.4 departure. — The owner amended §5.4, so it is no longer a departure.
  • toyos-net-ip/src/acd.rs:4 — "Every interval starts when its frame leaves." — False: DEFEND_INTERVAL and CONFLICT_RESET run from the conflicting packet's receipt (:149, :130).
  • PR body, decision 2 — "This departs from ip.md §5.4 … recorded in the track". — False since the amendment.
  • PR body, "What changed" — "its API and tests are unchanged". — False: RefusalLog::admit now takes a Counter, and toyos-net-tcp/tests/modern.rs changed.
  • PR body, "Unsure, and questions for the owner" — every item has been ruled on, and item 5 (Delete the test tiers: a run is the whole suite or its filter #625 in the queue) is stale.
Appendix: the green mutations and the three probes

Apply one with git apply, then run the baseline command above. Each one must turn a test red.

# R01
--- a/toyos-net-ip/src/counters.rs
+++ b/toyos-net-ip/src/counters.rs
@@ -125,10 +125,6 @@ impl Log {
         if !rule.logged() {
             return;
         }
-        if self.refusals >= limits::EVENTS {
-            self.count(Counter::IpEventOverflow);
-            return;
-        }
         self.refusals = self.refusals.saturating_add(1);
         self.events.push(Event::Refused(Refusal { rule, iface, peer }));
     }
# R02
--- a/toyos-net-ip/src/input.rs
+++ b/toyos-net-ip/src/input.rs
@@ -48,8 +48,7 @@ fn dispatch(i: &Interface, frame: &Frame<'_>) -> Option<Counter> {
 /// The acquisition exception (ruling H-1): before the interface holds a usable address, a
 /// unicast datagram to UDP port 68 in a frame to our MAC is admitted for the DHCP client alone.
 fn acquisition(i: &Interface, link: MacClass, packet: &Ipv4Packet<'_>) -> bool {
-    link == MacClass::Individual
-        && i.usable().next().is_none()
+    i.usable().next().is_none()
         && packet.protocol() == Protocol::Udp
         && !packet.is_fragment()
         && UdpDatagram::parse(packet).is_ok_and(|d| d.destination_port().get() == ACQUISITION_PORT)
# R03
--- a/toyos-dhcp/src/lib.rs
+++ b/toyos-dhcp/src/lib.rs
@@ -44,7 +44,7 @@ toyos_net_wire::counters! {
     NotReply = "dhcp.not-reply", logged;
     HardwareType = "dhcp.hardware-type", logged;
     ChaddrMismatch = "dhcp.chaddr-mismatch";
-    BootpReply = "dhcp.bootp-reply", logged;
+    BootpReply = "dhcp.bootp-reply";
     OptionTruncated = "dhcp.option-truncated", logged;
     NoEnd = "dhcp.no-end";
     OverloadInvalid = "dhcp.overload-invalid", logged;
# R04
--- a/toyos-net-ip/src/lib.rs
+++ b/toyos-net-ip/src/lib.rs
@@ -131,7 +131,7 @@ pub mod limits {
         pub const ANNOUNCE_NUM: u8 = 2;
         pub const ANNOUNCE_INTERVAL: Duration = Duration::from_secs(2);
         pub const MAX_CONFLICTS: u32 = 10;
-        pub const RATE_LIMIT_INTERVAL: Duration = Duration::from_secs(60);
+        pub const RATE_LIMIT_INTERVAL: Duration = Duration::from_secs(6);
         pub const DEFEND_INTERVAL: Duration = Duration::from_secs(10);
         /// The conflict count starts over after this long without a conflict.
         pub const CONFLICT_RESET: Duration = Duration::from_secs(600);
# R07
--- a/toyos-net-ip/src/nud.rs
+++ b/toyos-net-ip/src/nud.rs
@@ -205,9 +205,9 @@ fn make_room(i: &mut Interface, cx: &mut Cx<'_>) -> bool {
         return true;
     }
     let class = |n: &Neighbour| match &n.state {
-        Nud::Failed => Some(0),
+        Nud::Failed => Some(2),
         Nud::Unreachable(u) if u.quiescent() && !n.queued => Some(1),
-        Nud::Stale(_) => Some(2),
+        Nud::Stale(_) => Some(0),
         _ => None,
     };
     let victim = i.neighbours.iter().filter_map(|(a, n)| class(n).map(|c| (c, n.used, *a))).min().map(|(_, _, a)| a);
# R09
--- a/toyos-net-ip/src/input.rs
+++ b/toyos-net-ip/src/input.rs
@@ -144,10 +144,6 @@ impl Ip {
                 igmp::input(i, &mut cx, &packet, message.message());
                 None
             }
-            Protocol::Tcp if cast != Cast::Unicast => {
-                self.log.count(Counter::IpTcpNotUnicast);
-                None
-            }
             Protocol::Tcp => {
                 let segment = TcpSegment::parse(&packet).map_err(|e| self.log.wire(e.name())).ok()?;
                 Some(Delivery::Tcp(arrival, segment))
# R11
--- a/toyos-net-udp/src/lib.rs
+++ b/toyos-net-udp/src/lib.rs
@@ -253,7 +253,7 @@ impl Udp {
     fn refuse<T>(&mut self, rule: Counter, local: (Ipv4Addr, Port), peer: (Ipv4Addr, u16)) -> Result<T, Error> {
         self.count(rule);
         if rule.logged() {
-            if self.refusals.len() >= limits::EVENTS {
+            if false && self.refusals.len() >= limits::EVENTS {
                 self.count(Counter::EventOverflow);
             } else {
                 self.refusals.push(Refusal { rule, local, peer });
# R12
--- a/toyos-dhcp/src/lib.rs
+++ b/toyos-dhcp/src/lib.rs
@@ -424,7 +424,7 @@ impl Client {
         if !rule.logged() {
             return;
         }
-        if self.refusals.len() >= limits::EVENTS {
+        if false && self.refusals.len() >= limits::EVENTS {
             self.counters.add(Counter::EventOverflow, 1);
         } else {
             self.refusals.push(Refusal { rule, peer });
# R17
--- a/toyos-net-ip/src/counters.rs
+++ b/toyos-net-ip/src/counters.rs
@@ -84,7 +84,7 @@ toyos_net_wire::counters! {
     IcmpQuoteShort = "icmp.quote-short";
     IcmpRedirect = "icmp.redirect", logged;
     IcmpTimestampRequest = IcmpError::TimestampRequest.name(), logged;
-    IcmpSourceQuench = IcmpError::SourceQuench.name(), logged;
+    IcmpSourceQuench = IcmpError::SourceQuench.name();
     IcmpEchoRepliesSent = "icmp.echo-replies-sent";
     IcmpErrorsSent = "icmp.errors-sent";
     IgmpV1Query = "igmp.v1-query", logged;
# R18
--- a/toyos-dhcp/src/lib.rs
+++ b/toyos-dhcp/src/lib.rs
@@ -657,7 +657,7 @@ impl Client {
 
     /// A server's UDP payload and the IPv4 source it came from (§D3).
     pub fn receive(&mut self, now: Instant, payload: &[u8], from: Ipv4Addr, mut draw: impl FnMut() -> u32) -> Output {
-        let peer = Peer::From(from);
+        let peer = Peer::From(Ipv4Addr::UNSPECIFIED);
         let reply = match Reply::parse(payload, self.mac.get()) {
             Ok(reply) => reply,
             Err(rule) => {
# R19
--- a/toyos-dhcp/src/lib.rs
+++ b/toyos-dhcp/src/lib.rs
@@ -63,7 +63,7 @@ toyos_net_wire::counters! {
     YiaddrInvalid = "dhcp.yiaddr-invalid", logged;
     NoLeaseTime = "dhcp.no-lease-time", logged;
     LeaseZero = "dhcp.lease-zero", logged;
-    NoSubnetMask = "dhcp.no-subnet-mask", logged;
+    NoSubnetMask = "dhcp.no-subnet-mask";
     MaskInvalid = "dhcp.mask-invalid", logged;
     RouterInvalid = "dhcp.router-invalid", logged;
     DnsInvalid = "dhcp.dns-invalid", logged;
# R20
--- a/toyos-dhcp/src/lib.rs
+++ b/toyos-dhcp/src/lib.rs
@@ -53,7 +53,7 @@ toyos_net_wire::counters! {
     WrongDirection = "dhcp.wrong-direction", logged;
     Forcerenew = "dhcp.forcerenew", logged;
     MessageTypeUnsupported = "dhcp.message-type-unsupported", logged;
-    XidMismatch = "dhcp.xid-mismatch";
+    XidMismatch = "dhcp.xid-mismatch", logged;
     ClientIdMismatch = "dhcp.client-id-mismatch", logged;
     NoServerId = "dhcp.no-server-id", logged;
     ServerIdInvalid = "dhcp.server-id-invalid", logged;

P1 and P2 are in toyos-net-udp/tests/, using its common. Measured: P1 printed "1600 datagrams accepted … 1600 left", and P2 printed "yy".

#[test]
fn p1_close_parks_accepted_datagrams_past_every_bound() {
    let mut u = U::uf();
    let mut accepted = 0usize;
    for _ in 0..100 {
        let id = u.bind(ANY, 50_001).unwrap();
        for _ in 0..limits::TX_DATAGRAMS {
            u.udp.send_to(&mut u.ip, id, DNS, 53, &[7; 1_000]).unwrap();
            accepted += 1;
        }
        u.udp.close(id).unwrap();
    }
    let left = u.out().len();
    assert!(left <= limits::TX_DATAGRAMS, "{accepted} accepted, {left} held past any socket's bound");
}

#[test]
fn p2_a_reused_slot_takes_one_turn() {
    let mut u = U::uf();
    let x = u.bind(ANY, 50_001).unwrap();
    u.udp.send_to(&mut u.ip, x, DNS, 53, b"x").unwrap();
    u.udp.close(x).unwrap();
    assert_eq!(u.out_with(1).len(), 1);
    let y = u.bind(ANY, 50_002).unwrap();
    let z = u.bind(ANY, 50_003).unwrap();
    for _ in 0..2 {
        u.udp.send_to(&mut u.ip, y, DNS, 53, b"y").unwrap();
        u.udp.send_to(&mut u.ip, z, DNS, 53, b"z").unwrap();
    }
    let order: String = u.out_with(2).iter().map(|f| char::from(ip_of(f).unwrap().payload()[8])).collect();
    assert_eq!(order, "yz");
}

P3 is in toyos-net-ip/tests/, using its common. Measured: Some(Acquisition).

#[test]
fn p3_the_acquisition_exception_admits_unicast_only() {
    let mut h = H::bare();
    let ip = udp(R, Ipv4Addr::new(239, 9, 9, 9), 67, 68, b"offer");
    assert_eq!(cast_of(&h.frame(&eth(MAC_A, MAC_R, 0x0800, &ip))), None);
}

SEND BACK

Japabu and others added 5 commits September 30, 2026 16:21
…rithmetic, and pinned refusals

The review of #635 at ce7fcb9 upheld nine blockers; the spec owner's rulings for this round
bind beside them. Per decision:

- toyos-net-wire holds the IPv4 address classes and `Cidr` (addr.rs): `is_host`, `is_martian`,
  and a prefix built from an address and a length or a contiguous mask. toyos-net-ip,
  toyos-net-udp and toyos-dhcp read them from there; their copies are deleted. toyos-dhcp does not
  depend on toyos-net-ip, so the arithmetic moved down rather than across.
- The acquisition exception (C-3, ip.md §2.4 (5)) admits a destination that names one host only.
  A host destination can only arrive in a frame to our MAC (§4.1 step 3 refuses it in a group
  frame first), so the separate link check was dead and is gone.
- toyos-net-udp's close holds at most CLOSED_DATAGRAMS (one socket's queue) of accepted datagrams
  together, and counts the rest `udp.tx-discarded-on-close`: architecture §3.3 holds nothing
  without a bound, and nobody is left to refuse them to. A closed socket's turn leaves with it, and
  the closed sockets' datagrams take one turn among the sockets instead of going first.
- ip.md §5.4 as amended: the control queue holds at most 64 frames, released datagrams included.
  A released datagram that finds it full stays at the head of its neighbour's pending queue and
  enters as room appears; it is its neighbour's until it has left. An entry holding one is evicted
  after every other candidate (§6.8), dropping them as `nb.pending-evicted`, and is not idle: an
  idle lifetime that passed while it held them ends when the last leaves (NUD-29, NUD-30).
- IGMP caps a query's sources at one past the limit before merging, so a merge is bounded.
- tcp's unused re-exports of REFUSAL_LOG_INTERVAL and siphash24 go; its tests read siphash24 from
  toyos-net-wire. `Ip::wire_refusals` goes; `wire_counters` answers the same.

Tests the review asked for: MUST-57 in IN-09; the undrained-refusal bound in MOD-01, US-20 and
DH-38; DHCP refusals pinned with §D3.1's log flag as a literal and the refusal's peer; §13.2's
logged list as literals in MOD-02; ACD-08's 60 s as a literal; §6.8's full eviction order in
NUD-22; the acquisition exception's negative case, its probing case and `ip.acquisition-admitted`
in US-44; close's bound, slot reuse and turn in US-53; `udp.no-route` against
`udp.no-source-address` in US-24; DH-73's exactly one reason; NUD-28's Probe and Unreachable.
New scenarios OUT-08, NUD-29 and NUD-30 each have their test.

The track drops the §5.4 departure and the renew-unroutable departure (netd's at stage 5 now),
and records DH-64's announce request, close's bound, and stage 3's missing independent oracle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L
A closed socket's stale turn is consumed by the next transmit opportunity, so a slot reused only
after one never meets it: the check that close takes its turn away (the review's P2) passed
whether or not close did. The slot is now reused at once, and the mutation that leaves the turn
behind goes red.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L
@Japabu

Japabu commented Sep 30, 2026

Copy link
Copy Markdown
Collaborator Author

Review of #635 at 02dde43, round 4.

Ready for review.

  • CI host is SUCCESS at 02dde43: run 36741388865, job 109976375976, every step success.
  • The merge 02dde43 changes nothing in the five net crates, Cargo.toml, Cargo.lock or the track relative to 6a72869: git diff 6a7286939 02dde43f9 over them is empty. The PR body's mutation runs on 6a72869 therefore stand for this head.
  • These are pure host crates, so no hardware reading is owed.
  • The branch merges cleanly onto origin/main 6e1f166.

PROVENANCE: CLEAN.

  • Checked: the delta ce7fcb9..6a72869 over the five net crates' src and tests, 703 added lines.
  • Compared against: smoltcp 0.12.0, both ToyOSOrg/smoltcp checkouts (37ff3c5, 62cf979) and the crates of netstack3 46eb29a (349 .rs files).
  • Code: 16-, 20-, 24- and 40-token shingles, identifiers normalised, with at least 3 control-flow tokens. None are shared.
  • Comments: 6-word shingles. None are shared.
  • Identifiers: the shared ones are std names and plain words. The one shared literal is US-53's expected socket order "xyzxyz". In smoltcp it is a TCP test payload, so the match is a coincidence.

Round 2's BLOCKERs. Judged on the PR body's exits on 6a72869 (baseline exit 0, 1,044 passed). None was re-run here.

  • CLOSED — close() parked accepted datagrams without a bound. P1 exits 0. N01 exits 101 with US-53 red. US-53 now asserts 1,584 udp.tx-discarded-on-close and 16 out.
  • CLOSED — close() left the slot's turn behind. P2 exits 0. N02 exits 101 with US-53 red, since 6a72869 reuses the slot before any transmit.
  • CLOSED — the acquisition exception admitted a group destination. P3 exits 0. R02 exits 101 with US-44 red.
  • CLOSED — the MUST-57 gate was untested. R09 exits 101 with IN-09 red.
  • CLOSED — the undrained-refusal bound was untested. R01, R11 and R12 exit 101 with MOD-01, US-20 and DH-38 red.
  • CLOSED — DHCP log flags came from the crate and the peer was unpinned. R03, R18, R19 and R20 exit 101 (DH-43; DH-38–58; DH-57; DH-42). R17 exits 101 with MOD-02 red.
  • CLOSED — ACD-08 read RATE_LIMIT_INTERVAL from the crate. R04 exits 101 with ACD-08 red, against the 60,000 ms literal at toyos-net-ip/tests/acd.rs:167.
  • CLOSED — §6.8's order was pinned only for FAILED. R07 exits 101 with NUD-22 red.
  • CLOSED — dhcp and udp copied the address arithmetic. Neither toyos-dhcp/src nor toyos-net-udp/src keeps any mask, prefix or class arithmetic; both read toyos_net_wire::addr. toyos-net-tcp's own predicate is the fourth BLOCKER below.

The PR body's Unsure items, judged.

  • Released datagrams waiting at egress fail NUD-28 and §6.2 as amended: the first BLOCKER.
  • toyos-net-tcp's unicast() is a second address-class predicate beside addr::is_host: the fourth BLOCKER.

Growth.

  • git diff --shortstat origin/main...HEAD: 60 files, +13,036/−163.
    • src: +5,382/−149.
    • tests: +7,580/−12.
    • Manifests, lock and issue: +74/−2.
  • Since round 2 (ce7fcb9..6a72869, net crates): src +367/−243, tests +305/−71.
  • The production growth is accepted as stage 3.
  • Deletions owed:
    • tcp's unicast() (fourth BLOCKER).
    • Once the queue lives in the entry (first BLOCKER): Control::holds, the scans in Control::take, and Released::toward.

BLOCKER

  • toyos-net-ip/src/nud.rs:60-110, src/lib.rs:455-473 — the resolved states carry no pending queue.
    • §6.2 as amended gives REACHABLE, STALE, DELAY, PROBE and UNREACHABLE "the pending queue (released datagrams only)". NUD-28 asks for compile-fail doctests showing that "a resolved entry's queue holds only released datagrams and only drains".
    • Here the resolved variants carry no queue. Released datagrams wait in egress::Control, keyed by next hop. Control::release (egress.rs:112) accepts a datagram for any next hop in any state.
    • No doctest covers NUD-28's third clause. The four queued() compile_fail blocks pin the absence of the very field §6.2 names.
    • The PR body calls NUD-28 covered; it is not.
    • Fix: put a drain-only queue in each resolved variant, moved out of INCOMPLETE's pending at resolution. Add a compile_fail block that pushes into it, beside a twin that drains it. Egress keeps only the order.
  • toyos-net-ip/src/nud.rs:345, :333 — the idle rule is pinned on one side only.
    • NUD-30 pins that a STALE entry whose lifetime passed goes when its released datagrams drain (N07, N08).
    • Nothing pins that an entry whose lifetime has not passed stays. Nothing pins the UNREACHABLE (quiescent) half of §6.3's row either.
    • I expect patches A1 and A2 (appendix) to leave every test green.
    • A1 must turn NUD-29's first arm red: assert h.is_stale(b) after the h.out() at toyos-net-ip/tests/nud.rs:565.
    • A2 needs an UNREACHABLE twin of NUD-30.
  • toyos-net-ip/tests/acd.rs:52-59 — ACD-01 reads PROBE_WAIT, PROBE_MIN, PROBE_MAX and ANNOUNCE_WAIT from the crate.
    • Its only literal is the 200 ms ceiling. IP-D1's scaled values and decision 3's 114 ms floor are pinned by nothing.
    • Patch B (appendix) halves every probing interval, giving first use at 57–100 ms. I expect it to leave ACD-01, ACD-07, ACD-09 and CLK-08 green.
    • This is the defect class round 2 sent back for ACD-08.
    • Fix: assert 28,571,428 and 57,142,857 ns, and first use ≥ 114,285,713 ns, as literals.
  • toyos-net-tcp/src/stack.rs:174 — unicast() is a second "can this address name one host" predicate beside toyos_net_wire::addr::is_host.
    • The new module calls itself "the one place a net crate reads them from".
    • unicast() admits 0/8 and 127/8 remotes, which MUST-46 calls invalid and [ip]'s route lookup refuses anyway.
    • Fix: call is_host and delete unicast(). HS-35's three addresses stay refused.

NOTE

  • toyos-net-ip/src/nud.rs:210-217 — make_room calls Control::holds for every evictable entry. Each call scans up to 64 items plus waiting, so a full table costs up to 512 scans per creation. An on-link peer drives creations with ARP requests from fresh senders. The first BLOCKER's fix makes this O(1).
  • toyos-net-ip/src/acd.rs:76 — a probe dropped by the full control queue counts as sent (probed).
    • An address can therefore be verified with no probe on the wire, against RFC 5227 §2.1's MUST.
    • This round makes it more reachable: released datagrams now fill the 64 slots and take room ahead of [ip]'s own frames, and every interface shares the queue.
    • Fix: hold the probe the way released datagrams are held, or record it in the track.
  • toyos-net-ip/src/egress.rs:244 — a request queued before its neighbour resolved still leaves once the neighbour is REACHABLE. OUT-08's first frame is a broadcast request for REACHABLE B. §6.3 sends nothing to maintain REACHABLE.

REMOVE

  • toyos-net-ip/src/egress.rs:6 — "it stays at the head of its neighbour's pending queue" is false: it waits in Control::waiting, one FIFO shared by every neighbour.
  • toyos-net-wire/src/addr.rs:1 — "the one place a net crate reads them from" is false while toyos-net-tcp/src/stack.rs:174 stands.
  • PR body, "Scenario coverage" — "Tests: toyos-net-ip 248 plus 7 doctests, …" is a count the next landing moves.
  • PR body, decision 2 — the sentence ending "(NUD-28's doctests)" contradicts §6.2 as amended.
  • PR body, "Unsure" — this review rules on both items, and the body becomes main's record.
Appendix: the mutations

Apply one with git apply, then run cargo test -p toyos-net-ip -p toyos-net-udp -p toyos-dhcp -p toyos-net-wire -p toyos-net-tcp --no-fail-fast. Each must turn a test red.

# A1: the last released datagram's leaving deletes the entry even when its lifetime has not passed
--- a/toyos-net-ip/src/nud.rs
+++ b/toyos-net-ip/src/nud.rs
@@ -342,7 +342,7 @@
 /// a deadline armed but one whose idle lifetime passed while it held them, which goes now.
 pub(crate) fn drained(i: &mut Interface, cx: &mut Cx<'_>, addr: Ipv4Addr) {
     let Some(n) = i.neighbours.get(&addr) else { return };
-    let idle = matches!(n.state, Nud::Stale(_) | Nud::Unreachable(_)) && !n.queued && cx.timers.get(timer(cx, addr)).is_none();
+    let idle = matches!(n.state, Nud::Stale(_) | Nud::Unreachable(_)) && !n.queued;
     if idle {
         remove(i, cx, addr);
         route::refresh_active(i, cx);
# A2: a quiescent UNREACHABLE entry idles out while it holds released datagrams
--- a/toyos-net-ip/src/nud.rs
+++ b/toyos-net-ip/src/nud.rs
@@ -330,7 +330,7 @@
             cx.timers.arm(timer(cx, addr), now.after(IDLE_LIFETIME));
         }
         // Not idle while released datagrams are its: `drained` deletes it once they have left.
-        Nud::Stale(_) | Nud::Unreachable(_) if cx.control.holds(cx.iface, addr) => {}
+        Nud::Stale(_) if cx.control.holds(cx.iface, addr) => {}
         Nud::Stale(_) | Nud::Unreachable(_) | Nud::Failed => {
             remove(i, cx, addr);
             route::refresh_active(i, cx);
# B: every probing interval halved (first use 57 to 100 ms)
--- a/toyos-net-ip/src/lib.rs
+++ b/toyos-net-ip/src/lib.rs
@@ -120,7 +120,7 @@
         use core::time::Duration;
 
         const fn scaled(rfc_ms: u64) -> Duration {
-            Duration::from_nanos(rfc_ms.saturating_mul(1_000_000).saturating_mul(200) / 7_000)
+            Duration::from_nanos(rfc_ms.saturating_mul(1_000_000).saturating_mul(100) / 7_000)
         }
 
         pub const PROBE_WAIT: Duration = scaled(1_000);

SEND BACK

Japabu and others added 4 commits September 30, 2026 19:11
…ir queue

The resolved states carry the pending queue §6.2 gives them. REACHABLE,
STALE, DELAY, PROBE and UNREACHABLE hold a `Released` queue that only
drains; INCOMPLETE's `Pending` is the only one with `push`, and resolution
moves it into the resolved state. NUD-28's doctests now push into each
resolved queue (compile_fail) beside a twin that pushes into INCOMPLETE's
and drains every resolved one. The control queue keeps only the order:
a released datagram's turn names its entry by address and id, and the
datagram leaves from the entry's queue when the turn comes, to the MAC of
that moment. `Control::holds`, `Control::take` and `Released::toward` are
gone, and with them the scans `make_room` ran per candidate. An evicted
entry's turns name an id no entry holds, so they leave nothing and spend
no credit; the id keeps them from giving a later entry for the same
address an earlier place, which NUD-29's second arm now pins.

The idle rule is pinned on both sides. NUD-29's first arm asserts that B
stays STALE once 1 and 2 have left, since its lifetime has not passed.
NUD-30 gains a quiescent UNREACHABLE arm beside the STALE one.

ACD-01 asserts IP-D1's scaled constants (28,571,428 and 57,142,857 ns)
and the 114,285,713 ns floor as literals.

`toyos-net-tcp`'s connect refuses a remote that `addr::is_host` refuses,
and `unicast()` is gone; HS-35's three addresses stay refused, and 0/8 and
127/8 are refused too (MUST-46).

The spec owner's ruling on the ACD NOTE: a probe or an announcement that
finds the control queue full is not counted as sent. It waits for room,
as a released datagram's turn does, so an address is never verified
without its probes on the wire. A defence waits one at a time. ACD-18,
a proposed id, fills the control queue from a second interface during
probing, and again when a defence is due.

A request is built when it leaves, from the state of that moment:
broadcast in INCOMPLETE and UNREACHABLE, unicast in PROBE, and in
REACHABLE, STALE and DELAY not at all (§6.3). OUT-08's first frame, B's
request queued before the reply, no longer leaves.

The review's REMOVEs are deleted: egress's header clause and
`toyos-net-wire::addr`'s "the one place" clause.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L
A datagram resolution released waits in its entry's queue and leaves to
the MAC the entry holds when its turn comes. NBR-08 gains the arm that
pins it: R resolves to MAC R while the control queue is full, moves to
MAC X, and the datagram leaves to MAC X. Stamping the MAC at resolution
turns it red.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L
connect now refuses what `toyos_net_wire::addr::is_host` refuses, which
takes in 0/8 and 127/8 beside HS-35's broadcast, multicast and
unspecified remotes (MUST-46). HS-35 adds 127.0.0.1 and 0.1.2.3, so the
old predicate turns it red.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L
A turn whose entry was evicted stays in the control queue until the
transmit opportunity reaches it; it leaves nothing then. Until the
eviction counted it out of the interface's `held`, so repeated
last-resort evictions behind a stalled transmit could stack dead turns
without bound. `held` now counts pending datagrams and every turn not
yet reached, and a turn is spent when it is reached, so PENDING_TOTAL
bounds them together. NUD-29 gains an arm: after B's eviction the
interface takes 62 datagrams, not 64, until B's two turns are reached.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L
@Japabu

Japabu commented Sep 30, 2026

Copy link
Copy Markdown
Collaborator Author

Review of #635 at 3bd75d3, round 5.

Ready for review.

  • CI host is SUCCESS at 3bd75d3: run 36757379267, job 110030933808, every step success.
  • The PR body's gates and all 65 mutation exits were measured on 3bd75d3: baseline exit 0 with 1,045 passed, and every mutation exit 101. None was re-run here.
  • These are pure host crates, so no hardware reading is owed.

PROVENANCE: CLEAN.

  • Checked: the delta 02dde43..3bd75d3 over the five net crates' src and tests, 435 added lines.
  • Compared against:
    • smoltcp 0.12.0;
    • ToyOSOrg/smoltcp 37ff3c5 and 62cf979;
    • netstack3 at fuchsia 49c306b, 286 .rs files. Round 4's 46eb29a can no longer be fetched by that name.
  • Code: shingles with identifiers normalised and at least 3 control-flow tokens.
    • At 16 tokens, three are shared. Each is a generic let … else { return or ? chain.
    • At 20, 24 and 40 tokens, none are shared.
  • Comments: 6-word shingles. None are shared.
  • Identifiers: the delta's own names (Turn, take_released, released_mut, request_leaves, inherit, drained, defending) appear in neither source.

Round 4's BLOCKERs.

  • CLOSED — the resolved states carried no queue.
    • Each resolved variant now holds Released, which has pop and no push.
    • Control::holds, Control::take and Released::toward are gone.
    • N12 exits 101 with NUD-28's four push doctests red.
  • CLOSED — the idle rule was pinned on one side only. A1 exits 101 with NUD-29 red. A2 exits 101 with NUD-30's UNREACHABLE twin red.
  • CLOSED — ACD-01 read its constants from the crate.
    • It now asserts 28,571,428 and 57,142,857 ns and the 114,285,713 ns floor as literals (toyos-net-ip/tests/acd.rs:46, :62).
    • B exits 101 with ACD-01 red.
  • CLOSED — tcp kept a second host predicate. unicast() is deleted and connect calls is_host (toyos-net-tcp/src/stack.rs:557). N19 exits 101 with HS-35 red.
  • Round 4's NOTEs are answered:
    • make_room's scans are gone.
    • The probe dropped by a full queue is ruling 5 (ACD-18; N13, N14).
    • The request that left early is ruling 6 (N16).

The six rulings. Each is met.

  1. The acquisition exception while probing: toyos-net-ip/src/input.rs:53, pinned by US-44's "a probing address is not usable".
  2. Deletion when the queue drains: nud.rs:448; A1, N07r, N08r.
  3. The 200 ms bound: decision 3; B. See the NOTE on rulings 3 and 5.
  4. ip.acquisition-admitted: toyos-net-udp/tests/recv.rs:207-227; N11.
  5. ACD frames held: ACD-18; N13, N14.
  6. OUT-08: nud.rs:251; N16.

The four changes beyond the brief.

  • The request form chosen at departure.
    • Required for REACHABLE by ruling 6.
    • Allowed for the rest: §6.3 sends no request in STALE, DELAY or FAILED, and §6.4 keys the form on the state.
    • N16 (OUT-08) proves the drop.
    • Nothing proves the other half: a request queued in INCOMPLETE that leaves unicast in PROBE. See the NOTE.
  • Turns carry an entry id.
    • The effect is required: §5.4's FIFO order, and §6.8's drop of an evicted entry's datagrams, which must not let a later entry's datagram take their place.
    • The mechanism is allowed. N17 (NUD-29 arm 1) proves it.
    • It exists only because eviction leaves turns behind: see the BLOCKER.
  • Evicted turns count against PENDING_TOTAL.
    • Not allowed. §6.5 refuses a new datagram only "when PENDING_TOTAL datagrams are queued across all entries".
    • NUD-29 arm 2 pins a refusal with 62 queued. N20 proves the departure itself.
    • See the BLOCKER.
  • One defence waiting at a time.
    • Allowed. §8.4 (1) owes one announcement after a defended conflict, and the one still waiting leaves after both conflicts.
    • It is what bounds defences while egress is starved. N15 (ACD-18) proves it.

Growth.

  • git diff --shortstat origin/main...HEAD: 62 files, +13,247/−170.
    • src: +5,465/−155.
    • tests: +7,708/−13.
    • Manifests, lock and issue: +74/−2.
  • Since round 4: src +283/−206, tests +169/−42.
  • Accepted: the queue moved into the entry, and three kinds of frame now wait for room.
  • Deletion owed by the BLOCKER: Neighbour::id, Interface::entries, insert's counter, Turn::id and leave's id filter.

BLOCKER

  • toyos-net-ip/src/nud.rs:310-318, src/iface.rs:28, tests/nud.rs:589-601 — the evicted entry's turns stay in the control queue and count in held until they are reached. NUD-29 arm 2 pins a refusal (nb.pending-full) with only 62 datagrams queued.
    • Why: §6.5 refuses only at PENDING_TOTAL queued datagrams. The departure is asked of the owner in the body but is not recorded in the track.
    • Fix: in remove, drop the victim's turns: one retain over items and waiting for that interface and next hop, then refill, then take them out of held.
    • The retain is exact. Every other removal happens with the entry's queue empty, so every turn for an address is its current entry's.
    • This eviction is the last-resort path only. It is not the per-candidate scan round 4 retired.
    • Then no turn outlives its entry, and the id plumbing listed under Growth goes.
    • NUD-29 arm 2 then asserts PENDING_TOTAL. "remove leaves the turns" must turn arm 1 red.
    • Decision 2's evicted-turn sentences and the PENDING_TOTAL arm under "For the spec owner" go with the fix.
    • The other exit: the owner amends §6.5, and the departure is recorded in the track.

NOTE

  • toyos-net-ip/src/nud.rs:246-258 — no test sends a request queued in INCOMPLETE out in PROBE.
    • I expect 02dde43's to: Option<MacAddr>, fixed at queueing, to leave every test green.
    • Test: B INCOMPLETE with its request queued and no credit; an assertion from B gives STALE; a send gives DELAY; at +5,000, PROBE. With credit, the queued request leaves unicast to MAC B, and no broadcast request follows.
  • toyos-net-ip/src/nud.rs:251-254, :450 — a STALE entry can be left with no deadline.
    • The sequence: a request is queued behind its turns (negative advice), then an assertion with a new MAC gives STALE, and IDLE_LIFETIME passes without a transmit. drained then refuses the entry because n.queued is set, and request_leaves drops the request without calling drained.
    • The entry then lives until it is evicted, against §6.3's delete. This has been true since round 4's request_left path.
    • Fix: call drained from the drop branch.
  • toyos-net-ip/src/nud.rs:489-503 — every released datagram stays in its entry until its turn, including those whose turn found room.
    • §6.5's last sentence keeps only the datagrams that found the control queue full. So an eviction here drops datagrams whose frames the spec would send.
    • This follows the design round 4 asked for. It is for the spec owner.
  • ACD-18 — rulings 3 and 5 meet here: the address is still tentative at 1 s while a starved control queue holds the first probe. The owner should name a starved control queue as the second exception to 200 ms.
  • toyos-net-ip/src/nud.rs:169-178 — Nud::released() has one caller, releasing(). Fold it in.

REMOVE

  • PR body, decision 3 — "so first use waits 114 to 200 ms unless §8.3's rate limit holds the first probe back" is false under ACD-18.

SEND BACK

Japabu and others added 2 commits September 30, 2026 20:36
Round 5's review upheld a BLOCKER: an evicted entry's turns stayed in the
control queue and counted against PENDING_TOTAL until the transmit
opportunity reached them, so NUD-29 pinned a refusal with 62 datagrams
queued where ip §6.5 refuses only at 64.

`remove` now drops the entry's turns from the control queue, both from
the queue itself and from those waiting for room, refills it, and takes
them out of the interface's count. Every other removal happens with the
entry's queue empty, so the turns for an address are always its current
entry's. No turn outlives its entry, and the id that told a later
entry's turns from an evicted one's goes: `Neighbour::id`,
`Interface::entries`, `insert`'s counter, `Turn::id` and `leave`'s
filter. NUD-29's second arm now accepts PENDING_TOTAL datagrams while
the control queue still holds everything it held before the eviction.
Its first arm, a later entry at B's address behind a defence, now goes
red if the turns stay: the first of them would carry the new entry's
datagram out ahead of the defence.

A request queued behind released datagrams could strand a STALE entry:
if its idle lifetime passed while the datagrams waited, the last one's
leaving kept the entry because the request was still queued, and the
request's drop at egress deleted nothing. `request_leaves` now calls
`drained` when it drops a request, and `drained` checks the released
queue itself, so `leave` calls it after every datagram rather than
only after the last. NUD-30's third arm pins the deletion.

OUT-08's second arm pins the other half of the request form chosen at
departure: a request queued in INCOMPLETE and still queued when the
entry reaches PROBE leaves unicast to the cached MAC, and it is the
only request that leaves.

`Nud::released()` had one caller and is folded into `releasing()`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L
@Japabu

Japabu commented Sep 30, 2026

Copy link
Copy Markdown
Collaborator Author

Review of #635 at 8ea5e93, round 6.

Ready for review.

  • CI host is SUCCESS at 8ea5e93: run 36769980883, job 110073599404, every step success.
  • The PR body's gates and all 68 mutation exits were measured on 8ea5e93: baseline exit 0 with 1,045 passed, and every mutation exit 101. None was re-run here.
  • These are pure host crates, so no hardware reading is owed.
  • The merge 0610a07 changes nothing in the five net crates, the manifests, the lock or the track: git diff 3bd75d3bc 0610a0730 over them is empty.
  • The branch merges cleanly onto origin/main 815e089.

PROVENANCE: CLEAN.

  • Checked: the delta 3bd75d3..8ea5e93 over the five net crates' src and tests, 89 added lines.
  • Compared against:
    • smoltcp 0.12.0;
    • ToyOSOrg/smoltcp 37ff3c5 and 62cf979;
    • netstack3 46eb29a.
    • Together these are 582 .rs files.
  • Code: shingles with identifiers normalised.
    • With at least 3 control-flow tokens, none are shared at 16, 20 or 24 tokens.
    • Without that filter, two 20-token shingles are shared. Both are let …; assert!(… test chains. None are shared at 24.
  • Comments: 6-word shingles. None are shared.
  • Identifiers:
    • drop_turns, drained, request_leaves, released_mut and take_released appear in neither source.
    • releasing appears only as an English word, in two netstack3 TCP comments.

Round 5's BLOCKER.

  • CLOSED — an evicted entry's turns stayed queued and counted against PENDING_TOTAL.
    • remove now drops them and takes them out of held (toyos-net-ip/src/nud.rs:309-310).
    • N21 exits 101 with NUD-29's first arm red.
    • N22 exits 101 with the second arm red at 62. That arm now asserts PENDING_TOTAL.
    • Neighbour::id, Interface::entries, Turn::id and leave's id filter are gone.

Round 5's NOTEs.

  • Answered: OUT-08's second arm sends a request queued in INCOMPLETE out unicast in PROBE. N23 exits 101.
  • Answered: request_leaves calls drained, and NUD-30's third arm pins the deletion. N24 exits 101. See the NOTE on n.queued below.
  • Answered: drained reads the released queue itself. N25 exits 101.
  • Ruled: released datagrams stay in their entry until each one's turn (ruling 7). leave pops them there (nud.rs:424-434).
  • Ruled: a starved control queue is the second exception to 200 ms (ruling 3). Decision 3 now names both exceptions, and its false sentence is gone.
  • Done: Nud::released() is folded into releasing().

Growth.

  • git diff --shortstat origin/main...HEAD: 62 files, +13,289/−170.
    • src: +5,468/−155.
    • tests: +7,747/−13.
    • Manifests, lock and track: +74/−2.
  • Since round 5, in the net crates: src +45/−42, tests +44/−5.
  • Accepted: the id plumbing went, and one retain took its place.

BLOCKER

  • toyos-net-ip/src/egress.rs:138 — no test pins drop_turns's t.iface == iface. — Every remove runs drop_turns, including idle expiry and the end of a FAILED hold-down. Without the check, removing X on if0 also drops if1's turns for X. if1's datagrams then never leave. if0's held is cut by turns that were never its own. if1's held keeps them until its link goes down, because an eviction later finds no turns to subtract. No test holds turns for one address on two interfaces. Patch M-a (appendix) must turn a new NUD-29 arm red. The arm: use RTE-004's overlapping prefixes and put X on both interfaces. if1's X holds released datagrams whose turns wait. if0's X is then removed when its FAILED hold-down ends. A transmit must send if1's datagrams to X's MAC on if1.
  • toyos-net-ip/src/egress.rs:146 — no test pins drop_turns's refill. — NUD-29 evicts B only while B's turns wait outside the FIFO, so the retain never frees room in it. Without the refill, room freed in items goes to the next frame pushed, ahead of what waited (egress.rs:6-8, OUT-08). Patch M-b must turn a new NUD-29 arm red. The arm: the other entries are REACHABLE, and B resolves before the queue fills, so its turns are in the FIFO. 62 replies then fill it, and a defence waits. The send to the new neighbour evicts B. The defence must leave before the new entry's request.

NOTE

  • toyos-net-ip/src/nud.rs:441 — || n.queued keeps a drained entry whose idle lifetime passed until a request queued behind its datagrams is taken out.
    • Ruling 2 deletes the entry at drain.
    • fire (nud.rs:414-417) already deletes an idle entry whose request still waits. The tree therefore has two idle rules.
    • For UNREACHABLE, the fix loses the request a send queued, along with the entry. fire already loses it the same way when the deadline comes first.
    • This term is the only reason request_leaves calls drained (nud.rs:246).
    • NUD-30's third arm cannot fail on it. With patch M-c, B goes as 2 leaves, the orphaned request finds no entry, and every assertion still holds.
    • Fix: delete the term and that call, and assert that B is gone as 2 leaves. Decision 2's "or when a request queued behind them is taken out" goes with them.

REMOVE

  • PR body, "Gates" — "This round's arms went into existing tests, so the count is unchanged." This is review-round narration in main's record.
Appendix: the mutations

Apply one with git apply, then run cargo test -p toyos-net-ip -p toyos-net-udp -p toyos-dhcp -p toyos-net-wire -p toyos-net-tcp --no-fail-fast. M-a and M-b must each turn a test red. M-c is the NOTE's evidence. Each one applies to 8ea5e93 (git apply --check).

# M-a: drop_turns ignores the interface
--- a/toyos-net-ip/src/egress.rs
+++ b/toyos-net-ip/src/egress.rs
@@ -135,7 +135,7 @@
     pub fn drop_turns(&mut self, iface: IfIndex, next_hop: Ipv4Addr) -> usize {
         let mut dropped = 0usize;
         let mut keep = |item: &Item| {
-            let theirs = matches!(item, Item::Turn(t) if t.iface == iface && t.next_hop == next_hop);
+            let theirs = matches!(item, Item::Turn(t) if t.next_hop == next_hop);
             if theirs {
                 dropped = dropped.saturating_add(1);
             }
# M-b: the room drop_turns frees is not refilled
--- a/toyos-net-ip/src/egress.rs
+++ b/toyos-net-ip/src/egress.rs
@@ -143,7 +143,6 @@
         };
         self.items.retain(&mut keep);
         self.waiting.retain(&mut keep);
-        self.refill();
         dropped
     }
 
# M-c: a queued request does not keep a drained entry
--- a/toyos-net-ip/src/nud.rs
+++ b/toyos-net-ip/src/nud.rs
@@ -438,7 +438,7 @@
 /// datagrams, which goes once nothing of it waits there.
 fn drained(i: &mut Interface, cx: &mut Cx<'_>, addr: Ipv4Addr) {
     let Some(n) = i.neighbours.get(&addr) else { return };
-    let waits = n.state.releasing() || n.queued;
+    let waits = n.state.releasing();
     let idle = matches!(n.state, Nud::Stale(_) | Nud::Unreachable(_)) && !waits && cx.timers.get(timer(cx, addr)).is_none();
     if idle {
         remove(i, cx, addr);

SEND BACK

Ruling 2 deletes an entry whose idle lifetime passed while its queue
held released datagrams at drain. `drained` also kept it while a request
of its waited in the control queue (`|| n.queued`), which gave the tree
a second idle rule beside `fire`'s, which already deletes an idle entry
whose request still waits. The term goes, and with it the only reason
`request_leaves` called `drained`. A request that outlives its entry
finds none when it leaves and sends nothing, as it does after `fire`.

NUD-30's third arm now takes 1 and 2 out alone and asserts that B is
gone as 2 leaves, before the request behind them is taken out.

NUD-29 gains two arms for `Control::drop_turns`, which no test pinned:

- RTE-004's overlapping prefixes put X on both interfaces. X on if1
  holds 1 and 2 whose turns wait when X on if0 ends its FAILED
  hold-down; if1's datagrams must still leave to X on if1. This pins
  the `t.iface == iface` filter.
- B resolves while the control queue has room, so its turns are inside
  it; 62 replies fill it behind them and a defence waits. Evicting B
  frees their room, which is the defence's: it must leave ahead of the
  new entry's request. This pins the `refill` in `drop_turns`.

The two arms and the three earlier ones share their setup through
`b_and_511`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L
@Japabu

Japabu commented Sep 30, 2026

Copy link
Copy Markdown
Collaborator Author

Review of #635 at c1474cd, round 7. Scope: git diff 8ea5e93f6 c1474cd48.

Ready for review.

  • CI host is SUCCESS at c1474cd: run 36782250417, job 110115103055. The run's headSha is c1474cd, and none of its 6 steps failed.
  • The PR body's gates and all 70 mutation exits were measured on c1474cd: baseline exit 0 with 1,045 passed, and each mutation exit 101. I re-ran none of them.
  • These are pure host crates, so no hardware reading is owed.
  • The branch merges cleanly onto origin/main e754dc8: git merge-tree --write-tree exits 0.

PROVENANCE: CLEAN for this round's delta (72 added lines).

  • Compared against smoltcp 0.12.0, ToyOSOrg/smoltcp 37ff3c5 and 62cf979, and netstack3 at fuchsia main 1a0ae36 (286 .rs files). That is 634 files in all.
  • Code: shingles with identifiers normalised.
    • With at least 3 control-flow tokens, none are shared at 12, 16, 20 or 24 tokens.
    • Without that filter, 1 is shared at 24 tokens and 29 at 20. Each is a let x = Ipv4Addr::new(n, n, n, n); line, a use …::{…} list or a let …; assert!(…) chain.
  • Comments: none of the 143 six-word shingles are shared.
  • Identifiers: b_and_511, drop_turns and request_leaves appear in neither source. drained appears only as an English word, in four netstack3 comments.

Round 6's BLOCKERs.

  • CLOSED — no test pinned drop_turns' t.iface == iface (toyos-net-ip/src/egress.rs:138).
    • The new arm is at toyos-net-ip/tests/nud.rs:630-652. It uses RTE-004's prefixes (checked against tests/addr.rs:214-222). X on if1 is REACHABLE, holding 1 and 2 with their turns queued. X on if0 goes when its FAILED hold-down ends at 23,000.
    • M-a, the review's patch as written, exits 101 at tests/nud.rs:652.
    • Reading the code gives the same cause: without the check, remove(if0, X) drops if1's two turns, so 1 and 2 never leave.
  • CLOSED — no test pinned drop_turns' refill (egress.rs:146).
    • The new arm is at tests/nud.rs:616-628. B's two turns are inside the FIFO, 62 replies sit behind them, and a defence waits. The send to neighbour(9000) evicts B.
    • M-b exits 101 at tests/nud.rs:627.
    • Reading the code gives the same cause: without the refill, the new entry's request takes the freed room, and the defence enters behind it at the first pop.
    • Beyond the review's two patches, I expect no single-term mutation of drop_turns to survive:
      • deleting next_hop drops B's waiting turns in arm 0 (to_b);
      • deleting the waiting retain is N21 (arm 1);
      • deleting the items retain leaves the FIFO full, so the request is dropped and out.len() is 63 (:626);
      • the subtraction from held is N22.

Round 6's NOTE: done.

  • drained (src/nud.rs:438-445) no longer reads n.queued, and request_leaves' drop branch (:244-247) no longer calls it.
  • NUD-30's third arm now asserts that B is gone as 2 leaves (tests/nud.rs:713-715) and that the orphaned request sends nothing (:716).
  • Mc-r restores both the term and the call. It exits 101 on NUD-30.
  • The deleted call was dead once the term went. A STALE or UNREACHABLE entry with no deadline and an empty queue has already been removed by leave's call, so no path is lost.
  • Decision 2 no longer carries the request clause.

Growth.

  • This round: src +4/−6, tests +68/−19.
  • git diff --shortstat origin/main...HEAD: 62 files, +13,336/−170.
    • src: +5,465/−152.
    • tests: +7,796/−13.
    • Other: +75/−5.
  • Production shrank this round. The tests grew by two arms and a setup helper with two callers. Accepted.

BLOCKER

None.

NOTE

  • toyos-net-ip/src/nud.rs:438 — drained now has one caller, leave (:431). — An abstraction with one caller: fold its condition into leave and drop its doc comment and its second map lookup. The comment at :412 names it.
  • toyos-net-ip/tests/nud.rs:532 — b_and_511 names a count that limits::nud::TABLE_MAX owns. — The loop reads the limit but the name does not, so the name goes false when the limit moves.
  • toyos-net-ip/src/nud.rs:240, :305-313 — a Request item can outlive its entry, and a later entry at the same address can spend it. — remove drops the entry's turns but not its queued request. This delta adds drained as a third way an entry goes with its request still queued; fire and eviction were the first two. A new INCOMPLETE entry at that address sets queued, so the orphan leaves as its request, at the orphan's earlier FIFO slot, and the new entry's own item then sends nothing. The request's count, form and spacing still come from the live entry, so only §5.4's order is off. This is round 5's turn defect in a request. Fix: remove also drops Item::Request { iface, target: addr }, which does not count toward held, so no item outlives its entry. Or record it in the track.

REMOVE

None.

LAND AFTER NAMED CHANGES

Japabu and others added 2 commits October 1, 2026 00:16
`drained` had one caller, `leave`, which already holds the entry. Its
condition moves into `leave` over that borrow, so the second map lookup
and the doc comment go, and `fire`'s comment names `leave` instead.
Behaviour is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L
`b_and_511` stated a count that `limits::nud::TABLE_MAX` owns: its loop
reads the limit, and the name and its doc comment went false the moment
the limit moved. `b_in_a_full_table` and its doc name what it builds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L
`remove` dropped an entry's turns but left its queued `Item::Request`.
`fire`, eviction and `leave` each delete an entry whose request may
still wait, and a later INCOMPLETE entry at the same address sets
`queued` again. The orphan then left as that entry's request, from its
own earlier slot in the FIFO, and the new entry's item sent nothing.
Count, form and spacing still came from the live entry; ip.md §5.4's
order did not.

`Control::drop_turns` becomes `purge_entry`: it drops the entry's
requests with its turns, and still returns only the turns, since a
request never counted toward `held`. No item outlives its entry.

NUD-30's third arm goes on past B's deletion: a host asks for A, which
queues a reply, and then a datagram to B makes a new entry there. The
reply must leave first and the new entry's request after it. Without
this change the orphan leaves as B's request ahead of the reply, and
the test fails at the reply's assertion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L
@Japabu
Japabu added this pull request to the merge queue Oct 1, 2026
Merged via the queue into main with commit 9efafcc Oct 1, 2026
1 check passed
@Japabu
Japabu deleted the wt/toyos-netstack branch October 1, 2026 02:05
Japabu added a commit that referenced this pull request Oct 1, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L
Japabu added a commit that referenced this pull request Oct 1, 2026
…toyos-noredlist

#588 landed first, and its `usb_stick_left` arms `usb-transport-break`,
`usb-reset-moves`, `usb-reset-moves-after` and `usb-reset-moves-configured`
again, its T14 row (`usbbreak`) arms `usb-transport-break`, and its metal
record carries the `usbbreak` rows. Those stay, with `msc::transport_break`,
`msc::reset_moves`, `RECOVERING` and `Quiet::Staged`. Everything else this
branch deleted with `usb_transport_break` (90ecefe, b5c59cb, 402107b,
9ebf080) stays deleted: no test on either side arms it.

Conflicts, every hunk:

- `kernel/src/actuator.rs`, two hunks. First: main's eight USB rows. Kept
  `usb_transport_break`, `usb_reset_moves`, `usb_reset_moves_after` and
  `usb_reset_moves_configured`, with `usb_reset_moves`' doc losing its
  citation of the deleted `usb_transport_break`; dropped
  `usb_transport_offline`, `usb_reset_break`, `usb_transport_break_owed` and
  `usb_transport_break_flushed`, armed only by the deleted
  `usb_transport_break`, `log_flush_retry` and `partition_claim_departure`.
  Second: dropped `watch_window` (`blocking_read_window`, deleted in
  dc62efc), kept #634's `handler_post`.
- `kernel/src/drivers/xhci/wait/msc.rs`, fourteen hunks, misaligned by #588's
  rewrite of the file. Resolved as main's file with this branch's deletions
  applied to it: `Asks` and `bot`'s `asks` argument at its five call sites;
  `transport_break`'s owed and flushed half (`WROTE`, `FLUSHED`,
  `AFTER_A_FLUSH`, `wrote`, `flushed`, their calls in `MscDevice::wrote` and
  the flush, and `arm`'s argument), so `arm` reads `usb-transport-break`
  alone; `return_silent` and its hook in `served`; `reset_break` and the
  wrapper around `climb`, whose body `climb_until_in_step` carries again;
  `short_read` and its hold and release around the data phase; `staged` and
  its checks in `bot`, the bad signature, the withheld CBW, the gone port and
  the unanswered wait; the bind's `slow_return` stall and INQUIRY staging, and
  `slow_return` itself; `read_serial`'s short ask. `reset_moves`, its three
  holds and `RECOVERING` are main's as they stand.
- `kernel/src/watch.rs`, one hunk: dropped the `window` module
  (`watch-window`), kept #634's `handler_post` module.
- `toyos-sched/src/watch.rs`, one hunk: the same, `window` dropped and
  `handler_post` kept.
- `src/lib.rs`, one hunk: kept #652's `pub mod n2`, dropped `pub mod redlist`.
- `tests/common/power.rs`, one hunk: main's only change there renamed
  `transport_break_chain`'s doc to `usb_stick_left`; its one caller is the
  deleted `usb_transport_break`, so it stays deleted.
- `tests/common/usb.rs`, one hunk holding main's whole `usb_transport_break`
  region. Kept #588's `Held`, `usb_stick_left`,
  `a_stick_that_left_under_its_rung`, `transport_break_on_metal`,
  `transport_break_recovered` (which `tests/checks/usb.rs` judges) and
  `broke_on`; the rest is `usb_transport_break`'s and stays deleted:
  `a_read_whose_first_wait_spent_its_budget_goes_out_again`,
  `serial_short_is_not_read`, `cpu_of`, `line_with`, `Moved`,
  `a_stick_its_reset_moved_carries_on`, `abandoned_write_is_taken_offline`,
  `no_command_was_refused`, `port_gone_is_left_to_the_teardown`,
  `control_requests`, `command_blocks`,
  `every_reset_is_followed_by_a_test_unit_ready`, `is_a_rungs_configuration`,
  `every_port_reset_is_followed_by_a_test_unit_ready`,
  `every_reset_is_followed_by_both_clears` and `transport_gives_up`.
- `tests/toyos.rs`, five hunks. `MACHINE_TESTS`: kept `usb_stick_left` and
  #634's `handler_post_without_a_pass`; dropped `usb_transport_break`,
  `xhci_full_speed_device` (5e42235), `blocking_read_window` and
  `user_copy_races_munmap` (7ea6be1). `METAL`: kept #588's `usb_stick_left`
  row. Dispatch: kept `usb_stick_left` and `handler_post_without_a_pass`;
  dropped `usb_transport_break`, `xhci_full_speed_device` and
  `smp_failed_ap_leaves_no_hole` (aedcf17).

Outside the conflicts:

- `kernel/src/drivers/xhci/wait/mod.rs`: `Quiet::Staged` and its line come
  back, which `transport_break::take` answers with; the auto-merge had taken
  9ebf080's deletion. The `reset_break` and `return_silent` hooks stay
  deleted.
- `src/metal.rs`: `FLASHABLE`'s `usb-transport-break` row comes back
  (b5c59cb took it), since `usb_stick_left`'s T14 arm flashes it.
- `issues/kernel/a-held-disk-waits-for-a-pass-no-cpu-takes-when-every-cpu-is-in-a-call-on-it.md`:
  records `4f2bea143`, this merge's second parent, as the tree holding #588's
  version of `usb_transport_break`, in place of the instruction to record it.
- `issues/boot-media/logd-ends-the-boots-log-on-one-refused-create-and-nothing-durable-says-so.md`:
  drops its citation of `power::transport_break_chain`, deleted in 90ecefe.

The `rust` gitlink is main's.

On this tree: `cargo run -- --build-only` exit 0; the kernel's `cargo check`
for x86-64 and AArch64, bare, with `boot-actuators` and with
`boot-actuators,test-actuators`, exit 0 each; `cargo test --test toyos-build
--no-run` exit 0, no warning.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016t9wjdQkB8SH7bmfUoiy6L
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant