diff --git a/doc/traffic-manager.md b/doc/traffic-manager.md new file mode 100644 index 0000000..02d3b8c --- /dev/null +++ b/doc/traffic-manager.md @@ -0,0 +1,361 @@ +# Traffic Manager: VOQ + fabric + strict-priority egress + +This document describes the high-fidelity switch Traffic Manager (TM) added to +the P4sim V1model core, the switch/channel changes that support it, the +configuration surface, and the examples and tests used to validate it. + +The TM replaces the old output-only priority scheduler (`NSQueueingLogicPriRL`) +with an input-buffered **Virtual Output Queue (VOQ)** stage, a **fabric +scheduler** that matches inputs to outputs, and a per-output **strict-priority +egress** stage that serialises frames onto the wire at line rate. + +The whole path is **opt-in and additive**: with `EnableVoqFabric=false` (the +default) the switch behaves exactly as before. Nothing on the legacy path was +removed. + +--- + +## 1. Concepts + +Three orthogonal ideas are easy to conflate, so we separate them explicitly: + +| Concept | What it controls | Where it lives in the TM | +| --- | --- | --- | +| **VOQ** | *Queue organisation / buffering* — which queue a packet enters and where it waits on the **input** side, indexed per `[inPort][outPort][priority]`. Avoids head-of-line blocking. | `EnqueueToVoq()` + `m_voq` | +| **Fabric scheduling** | *Matching* — when several VOQs want to move, which input is granted to which output this round. | `RunFabricScheduler()` (priority-first maximal matching) | +| **Strict Priority (SP)** | *Scheduling on the output* — when several egress queues of a port hold packets, which priority is served first. | `SelectEgressPriority()` | +| **FIFO** | *Ordering within one queue* — packets leave a single queue in arrival order. | `std::deque` per `[out][prio]` | + +The egress side is therefore **strict priority across the 8 priority queues of a +port, FIFO within each queue** — i.e. per output port it behaves like a `pfifo` +(strict-priority + FIFO) scheduler. Priority is a 3-bit field, so there are 8 +levels; **higher value = higher priority** (7 is highest). + +--- + +## 2. Datapath + +``` + ingress pipeline + │ (outPort, priority chosen by the P4 program) + ▼ + EnqueueToVoq ──► VOQ[inPort][outPort][priority] (input-buffered) + │ + ▼ + fabric scheduler (priority-first + grants in→out maximal matching) + │ + ▼ + egress[outPort][priority] (per-port SP + FIFO) + │ + ▼ + egress pipeline + deparse + │ + ▼ + TransmitCallback ──► ns-3 NetDevice ──► wire + ▲ │ + └──── NotifyEgressTxComplete ◄───────┘ + (frame finished serialising) +``` + +### Fabric policy + +The default matching is **priority-first maximal matching**: for priority 7 down +to 0, for each still-unused input port, grant the first unused output port that +has a non-empty VOQ at that priority. One input grants at most one packet and one +output receives at most one packet per round. The policy is a single overridable +method (`DoRunFabricScheduler()`) so iSLIP / round-robin can be dropped in +without touching the rest of the TM. + +### Event-driven vs manual + +- **Manual (default object mode):** call `RunFabricScheduler()` / + `DequeueFromVoq()` yourself; nothing touches the ns-3 event queue. Used by the + low-level unit tests. +- **Event-driven (`EventDriven=true`):** `EnqueueToVoq()` arms a self-clocking + fabric+egress loop via `Simulator::Schedule` (no threads) that moves packets + VOQ → egress → wire, honouring the fabric/port rates and the + arbitration/pipeline delays. This is the mode the V1model core uses. + +### Completion-driven egress and the channel signal + +By default the egress scheduler self-clocks each port's serialisation at +`PortRate`. With `EgressCompletionDriven=true` (the mode the core uses) the flow +is decoupled instead: + +1. `EgressServiceEvent()` hands the frame to the `TransmitCallback` **first**, + then waits — it does **not** immediately count the frame or re-arm the port. +2. The datapath (the switched-Ethernet channel / PHY) runs the egress pipeline + and decides what became of the frame, then calls + `NotifyEgressTxComplete(outPort, outcome)` with a `TmTxOutcome`. +3. Only on that signal does the TM account for the frame by outcome, clear the + in-flight slot, and serve the next frame: + - `TRANSMITTED` — the frame reached the wire: count it toward + `totalTransmitted`, `perPriorityTransmitted`, `perPortTxBytes`. + - `DROPPED` — the egress pipeline `drop()`ped it, or the datapath could not + send it (channel busy / bad port): count it as a drop + (`totalDropped`, `dropsByReason[EGRESS_POST_DEQUEUE_DROP]`). + - `RECIRCULATED` — the frame went back to ingress instead of onto the wire: + count it in `totalRecirculated`. It is sent for real on a later pass, so + counting it as transmitted here would double-count it. + +This separates *"which packet goes next"* (the TM's decision) from *"when the +wire is free"* (the PHY's), and guarantees `totalTransmitted` counts a frame only +**after** it is actually on the wire — never a frame that was dropped or +recirculated after leaving the egress queue. + +The accounting is split accordingly: **queue-residence** work (buffer release, +egress/total delay, egress-dequeue trace) happens at dequeue; **on-wire** +counters happen on completion. + +### Switched-Ethernet channel: propagation no longer blocks the next frame + +`SwitchedEthernetChannel` models a full-duplex link with independent per-source +state. The sender is freed after **serialisation** (`txTime`), not after +**propagation** (`txTime + delay`): once a frame's last bit has left the port the +port returns to `IDLE` and may start the next frame, while the in-flight frame +keeps propagating toward the receiver. `GetState()` still reports `PROPAGATING` +for observers, and `IsBusy()` (which gates `TransmitStart`) is true only while a +frame is actively serialising. This keeps back-to-back frames flowing at line +rate instead of paying the propagation delay per frame. + +--- + +## 3. Enabling the TM + +Set one attribute on the P4 switch device before the simulation starts: + +```cpp +P4Helper p4; +p4.SetDeviceAttribute("JsonPath", StringValue(jsonPath)); +p4.SetDeviceAttribute("FlowTablePath", StringValue(flowTablePath)); +p4.SetDeviceAttribute("P4SwitchArch", UintegerValue(0)); // V1model +p4.SetDeviceAttribute("EnableVoqFabric", BooleanValue(true)); // ← opt in +``` + +The core then creates the TM at start-up, sizes it to the number of attached +ports, seeds `PortRate` from the egress channel's `DataRate`, wires the +`TransmitCallback` to the egress send path, and enables `EventDriven` + +`EgressCompletionDriven`. The TM is disposed before the core is destroyed. + +--- + +## 4. Configuration surface + +All knobs are ns-3 attributes on `ns3::P4TrafficManager` (buffer limits are in +**bytes**; `0` means unlimited): + +| Attribute | Type | Default | Meaning | +| --- | --- | --- | --- | +| `NumPorts` | uint32 | 0 | Number of ports N; allocates N·N·8 VOQs. Set by the core. | +| `GlobalBufferLimit` | uint64 | 0 | Total bytes across VOQ + egress. | +| `InputBufferLimit` | uint64 | 0 | Per-input-port bytes. | +| `VoqLimit` | uint64 | 0 | Per-VOQ `[in][out][prio]` bytes. | +| `EgressPortLimit` | uint64 | 0 | Per-output-port egress bytes. | +| `EgressQueueLimit` | uint64 | 0 | Per-egress-queue `[out][prio]` bytes. | +| `PortRate` | DataRate | 1Gbps | Output-port serialisation rate. | +| `FabricRate` | DataRate | 10Gbps | Fabric transfer rate. | +| `IngressPipelineDelay` | Time | 0 | Fixed ingress processing delay (applied before the fabric round). | +| `FabricArbitrationDelay` | Time | 0 | Fixed fabric arbitration delay per round. | +| `EgressPipelineDelay` | Time | 0 | Fixed egress processing delay. | +| `EventDriven` | bool | false | Self-clock the fabric+egress loop via ns-3 events. | +| `EgressCompletionDriven` | bool | false | Wait for `NotifyEgressTxComplete()` before counting a frame transmitted (requires `EventDriven`). | + +Switch-level knob (on `ns3::P4SwitchNetDevice`): + +| Attribute | Type | Default | Meaning | +| --- | --- | --- | --- | +| `EnableVoqFabric` | bool | false | Route egress through the VOQ+fabric TM instead of the legacy output-queued path. | + +--- + +## 5. Statistics, traces, and drop reasons + +`GetStats()` returns a cumulative `TmStats`: + +| Counter | Meaning | +| --- | --- | +| `totalReceived` | packets offered to `EnqueueToVoq` | +| `totalVoqEnqueued` | accepted into a VOQ | +| `totalMovedToEgress` | dequeued from VOQ (granted by the fabric) | +| `totalEgressEnqueued` | accepted into an egress queue | +| `totalTransmitted` | serialised onto the wire (counted **after** transmit) | +| `totalRecirculated` | recirculated after egress dequeue (not transmitted) | +| `totalDropped` | dropped for any reason | +| `dropsByReason[6]` | per-`TmDropReason` breakdown | +| `perPriorityTransmitted[8]` | transmitted per priority level | +| `perPortTxBytes[]` | bytes transmitted per output port | +| `AvgVoqDelay/AvgEgressDelay/AvgTotalDelay`, `maxQueueingDelay` | delay accumulators | + +Trace sources: `VoqEnqueue`, `VoqDequeue`, `EgressEnqueue`, `EgressDequeue`, +`Drop`, `VoqWaitingDelay`, `EgressWaitingDelay`, `TotalDelay`. + +Drop reasons (`TmDropReason`). Reasons 1–5 are admission drops, checked in +admission order; reason 6 is a post-dequeue drop reported by the datapath: + +1. `VOQ_GLOBAL_BUFFER_FULL` — global buffer would overflow +2. `VOQ_INPUT_BUFFER_FULL` — per-input-port buffer would overflow +3. `VOQ_QUEUE_FULL` — the target `VOQ[in][out][prio]` would overflow +4. `EGRESS_PORT_BUFFER_FULL` — per-output-port egress buffer would overflow +5. `EGRESS_QUEUE_FULL` — the target egress queue would overflow +6. `EGRESS_POST_DEQUEUE_DROP` — dropped after egress dequeue: the egress + pipeline `drop()`ped the frame, or the datapath could not send it + +--- + +## 6. Validation + +Three levels of validation were run: a unit test suite (TM logic in isolation), +an end-to-end integration example (real P4 program through the switch), and a +throughput benchmark (goodput vs line rate). All results below are from the +current branch. + +### 6.1 Unit test suite — `test/p4-traffic-manager-test-suite.cc` + +9 QUICK cases covering enqueue/dequeue, priority scheduling, VOQ isolation for a +shared output, fabric matching, finite-buffer drops, delay measurement, +event-driven drain, egress strict priority, and egress drop: + +``` +$ ./test.py -s p4-traffic-manager +[1/1] PASS: TestSuite p4-traffic-manager +1 of 1 tests passed (1 passed, 0 skipped, 0 failed, 0 crashed, 0 valgrind errors) +``` + +Cases: `TmBasicEnqueueDequeueTest`, `TmPrioritySchedulingTest`, +`TmVoqSameOutputTest`, `TmFabricMatchingTest`, `TmBufferDropTest`, +`TmDelayMeasurementTest`, `TmEventDrivenDrainTest`, `TmEgressStrictPriorityTest`, +`TmEgressDropTest`. + +### 6.2 Integration example — `examples/p4-voq-fabric-integration.cc` + +Two hosts and one V1model switch running the `simple_v1model` IPv4-forwarding +program. The example is self-validating (non-zero exit on failure) and runs both +datapaths so the additive contract is checked directly. + +``` +$ ./ns3 run "p4-voq-fabric-integration --run=voq" + rxBytes=296000 tmPresent=1 tmReceived=298 tmVoqEnqueued=298 tmTransmitted=298 tmDropped=0 + [PASS] V1model core exists + [PASS] Traffic Manager created when EnableVoqFabric=true + [PASS] Sink received data over the VOQ datapath + [PASS] Packets entered a VOQ + [PASS] TM serialised packets onto the wire + [PASS] Transmitted count does not exceed VOQ-enqueued count + [PASS] VOQ-enqueued count does not exceed offered count +=== ALL CHECKS PASSED (0 failure(s)) === + +$ ./ns3 run "p4-voq-fabric-integration --run=legacy" + rxBytes=296000 tmPresent=0 + [PASS] V1model core exists + [PASS] No Traffic Manager created when disabled (additive contract) + [PASS] Sink received data over the legacy datapath +=== ALL CHECKS PASSED (0 failure(s)) === +``` + +Both datapaths deliver the same 296 000 bytes; the VOQ path additionally shows +`received == voqEnqueued == transmitted` with zero drops (nothing lost inside the +TM), and the legacy path confirms no TM is created when the feature is off. + +### 6.3 Throughput benchmark — `examples/p4-voq-fabric-throughput.cc` + +A single saturating UDP flow (offered at 1.2× the egress line rate) is pushed +host0 → switch → host1. The topology uses a fast ingress link (10 Gbps) so the +sender NIC is never the limiter, and the switch egress port is the sole +bottleneck. Goodput at the sink is compared against the link's line rate; the +header-overhead ceiling is `payload / (payload + 14 + 20 + 8)` ≈ 97.09 % for a +1400-byte payload. + +``` +$ ./ns3 run "p4-voq-fabric-throughput --linkRate=100Mbps" + [TM] received=2059 voqEnq=2059 transmitted=2059 dropped=0 + goodput=97.13 Mbps of 100.00 Mbps line (97.13% of line; header-overhead ceiling ~97.09%) + [PASS] goodput >= 80.00% of line rate + +$ ./ns3 run "p4-voq-fabric-throughput --linkRate=1000Mbps" + [TM] received=20574 voqEnq=20574 transmitted=20574 dropped=0 + goodput=970.92 Mbps of 1000.00 Mbps line (97.09% of line; header-overhead ceiling ~97.09%) + [PASS] goodput >= 80.00% of line rate +``` + +| Egress line rate | Goodput | % of line | Header-overhead ceiling | Drops | +| --- | --- | --- | --- | --- | +| 100 Mbps | 97.13 Mbps | 97.13 % | ~97.09 % | 0 | +| 1000 Mbps | 970.92 Mbps | 97.09 % | ~97.09 % | 0 | + +The delivered goodput sits right at the header-overhead ceiling at both rates, +confirming the completion-driven egress serialises at true line rate with no +artificial timer bottleneck and no internal loss. + +### 6.4 Strict-priority demo — `examples/p4-voq-fabric-priority.cc` + +Two saturating UDP flows from two **separate sender hosts** converge on one +receiver through a switch running the `qos` P4 program, which classifies by UDP +destination port (`dport 4000 → priority 3` HIGH, `dport 2000 → priority 1` +LOW). Each flow enters on its own ingress port — so each has its own host NIC +and its own VOQ — and they contend only inside the switch, at the shared +oversubscribed output port. A finite egress buffer turns the excess into +Traffic-Manager drops rather than unbounded queueing. + +``` +$ ./ns3 run "p4-voq-fabric-priority" + egressLink=100Mbps perFlow=0.7x egress line (combined 1.4x) + HIGH=dport 4000 (prio 3) from host0 LOW=dport 2000 (prio 1) from host1 + HIGH: rx=1678600 B ~67.14 Mbps (offered ~70.00 Mbps, retained 95.92%) + LOW : rx=739200 B ~29.57 Mbps (offered ~70.00 Mbps) + [TM] received=2421 transmitted prio3=1199 prio1=528 dropped=690 + [PASS] Both priority classes carried some traffic + [PASS] HIGH priority delivered more than LOW under congestion + [PASS] TM transmitted more prio-3 frames than prio-1 frames + [PASS] HIGH priority protected (retained offered load) + [PASS] TM dropped the excess low-priority load (port oversubscribed) +=== STRICT PRIORITY OBSERVED (0 failure(s)) === +``` + +| Class | Priority | Offered | Delivered | Result | +| --- | --- | --- | --- | --- | +| HIGH | 3 | ~70 Mbps | ~67.1 Mbps (95.9 %) | protected — served in full | +| LOW | 1 | ~70 Mbps | ~29.6 Mbps | throttled to leftover (~line − HIGH) | + +With the port oversubscribed at 1.4×, the HIGH class keeps essentially all of its +offered load while the LOW class is squeezed to the ~30 Mbps the link has left +after HIGH is served, and 690 excess low-priority frames are dropped — exactly +the strict-priority contract. + +--- + +## 7. How to run + +From the ns-3 root (with this module in `contrib/p4sim`): + +```bash +# Unit tests +./test.py -s p4-traffic-manager + +# End-to-end integration check (both datapaths) +./ns3 run "p4-voq-fabric-integration --run=voq" +./ns3 run "p4-voq-fabric-integration --run=legacy" + +# Throughput benchmark (one link rate per invocation — bmv2 cannot be +# re-initialised within a single process) +./ns3 run "p4-voq-fabric-throughput --linkRate=100Mbps" +./ns3 run "p4-voq-fabric-throughput --linkRate=1000Mbps" + +# Strict-priority demo (HIGH vs LOW flow on a congested output) +./ns3 run "p4-voq-fabric-priority" +``` + +--- + +## 8. Source map + +| File | Role | +| --- | --- | +| `utils/p4-traffic-manager.{h,cc}` | VOQ, fabric scheduler, egress SP, stats/traces, completion signal | +| `model/p4-core-v1model.{h,cc}` | Creates/wires/disposes the TM; opt-in egress branch; transmit + completion glue | +| `model/p4-switch-net-device.{h,cc}` | `EnableVoqFabric` attribute; propagates it to the core | +| `model/switched-ethernet-channel.{h,cc}` | Full-duplex link; sender freed after serialisation, not propagation | +| `test/p4-traffic-manager-test-suite.cc` | 9-case unit suite | +| `examples/p4-voq-fabric-integration.cc` | End-to-end additive-contract check | +| `examples/p4-voq-fabric-throughput.cc` | Near-line-rate goodput benchmark | +| `examples/p4-voq-fabric-priority.cc` | Strict-priority demo (HIGH protected, LOW throttled) | diff --git a/examples/CMakeLists.txt b/examples/CMakeLists.txt index 4fa9ef8..b68423a 100644 --- a/examples/CMakeLists.txt +++ b/examples/CMakeLists.txt @@ -39,6 +39,20 @@ build_lib_example( LIBRARIES_TO_LINK ${P4SIM_CSMA_LIBS} ) +# 2 hosts, 1 switch — throughput benchmark for the VOQ + fabric datapath +build_lib_example( + NAME p4-voq-fabric-throughput + SOURCE_FILES p4-voq-fabric-throughput.cc + LIBRARIES_TO_LINK ${P4SIM_CSMA_LIBS} +) + +# 2 hosts, 1 switch — strict-priority demo (HIGH vs LOW flow) over the VOQ path +build_lib_example( + NAME p4-voq-fabric-priority + SOURCE_FILES p4-voq-fabric-priority.cc + LIBRARIES_TO_LINK ${P4SIM_CSMA_LIBS} +) + # # 3 hosts, 3 routers (line topology) — L3 forwarding # build_lib_example( # NAME p4-l3-router diff --git a/examples/p4-voq-fabric-priority.cc b/examples/p4-voq-fabric-priority.cc new file mode 100644 index 0000000..d7fa3ec --- /dev/null +++ b/examples/p4-voq-fabric-priority.cc @@ -0,0 +1,259 @@ +/* + * Copyright (c) 2025 TU Dresden + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation; + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + * + * Authors: Vineet Goel + */ + +/** + * Strict-priority demonstration for the VOQ + fabric Traffic Manager. + * + * Two saturating UDP flows from two different sender hosts converge on one + * receiver through a V1model switch running the `qos` P4 program, which + * classifies packets by UDP destination port and writes + * standard_metadata.priority (aliased to intrinsic_metadata.priority, which the + * Traffic Manager reads): + * + * dport 4000 -> priority 3 (HIGH) + * dport 2000 -> priority 1 (LOW) + * + * hostH (10.1.1.1, port 0) ── HIGH (prio 3) ─┐ + * ├─► switch ─►[bottleneck]─► hostR + * hostL (10.1.1.2, port 1) ── LOW (prio 1) ─┘ (10.1.1.3, port 2) + * + * The HIGH and LOW flows enter on *separate* ingress ports, so each has its own + * host NIC and its own Virtual Output Queue (VOQ[in][out=2][prio]); they only + * contend inside the switch, at the shared output port 2. That output link is + * the bottleneck and each flow is offered at 0.7x its line rate, so their + * combined 1.4x oversubscribes it. + * + * The switch egress runs strict priority across the 8 per-port queues and the + * fabric grants priority-first, so the HIGH class is protected (served in full) + * while the LOW class is throttled to the leftover capacity. A finite egress + * buffer (set via P4TrafficManager attribute defaults) makes the excess + * low-priority load visible as Traffic-Manager drops instead of unbounded + * queueing. + * + * ./ns3 run p4-voq-fabric-priority + * + * Exit code 0 = strict priority observed (HIGH protected, HIGH > LOW, excess + * dropped); non-zero = a check failed. + */ + +#include "ns3/applications-module.h" +#include "ns3/core-module.h" +#include "ns3/data-rate.h" +#include "ns3/format-utils.h" +#include "ns3/internet-module.h" +#include "ns3/network-module.h" +#include "ns3/p4-core-v1model.h" +#include "ns3/p4-helper.h" +#include "ns3/p4-switch-net-device.h" +#include "ns3/p4-traffic-manager.h" +#include "ns3/packet-sink.h" +#include "ns3/switched-ethernet-channel.h" +#include "ns3/switched-ethernet-helper.h" + +#include +#include +#include +#include + +using namespace ns3; + +NS_LOG_COMPONENT_DEFINE("P4VoqFabricPriority"); + +namespace +{ + +int g_failures = 0; + +void +Check(bool cond, const std::string& what) +{ + std::cout << " [" << (cond ? "PASS" : "FAIL") << "] " << what << "\n"; + if (!cond) + { + ++g_failures; + } +} + +} // namespace + +int +main(int argc, char* argv[]) +{ + std::string egressLink = "100Mbps"; // bottleneck: switch(port 2) -> receiver + std::string ingressLink = "10Gbps"; // kept fast: senders -> switch + uint32_t pktSize = 1400; // UDP payload bytes + double perFlowFactor = 0.7; // each flow offered = factor * egress line + double flowDuration = 0.2; // seconds of saturating traffic + uint32_t egressBufferBytes = 65536; // finite per-output egress buffer + double protectThreshold = 0.85; // HIGH must retain >= this fraction of offered + + CommandLine cmd; + cmd.AddValue("egressLink", "Bottleneck egress link rate switch->receiver", egressLink); + cmd.AddValue("ingressLink", "Ingress link rate senders->switch (kept fast)", ingressLink); + cmd.AddValue("pktSize", "UDP payload size in bytes", pktSize); + cmd.AddValue("perFlowFactor", "Per-flow offered load as a multiple of the egress line", perFlowFactor); + cmd.AddValue("flowDuration", "Duration of the saturating flows (s)", flowDuration); + cmd.AddValue("egressBufferBytes", "Per-output egress buffer limit in bytes (0 = unlimited)", egressBufferBytes); + cmd.AddValue("protectThreshold", "HIGH must retain >= this fraction of its offered load", protectThreshold); + cmd.Parse(argc, argv); + + const uint64_t egressBps = DataRate(egressLink).GetBitRate(); + const uint64_t perFlowBps = static_cast(egressBps * perFlowFactor); + std::ostringstream perFlowRate; + perFlowRate << perFlowBps << "bps"; + + std::cout << "=== VOQ+fabric strict-priority demo ===\n" + << " egressLink=" << egressLink << " ingressLink=" << ingressLink + << " pktSize=" << pktSize << " perFlow=" << perFlowFactor + << "x egress line (combined " << (2 * perFlowFactor) << "x)\n" + << " HIGH=dport 4000 (prio 3) from host0 LOW=dport 2000 (prio 1) from host1\n" + << " egressBuffer=" << egressBufferBytes << " B\n"; + + // A finite egress buffer turns the oversubscribed low-priority backlog into + // Traffic-Manager drops. The core creates the TM with CreateObject, so the + // attribute default set here is picked up (the core only overrides port + // count, rate, and the event-driven flags — not the buffer limits). + if (egressBufferBytes > 0) + { + Config::SetDefault("ns3::P4TrafficManager::EgressPortLimit", + UintegerValue(egressBufferBytes)); + } + + // ---- Topology: host0 (HIGH), host1 (LOW) -> switch -> host2 (receiver) ---- + NodeContainer terminals; + terminals.Create(3); // 0 = HIGH sender, 1 = LOW sender, 2 = receiver + Ptr switchNode = CreateObject(); + + InternetStackHelper internet; + internet.Install(terminals); + internet.Install(switchNode); + + Ipv4AddressHelper ipv4Addr; + ipv4Addr.SetBase("10.1.1.0", "255.255.255.0"); + + const std::string p4Dir = GetP4ExamplePath() + "/qos"; + + P4Helper p4; + p4.SetDeviceAttribute("JsonPath", StringValue(p4Dir + "/qos.json")); + p4.SetDeviceAttribute("FlowTablePath", StringValue(p4Dir + "/flowtable_priority.txt")); + p4.SetDeviceAttribute("P4SwitchArch", UintegerValue(0)); // V1model + p4.SetDeviceAttribute("SwitchRate", UintegerValue(10000)); + p4.SetDeviceAttribute("EnableVoqFabric", BooleanValue(true)); + Ptr sw = DynamicCast(p4.Install(switchNode).Get(0)); + + SwitchedEthernetHelper eth; + eth.SetChannelAttribute("DataRate", StringValue(ingressLink)); + eth.SetChannelAttribute("Delay", StringValue("1us")); + NetDeviceContainer hostDevs = eth.Install(sw, terminals); + + // The qos flowtable rewrites the destination MAC on forwarding: + // 10.1.1.1 -> port 0, MAC ...:01 (host0, HIGH sender) + // 10.1.1.2 -> port 1, MAC ...:03 (host1, LOW sender) + // 10.1.1.3 -> port 2, MAC ...:05 (host2, receiver) + hostDevs.Get(0)->SetAddress(Mac48Address("00:00:00:00:00:01")); + hostDevs.Get(1)->SetAddress(Mac48Address("00:00:00:00:00:03")); + hostDevs.Get(2)->SetAddress(Mac48Address("00:00:00:00:00:05")); + ipv4Addr.Assign(hostDevs.Get(0)); + ipv4Addr.Assign(hostDevs.Get(1)); + ipv4Addr.Assign(hostDevs.Get(2)); + + // Slow down only the egress link switch(port 2) -> receiver so it is the + // sole bottleneck; the ingress links keep the fast ingressLink rate. + Ptr egressCh = sw->GetPortChannel(2); + egressCh->SetAttribute("DataRate", DataRateValue(DataRate(egressLink))); + + // ---- Two competing UDP flows -> receiver (host2) ---- + const uint16_t highPort = 4000; // qos: prio 3 (HIGH) + const uint16_t lowPort = 2000; // qos: prio 1 (LOW) + Ipv4Address rxAddr = terminals.Get(2)->GetObject()->GetAddress(1, 0).GetLocal(); + + PacketSinkHelper highSink("ns3::UdpSocketFactory", + InetSocketAddress(Ipv4Address::GetAny(), highPort)); + PacketSinkHelper lowSink("ns3::UdpSocketFactory", + InetSocketAddress(Ipv4Address::GetAny(), lowPort)); + ApplicationContainer highSinkApp = highSink.Install(terminals.Get(2)); + ApplicationContainer lowSinkApp = lowSink.Install(terminals.Get(2)); + highSinkApp.Start(Seconds(1.0)); + lowSinkApp.Start(Seconds(1.0)); + highSinkApp.Stop(Seconds(2.0 + flowDuration + 1.0)); + lowSinkApp.Stop(Seconds(2.0 + flowDuration + 1.0)); + + auto makeFlow = [&](uint32_t senderIdx, uint16_t dport) { + OnOffHelper onOff("ns3::UdpSocketFactory", InetSocketAddress(rxAddr, dport)); + onOff.SetAttribute("PacketSize", UintegerValue(pktSize)); + onOff.SetAttribute("DataRate", StringValue(perFlowRate.str())); + onOff.SetAttribute("OnTime", StringValue("ns3::ConstantRandomVariable[Constant=1]")); + onOff.SetAttribute("OffTime", StringValue("ns3::ConstantRandomVariable[Constant=0]")); + ApplicationContainer app = onOff.Install(terminals.Get(senderIdx)); + app.Start(Seconds(2.0)); + app.Stop(Seconds(2.0 + flowDuration)); + return app; + }; + makeFlow(0, highPort); // host0 -> HIGH + makeFlow(1, lowPort); // host1 -> LOW + + Simulator::Stop(Seconds(2.0 + flowDuration + 1.0)); + Simulator::Run(); + + // ---- Capture results while the core / TM still exist ---- + const uint64_t highRx = DynamicCast(highSinkApp.Get(0))->GetTotalRx(); + const uint64_t lowRx = DynamicCast(lowSinkApp.Get(0))->GetTotalRx(); + + uint64_t txPrioHigh = 0; + uint64_t txPrioLow = 0; + uint64_t tmReceived = 0; + uint64_t tmDropped = 0; + P4CoreV1model* core = sw->GetV1ModelCore(); + Ptr tm = core ? core->GetTrafficManager() : nullptr; + if (tm) + { + const auto& s = tm->GetStats(); + txPrioHigh = s.perPriorityTransmitted[3]; + txPrioLow = s.perPriorityTransmitted[1]; + tmReceived = s.totalReceived; + tmDropped = s.totalDropped; + } + + Simulator::Destroy(); + + // ---- Results ---- + const double highMbps = highRx * 8.0 / flowDuration / 1e6; + const double lowMbps = lowRx * 8.0 / flowDuration / 1e6; + const double offeredMbps = perFlowBps / 1e6; + const double highRetained = (offeredMbps > 0) ? (highMbps / offeredMbps) : 0.0; + + std::cout << std::fixed << std::setprecision(2) + << " HIGH: rx=" << highRx << " B ~" << highMbps << " Mbps (offered ~" + << offeredMbps << " Mbps, retained " << (highRetained * 100.0) << "%)\n" + << " LOW : rx=" << lowRx << " B ~" << lowMbps << " Mbps (offered ~" << offeredMbps + << " Mbps)\n" + << " [TM] received=" << tmReceived << " transmitted prio3=" << txPrioHigh + << " prio1=" << txPrioLow << " dropped=" << tmDropped << "\n"; + + Check(tm != nullptr, "Traffic Manager active on the switch"); + Check(highRx > 0 && lowRx > 0, "Both priority classes carried some traffic"); + Check(highRx > lowRx, "HIGH priority delivered more than LOW under congestion"); + Check(txPrioHigh > txPrioLow, "TM transmitted more prio-3 frames than prio-1 frames"); + Check(highRetained >= protectThreshold, "HIGH priority protected (retained offered load)"); + Check(tmDropped > 0, "TM dropped the excess low-priority load (port oversubscribed)"); + + std::cout << "=== " << (g_failures == 0 ? "STRICT PRIORITY OBSERVED" : "CHECKS FAILED") << " (" + << g_failures << " failure(s)) ===\n"; + return g_failures == 0 ? 0 : 1; +} diff --git a/examples/p4-voq-fabric-throughput.cc b/examples/p4-voq-fabric-throughput.cc new file mode 100644 index 0000000..b3c13b4 --- /dev/null +++ b/examples/p4-voq-fabric-throughput.cc @@ -0,0 +1,221 @@ +/* + * Copyright (c) 2025 TU Dresden + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation; + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + * + * Authors: Vineet Goel + */ + +/** + * Throughput benchmark for the VOQ + fabric Traffic Manager datapath. + * + * A single saturating UDP flow (offered load above link capacity) is sent + * host0 -> host1 through a V1model switch running IPv4 forwarding, and the + * goodput received at the sink is compared against the link's line rate. + * + * Because the VOQ datapath serialises each egress port at the port's own line + * rate (PortRate, seeded from the channel) and is driven by the transmit + * completion signal rather than a fixed timer, the delivered goodput should + * sit just below line rate (the small gap is Ethernet/IP/UDP header overhead), + * with no artificial timer bottleneck. + * + * One link rate per invocation (bmv2 cannot be re-initialised in a process): + * ./ns3 run "p4-voq-fabric-throughput --linkRate=100Mbps" + * ./ns3 run "p4-voq-fabric-throughput --linkRate=1000Mbps" + * + * Exit code 0 = goodput reached the near-line-rate threshold; non-zero = below. + */ + +#include "ns3/applications-module.h" +#include "ns3/core-module.h" +#include "ns3/data-rate.h" +#include "ns3/format-utils.h" +#include "ns3/internet-module.h" +#include "ns3/network-module.h" +#include "ns3/switched-ethernet-channel.h" +#include "ns3/p4-core-v1model.h" +#include "ns3/p4-helper.h" +#include "ns3/p4-switch-net-device.h" +#include "ns3/p4-traffic-manager.h" +#include "ns3/packet-sink.h" +#include "ns3/switched-ethernet-helper.h" + +#include +#include +#include +#include + +using namespace ns3; + +NS_LOG_COMPONENT_DEFINE("P4VoqFabricThroughput"); + +namespace +{ + +uint64_t g_rxBytes = 0; +double g_firstRx = -1.0; +double g_lastRx = 0.0; + +void +RxTrace(uint32_t payloadSize, Ptr pkt, const Address&) +{ + // Count only full data packets (skip stray/short frames). + if (pkt->GetSize() != payloadSize) + { + return; + } + double now = Simulator::Now().GetSeconds(); + if (g_firstRx < 0.0) + { + g_firstRx = now; + } + g_lastRx = now; + g_rxBytes += pkt->GetSize(); +} + +} // namespace + +int +main(int argc, char* argv[]) +{ + std::string linkRate = "1000Mbps"; // egress (bottleneck) link switch -> host1 + std::string hostLinkRate = "10Gbps"; // ingress link host0 -> switch (kept fast) + uint32_t pktSize = 1400; // UDP payload bytes + double offeredFactor = 1.2; // offered load = offeredFactor * linkRate + double flowDuration = 0.2; // seconds of saturating traffic + bool voq = true; // use the VOQ + fabric datapath + double passThreshold = 0.80; // PASS if goodput >= threshold * linkRate + + CommandLine cmd; + cmd.AddValue("linkRate", "Egress (bottleneck) link rate switch->host1", linkRate); + cmd.AddValue("hostLinkRate", "Ingress link rate host0->switch (kept fast)", hostLinkRate); + cmd.AddValue("pktSize", "UDP payload size in bytes", pktSize); + cmd.AddValue("offeredFactor", "Offered load as a multiple of the egress link rate", offeredFactor); + cmd.AddValue("flowDuration", "Duration of the saturating flow (s)", flowDuration); + cmd.AddValue("voq", "Use the VOQ+fabric datapath (else legacy)", voq); + cmd.AddValue("passThreshold", "PASS if goodput >= threshold * linkRate", passThreshold); + cmd.Parse(argc, argv); + + const uint64_t linkBps = DataRate(linkRate).GetBitRate(); + const uint64_t offeredBps = static_cast(linkBps * offeredFactor); + std::ostringstream offeredRate; + offeredRate << offeredBps << "bps"; + + // The ingress link host0->switch is kept fast so the host NIC (which has no + // tx queue and drops on a busy channel) never becomes the limiter; the + // switch's egress port switch->host1 is the sole bottleneck under test. + std::cout << "=== VOQ+fabric throughput benchmark ===\n" + << " egressLink=" << linkRate << " ingressLink=" << hostLinkRate + << " datapath=" << (voq ? "VOQ+fabric" : "legacy") << " pktSize=" << pktSize + << " offered=" << offeredFactor << "x egress line\n"; + + // ---- Topology: host0 -> switch -> host1 ---- + NodeContainer terminals; + terminals.Create(2); + Ptr switchNode = CreateObject(); + + InternetStackHelper internet; + internet.Install(terminals); + internet.Install(switchNode); + + Ipv4AddressHelper ipv4Addr; + ipv4Addr.SetBase("10.1.1.0", "255.255.255.0"); + + const std::string p4Dir = GetP4ExamplePath() + "/simple_v1model"; + + P4Helper p4; + p4.SetDeviceAttribute("JsonPath", StringValue(p4Dir + "/simple_v1model.json")); + p4.SetDeviceAttribute("FlowTablePath", StringValue(p4Dir + "/flowtable_0.txt")); + p4.SetDeviceAttribute("P4SwitchArch", UintegerValue(0)); + p4.SetDeviceAttribute("SwitchRate", UintegerValue(10000)); + p4.SetDeviceAttribute("EnableVoqFabric", BooleanValue(voq)); + Ptr sw = DynamicCast(p4.Install(switchNode).Get(0)); + + SwitchedEthernetHelper eth; + eth.SetChannelAttribute("DataRate", StringValue(hostLinkRate)); + eth.SetChannelAttribute("Delay", StringValue("1us")); + NetDeviceContainer hostDevs = eth.Install(sw, terminals); + + for (uint32_t i = 0; i < terminals.GetN(); ++i) + { + std::ostringstream mac; + mac << "00:00:00:00:00:" << std::hex << std::setfill('0') << std::setw(2) << (i + 1); + hostDevs.Get(i)->SetAddress(Mac48Address(mac.str().c_str())); + ipv4Addr.Assign(hostDevs.Get(i)); + } + + // Slow down only the egress link switch(port 1) -> host1 so it is the + // bottleneck; the ingress link keeps the fast hostLinkRate set above. + Ptr egressCh = sw->GetPortChannel(1); + egressCh->SetAttribute("DataRate", DataRateValue(DataRate(linkRate))); + + // ---- Saturating UDP flow host0 -> host1 ---- + const uint16_t serverPort = 9000; + Ipv4Address serverAddr = terminals.Get(1)->GetObject()->GetAddress(1, 0).GetLocal(); + + PacketSinkHelper sink("ns3::UdpSocketFactory", + InetSocketAddress(Ipv4Address::GetAny(), serverPort)); + ApplicationContainer sinkApp = sink.Install(terminals.Get(1)); + sinkApp.Start(Seconds(1.0)); + sinkApp.Stop(Seconds(2.0 + flowDuration + 1.0)); + + OnOffHelper onOff("ns3::UdpSocketFactory", InetSocketAddress(serverAddr, serverPort)); + onOff.SetAttribute("PacketSize", UintegerValue(pktSize)); + onOff.SetAttribute("DataRate", StringValue(offeredRate.str())); + onOff.SetAttribute("OnTime", StringValue("ns3::ConstantRandomVariable[Constant=1]")); + onOff.SetAttribute("OffTime", StringValue("ns3::ConstantRandomVariable[Constant=0]")); + ApplicationContainer clientApp = onOff.Install(terminals.Get(0)); + clientApp.Start(Seconds(2.0)); + clientApp.Stop(Seconds(2.0 + flowDuration)); + + sinkApp.Get(0)->TraceConnectWithoutContext("Rx", MakeBoundCallback(&RxTrace, pktSize)); + + Simulator::Stop(Seconds(2.0 + flowDuration + 1.0)); + Simulator::Run(); + + if (voq) + { + P4CoreV1model* core = sw->GetV1ModelCore(); + Ptr tm = core ? core->GetTrafficManager() : nullptr; + if (tm) + { + const auto& s = tm->GetStats(); + std::cout << " [TM] received=" << s.totalReceived << " voqEnq=" << s.totalVoqEnqueued + << " transmitted=" << s.totalTransmitted << " dropped=" << s.totalDropped + << "\n"; + } + } + + Simulator::Destroy(); + + // ---- Results ---- + const double window = (g_lastRx > g_firstRx) ? (g_lastRx - g_firstRx) : flowDuration; + const double goodputMbps = (window > 0) ? (g_rxBytes * 8.0 / window / 1e6) : 0.0; + const double linkMbps = linkBps / 1e6; + const double pctOfLink = (linkMbps > 0) ? (goodputMbps / linkMbps * 100.0) : 0.0; + + // Header-overhead ceiling for reference: payload / (payload + Eth+IP+UDP). + const double ceilingPct = 100.0 * pktSize / (pktSize + 14 + 20 + 8); + + std::cout << std::fixed << std::setprecision(2) << " rxBytes=" << g_rxBytes + << " window=" << window << "s\n" + << " goodput=" << goodputMbps << " Mbps of " << linkMbps << " Mbps line (" + << pctOfLink << "% of line; header-overhead ceiling ~" << ceilingPct << "%)\n"; + + const bool pass = pctOfLink >= passThreshold * 100.0; + std::cout << " [" << (pass ? "PASS" : "FAIL") << "] goodput >= " << (passThreshold * 100.0) + << "% of line rate\n" + << "=== " << (pass ? "NEAR LINE RATE" : "BELOW THRESHOLD") << " ===\n"; + return pass ? 0 : 1; +} diff --git a/examples/p4src/qos/flowtable_priority.txt b/examples/p4src/qos/flowtable_priority.txt new file mode 100644 index 0000000..a26746b --- /dev/null +++ b/examples/p4src/qos/flowtable_priority.txt @@ -0,0 +1,14 @@ +table_set_default ipv4_nhop drop +table_set_default arp_simple drop +table_add ipv4_nhop ipv4_forward 0x0a010101 => 00:00:00:00:00:01 0x0 +table_add ipv4_nhop ipv4_forward 0x0a010102 => 00:00:00:00:00:03 0x1 +table_add ipv4_nhop ipv4_forward 0x0a010103 => 00:00:00:00:00:05 0x2 +table_add arp_simple set_arp_nhop 0x0a010101 => 0x0 +table_add arp_simple set_arp_nhop 0x0a010102 => 0x1 +table_add arp_simple set_arp_nhop 0x0a010103 => 0x2 +table_add udp_priority set_priority 2000 => 0x1 +table_add udp_priority set_priority 3000 => 0x2 +table_add udp_priority set_priority 4000 => 0x3 +table_add tcp_priority set_priority 2000 => 0x1 +table_add tcp_priority set_priority 3000 => 0x2 +table_add tcp_priority set_priority 4000 => 0x3