diff --git a/issues/a-bar-behind-a-bridge-with-no-free-2-mib-of-its-windows-is-not-handed-over.md b/issues/a-bar-behind-a-bridge-with-no-free-2-mib-of-its-windows-is-not-handed-over.md new file mode 100644 index 0000000000..7908b67293 --- /dev/null +++ b/issues/a-bar-behind-a-bridge-with-no-free-2-mib-of-its-windows-is-not-handed-over.md @@ -0,0 +1,52 @@ +--- +status: open +kind: defect +opened: 2026-10-12 +--- + +# A BAR behind a bridge with no free 2 MiB of its windows is not handed over + +A claim moves each BAR it maps onto a 2 MiB window of its own +(`place_bar`, `kernel/src/pcidev/mod.rs`), because a 2 MiB page is the only +one this kernel maps into a process. Behind a bridge the window is cut only +where every bridge on the function's path forwards +(`placement::behind_bridges`, `kernel/pci/src/placement.rs`); where none of +that is free, the BAR stays where firmware put it and is handed to nobody, +named `NoRoutedRun`, and a function with no other mappable BAR is refused. +A bridge firmware sized to its BARs leaves no such 2 MiB. Where the BAR is +also the one holding the function's MSI-X table, a claim withholds it and +the function is refused `NoMappableBar` before placement; a claim that +handed the table's BAR over, as #921 does, would meet `NoRoutedRun`: + +- **The AMD bench**: the AX200 at 01:00.0 has BAR 0 at 0xd0600000, behind + bridge 00:02.2, which forwards 0xd0600000..0xd0700000 (#884's E13 §1, + https://github.com/ToyOSOrg/evidence/blob/2eb315326d1b4d0363b4c918da8dd2bfaabd1234/e13-amd-ax200-bar.log); + the xHCI at 03:00.4 is behind 00:08.1, whose 0xd0000000..0xd0500000 + bus 3 fills. Its one BAR holds its MSI-X table, which a claim withholds, + so it is refused `NoMappableBar` before placement; a claim that handed + the table's BAR over, as #921 does, would meet `NoRoutedRun` there. +- **The T14**, as S1 read it (lines 111 to 116, + https://github.com/ToyOSOrg/evidence/blob/cd368b25b2259a76b80dc31f4219dac1c91d84dc/s1/wifi-s1-a-kernel.log): + the AX210 at 09:00.0 has BAR 0 at 0xbc300000, 16 KiB, its MSI-X table + in it (`issues/there-is-no-wifi.md`, S1's read-out), behind root port + 00:1c.4, which forwards 0xbc300000..0xbc400000. + +Two placement issues rest on placement as it is, and P1 takes them up with +this one: + +- `issues/nothing-reds-when-the-bar-survey-drops-an-assigned-bar.md` asks for + a guest that reds when the forwarded-range push is deleted. +- `issues/no-boot-reaches-the-retry-that-undoes-a-placement-nothing-answered.md` + wants a function behind a `pcie-root-port` to settle at a later candidate; + `bar_map_behind_a_bridge` now settles one at the first. + +Owner: P1 of `issues/there-is-no-wifi.md`. + +**Exit**: +- a BAR inside every window on its path to the root is mapped in place, in + 4 KiB pages; +- a BAR that shares a 4 KiB page with another function's is moved within + those windows, never out of them, or is refused by name; +- the T14's AX210 is claimed with its BAR in place; +- each of the two placement issues above closed, or its exit rewritten to the + placement P1 lands. diff --git a/issues/pcidev-moves-a-bar-behind-a-bridge-out-of-the-range-the-bridge-forwards.md b/issues/pcidev-moves-a-bar-behind-a-bridge-out-of-the-range-the-bridge-forwards.md deleted file mode 100644 index 79f0da9cbc..0000000000 --- a/issues/pcidev-moves-a-bar-behind-a-bridge-out-of-the-range-the-bridge-forwards.md +++ /dev/null @@ -1,61 +0,0 @@ ---- -status: open -kind: defect -opened: 2026-10-12 ---- - -# pcidev moves a BAR behind a bridge out of the range the bridge forwards - -Before a hand-over, `kernel/src/pcidev/mod.rs` re-places each BAR onto a -2 MiB boundary of its own (`place_bar`). The candidates come from -`placement::reserve` (`kernel/pci/src/placement.rs`), and a candidate is one -of the free runs: a gap left by everything the machine decodes, inside a -root bridge's window. - -The range every bridge forwards to its secondary bus is part of that -"everything" (the `forwarded()` loop that feeds `placement::free_runs`). So no -candidate ever lies inside the range a function's own bridge forwards. A -function on a secondary bus decodes at none of the `MAX_CANDIDATES` (8) -addresses it is offered, and its claim is refused. - -Both benches put their radio there: - -- **The AMD bench**, as `main`'s kernel logged it in #884's metal run (E13 §1, - https://github.com/ToyOSOrg/evidence/blob/2eb315326d1b4d0363b4c918da8dd2bfaabd1234/e13-amd-ax200-bar.log): - the AX200 at 01:00.0 has BAR 0 at 0xd0600000, behind bridge 00:02.2, which - forwards 0xd0600000..0xd0700000; no free run reaches into that range. -- **The T14**, as S1 read it (lines 111 to 116, - https://github.com/ToyOSOrg/evidence/blob/cd368b25b2259a76b80dc31f4219dac1c91d84dc/s1/wifi-s1-a-kernel.log): - the AX210 at 09:00.0 has BAR 0 at 0xbc300000, 16 KiB, behind root port - 00:1c.4, which forwards 0xbc300000..0xbc400000. - -The Wi-Fi demo's run on the AMD bench took this path, as its commit messages -record (E13 §2): pcidev moved the AX200's BAR 0 onto eight 2 MiB windows from -0x450200000, all outside the 1 MiB that 00:02.2 forwards; every moved BAR read -0xffffffff, and the claim was refused. The demo then kept the BAR in place, -untranslated and unremapped, which `main` never does. - -Two placement issues rest on placement as it is, and P1 takes them up with -this one: - -- `issues/nothing-reds-when-the-bar-survey-drops-an-assigned-bar.md` asks for - a guest that reds when the forwarded-range push is deleted. That push is - what puts every candidate outside its bridge. Its "No machine in reach puts - a BAR it hands over behind a bridge" holds only until a radio is handed - over: both benches put theirs behind one. -- `issues/no-boot-reaches-the-retry-that-undoes-a-placement-nothing-answered.md` - wants a function behind a `pcie-root-port` to settle at a later candidate. - Under this issue no candidate lies inside a port's window, so none ever - answers there. - -Owner: P1 of `issues/there-is-no-wifi.md`. - -**Exit**: -- a BAR inside every window on its path to the root is mapped in place, in - 4 KiB pages; -- a BAR that shares a 4 KiB page with another function's is moved within - those windows, never out of them, or is refused by name; -- a host test places a BAR behind a bridge; -- the T14's AX210 is claimed with its BAR in place; -- each of the two placement issues above closed, or its exit rewritten to the - placement P1 lands. diff --git a/issues/there-is-no-wifi.md b/issues/there-is-no-wifi.md index 7873c9da97..bad0a53cce 100644 --- a/issues/there-is-no-wifi.md +++ b/issues/there-is-no-wifi.md @@ -148,7 +148,7 @@ page included.** Owner: the orchestrator. - `msix-exclusive-bar=on` gone from `tests/common/qemu.rs` and `build/qemu.rs`, so the NVMe guests drive QEMU's default controller through its table in BAR 0; - - `issues/pcidev-moves-a-bar-behind-a-bridge-out-of-the-range-the-bridge-forwards.md` + - `issues/a-bar-behind-a-bridge-with-no-free-2-mib-of-its-windows-is-not-handed-over.md` closed, and each of the two placement issues it names closed or its exit rewritten to the placement P1 lands; - the exit of diff --git a/kernel/pci/src/bridge.rs b/kernel/pci/src/bridge.rs index e22c8040d9..f272df6f47 100644 --- a/kernel/pci/src/bridge.rs +++ b/kernel/pci/src/bridge.rs @@ -22,6 +22,74 @@ pub const PREFETCH_BASE: u64 = 0x24; /// removed. pub const HEADER_TYPE_BRIDGE: u8 = 1; +/// Byte offsets of a Type 1 header's Secondary and Subordinate Bus Number +/// registers (§3.2.5.3, §3.2.5.4): the bus directly below the bridge, and +/// the highest bus number anywhere below it. +pub const SECONDARY_BUS: u64 = 0x19; +pub const SUBORDINATE_BUS: u64 = 0x1A; + +/// One bridge, as the path to a function behind it reads it: the bus it is +/// on, and its bus numbers and windows as its registers answer, which a +/// bridge firmware never configured (both bus numbers reset to 0) or a +/// hostile switch may answer with anything. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct Bridge { + pub bus: u8, + pub secondary: u8, + pub subordinate: u8, + pub memory: Option, + pub prefetchable: Option, +} + +/// Why a function's path to its root bus cannot be named. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum PathRefused { + /// Two bridges each name this bus as their secondary one, and requests + /// for it reach the function through at most one of them. + Ambiguous { bus: u8 }, +} + +/// The bridges from a function's bus up to its root bus, nearest first; empty +/// for a function on a root bus. +/// +/// **A step is a bridge whose secondary bus is the one reached, which sits on +/// a lower bus and whose subordinate range holds the function's** (§3.2.5.3, +/// §3.2.5.4): anything else is not a bridge requests for that bus pass, and +/// is passed over, so an unconfigured or lying bridge moves no function off +/// its root bus. Each step lowers the bus, so the walk ends; two bridges that +/// could each be the step refuse the walk. +pub fn path(bridges: &[Bridge], bus: u8) -> impl Iterator> { + let mut at = Some(bus); + core::iter::from_fn(move || { + let reached = at?; + let mut parents = bridges.iter().filter(|b| { + b.secondary == reached && b.bus < b.secondary && b.secondary <= bus && bus <= b.subordinate + }); + let step = match (parents.next(), parents.next()) { + (None, _) => None, + (Some(parent), None) => Some(Ok(parent)), + (Some(_), Some(_)) => Some(Err(PathRefused::Ambiguous { bus: reached })), + }; + at = match step { + Some(Ok(parent)) => Some(parent.bus), + _ => None, + }; + step + }) +} + +impl Bridge { + /// Whether this bridge forwards all of `at..end` to its secondary side + /// through a window a BAR of this kind may sit in (§3.2.5.6 to + /// §3.2.5.8). The prefetchable window may be read ahead and its writes + /// merged, so only a prefetchable BAR belongs there; the memory window + /// takes either kind. + pub fn forwards(&self, at: u64, end: u64, prefetchable: bool) -> bool { + let holds = |window: Option| window.is_some_and(|w| w.start <= at && end <= w.end); + holds(self.memory) || (prefetchable && holds(self.prefetchable)) + } +} + /// The granularity both windows are expressed in: bits 31:20. const GRANULE: u64 = 1 << 20; @@ -161,4 +229,63 @@ mod tests { ); assert_eq!(prefetch(0x0000_0010, 0, 0), None); } + + fn port(bus: u8, secondary: u8, subordinate: u8) -> Bridge { + Bridge { bus, secondary, subordinate, memory: None, prefetchable: None } + } + + fn walked(bridges: &[Bridge], bus: u8) -> std::vec::Vec> { + path(bridges, bus).map(|step| step.copied()).collect() + } + + /// The T14's shape: every claimed function on bus 0, and root ports with + /// buses below them. A port firmware never configured reads both bus + /// numbers 0, and a hostile one may claim every bus; neither moves a + /// bus-0 function off its root bus. + #[test] + fn a_function_on_the_root_bus_has_no_path_whatever_a_bridge_says() { + let real = port(0, 9, 9); + assert_eq!(walked(&[real], 0), []); + assert_eq!(walked(&[real, port(0, 0, 0)], 0), []); + assert_eq!(walked(&[real, port(0, 0, 0xff)], 0), []); + assert_eq!(walked(&[real, port(0, 0, 0xff)], 9), [Ok(real)]); + } + + /// A walk climbs one bus per step to the root bus, each step a bridge on + /// a lower bus whose subordinate range holds the function's bus; a bridge + /// that claims a bus from above it or a range not holding the function is + /// no step. + #[test] + fn a_path_climbs_from_the_functions_bus_to_the_root() { + let upstream = port(0, 2, 4); + let downstream = port(2, 3, 3); + assert_eq!(walked(&[upstream, downstream], 3), [Ok(downstream), Ok(upstream)]); + assert_eq!(walked(&[upstream, port(5, 3, 3)], 3), []); + assert_eq!(walked(&[upstream, port(2, 3, 2)], 3), []); + } + + /// Two bridges naming one secondary bus refuse the walk for a function on + /// it, and for no function elsewhere. + #[test] + fn a_bus_two_bridges_claim_refuses_its_functions_alone() { + let one = port(0, 3, 3); + let other = port(1, 3, 3); + let elsewhere = port(0, 5, 5); + assert_eq!(walked(&[one, other, elsewhere], 3), [Err(PathRefused::Ambiguous { bus: 3 })]); + assert_eq!(walked(&[one, other, elsewhere], 5), [Ok(elsewhere)]); + } + + /// The memory window forwards either kind of BAR; the prefetchable window + /// forwards a prefetchable BAR alone, and only an extent wholly inside. + #[test] + fn forwarding_holds_the_bars_kind_and_its_whole_extent() { + let memory = Window { start: 0xd000_0000, end: 0xd040_0000 }; + let prefetch = Window { start: 0xe000_0000, end: 0xe040_0000 }; + let b = Bridge { memory: Some(memory), prefetchable: Some(prefetch), ..port(0, 1, 1) }; + assert!(b.forwards(0xd000_0000, 0xd020_0000, false)); + assert!(b.forwards(0xd000_0000, 0xd020_0000, true)); + assert!(b.forwards(0xe000_0000, 0xe020_0000, true)); + assert!(!b.forwards(0xe000_0000, 0xe020_0000, false)); + assert!(!b.forwards(0xd030_0000, 0xd050_0000, true)); + } } diff --git a/kernel/pci/src/placement.rs b/kernel/pci/src/placement.rs index 07d2c35f8b..72e6eb3a6b 100644 --- a/kernel/pci/src/placement.rs +++ b/kernel/pci/src/placement.rs @@ -10,7 +10,7 @@ use toyos_abi::boot::RootBridgeWindow; -use crate::bridge::Window; +use crate::bridge::{Bridge, Window}; /// Where the fixed platform devices start on a PC: the I/O APIC, the HPET and /// the LAPIC window are at and above this, so a 32-bit run may not reach it. @@ -32,6 +32,11 @@ pub const WIDE_FLOOR: u64 = 1 << 32; pub fn free_runs(taken: &mut [Window], wide: bool) -> impl Iterator + '_ { let (floor, ceiling) = if wide { (WIDE_FLOOR, u64::MAX) } else { (0, PLATFORM_MMIO) }; taken.sort_unstable_by_key(|extent| extent.start); + gaps(taken, floor, ceiling) +} + +/// What sorted `taken` leaves free of `floor..ceiling`, lowest first. +fn gaps(taken: &[Window], floor: u64, ceiling: u64) -> impl Iterator + '_ { let mut at = floor; let mut rest = taken.iter(); core::iter::from_fn(move || { @@ -104,6 +109,50 @@ pub fn reserve( None } +/// A `span`-aligned address for a BAR behind the bridges of `path` (nearest +/// first, from [`crate::bridge::path`]), inside one of `windows` and free of +/// `taken`, and that window's base; `None` where there is none. +/// +/// **Only what every bridge on the path forwards is offered** (PCI-to-PCI +/// Bridge Architecture §3.2.5.6 to §3.2.5.8): an address outside a window of +/// one of them is never forwarded to the function, and a read of it does not +/// come back on real hardware. Candidates come from the nearest bridge's +/// windows of the BAR's kind, below 4 GiB for a 32-bit BAR, and a +/// prefetchable BAR tries the prefetchable window first, so it leaves the +/// memory window to BARs that may use nothing else. A run whose first such +/// address an ancestor does not also forward is passed over unread: firmware +/// nests a bridge's windows inside its parent's, and a machine that did not +/// is offered nothing there. `taken` is every extent something else decodes +/// in there: the firmware memory map, every BAR, every window cut or being +/// tried, and the windows of every bridge off the path. +pub fn behind_bridges( + path: &[Bridge], + prefetchable: bool, + wide: bool, + taken: &mut [Window], + windows: &[RootBridgeWindow], + span: u64, +) -> Option<(u64, u64)> { + let parent = path.first()?; + let reach = if wide { u64::MAX } else { WIDE_FLOOR }; + taken.sort_unstable_by_key(|extent| extent.start); + let kinds = if prefetchable { [parent.prefetchable, parent.memory] } else { [parent.memory, None] }; + for window in kinds.into_iter().flatten() { + let mut first: Option<(u64, u64)> = None; + for run in gaps(taken, window.start, window.end.min(reach)) { + let Some((at, base)) = offered(&run, windows, span) else { continue }; + let routed = path.iter().all(|bridge| bridge.forwards(at, at + span, prefetchable)); + if routed && first.is_none_or(|(held, _)| at < held) { + first = Some((at, base)); + } + } + if first.is_some() { + return first; + } + } + None +} + /// Put a reservation back. /// /// A run hands its addresses out upward and holds one extent, so the only one @@ -389,4 +438,108 @@ mod tests { release(&mut runs, second, 2 * MIB); assert_eq!(reserve(&mut runs, &T14_WINDOWS, 2 * MIB), Some(second)); } + + /// The AMD bench's bus 3 behind its port `00:08.1`, as its boot read it: + /// the port forwards `0xd0000000..0xd0500000` and, prefetchable, + /// `0xfc90000000..0xfca0200000`, and firmware packed every bus-3 BAR into + /// them. The declaration is the bench's own loader list. + mod behind { + use super::*; + + const MIB: u64 = 1024 * 1024; + const PORT: Bridge = Bridge { + bus: 0, + secondary: 3, + subordinate: 3, + memory: Some(Window { start: 0xd000_0000, end: 0xd050_0000 }), + prefetchable: Some(Window { start: 0xfc_9000_0000, end: 0xfc_a020_0000 }), + }; + const DECLARED: [RootBridgeWindow; 3] = [ + RootBridgeWindow { base: 0xd000_0000, length: 0x70_0000 }, + RootBridgeWindow { base: 0xfc_9000_0000, length: 0x1020_0000 }, + RootBridgeWindow { base: 0x4_5020_0000, length: 0xf8_3fe0_0000 }, + ]; + + fn bus_3() -> std::vec::Vec { + let bar = |start, bytes| Window { start, end: start + bytes }; + std::vec![ + bar(0xd000_0000, MIB), + bar(0xd010_0000, MIB), + bar(0xd020_0000, MIB), + bar(0xd030_0000, MIB), + bar(0xd040_0000, 512 * 1024), + bar(0xd048_0000, 256 * 1024), + bar(0xd04c_0000, 32 * 1024), + bar(0xd04c_8000, 16 * 1024), + bar(0xd04c_c000, 8 * 1024), + bar(0xd04c_e000, 8 * 1024), + bar(0xfc_9000_0000, 256 * MIB), + bar(0xfc_a000_0000, 2 * MIB), + ] + } + + /// The bench's xHCI at `03:00.4`, a 64-bit non-prefetchable BAR: no + /// 2 MiB of the port's memory window is free, so nothing is offered, + /// and in particular not `0x450200000`, which the declaration names + /// and the port does not forward. + #[test] + fn a_packed_bridge_window_offers_nothing_and_never_the_roots_space() { + assert_eq!(behind_bridges(&[PORT], false, true, &mut bus_3(), &DECLARED, 2 * MIB), None); + assert_eq!(behind_bridges(&[PORT], true, true, &mut bus_3(), &DECLARED, 2 * MIB), None); + } + + /// The same port forwarding 2 MiB more: that address, and nothing + /// outside the window. + #[test] + fn a_free_run_inside_the_path_is_offered() { + let port = Bridge { memory: Some(Window { start: 0xd000_0000, end: 0xd080_0000 }), ..PORT }; + let declared = [RootBridgeWindow { base: 0xd000_0000, length: 0x80_0000 }]; + assert_eq!( + behind_bridges(&[port], false, true, &mut bus_3(), &declared, 2 * MIB), + Some((0xd060_0000, 0xd000_0000)) + ); + } + + /// A prefetchable BAR takes the prefetchable window first, and the + /// memory window where that one is full; a non-prefetchable BAR never + /// takes the prefetchable window, and a 32-bit one never above 4 GiB. + #[test] + fn the_bars_kind_picks_the_window() { + let both = Bridge { memory: Some(Window { start: 0xd000_0000, end: 0xd080_0000 }), ..PORT }; + let declared = [DECLARED[1], RootBridgeWindow { base: 0xd000_0000, length: 0x80_0000 }]; + let mut taken = [Window { start: 0xfc_9000_0000, end: 0xfc_a000_0000 }]; + assert_eq!( + behind_bridges(&[both], true, true, &mut taken, &declared, 2 * MIB), + Some((0xfc_a000_0000, 0xfc_9000_0000)) + ); + let mut full = [Window { start: 0xfc_9000_0000, end: 0xfc_a020_0000 }]; + assert_eq!( + behind_bridges(&[both], true, true, &mut full, &declared, 2 * MIB), + Some((0xd000_0000, 0xd000_0000)) + ); + let prefetchable_only = Bridge { memory: None, ..PORT }; + assert_eq!(behind_bridges(&[prefetchable_only], false, true, &mut taken, &DECLARED, 2 * MIB), None); + assert_eq!(behind_bridges(&[prefetchable_only], true, false, &mut taken, &DECLARED, 2 * MIB), None); + } + + /// Every bridge on the path has to forward the address: a port whose + /// window lies outside its parent's is offered nothing. + #[test] + fn an_address_one_bridge_on_the_path_does_not_forward_is_never_offered() { + let free = Window { start: 0xd000_0000, end: 0xd040_0000 }; + let declared = [RootBridgeWindow { base: 0xd000_0000, length: 0x40_0000 }]; + let parent = Bridge { + bus: 0, + secondary: 2, + subordinate: 3, + memory: Some(Window { start: 0xe000_0000, end: 0xe040_0000 }), + prefetchable: None, + }; + let child = Bridge { bus: 2, secondary: 3, subordinate: 3, memory: Some(free), prefetchable: None }; + assert_eq!(behind_bridges(&[child], false, false, &mut [], &declared, 2 * MIB), Some((0xd000_0000, 0xd000_0000))); + assert_eq!(behind_bridges(&[child, parent], false, false, &mut [], &declared, 2 * MIB), None); + assert_eq!(behind_bridges(&[], false, false, &mut [], &declared, 2 * MIB), None); + } + } + } diff --git a/kernel/src/drivers/pci.rs b/kernel/src/drivers/pci.rs index 3e8e7e5175..8fdc8168b5 100644 --- a/kernel/src/drivers/pci.rs +++ b/kernel/src/drivers/pci.rs @@ -442,17 +442,27 @@ impl PciDevice { /// may hand out, and reading them costs the machine nothing — unlike /// `bar_size`, which takes memory decode off for the length of its probe. pub fn forwarded(&self) -> Vec { + self.bridge().map_or_else(Vec::new, |b| b.memory.into_iter().chain(b.prefetchable).collect()) + } + + /// This function as a bridge: the buses below it and the windows it + /// forwards to them, read and never probed; `None` for a function that is + /// not a bridge. + pub fn bridge(&self) -> Option { if self.read_config_u8(HEADER_TYPE) & !MULTI_FUNCTION != bridge::HEADER_TYPE_BRIDGE { - return Vec::new(); + return None; } - let mut out = Vec::new(); - out.extend(bridge::window(self.read_config_u32(bridge::MEMORY_BASE))); - out.extend(bridge::prefetch( - self.read_config_u32(bridge::PREFETCH_BASE), - self.read_config_u32(bridge::PREFETCH_BASE_UPPER), - self.read_config_u32(bridge::PREFETCH_LIMIT_UPPER), - )); - out + Some(bridge::Bridge { + bus: self.bus, + secondary: self.read_config_u8(bridge::SECONDARY_BUS), + subordinate: self.read_config_u8(bridge::SUBORDINATE_BUS), + memory: bridge::window(self.read_config_u32(bridge::MEMORY_BASE)), + prefetchable: bridge::prefetch( + self.read_config_u32(bridge::PREFETCH_BASE), + self.read_config_u32(bridge::PREFETCH_BASE_UPPER), + self.read_config_u32(bridge::PREFETCH_LIMIT_UPPER), + ), + }) } pub fn capabilities(&self) -> CapabilityIter<'_> { diff --git a/kernel/src/pcidev/mod.rs b/kernel/src/pcidev/mod.rs index c0c5fe0f75..023730a589 100644 --- a/kernel/src/pcidev/mod.rs +++ b/kernel/src/pcidev/mod.rs @@ -33,7 +33,10 @@ //! window of the declaration [`Apertures`] holds: the address firmware put in //! the BAR is held against those windows here, no other address is ever //! offered ([`placement::reserve`]), and with no declaration held no function -//! is handed over. +//! is handed over. Behind a bridge an address is offered only where every +//! bridge on the path forwards it ([`placement::behind_bridges`]); where none +//! of that is free the BAR stays where firmware put it and is mapped to +//! nobody, since its page there is not its own. //! //! **Inside one, the function itself is the proof.** [`place_bar`] reads one //! dword through the BAR where firmware put it, moves the BAR onto the @@ -128,7 +131,7 @@ use toyos_abi::inventory::Arrival; use toyos_abi::pci::{DeviceIrqRecord, PciFunctionInfo, BARS}; use toyos_abi::syscall::{PciId, RegWidth, SyscallError}; use toyos_dma::Register; -use toyos_pci::bridge::Window; +use toyos_pci::bridge::{self, Bridge, Window}; use toyos_pci::{af, aperture, bar, express, msix, placement, pm, probe}; use crate::device::{Claim, ClaimError}; @@ -370,6 +373,15 @@ struct Machine { /// which is not the same claim as reaching the bus. low: Vec, high: Vec, + /// Every bridge on the bus, read in [`publish`]: what a function behind + /// one may be placed inside ([`placement::behind_bridges`]). + bridges: Vec, + /// The firmware memory map's extents, which no candidate behind a bridge + /// may reach any more than a run may. + map: Vec, + /// Addresses behind a bridge a claim is probing, kept from every other + /// claim until the probe settles. + trying: Vec, /// What this kernel holds of the memory the root bridges decode: the /// necessary condition for any read this module issues. apertures: Apertures, @@ -410,6 +422,9 @@ static MACHINE: Lock = Lock::new(Machine { kernel_driven: Vec::new(), low: Vec::new(), high: Vec::new(), + bridges: Vec::new(), + map: Vec::new(), + trying: Vec::new(), apertures: Apertures::Unknown, windows: Vec::new(), resetting: Vec::new(), @@ -539,8 +554,9 @@ pub fn note_kernel_driver(pci: &PciDevice) { /// driver mid-transfer must not meet that. pub fn publish(devices: &[PciDevice], segment: u16, maps: &[MemoryMapEntry], firmware: &[RootBridgeWindow]) { let mut decoded = Vec::new(); - let mut taken: Vec = - maps.iter().map(|entry| Window { start: entry.start, end: entry.end }).collect(); + let bridges: Vec = devices.iter().filter_map(PciDevice::bridge).collect(); + let map: Vec = maps.iter().map(|entry| Window { start: entry.start, end: entry.end }).collect(); + let mut taken = map.clone(); for device in devices { // Bounded by the header's own declaration: a bridge has two BAR slots // and four registers past them that are not BARs, and `bar_size` below @@ -634,6 +650,8 @@ pub fn publish(devices: &[PciDevice], segment: u16, maps: &[MemoryMapEntry], fir machine.segment = segment; machine.decoded = decoded; machine.apertures = apertures; + machine.bridges = bridges; + machine.map = map; machine.low = low; machine.high = high; } @@ -690,6 +708,12 @@ enum Refusal { /// This machine offered the BAR no address of that width at all: no free /// run holds a span-aligned one inside a window firmware declared. NoRun { wide: bool }, + /// The function is behind a bridge, and no 2 MiB the bridges on its path + /// forward to it is free, so the BAR stays where firmware put it. + NoRoutedRun, + /// Two bridges each name this bus as their secondary one, so no path to + /// the function can be named. + PathAmbiguous { bus: u8 }, /// Every address this machine offered was tried and the function answered /// at none of them. NoPlacement { wide: bool, asked: usize }, @@ -762,6 +786,12 @@ impl core::fmt::Display for Refusal { this bus's assigned BARs and its bridges' forwarded ranges and inside a window \ firmware declared" ), + Self::NoRoutedRun => write!(f, "no 2 MiB the bridges above it forward is free"), + Self::PathAmbiguous { bus } => write!( + f, + "two bridges each name bus {bus:#04x} as their secondary one, so nothing says \ + which forwards to it" + ), Self::NoPlacement { wide, asked } => write!( f, "its BAR was moved onto {asked} {}-bit address(es) inside the windows firmware \ @@ -999,7 +1029,9 @@ fn place_bars( // is that window's loss and not the function's. Err(why @ (Refusal::BarUnassigned(_) | Refusal::BarUnrouted(_) - | Refusal::BarReferenceEmpty(_))) => { + | Refusal::BarReferenceEmpty(_) + | Refusal::NoRoutedRun + | Refusal::PathAmbiguous { .. })) => { log!( "pcidev: PCI {:02x}:{:02x}.{} keeps BAR {index} where firmware put it and \ hands it to nobody — {why}", @@ -1319,8 +1351,9 @@ fn probe_dword(at: u64, span: u64, offset: u64) -> u32 { crate::mm::paging::map_mmio(at, span, MmioPolicy::Uncacheable).read_u32(offset) } -/// Move BAR `index` onto a 2 MiB boundary inside a window firmware declared, -/// and answer where. +/// Move BAR `index` onto a 2 MiB boundary inside a window firmware declared +/// and, behind a bridge, inside what every bridge on its path forwards, and +/// answer where. /// /// **Only an address inside a window firmware declared is ever read**, because /// a load no bridge forwards does not come back on real hardware — the address @@ -1329,8 +1362,8 @@ fn probe_dword(at: u64, span: u64, offset: u64) -> u32 { /// Memory decode is off across every write, so nothing can read through a BAR /// that is half-programmed, and the address is read back off the register /// rather than assumed. A candidate the machine refuses leaves the register -/// holding what firmware left in it, and a walk that ends in a refusal gives -/// every address it took back to the run it came out of. +/// holding what firmware left in it, and a walk gives back every address it +/// took and did not keep. fn place_bar(pci: &PciDevice, id: PciId, index: u8, size: u64, windows: &[RootBridgeWindow]) -> Result { let offset = bar::BASE + index as u64 * 4; let low = pci.read_config_u32(offset); @@ -1362,7 +1395,8 @@ fn place_bar(pci: &PciDevice, id: PciId, index: u8, size: u64, windows: &[RootBr // A BAR firmware assigned no address has no such reference, and reading one // at zero would take low RAM for the function's answer — so it is refused // by name rather than settled against a number that is nothing's. - let was = pci.memory_bar(index).map_err(|_| Refusal::BarUnplaceable(index))?.address(); + let firmware = pci.memory_bar(index).map_err(|_| Refusal::BarUnplaceable(index))?; + let was = firmware.address(); if was == 0 { return Err(Refusal::BarUnassigned(index)); } @@ -1377,17 +1411,17 @@ fn place_bar(pci: &PciDevice, id: PciId, index: u8, size: u64, windows: &[RootBr } let who = alloc::format!("PCI {:02x}:{:02x}.{}", pci.bus, pci.dev, pci.func); + let source = Source::of(pci.bus, wide, firmware.prefetchable()).map_err(|bridge::PathRefused::Ambiguous { bus }| Refusal::PathAmbiguous { bus })?; let mut refused = [None; MAX_CANDIDATES]; let mut asked = 0usize; while asked < MAX_CANDIDATES { - let Some(candidate) = with_runs(wide, |runs| placement::reserve(runs, windows, span)) - else { + let Some(candidate) = source.take(windows, span) else { break; }; - let at = candidate.at; + let at = candidate.at(); let Ok(placed) = bar::placement(index, low, at, size) else { refused[asked] = Some(candidate); - give_back(wide, &refused, span); + give_back(&refused, span); return Err(Refusal::BarUnplaceable(index)); }; pci.set_memory_decode(false); @@ -1401,18 +1435,19 @@ fn place_bar(pci: &PciDevice, id: PciId, index: u8, size: u64, windows: &[RootBr if !matches!(pci.memory_bar(index), Ok(memory) if memory.address() == at) { restore(); refused[asked] = Some(candidate); - give_back(wide, &refused, span); + give_back(&refused, span); return Err(Refusal::BarUnplaceable(index)); } let after = probe_dword(at, span, reference); if after == signature { alone_in_its_page(pci, index, at, span); cut(pci, index, at, span); + give_back(&refused, span); log!( "pcidev: {who} BAR {index} ({size:#x} bytes) placed at {at:#x} — inside \ firmware's mem {:#x}; its +{reference:#x} dword answers {after:#010x} there, \ and answered {signature:#010x} from {was:#x}, where firmware put it", - candidate.window + candidate.window() ); return Ok(at); } @@ -1426,8 +1461,80 @@ fn place_bar(pci: &PciDevice, id: PciId, index: u8, size: u64, windows: &[RootBr refused[asked] = Some(candidate); asked += 1; } - give_back(wide, &refused, span); - Err(if asked == 0 { Refusal::NoRun { wide } } else { Refusal::NoPlacement { wide, asked } }) + give_back(&refused, span); + Err(match (asked, &source) { + (0, Source::Behind { .. }) => Refusal::NoRoutedRun, + (0, Source::Runs { .. }) => Refusal::NoRun { wide }, + _ => Refusal::NoPlacement { wide, asked }, + }) +} + +/// Where a BAR's candidate addresses come from: this machine's free runs for a +/// function on a root bus, and the windows of the bridges on its path, nearest +/// first, for one behind a bridge, which no root-level run lies in. +enum Source { + Runs { wide: bool }, + Behind { path: Vec, wide: bool, prefetchable: bool }, +} + +/// One address [`Source::take`] handed out, and where it goes back to. +#[derive(Clone, Copy)] +enum Candidate { + Run { wide: bool, reservation: placement::Reservation }, + Behind { at: u64, window: u64 }, +} + +impl Candidate { + fn at(self) -> u64 { + match self { + Self::Run { reservation, .. } => reservation.at, + Self::Behind { at, .. } => at, + } + } + + /// The base of the declared root window the address is inside. + fn window(self) -> u64 { + match self { + Self::Run { reservation, .. } => reservation.window, + Self::Behind { window, .. } => window, + } + } +} + +impl Source { + fn of(bus: u8, wide: bool, prefetchable: bool) -> Result { + let path = bridge::path(&MACHINE.lock().bridges, bus).map(|step| step.copied()).collect::, _>>()?; + Ok(if path.is_empty() { Self::Runs { wide } } else { Self::Behind { path, wide, prefetchable } }) + } + + /// The next address to try, already out of reach of every other claim. + fn take(&self, windows: &[RootBridgeWindow], span: u64) -> Option { + match self { + &Self::Runs { wide } => with_runs(wide, |runs| placement::reserve(runs, windows, span)) + .map(|reservation| Candidate::Run { wide, reservation }), + Self::Behind { path, wide, prefetchable } => { + let mut machine = MACHINE.lock(); + let mut taken: Vec = machine + .map + .iter() + .copied() + .chain(machine.decoded.iter().map(|&(_, start, end)| Window { start, end })) + .chain(machine.windows.iter().map(|&(_, _, at, span)| Window { start: at, end: at + span })) + .chain(machine.trying.iter().copied()) + .chain( + machine + .bridges + .iter() + .filter(|bridge| !path.contains(bridge)) + .flat_map(|bridge| bridge.memory.into_iter().chain(bridge.prefetchable)), + ) + .collect(); + let (at, window) = placement::behind_bridges(path, *prefetchable, *wide, &mut taken, windows, span)?; + machine.trying.push(Window { start: at, end: at + span }); + Some(Candidate::Behind { at, window }) + } + } + } } /// This machine's free runs of one width, under the lock that hands them out. @@ -1442,14 +1549,18 @@ fn with_runs(wide: bool, f: impl FnOnce(&mut Vec) -> T) -> T { f(if wide { &mut machine.high } else { &mut machine.low }) } -/// Put back every address a refused walk took, newest first: a run gives back -/// only the address it handed out last. -fn give_back(wide: bool, refused: &[Option], span: u64) { - with_runs(wide, |runs| { - for candidate in refused.iter().rev().flatten() { - placement::release(runs, *candidate, span); +/// Put back every address a walk took and did not keep, newest first: a run +/// gives back only the address it handed out last. +fn give_back(refused: &[Option], span: u64) { + let mut machine = MACHINE.lock(); + for candidate in refused.iter().rev().flatten() { + match *candidate { + Candidate::Run { wide, reservation } => { + placement::release(if wide { &mut machine.high } else { &mut machine.low }, reservation, span) + } + Candidate::Behind { at, .. } => machine.trying.retain(|tried| tried.start != at), } - }); + } } /// Where this BAR was already put, if a claim before this one put it there. @@ -1479,13 +1590,12 @@ fn was_cut(pci: &PciDevice, index: u8) -> bool { MACHINE.lock().windows.iter().any(|(w, i, _, _)| *w == who && *i == index) } -/// Record `at .. at + span` as this BAR's for the life of the boot. -/// -/// The address is already out of the runs: [`placement::reserve`] took it there -/// under [`with_runs`], which is what keeps a second claim from being offered -/// it. +/// Record `at .. at + span` as this BAR's for the life of the boot, where no +/// later claim is offered it. fn cut(pci: &PciDevice, index: u8, at: u64, span: u64) { - MACHINE.lock().windows.push((requester(pci), index, at, span)); + let mut machine = MACHINE.lock(); + machine.trying.retain(|tried| tried.start != at); + machine.windows.push((requester(pci), index, at, span)); } /// Nothing else on this machine decodes inside the page that is about to be diff --git a/tests/common/qemu.rs b/tests/common/qemu.rs index 4135551468..010e176493 100644 --- a/tests/common/qemu.rs +++ b/tests/common/qemu.rs @@ -691,6 +691,10 @@ pub enum Profile { /// the T14's two are, because a claim never maps the BAR that holds an /// MSI-X table and QEMU puts this one's in its registers' BAR. HeadlessUsbSpare, + /// [`Profile::Headless`] with its virtio NIC behind a `pcie-root-port`: + /// a function a claim may take whose BAR has to land inside the port's + /// windows, as every function behind a bridge on a laptop does. + HeadlessNicBehindPort, /// [`Profile::Headless`] with two of QEMU's `edu`, which the kernel drives /// neither of: two identical functions, each a `drives` row's to start a /// process on. @@ -774,6 +778,7 @@ impl Profile { | Self::HeadlessNoUsb | Self::HeadlessVirtioGpu | Self::HeadlessUsbSpare + | Self::HeadlessNicBehindPort | Self::HeadlessTwoEdu | Self::Metal | Self::MetalNoX2apic @@ -888,6 +893,8 @@ impl Virtio { enum Nic { Absent, Virtio, + /// The virtio NIC on the secondary bus of a root port of its own. + BehindPort, } /// Everything a profile decides about the machine, in one table. A new @@ -1057,6 +1064,7 @@ impl Profile { Self::HeadlessNoIommu => Shape { iommu: None, ..Self::Headless.shape() }, Self::HeadlessNoUsb => Shape { xhci: &[], usb: &[], ..Self::Headless.shape() }, Self::HeadlessVirtioGpu => Shape { virtio_gpu: true, ..Self::Headless.shape() }, + Self::HeadlessNicBehindPort => Shape { nic: Nic::BehindPort, ..Self::Headless.shape() }, Self::HeadlessTwoEdu => Shape { probes: Probes::Edus, ..Self::Headless.shape() }, Self::HeadlessUsbSpare => Shape { xhci: &[XHCI_DEFAULT, XHCI_SPARE], @@ -2335,6 +2343,12 @@ fn qemu_command( "virtio-net-pci-non-transitional,netdev=net0{platform}" )); } + Nic::BehindPort => { + qemu.arg("-device").arg("pcie-root-port,id=nicport,chassis=9"); + qemu.arg("-netdev").arg("user,id=net0").arg("-device").arg(format!( + "virtio-net-pci-non-transitional,netdev=net0,bus=nicport{platform}" + )); + } } if shape.virtio.present() { if shape.virtio.sound() { diff --git a/tests/toyos.rs b/tests/toyos.rs index 748bf0beed..9e6b2e8cf5 100644 --- a/tests/toyos.rs +++ b/tests/toyos.rs @@ -413,6 +413,12 @@ const MACHINE_TESTS: &[&str] = &[ // function is its I219, and a kernel that dies on this takes the bench's // machine down with it. "bar_map_again", + // The same claim with the NIC behind a root port: the BAR has to land + // inside the windows the port forwards, which a host test of the rule + // cannot say the kernel reads off a real bridge. The T14's claimed + // functions are all on bus 0; the AMD bench's xHCI that is not has its + // MSI-X table in its one BAR. + "bar_map_behind_a_bridge", // A region that is not memory the kernel allocated, lent as a window: the // kernel describes the BAR of a function no program of the boot holds, // which on the T14 is its I219, as `bar_map_again` says. @@ -3729,6 +3735,7 @@ fn run_machine_test(name: &str, test_config: &Path) -> Result<(), String> { "machine_shutdown_wire_kept" => power::machine_shutdown_wire_held(test_config, true), "machine_shutdown_wire_at_the_seal" => power::machine_shutdown_wire_at_the_seal(test_config), "bar_map_again" => bar_map_again(test_config), + "bar_map_behind_a_bridge" => bar_map_behind_a_bridge(test_config), "lent_device_memory_refused" => lent_device_memory_refused(test_config), "console_image_boots" => console_image_boots(), "desktop_without_x2apic" => desktop_without_x2apic(), @@ -4075,6 +4082,52 @@ fn bar_map_again(test_config: &Path) -> Result<(), String> { Ok(()) } +/// The virtio NIC behind a root port is claimed and its BAR mapped, and the +/// kernel put the BAR inside a window the port forwards. +fn bar_map_behind_a_bridge(test_config: &Path) -> Result<(), String> { + const JOB: &str = "bar_map_again"; + const FORWARDS: &str = "pcidev: PCI 00:"; + const PLACED: &str = "pcidev: PCI 01:00.0 BAR "; + let bin = qemu::build_toyos_bin(qemu::SUITE_ARCH, &compile::repo_root().join("tests/guest/rust-tests"), JOB); + let options = BootOptions { profile: qemu::Profile::HeadlessNicBehindPort, ..Default::default() }; + let mut qemu = QemuInstance::boot_with_options(test_config, &[], &[(JOB.to_string(), bin)], options); + let boot = serial::Serial::boot(&qemu); + boot.must_be_clean()?; + let range = |line: &str| -> Option<(u64, u64)> { + let (from, to) = line.split_once("..")?; + let hex = |t: &str| u64::from_str_radix(t.trim().strip_prefix("0x")?, 16).ok(); + Some((hex(from.rsplit(' ').next()?)?, hex(to.split(' ').next()?)?)) + }; + let forwarded: Vec<(u64, u64)> = boot + .text() + .lines() + .filter(|l| l.contains(FORWARDS) && l.contains(" forwards ")) + .filter_map(|l| range(l.split_once(" forwards ")?.1)) + .collect(); + if forwarded.is_empty() { + return Err(format!("no root port said what it forwards:\n{}", boot.text())); + } + let result = qemu.run_test("test_rs_bar_map_again", Duration::from_secs(60)); + if let Some(why) = &result.error { + return Err(format!("{why}\nthe job said:\n{}", result.stdout)); + } + if result.exit_code != Some(0) { + return Err(format!("the job ended {:?}:\n{}", result.exit_code, result.stdout)); + } + let Some(placed) = result.serial.lines().find(|l| l.contains(PLACED) && l.contains(" placed at ")) else { + return Err(format!("the NIC's BAR was never placed:\n{}", result.serial)); + }; + let at = placed + .split_once(" placed at ") + .and_then(|(_, rest)| u64::from_str_radix(rest.split_whitespace().next()?.strip_prefix("0x")?, 16).ok()) + .ok_or_else(|| format!("an unreadable placement: {placed:?}"))?; + if !forwarded.iter().any(|(start, end)| *start <= at && at < *end) { + return Err(format!("{placed:?} is inside none of what the port forwards, {forwarded:x?}")); + } + eprintln!(" [claims] {}", placed.trim()); + Ok(()) +} + /// The kernel answers a lent window that is the virtio NIC's BAR as no memory /// it allocated, by `fstat`, so no reader maps it as a region. fn lent_device_memory_refused(test_config: &Path) -> Result<(), String> {