Skip to content

Commit b9132c3

Browse files
torvaldstorvalds
committed
Merge tag 'cxl-for-5.18' of git://git.kernel.org/pub/scm/linux/kernel/git/cxl/cxl
Pull CXL (Compute Express Link) updates from Dan Williams: "This development cycle extends the subsystem to discover CXL resources throughout a CXL/PCIe switch topology and respond to hot add/remove events anywhere in that topology. This is more foundational infrastructure in preparation for dynamic memory region provisioning support. Recall that CXL memory regions, as the new "Theory of Operation" section of Documentation/driver-api/cxl/memory-devices.rst describes, bring storage volume striping semantics to memory. The hot add/remove behavior is validated with extensions to the cxl_test unit test environment and this test in the cxl-cli test suite: https://github.com/pmem/ndctl/blob/djbw/for-74/cxl/test/cxl-topology.sh Summary: - Add a driver for 'struct cxl_memdev' objects responsible for CXL.mem operation as distinct from 'cxl_pci' mailbox operations. Its primary responsibility is enumerating an endpoint 'struct cxl_port' and all the 'struct cxl_port' instances between an endpoint and the CXL platform root. - Add a driver for 'struct cxl_port' objects responsible for enumerating and operating all Host-managed Device Memory (HDM) decoder resources between the platform-level CXL memory description, all intervening host bridges / switches, and the HDM resources in endpoints. - Update the cxl_pci driver to validate CXL.mem operation precursors to HDM decoder operation like ready-polling, and legacy CXL 1.1 DVSEC based CXL.mem configuration. - Add basic lockdep coverage for usage of device_lock() on CXL subsystem objects similar to what exists for LIBNVDIMM. Include a compile-time switch for which subsystem to validate at run-time. - Update cxl_test to emulate a one level switch topology. - Document a "Theory of Operation" for the subsystem. - Add 'numa_node' and 'serial' attributes to cxl_memdev sysfs - Include miscellaneous fixes for spec / QEMU CXL emulation compatibility and static analysis reports" * tag 'cxl-for-5.18' of git://git.kernel.org/pub/scm/linux/kernel/git/cxl/cxl: (48 commits) cxl/core/port: Fix NULL but dereferenced coccicheck error cxl/port: Hold port reference until decoder release cxl/port: Fix endpoint refcount leak cxl/core: Fix cxl_device_lock() class detection cxl/core/port: Fix unregister_port() lock assertion cxl/regs: Fix size of CXL Capability Header Register cxl/core/port: Handle invalid decoders cxl/core/port: Fix / relax decoder target enumeration tools/testing/cxl: Add a physical_node link tools/testing/cxl: Enumerate mock decoders tools/testing/cxl: Mock one level of switches tools/testing/cxl: Fix root port to host bridge assignment tools/testing/cxl: Mock dvsec_ranges() cxl/core/port: Add endpoint decoders cxl/core: Move target_list out of base decoder attributes cxl/mem: Add the cxl_mem driver cxl/core/port: Add switch port enumeration cxl/memdev: Add numa_node attribute cxl/pci: Emit device serial number cxl/pci: Implement wait for media active ...
2 parents b14ffae + 05e8155 commit b9132c3

32 files changed

Lines changed: 3725 additions & 1225 deletions

File tree

‎Documentation/ABI/testing/sysfs-bus-cxl‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,12 @@
1+
What: /sys/bus/cxl/flush
2+
Date: Januarry, 2022
3+
KernelVersion: v5.18
4+
Contact: linux-cxl@vger.kernel.org
5+
Description:
6+
(WO) If userspace manually unbinds a port the kernel schedules
7+
all descendant memdevs for unbind. Writing '1' to this attribute
8+
flushes that work.
9+
110
What: /sys/bus/cxl/devices/memX/firmware_version
211
Date: December, 2020
312
KernelVersion: v5.12
@@ -25,6 +34,24 @@ Description:
2534
identically named field in the Identify Memory Device Output
2635
Payload in the CXL-2.0 specification.
2736

37+
What: /sys/bus/cxl/devices/memX/serial
38+
Date: January, 2022
39+
KernelVersion: v5.18
40+
Contact: linux-cxl@vger.kernel.org
41+
Description:
42+
(RO) 64-bit serial number per the PCIe Device Serial Number
43+
capability. Mandatory for CXL devices, see CXL 2.0 8.1.12.2
44+
Memory Device PCIe Capabilities and Extended Capabilities.
45+
46+
What: /sys/bus/cxl/devices/memX/numa_node
47+
Date: January, 2022
48+
KernelVersion: v5.18
49+
Contact: linux-cxl@vger.kernel.org
50+
Description:
51+
(RO) If NUMA is enabled and the platform has affinitized the
52+
host PCI device for this memory device, emit the CPU node
53+
affinity for this device.
54+
2855
What: /sys/bus/cxl/devices/*/devtype
2956
Date: June, 2021
3057
KernelVersion: v5.14
@@ -34,6 +61,15 @@ Description:
3461
the same value communicated in the DEVTYPE environment variable
3562
for uevents for devices on the "cxl" bus.
3663

64+
What: /sys/bus/cxl/devices/*/modalias
65+
Date: December, 2021
66+
KernelVersion: v5.18
67+
Contact: linux-cxl@vger.kernel.org
68+
Description:
69+
CXL device objects export the modalias attribute which mirrors
70+
the same value communicated in the MODALIAS environment variable
71+
for uevents for devices on the "cxl" bus.
72+
3773
What: /sys/bus/cxl/devices/portX/uport
3874
Date: June, 2021
3975
KernelVersion: v5.14

‎Documentation/driver-api/cxl/memory-devices.rst‎

Lines changed: 313 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,303 @@ that optionally define a device's contribution to an interleaved address
1414
range across multiple devices underneath a host-bridge or interleaved
1515
across host-bridges.
1616

17+
CXL Bus: Theory of Operation
18+
============================
19+
Similar to how a RAID driver takes disk objects and assembles them into a new
20+
logical device, the CXL subsystem is tasked to take PCIe and ACPI objects and
21+
assemble them into a CXL.mem decode topology. The need for runtime configuration
22+
of the CXL.mem topology is also similar to RAID in that different environments
23+
with the same hardware configuration may decide to assemble the topology in
24+
contrasting ways. One may choose performance (RAID0) striping memory across
25+
multiple Host Bridges and endpoints while another may opt for fault tolerance
26+
and disable any striping in the CXL.mem topology.
27+
28+
Platform firmware enumerates a menu of interleave options at the "CXL root port"
29+
(Linux term for the top of the CXL decode topology). From there, PCIe topology
30+
dictates which endpoints can participate in which Host Bridge decode regimes.
31+
Each PCIe Switch in the path between the root and an endpoint introduces a point
32+
at which the interleave can be split. For example platform firmware may say at a
33+
given range only decodes to 1 one Host Bridge, but that Host Bridge may in turn
34+
interleave cycles across multiple Root Ports. An intervening Switch between a
35+
port and an endpoint may interleave cycles across multiple Downstream Switch
36+
Ports, etc.
37+
38+
Here is a sample listing of a CXL topology defined by 'cxl_test'. The 'cxl_test'
39+
module generates an emulated CXL topology of 2 Host Bridges each with 2 Root
40+
Ports. Each of those Root Ports are connected to 2-way switches with endpoints
41+
connected to those downstream ports for a total of 8 endpoints::
42+
43+
# cxl list -BEMPu -b cxl_test
44+
{
45+
"bus":"root3",
46+
"provider":"cxl_test",
47+
"ports:root3":[
48+
{
49+
"port":"port5",
50+
"host":"cxl_host_bridge.1",
51+
"ports:port5":[
52+
{
53+
"port":"port8",
54+
"host":"cxl_switch_uport.1",
55+
"endpoints:port8":[
56+
{
57+
"endpoint":"endpoint9",
58+
"host":"mem2",
59+
"memdev":{
60+
"memdev":"mem2",
61+
"pmem_size":"256.00 MiB (268.44 MB)",
62+
"ram_size":"256.00 MiB (268.44 MB)",
63+
"serial":"0x1",
64+
"numa_node":1,
65+
"host":"cxl_mem.1"
66+
}
67+
},
68+
{
69+
"endpoint":"endpoint15",
70+
"host":"mem6",
71+
"memdev":{
72+
"memdev":"mem6",
73+
"pmem_size":"256.00 MiB (268.44 MB)",
74+
"ram_size":"256.00 MiB (268.44 MB)",
75+
"serial":"0x5",
76+
"numa_node":1,
77+
"host":"cxl_mem.5"
78+
}
79+
}
80+
]
81+
},
82+
{
83+
"port":"port12",
84+
"host":"cxl_switch_uport.3",
85+
"endpoints:port12":[
86+
{
87+
"endpoint":"endpoint17",
88+
"host":"mem8",
89+
"memdev":{
90+
"memdev":"mem8",
91+
"pmem_size":"256.00 MiB (268.44 MB)",
92+
"ram_size":"256.00 MiB (268.44 MB)",
93+
"serial":"0x7",
94+
"numa_node":1,
95+
"host":"cxl_mem.7"
96+
}
97+
},
98+
{
99+
"endpoint":"endpoint13",
100+
"host":"mem4",
101+
"memdev":{
102+
"memdev":"mem4",
103+
"pmem_size":"256.00 MiB (268.44 MB)",
104+
"ram_size":"256.00 MiB (268.44 MB)",
105+
"serial":"0x3",
106+
"numa_node":1,
107+
"host":"cxl_mem.3"
108+
}
109+
}
110+
]
111+
}
112+
]
113+
},
114+
{
115+
"port":"port4",
116+
"host":"cxl_host_bridge.0",
117+
"ports:port4":[
118+
{
119+
"port":"port6",
120+
"host":"cxl_switch_uport.0",
121+
"endpoints:port6":[
122+
{
123+
"endpoint":"endpoint7",
124+
"host":"mem1",
125+
"memdev":{
126+
"memdev":"mem1",
127+
"pmem_size":"256.00 MiB (268.44 MB)",
128+
"ram_size":"256.00 MiB (268.44 MB)",
129+
"serial":"0",
130+
"numa_node":0,
131+
"host":"cxl_mem.0"
132+
}
133+
},
134+
{
135+
"endpoint":"endpoint14",
136+
"host":"mem5",
137+
"memdev":{
138+
"memdev":"mem5",
139+
"pmem_size":"256.00 MiB (268.44 MB)",
140+
"ram_size":"256.00 MiB (268.44 MB)",
141+
"serial":"0x4",
142+
"numa_node":0,
143+
"host":"cxl_mem.4"
144+
}
145+
}
146+
]
147+
},
148+
{
149+
"port":"port10",
150+
"host":"cxl_switch_uport.2",
151+
"endpoints:port10":[
152+
{
153+
"endpoint":"endpoint16",
154+
"host":"mem7",
155+
"memdev":{
156+
"memdev":"mem7",
157+
"pmem_size":"256.00 MiB (268.44 MB)",
158+
"ram_size":"256.00 MiB (268.44 MB)",
159+
"serial":"0x6",
160+
"numa_node":0,
161+
"host":"cxl_mem.6"
162+
}
163+
},
164+
{
165+
"endpoint":"endpoint11",
166+
"host":"mem3",
167+
"memdev":{
168+
"memdev":"mem3",
169+
"pmem_size":"256.00 MiB (268.44 MB)",
170+
"ram_size":"256.00 MiB (268.44 MB)",
171+
"serial":"0x2",
172+
"numa_node":0,
173+
"host":"cxl_mem.2"
174+
}
175+
}
176+
]
177+
}
178+
]
179+
}
180+
]
181+
}
182+
183+
In that listing each "root", "port", and "endpoint" object correspond a kernel
184+
'struct cxl_port' object. A 'cxl_port' is a device that can decode CXL.mem to
185+
its descendants. So "root" claims non-PCIe enumerable platform decode ranges and
186+
decodes them to "ports", "ports" decode to "endpoints", and "endpoints"
187+
represent the decode from SPA (System Physical Address) to DPA (Device Physical
188+
Address).
189+
190+
Continuing the RAID analogy, disks have both topology metadata and on device
191+
metadata that determine RAID set assembly. CXL Port topology and CXL Port link
192+
status is metadata for CXL.mem set assembly. The CXL Port topology is enumerated
193+
by the arrival of a CXL.mem device. I.e. unless and until the PCIe core attaches
194+
the cxl_pci driver to a CXL Memory Expander there is no role for CXL Port
195+
objects. Conversely for hot-unplug / removal scenarios, there is no need for
196+
the Linux PCI core to tear down switch-level CXL resources because the endpoint
197+
->remove() event cleans up the port data that was established to support that
198+
Memory Expander.
199+
200+
The port metadata and potential decode schemes that a give memory device may
201+
participate can be determined via a command like::
202+
203+
# cxl list -BDMu -d root -m mem3
204+
{
205+
"bus":"root3",
206+
"provider":"cxl_test",
207+
"decoders:root3":[
208+
{
209+
"decoder":"decoder3.1",
210+
"resource":"0x8030000000",
211+
"size":"512.00 MiB (536.87 MB)",
212+
"volatile_capable":true,
213+
"nr_targets":2
214+
},
215+
{
216+
"decoder":"decoder3.3",
217+
"resource":"0x8060000000",
218+
"size":"512.00 MiB (536.87 MB)",
219+
"pmem_capable":true,
220+
"nr_targets":2
221+
},
222+
{
223+
"decoder":"decoder3.0",
224+
"resource":"0x8020000000",
225+
"size":"256.00 MiB (268.44 MB)",
226+
"volatile_capable":true,
227+
"nr_targets":1
228+
},
229+
{
230+
"decoder":"decoder3.2",
231+
"resource":"0x8050000000",
232+
"size":"256.00 MiB (268.44 MB)",
233+
"pmem_capable":true,
234+
"nr_targets":1
235+
}
236+
],
237+
"memdevs:root3":[
238+
{
239+
"memdev":"mem3",
240+
"pmem_size":"256.00 MiB (268.44 MB)",
241+
"ram_size":"256.00 MiB (268.44 MB)",
242+
"serial":"0x2",
243+
"numa_node":0,
244+
"host":"cxl_mem.2"
245+
}
246+
]
247+
}
248+
249+
...which queries the CXL topology to ask "given CXL Memory Expander with a kernel
250+
device name of 'mem3' which platform level decode ranges may this device
251+
participate". A given expander can participate in multiple CXL.mem interleave
252+
sets simultaneously depending on how many decoder resource it has. In this
253+
example mem3 can participate in one or more of a PMEM interleave that spans to
254+
Host Bridges, a PMEM interleave that targets a single Host Bridge, a Volatile
255+
memory interleave that spans 2 Host Bridges, and a Volatile memory interleave
256+
that only targets a single Host Bridge.
257+
258+
Conversely the memory devices that can participate in a given platform level
259+
decode scheme can be determined via a command like the following::
260+
261+
# cxl list -MDu -d 3.2
262+
[
263+
{
264+
"memdevs":[
265+
{
266+
"memdev":"mem1",
267+
"pmem_size":"256.00 MiB (268.44 MB)",
268+
"ram_size":"256.00 MiB (268.44 MB)",
269+
"serial":"0",
270+
"numa_node":0,
271+
"host":"cxl_mem.0"
272+
},
273+
{
274+
"memdev":"mem5",
275+
"pmem_size":"256.00 MiB (268.44 MB)",
276+
"ram_size":"256.00 MiB (268.44 MB)",
277+
"serial":"0x4",
278+
"numa_node":0,
279+
"host":"cxl_mem.4"
280+
},
281+
{
282+
"memdev":"mem7",
283+
"pmem_size":"256.00 MiB (268.44 MB)",
284+
"ram_size":"256.00 MiB (268.44 MB)",
285+
"serial":"0x6",
286+
"numa_node":0,
287+
"host":"cxl_mem.6"
288+
},
289+
{
290+
"memdev":"mem3",
291+
"pmem_size":"256.00 MiB (268.44 MB)",
292+
"ram_size":"256.00 MiB (268.44 MB)",
293+
"serial":"0x2",
294+
"numa_node":0,
295+
"host":"cxl_mem.2"
296+
}
297+
]
298+
},
299+
{
300+
"root decoders":[
301+
{
302+
"decoder":"decoder3.2",
303+
"resource":"0x8050000000",
304+
"size":"256.00 MiB (268.44 MB)",
305+
"pmem_capable":true,
306+
"nr_targets":1
307+
}
308+
]
309+
}
310+
]
311+
312+
...where the naming scheme for decoders is "decoder<port_id>.<instance_id>".
313+
17314
Driver Infrastructure
18315
=====================
19316

@@ -28,6 +325,14 @@ CXL Memory Device
28325
.. kernel-doc:: drivers/cxl/pci.c
29326
:internal:
30327

328+
.. kernel-doc:: drivers/cxl/mem.c
329+
:doc: cxl mem
330+
331+
CXL Port
332+
--------
333+
.. kernel-doc:: drivers/cxl/port.c
334+
:doc: cxl port
335+
31336
CXL Core
32337
--------
33338
.. kernel-doc:: drivers/cxl/cxl.h
@@ -36,10 +341,16 @@ CXL Core
36341
.. kernel-doc:: drivers/cxl/cxl.h
37342
:internal:
38343

39-
.. kernel-doc:: drivers/cxl/core/bus.c
344+
.. kernel-doc:: drivers/cxl/core/port.c
40345
:doc: cxl core
41346

42-
.. kernel-doc:: drivers/cxl/core/bus.c
347+
.. kernel-doc:: drivers/cxl/core/port.c
348+
:identifiers:
349+
350+
.. kernel-doc:: drivers/cxl/core/pci.c
351+
:doc: cxl core pci
352+
353+
.. kernel-doc:: drivers/cxl/core/pci.c
43354
:identifiers:
44355

45356
.. kernel-doc:: drivers/cxl/core/pmem.c

0 commit comments

Comments
 (0)