Add basic low-level operatations for CXL-capable PCI devices

[gonzoua]

Description

The pull request introduced a new CxlIo protocol that provides a basic operations to communicate with CXL-capable devices:

• Read area of register block of certain type
• Write area of register block of certain type
• Perform a DOE transaction: send a request and read a response.

CxlDxe driver implements the protocol and binds it to CXL-capable PCI devices.

Driver code is partially based on two patchsets:

• Ssir/v6 uefi cxl driver fmp final p5 #11214
• https://edk2.groups.io/g/rfc/message/824

The patchsets' authors are attributed in the commit message.

On the Google's side @NicholasGraves and Ryan Heise did most of the work on the CxlIo design and implementation. The development history was squashed for brevity.

The PR also introduces cxl command for a basic introspection of CXL devices:

• List all CXL devices
• Dump CDAT structures for a specified device

• Breaking change?
• Impacts security?
• Includes tests?

How This Was Tested

Build

host$ docker run --rm -ti -v "${HOME}":"${HOME}" -e EDK2_DOCKER_USER_HOME="${HOME}" \
  -v $(pwd):/host -w /host --name edk2-ubuntu-dev \
  ghcr.io/tianocore/containers/ubuntu-22-dev:latest\
 /bin/bash
docker$ export WORKSPACE=$(pwd)
docker$ export EDK_TOOLS_PATH=$(pwd)/BaseTools
docker$ source edksetup.sh]
docker$ make -j$(nproc) -C BaseTools
docker$ build -t GCC -a X64 -p OvmfPkg/OvmfPkgX64.dsc -D FD_SIZE_4MB -D NVME

Test

Start QEMU:

host$ export CODE=Build/OvmfX64/DEBUG_GCC/FV/OVMF_CODE.fd
host$ export VARS=Build/OvmfX64/DEBUG_GCC/FV/OVMF_VARS.fd

host$ qemu-system-x86_64 -drive if=none,id=code,format=raw,file=${CODE},readonly=on \
  -drive if=none,id=vars,format=raw,file=${VARS},snapshot=on \
  -drive file=fat:rw:root,id=fat1,format=vvfat \
  -machine q35,cxl=on,pflash0=code,pflash1=vars \
  -object memory-backend-ram,id=m1,size=1G \
  -object memory-backend-ram,id=m2,size=512M \
  -object memory-backend-file,id=cxl-lsa0,share=on,mem-path=lsa0.raw,size=4k \
  -object memory-backend-file,id=cxl-lsa1,share=on,mem-path=lsa1.raw,size=4k \
  -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \
  -device cxl-rp,port=0,bus=cxl.1,id=root_port_cxl1,chassis=0,slot=2 \
  -device cxl-rp,port=1,bus=cxl.1,id=root_port_cxl2,chassis=1,slot=2 \
  -device cxl-type3,bus=root_port_cxl1,memdev=m1,id=cxl-mem0,lsa=cxl-lsa0 \
  -device cxl-type3,bus=root_port_cxl2,memdev=m2,id=cxl-mem1,lsa=cxl-lsa1 \
  -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=10G \
  -display none -nographic

Navigate menu and start UEFI shell, in shell use the following commands

Shell> cxl
    00   0D   00    00
             Vendor 8086 Device 0D93
    00   0E   00    00
             Vendor 8086 Device 0D93
Shell> cxl 0d 00 00
Device Scoped Memory Affinity (DSMAS):
   Type          00
   DSMADHandle   00
   Flags         04
   DPABase       0
   DPALength     40000000
...
Shell> cxl 0e 00 00
Device Scoped Memory Affinity (DSMAS):
   Type          00
   DSMADHandle   00
   Flags         04
   DPABase       0
   DPALength     20000000

Integration Instructions

N/A

[ardbiesheuvel]

Thanks for this contribution.

High level feedback before even having looked at the actual code: MdePkg is meant for protocols and other definitions that are derived from an external (i.e., non-edk2) spec, e.g., UEFI, PI, ACPI, etc. As I understand from the commit log, the CxlIo protocol being proposed here is not defined in such a spec today.

So there are two options:

• propose/contribute the CXL I/O protocol to UEFI or CXL or another spec (which may take a long time),
• move the code to MdeModulePkg, and omit EFI_ prefixes for identifiers (use EDKII_ instead)

I think the second option is more appropriate here, and we can always start a parallel track to promote the protocol once it lands in one of the industry specifications.

(Btw no need to go and repaint everything right way - we can take a bit of time to review and discuss the implementation first)

[leiflindholm]

@gonzoua can you please add some comments as to what you're changing with each update?

[gonzoua]

@leiflindholm Do you mean comments for PR iterations (force-pushes)?

Series of force-pushes up until commit 045be1f was a CI faliures clean-up: there was a number of issues with Doxygen comments and implicit int types conversion that caused VS build to fail.

Commit 3ee87dd push - addressed @ardbiesheuvel's comment. I moved the CxlIo protocol to MdeModulePkg and switched it to using EDKII prefix/namespace.

Commit 198c86b push - commit message fix, I forgot to change MdePkg to MdeModulePkg in the previous modification.