[Bug]: SIGSEGV in VBoxDxVk.so (D3D11CommonTexture::GetPlaneCount) - NULL m_image deref when vmsvga3d creates a view for a STAGING texture

[zgyhim]

Version

7.2.16 (Other or before v7.1.0 — not listed in the template)

Host OS Type

Linux

Host OS name + version

Arch Linux (rolling), kernel 7.1.9-arch1-2, X11 session (Xorg on Intel iGPU)

Host Architecture

x86

Guest OS Type

Windows

Guest Architecture

x86

Guest OS name + version

Windows 10 build 19041 (2004/20H1, NOT 22H2 — 22H2 is build 19045), VirtualBox Guest Additions 7.2.8 in-guest

Component

3D Support

What happened?

The VirtualBoxVM process segfaults (SIGSEGV, SEGV_MAPERR) deterministically in the VBoxDxVk.so translation layer while running a Windows 10 guest with 3D acceleration enabled. The crash happens in the VMSVGA FIFO thread and is triggered from the host-side vmsvga3d code (VBoxDD.so). It is fully reproducible (crashed twice at the exact same instruction) and appears to be triggered shortly after the guest submits a 3D/present workload; both times ~8–9 minutes into a session.

Host kernel/coredump:

kernel: VMSVGA FIFO[11130]: segfault at 44 ip 00007ffb5fd2dd2c ... in VBoxDxVk.so[12dd2c,7ffb5fc2c000+2a8000] likely on CPU 4
systemd-coredump: Process 11062 (VirtualBoxVM) ... terminated abnormally with signal 11/SEGV
Stack trace of thread 11130 (VMSVGA FIFO):
#0  VBoxDxVk.so + 0x12dd2c
#1  VBoxDxVk.so + 0x8b939
#2  VBoxDxVk.so + 0x929b4
#3  VBoxDD.so + 0x12b73b
... (VBoxDD.so vmsvga FIFO path)

Fault address is 0x44 (NULL + 0x44).

Root cause (source + disassembly)

Crash site, disassembled from the shipped VBoxDxVk.so:

12dd20 <dxvk::D3D11CommonTexture::GetPlaneCount() const>:
12dd25  mov rax, QWORD PTR [rdi+0x80]   ; this->m_image
12dd2c  mov edi, DWORD PTR [rax+0x44]   ; <-- m_image == NULL -> read 0x44 -> SEGV

Corresponding source (src/libs/dxvk-2.6.2/src/d3d11/d3d11_texture.cpp):

uint32_t D3D11CommonTexture::GetPlaneCount() const {
  return vk::getPlaneCount(m_image->formatInfo()->aspectMask);   // m_image is NULL
}

m_image is NULL because the constructor skips Vulkan image creation for D3D11_USAGE_STAGING textures (d3d11_texture.cpp):

// Skip image creation if possible
if (m_mapMode == D3D11_COMMON_TEXTURE_MAP_MODE_STAGING)
  return;

Trigger path (host side)

The vmsvga3d host code creates a D3D11_USAGE_STAGING texture when the guest surface carries SVGA3D_SURFACE_STAGING_UPLOAD / STAGING_DOWNLOAD flags (src/VBox/Devices/Graphics/DevVGA-SVGA3d-dx-dx11.cpp):

if (pSurface->f.surfaceFlags & (SVGA3D_SURFACE_STAGING_UPLOAD | SVGA3D_SURFACE_STAGING_DOWNLOAD))
    Usage = D3D11_USAGE_STAGING;

Such a STAGING surface later enters a view-creation path (e.g. vmsvga3dBackDXPresentBlt creates an RTV for pDstResource and an SRV for pSrcResource, and DX surface copy/readback paths). D3D11 forbids creating views on STAGING resources, but the VBox host code does not validate this. On the dxvk side, D3D11Device::CreateShaderResourceView calls GetViewPlaneIndex() unconditionally as its first step, before any bind-flag/usage check (src/libs/dxvk-2.6.2/src/d3d11/d3d11_device.cpp):

uint32_t plane = GetViewPlaneIndex(pResource, pDesc ? pDesc->Format : DXGI_FORMAT_UNKNOWN);
...
uint32_t planeCount = texture->GetPlaneCount();   // NULL m_image -> crash

So an illegal STAGING-view request turns into a hard process crash instead of E_INVALIDARG.

Environment notes

• Host is a hybrid-GPU laptop: Intel TigerLake-LP GT2 [Iris Xe Graphics] (used by VirtualBox; glxinfo renders on it) + NVIDIA TU117M [GeForce MX450] (PRIME offload, idle). CPU: 11th Gen Intel Core i5-1135G7. Mesa 26.2.1-arch1.1.
• VM config: graphicscontroller=vboxsvga, accelerate-3d=on, 256 MB VRAM.
• Upgraded the host VirtualBox package to 7.2.16 before the second crash (the only installed extension pack is the VNC 7.2.16 pack; no Oracle PUEL extension pack is installed). The exact same crash (same 0x12dd2c offset) recurred. The host CPU was ~95 °C at the time, but since both crashes are at the identical instruction this is a deterministic bug, not thermal instability.

How can we reproduce this?

1. Host: Arch Linux with Mesa on Intel Iris Xe.
2. Create a Windows 10 guest with --graphicscontroller vboxsvga --accelerate-3d on.
3. Run a 3D workload (desktop composition / video / games) in the guest for a few minutes; the guest vmsvga driver submits STAGING_UPLOAD/DOWNLOAD surfaces that later get used in present-blit / copy+readback paths.
4. VM dies with SIGSEGV in VBoxDxVk.so (D3D11CommonTexture::GetPlaneCount), VMSVGA FIFO thread.

A reliable standalone repro inside the guest is not trivial to pin down without driver-level tracing, but the code path above is deterministic: creating any D3D11 view on a STAGING vmsvga surface hits the NULL m_image dereference.

Suggested fixes

1. dxvk side: guard D3D11CommonTexture::GetPlaneCount() (and GetImage()->info() in CheckViewCompatibility) against a NULL m_image, and/or have D3D11Device::CreateShaderResourceView/RTV/UAV return E_INVALIDARG for D3D11_USAGE_STAGING resources (as the Windows runtime does) before calling GetViewPlaneIndex.
2. VBox host side: reject view creation for surfaces created with D3D11_USAGE_STAGING in the vmsvga3d present/copy paths.

I can attach the full VBox.log (ends abruptly at the crash, no clean shutdown) and the coredump backtraces on request.