[Bug]: Recording fails on niri + NVIDIA: every DMA-BUF frame is marked corrupted #1503

Search existing issues

• I have searched the existing issues

Describe the bug

Screen recording still fails on niri + NVIDIA. The cursor fallback from #1363 works, but after the portal stream starts, every captured frame is marked as corrupted and the recording never starts.

The important part is that the same niri + NVIDIA + GNOME portal setup works in OBS: OBS can record the same monitor through the same portal, with the cursor visible. So the portal and the compositor are able to provide usable frames. OpenScreen is the only application where the frames arrive but are all rejected as corrupted.

This looks like the DMA-BUF issue described in #1244.

Expected behavior

The recording should start and capture the selected monitor.

To Reproduce

1. Use niri on Wayland with proprietary NVIDIA drivers.
2. Launch Openscreen-Linux-2.2.2.AppImage.
3. Select a display and start recording.
4. The recording starts for a split second, and the monitor is selected in the xdg-desktop-portal-gnome interface. After selection, recording does not even begin, and errors appear in the logs.

Screenshots

No response

OS

Linux

OS Version

NixOS

Other OS

No response

Browser

Other

Browser Version

159.0-1

Other Browser

Librewolf

Device Type

Desktop

Other Device

No response

Additional context

On NixOS, the AppImage initially failed before capture started because the PipeWire helper could not load libva-drm.so.2 via dlopen:

[capture-linux] implib-gen: libva-drm.so.2: failed to load library
'libva-drm.so.2' via dlopen: libva-drm.so.2: cannot open shared object file:
No such file or directory

openscreen-pipewire-helper: libva-drm.so.2.init.c:122: load_library:
Assertion `0 && "Assertion in generated code"' failed.

Could not prepare the native Linux capture:
Error: the Linux capture helper exited (code=null, signal=SIGABRT)

This was an environment issue on NixOS, not an OpenScreen bug: exposing the NixOS nix-ld library path fixed it:

env LD_LIBRARY_PATH=/run/current-system/sw/share/nix-ld/lib \
  appimage-run ./Openscreen-Linux-2.2.2.AppImage

After that, libva-drm.so.2 loads and the PipeWire/portal capture starts. However, recording still fails because every DMA-BUF frame is rejected as corrupted:

[capture-linux] [AVHWFramesContext @ 0x...] Unsupported format: bgr0.

[capture-linux] portal stream started
{"width":2560,"height":1080,"sourceKind":"monitor"}

[capture-linux] frame-dropped: the compositor's buffer failed validation
(producer-marked-frame-corrupted), so this frame was dropped:
dataType=DmaBuf stride=10240 width=2560 height=1080
chunkOffset=0 chunkSize=0 available=11796480

The recording never receives a usable frame and aborts after 10 seconds:

Failed to start native Linux recording: Error: OpenScreen could not start
screen capture: the screen share delivered no frame within 10 s.

This appears to be the same underlying issue as #1244: on proprietary NVIDIA, the VAAPI import probe fails, and the helper may advertise DMA-BUF modifiers that the NVIDIA driver cannot import or render. The same niri + NVIDIA + GNOME portal setup works in OBS, which records the same monitor successfully.

A shared-memory (memfd) fallback would likely make recording work here, even if it requires a CPU copy.