[GSD-13433] [xe] Small-BAR configs are rejected while i915 allows them — could this be a debug key?

[s-tory]

Summary

On the xe driver, Level Zero refuses to initialize a discrete GPU when a small BAR is detected, and the device disappears from enumeration entirely. On i915 the same condition is explicitly allowed. The difference is a hardcoded constant in each IoctlHelper, not a policy that can be overridden.

I would like to ask whether this asymmetry is intentional for xe, and whether it would be reasonable to gate it behind a debug key so that small-BAR configurations can at least be attempted.

Where this happens

shared/source/os_interface/linux/drm_neo.cpp:

} else if (getMemoryInfo()->isSmallBarDetected()) {
    IoFunctions::fprintf(stderr, "WARNING: Resizable BAR not detected for device %s\n", getPciPath().c_str());
    if (!ioctlHelper->isSmallBarConfigAllowed()) {
        return -1;
    }
}

smallBarDetected is computed in shared/source/os_interface/linux/memory_info.cpp:

smallBarDetected = std::ranges::any_of(localMemoryRegions,
                                       [](const MemoryRegion &region) {
                                           return (region.cpuVisibleSize && region.cpuVisibleSize < region.probedSize);
                                       });

And the two implementations differ:

// shared/source/os_interface/linux/ioctl_helper.h  (IoctlHelperI915)
bool isSmallBarConfigAllowed() const override { return true; }

// shared/source/os_interface/linux/xe/ioctl_helper_xe.h  (IoctlHelperXe)
bool isSmallBarConfigAllowed() const override { return false; }

So on i915 a small-BAR device continues past the check with a warning, and on xe it returns -1 and is never enumerated. Since Battlemage and later are xe-only, there is no way to fall back to the permissive path.

My case

An Arc B580 in a Thunderbolt/USB4 enclosure. The card enumerates on the PCI bus, binds to xe, and its edge connector trains at the expected 16GT/s x4 for a USB4 tunnel. Only the BAR stays small:

Device         0000:54:00.0  (Intel Arc B580, 12GB)
BAR2           0x2030000000, 256 MiB
resize caps    0x7f00           (256MB .. 16GB advertised)
driver         xe
NEO            26.31.39395.13
kernel         7.0.0-31

$ lspci -vv
  B580  BAR 2: current size: 256MB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB
  iGPU  BAR 2: current size: 256MB, supported: 256MB

The BAR cannot be grown on this host. Firmware reserves only a few hundred MB of prefetchable space per tunneled PCIe root port (I measured 448 MiB and 711 MiB across boots), and the BAR lands at the start of that window, so it is only 256MB-aligned and pci_resize_resource rejects every size from 512MB upward. I raised this separately on linux-pci, and Mika Westerberg confirmed the shape of it:

Typically there is just certain amount of resources allocated for each PCIe root port that gets tunneled so the way to do this is to increase that in the BIOS. Having said that most of the vendors don't actually allow it to be changed.

https://lore.kernel.org/linux-pci/20260911090800.GZ106095@black.igk.intel.com/

I have filed a request with the laptop vendor as well, but that path is slow and may not land at all. Meanwhile the card is completely unusable for compute even though nothing else about it is wrong.

Why I think a small BAR may be acceptable for my workload

I understand the reasoning in the FAQ — that the driver does not initialize without Resizable BAR in order to avoid a subpar experience. For interactive and graphics workloads that makes sense to me.

My workload is fine-tuning imitation-learning policies for robot manipulation — action-chunking transformers today, vision-language-action models next. Weights and activations live in device memory for the duration of training; the host writes batches in and reads scalars out. It is close to the least BAR-sensitive thing one can run on a GPU.

The integrated Arc 140V on the same machine has a 256MB BAR that cannot be resized at all (supported: 256MB), and it fine-tunes these policies without trouble — that is measured, at batch sizes up to 36 for my input shape. It is an integrated GPU sharing system memory rather than a discrete card with its own local memory, so it is not a strict equivalent. It is simply the closest evidence I have that a 256MB aperture is not by itself fatal to this kind of work.

I fully accept that performance may be poor, possibly much worse than a resizable-BAR configuration. I would like to be able to measure that rather than be unable to start.

Questions

1. Is isSmallBarConfigAllowed() == false for xe a deliberate policy decision, or does it reflect a functional limitation in the xe path that does not exist on i915?
2. If it is policy, would you consider gating it behind a debug key — something like AllowSmallBarConfig read via NEOReadDebugKeys=1 — so that users can opt in and accept the performance consequences?
3. If it is a functional limitation, is there a description of what breaks, so that it is clear this is not worth pursuing?

I checked the debug variables in shared/source/debug_settings/debug_variables_base.inl and could not find an existing key that reaches this branch; ForceLocalMemoryAccessMode, EnableLocalMemory and OverrideGpuAddressSpace all leave the check in place, which matches the code above since the branch does not consult debugManager.

Thanks for reading, and thanks for the runtime.

[s-tory]

Follow-up: I answered my own question 1 experimentally — it looks functional, not just policy

After filing this I patched a copy of the shipped libze_intel_gpu.so to turn the je at the isSmallBarConfigAllowed() check into a 6-byte nop, and loaded it via ZE_ENABLE_ALT_DRIVERS. The system library was not modified. The intent was only to see how it fails past the gate, since that seemed more useful to you than asking.

The relevant code, for reference:

2630fe:  mov  0x170(%rbx),%rdi        ; ioctlHelper
263105:  mov  (%rdi),%rax             ; vtable
263108:  call *0x3d8(%rax)            ; isSmallBarConfigAllowed()
26310e:  test %al,%al
263110:  je   262b20                  ; <- patched to nop

Results on the same hardware as above (Arc B580, 256MB BAR, xe):

configuration	result
patched gate only	SIGBUS during init
patched gate + ForceLocalMemoryAccessMode=3	SIGBUS during init
patched gate + EnableLocalMemory=0	device enumerates, then aborts on first work submission

The third case is the interesting one. The device is enumerated and reports itself correctly:

xpu:0  Intel(R) Arc(TM) B580 Graphics   11.93 GiB   EU=160

But the first attempt to actually submit work aborts:

Abort was called at 90 line in file:
./shared/source/command_stream/linear_stream.h

So enumeration succeeds and nothing executes. A SIGBUS in the default path is also consistent with something mapping beyond the visible aperture rather than with a policy guard.

This reads to me as a genuine functional gap in the xe path rather than a conservative default, which answers my question 1 in the direction of "functional limitation". I no longer think simply exposing a debug key would be correct — it would hand users a configuration that cannot run.

That shifts what I am actually asking. Rather than an override flag, the useful thing would be:

1. Whether small-BAR support on xe is something you would consider implementing, given that i915 has it (IoctlHelperI915::isSmallBarConfigAllowed() == true) and that Battlemage and later are xe-only, so there is no fallback path for these cards.
2. If not, whether the failure mode could be made explicit — the current message is a WARNING followed by silent removal from enumeration, which reads as "this might still work" when it cannot. Something stating that the device is unsupported in this configuration would have saved me a day of assuming the problem was on the PCI side.

Point 2 is minor but cheap, and it is the part that cost me the most time.

Happy to run anything else that would be useful — this is a reproducible small-BAR setup, which I gather is not that common to have on hand.