Repository navigation
fix(gpu): select one CDI GPU by default for Docker and Podman #1477
Description
Description
Update Docker and Podman GPU sandbox defaults so --gpu prefers one CDI GPU device instead of defaulting to nvidia.com/gpu=all.
This is part of the GPU roadmap in #1444. --gpu means the active driver's default GPU behavior, and for GPU-enabled drivers that default should inject or allocate one suitable GPU when the runtime supports individual device selection.
Context
Parent roadmap: #1444
Current local-container behavior maps a GPU request with no explicit gpu_device to nvidia.com/gpu=all through the shared CDI helper. That makes Docker and Podman inconsistent with Kubernetes and VM behavior, where a default GPU request maps to one GPU.
Docker has priority for implementation because OpenShell's Docker GPU path and CDI discovery are more mature today. Podman should be handled in the same task, but may require additional runtime support or an out-of-band CDI device discovery path. Upstream Podman behavior such as podman-container-tools/podman#28712 may be relevant.
Proposed Scope
- Define local-container default GPU selection semantics for Docker and Podman.
- Change Docker default
--gpubehavior to prefer one CDI GPU device instead ofnvidia.com/gpu=all. - Change Podman default
--gpubehavior to prefer one CDI GPU device instead ofnvidia.com/gpu=all. - Prefer runtime-reported CDI inventory when available.
- Preserve explicit
--gpu-devicebehavior as a driver-native advanced option. - Do not add multi-GPU count support in this task.
- Do not require OpenShell-managed GPU assignment/exclusivity tracking in this task.
Target Behavior
Default GPU selection should use this order:
- If the runtime reports individual CDI GPU devices, select one individual device.
- If reliable CDI inventory is unavailable but individual device IDs are expected to work, fall back to
nvidia.com/gpu=0. - If the runtime/platform only reports or supports
nvidia.com/gpu=all, such as some WSL2-based setups, usenvidia.com/gpu=allas a compatibility fallback.
Additional behavior:
openshell sandbox create --gpu ...on Docker injects one CDI GPU device when individual device selection is available.openshell sandbox create --gpu ...on Podman injects one CDI GPU device when individual device selection is available.openshell sandbox create --gpu --gpu-device nvidia.com/gpu=0 ...continues to pass the explicit CDI device ID through.- The fallback to
nvidia.com/gpu=allshould be intentional and documented, not the default for platforms with individual device selection. - Non-zero
gpu_countremains unsupported unless a driver explicitly implements count-based allocation.
Out of Scope
This task fixes default GPU device selection cardinality. It does not require OpenShell to track active GPU assignments or prevent two OpenShell sandboxes from selecting the same default GPU.
If multiple sandboxes are created concurrently, selecting the same default fallback device is acceptable until a separate allocation/exclusivity task is implemented.
Open Questions
- Where should CDI inventory discovery live: shared OpenShell core helper, driver-specific code, or both?
- What should Podman use as the authoritative CDI device inventory source before runtime-level enumeration is reliable?
- Should assignment/exclusivity tracking be added later at the driver level or as part of a broader resource allocation model?
Definition of Done
- Docker default
--gpuprefers one individual CDI GPU device when available. - Podman default
--gpuprefers one individual CDI GPU device when available. - If reliable CDI inventory is unavailable and individual IDs are expected to work, default selection falls back to
nvidia.com/gpu=0. - If individual selection is unavailable,
nvidia.com/gpu=allremains available as a documented compatibility fallback. - Explicit
--gpu-devicepass-through behavior is preserved for Docker and Podman. - Tests cover individual-device default selection, fallback selection, and explicit device pass-through.
- Docs describe Docker/Podman default GPU behavior, compatibility fallback behavior, and
--gpu-deviceas an advanced driver-native option.
Metadata
Metadata
Assignees
Type
Projects
- Status
Activity
elezar commentedon Jun 1, 2026
Implementation Plan
Issue type:
fixComplexity: Medium
Confidence: High - Docker uses daemon inventory and Podman local Linux
inventory is derived directly from
/dev/nvidiaN; remote Podman inventory isexplicitly out of scope.
Summary
Update local-container GPU selection so a bare
--gpurequest resolves to oneCDI GPU device by default instead of
nvidia.com/gpu=all, usingdriver-owned round-robin selection to rotate default requests across available
IDs without tracking active ownership. Docker and Podman share CDI selection
helpers, while inventory collection stays driver-specific: Docker uses
daemon-reported CDI inventory, and local Linux Podman maps
/dev/nvidiaNnodes to
nvidia.com/gpu=N.nvidia.com/gpu=allremains an intentionalcompatibility path for all-only runtimes. Docker indexed IDs and UUID-style
IDs are treated as alternate naming families for the same device pool and are
not mixed.
Scope
crates/openshell-core/src/gpu.rs: Add shared CDI GPU inventorynormalization, fallible default-selection, and concurrency-safe round-robin
cursor helpers.
crates/openshell-driver-docker/src/lib.rs: Capture Docker daemon CDI GPUinventory from
docker.info()and use it when building CDI device requests.Treat missing, null, empty, or non-GPU Docker inventory as a clear
precondition error for bare
--gpuwhile preserving explicit--gpu-devicepass-through when Docker CDI support is enabled. Normalizemixed indexed/UUID-style CDI IDs to one available naming family, preferring
indexed IDs when present. Add a Docker-owned round-robin cursor and ensure
validation/preview paths do not advance it.
crates/openshell-driver-docker/src/tests.rs: Cover individual defaultselection, missing/null inventory failure, all-only fallback, known-empty
failure, mixed indexed/UUID-style normalization, UUID-style-only selection,
cursor advancement/wrapping, validation no-advance behavior, and explicit
device pass-through.
crates/openshell-driver-podman/src/driver.rsandcrates/openshell-driver-podman/src/container.rs: Enumerate local/dev/nvidiaNnodes, map them tonvidia.com/gpu=N, carry that inventoryinto a driver-owned round-robin cursor and a fallible inventory-aware
container spec builder. The inventory helper should be unit-testable with an
injectable device root and ignore non-numeric NVIDIA control/helper nodes.
crates/openshell-driver-podman/src/container.rs/driver tests: Coverlocal
/dev/nvidiaNinventory mapping, empty-inventory rejection, defaultselection, cursor advancement/wrapping, validation no-advance behavior, and
explicit pass-through.
e2e/rust/tests/gpu_device_selection.rs: Update optional GPU e2eexpectations so Docker uses daemon CDI inventory and Podman uses local
/dev/nvidiaNinventory for the default device.compatibility path, missing-inventory failure behavior, and
--gpu-deviceoverride behavior.
Implementation Steps
openshell-core.construction, normalizing indexed and UUID-style aliases into one available
naming family, and add Docker driver-owned round-robin selection.
/dev/nvidiaNinventory enumeration and Podmandriver-owned round-robin selection.
Docker mixed-name-family normalization, explicit pass-through,
cursor advancement/wrapping, validation no-advance behavior, Docker's
missing/empty inventory failure paths, and Podman's local empty-inventory
rejection.
Podman local
/dev/nvidiaNinventory.Test Plan
openshell-core, Docker driver tests, and Podmandriver/container tests.
builders.
mise run e2e:docker:gpuandmise run e2e:podman:gpuonly in aCDI-capable GPU environment; otherwise note the hardware gate.
Risks & Constraints
task, Podman inventory is local Linux only via
/dev/nvidiaN; future Podmanremote CDI reporting can replace or augment that later.
DiscoveredDevices, known-empty Docker inventory, and emptylocal Podman inventory fail clearly for bare default requests instead of
falling back.
GetCapabilitiesResponse.gpu_countstays0; this task does not implementpublic count reporting.
existing sandbox, and separate gateway/driver processes can choose the same
ID.
Documentation Impact
docs/reference/gateway-config.mdxupdate expected.Revision 1 - initial plan
elezar commentedon Jun 2, 2026
Implementation Complete
PR: #1675
What was built
Docker and Podman now use driver-owned CDI GPU inventory to select one default NVIDIA CDI GPU for bare
--gpurequests. Explicit--gpu-devicevalues continue to pass through unchanged.Tests
--no-run.mise run pre-commitpassed.GPU e2e execution was environment-blocked locally: Docker reports only
docker.com/gpu=webgpu, Podman is not installed, and/dev/nvidia0is absent.Docs updated
docs/sandboxes/manage-sandboxes.mdxdocs/reference/sandbox-compute-drivers.mdxcrates/openshell-driver-docker/README.mdcrates/openshell-driver-podman/README.mdThe issue will auto-close when the PR is merged.