bug(gpu): cuInit segfaults in DGX OS 7.5 sandbox while plain CDI passes #2343

Activity

1. github-actions

   added

   state:triage-neededOpened without agent diagnostics and needs triageOpened without agent diagnostics and needs triage

   on Jul 17, 2026
2. elezar

   self-assigned this

   on Jul 17, 2026
3. elezar

   added

   arch:arm64Bug affects arm64 architectureBug affects arm64 architecture

   area:compute

   area:policyPolicy engine and policy lifecycle workPolicy engine and policy lifecycle work

   area:sandboxSandbox runtime and isolation workSandbox runtime and isolation work

   os:linuxIssue affects Linux hostsIssue affects Linux hosts

   spike

   test:e2e-gpuRequires GPU end-to-end coverageRequires GPU end-to-end coverage

   topic:compatibilityCompatibility-related workCompatibility-related work

   and removed

   state:triage-neededOpened without agent diagnostics and needs triageOpened without agent diagnostics and needs triage

   on Jul 17, 2026
4. elezar commented on Jul 17, 2026

   

   elezar

   on Jul 17, 2026

   Member

   More actions

   📋 triage-agent

   Triage Assessment

   Classification: needs-investigation

   Summary

   The report has substantive diagnostics and strong controls: host CUDA and plain Docker CDI pass, while an OpenShell GPU sandbox using the same injected driver library reaches cuDriverGetVersion but segfaults in cuInit(0). I do not see this as a duplicate of #1486 because #1486 covered the generic /proc and static GPU baseline failure; this report says the /proc/.../comm write and Ubuntu control path now pass.

   Investigation

   • crates/openshell-driver-docker/src/lib.rs selects Docker CDI devices and submits DeviceRequest { driver: "cdi" }; it does not derive inner sandbox policy from the selected CDI spec.
   • crates/openshell-sandbox/src/lib.rs currently detects GPU presence via /dev/nvidiactl or /dev/dxg, adds static GPU paths, enumerates /dev/nvidia[0-9]*, and promotes /proc to read-write for CUDA compatibility.
   • architecture/security-policy.md documents that GPU baseline enrichment is path-based and startup-time, and architecture/plans/cdi-derived-gpu-filesystem-policy-issue.md already identifies the design gap: non-standard CDI layouts can inject correct devices, mounts, libraries, or groups into the outer container while the OpenShell inner workload still lacks the corresponding Landlock or process setup.
   • Existing GPU e2e coverage (e2e/gpu/README.md, e2e/rust/tests/gpu/workloads.rs) validates the generic cuda-basic path, but not this DGX OS 7.5 / GB300 / ARM64 stack or CDI-derived platform shapes.

   Recommendation

   Route as a spike for platform-specific GPU sandbox investigation. The spike should collect the selected CDI spec/container edits from DGX OS 7.5, compare the OpenShell effective filesystem/process setup against a passing plain Docker CDI container, and decide whether the fix belongs in CDI-derived sandbox requirement enrichment, CUDA runtime diagnostics, or a separate DGX OS compatibility path. Do not apply state:agent-ready until a human approves the investigation or a concrete build issue.
5. elezar

   added

   area:supervisorProxy and routing-path workProxy and routing-path work

   state:review-readyReady for human reviewReady for human review

   on Jul 17, 2026
6. elezar commented on Jul 17, 2026

   

   elezar

   on Jul 17, 2026

   Member

   More actions

   spike-investigation

   Investigation Summary

   #2343 looks like a concrete DGX OS 7.5 / GB300 reproducer for the broader CDI-derived sandbox requirement gap tracked in #1606, plus a smaller sandbox exec diagnostic issue.

   The #1486 / #1522 /proc failure mode does not appear to be the whole explanation here. The reporter proves /proc/<pid>/task/<tid>/comm writes work, libcuda.so.1 loads, and cuDriverGetVersion succeeds before cuInit(0) segfaults. The host and plain Docker CDI controls also pass with the same driver library and UID/GID, so the remaining boundary is the OpenShell inner sandbox setup on this DGX kernel/driver/CDI stack.

   Code Findings

   • crates/openshell-driver-docker/src/lib.rs detects Docker CDI support from Info.CDISpecDirs, resolves opaque CDI GPU IDs through resolve_gpu_cdi_devices, and passes them only to Docker as DeviceRequest { driver: "cdi" }.
   • The Docker create/start flow does not pass the selected CDI IDs or daemon CDISpecDirs to the supervisor. After Docker applies CDI to the outer container, OpenShell has no supervisor-side source of truth for CDI containerEdits.
   • crates/openshell-sandbox/src/lib.rs enriches the inner Landlock policy from a static GPU baseline: /dev/nvidiactl, /dev/nvidia-uvm, /dev/nvidia-uvm-tools, /dev/nvidia-modeset, /dev/dxg, /proc, plus /dev/nvidia[0-9]* discovered by enumerate_gpu_device_nodes.
   • That static baseline does not derive CDI deviceNodes, mounts[].containerPath, or additionalGIDs. It also does not enumerate paths such as /dev/nvidia-caps/*; the VM GPU init script does create /dev/nvidia-caps/nvidia-cap1 and nvidia-cap2, which is a useful local signal that GPU platforms can need device nodes outside the container baseline.
   • crates/openshell-supervisor-process/src/process.rs drops privileges with initgroups, setgid, and setuid, but there is no path to merge CDI additionalGIDs into the sandbox workload credentials.
   • crates/openshell-supervisor-process/src/ssh.rs collapses signaled SSH exec children to exit status 1 because both PTY and no-PTY waiters use status.and_then(|s| s.code()).unwrap_or(1). The entrypoint path preserves signals through ProcessStatus::code() as 128 + signal, so openshell sandbox exec is currently less diagnostic for a SIGSEGV than the entrypoint path.

   Most Likely Failure Class

   The leading hypothesis is that DGX OS 7.5 CDI injects one or more required CUDA initialization resources that are present in the outer Docker CDI container but absent from the OpenShell inner policy or credentials. Candidate resource classes are:

   • platform-specific device nodes, for example nvidia-caps or DGX/NVSwitch/fabric-related nodes;
   • per-file driver/library bind mounts whose container destinations are not covered by the existing static baseline under Landlock mount-boundary behavior;
   • CDI additionalGIDs, if the selected spec requires supplemental groups not present after OpenShell drops to the sandbox user.

   This fits the observed call boundary: cuDriverGetVersion can succeed with only libcuda.so.1, while cuInit(0) can touch more devices, helper libraries, sockets, or driver paths.

   Recommended Next Diagnostics On A DGX Host

   To confirm the exact missing resource, capture the selected CDI metadata and compare it with the OpenShell effective sandbox view:

   docker info --format "{{json .CDISpecDirs}}"
   docker info --format "{{json .DiscoveredDevices}}"
   grep -R "nvidia.com/gpu" /etc/cdi /var/run/cdi 2>/dev/null

   For the selected CDI spec, inspect only the relevant containerEdits.deviceNodes, containerEdits.mounts, and containerEdits.additionalGIDs entries. Then compare plain Docker CDI versus openshell sandbox exec for:

   find /dev -maxdepth 2 \( -name "nvidia*" -o -name "nv*" \) -ls 2>/dev/null
   LD_DEBUG=libs python3 -c "import ctypes; lib=ctypes.CDLL(\"libcuda.so.1\"); print(lib.cuDriverGetVersion); lib.cuInit(0)"

   If available, a file-focused trace around cuInit would be even better:

   strace -f -o /tmp/cuinit.strace -e trace=file,process python3 -c "import ctypes; lib=ctypes.CDLL(\"libcuda.so.1\"); lib.cuInit(0)"

   Recommended Engineering Direction

   • Treat bug(gpu): cuInit segfaults in DGX OS 7.5 sandbox while plain CDI passes #2343 as a hardware/platform validation case for feat(gpu): derive sandbox access requirements from CDI specs #1606: Docker should preserve Info.CDISpecDirs, mount CDI specs supervisor-only, upload a versioned cdi-context.json after create_container and before start_container, and let the supervisor resolve selected CDI IDs into Landlock paths and supplemental groups.
   • The supervisor should derive device nodes as read-write paths, mount destinations as read-only by default, and additionalGIDs as supplemental groups before privilege drop, while keeping fail-closed diagnostics for missing specs, unsafe paths, writable directory mounts, and custom policy conflicts.
   • Add a separate small fix in the supervisor SSH exec path so signal deaths are surfaced as 128 + signal and ideally as an SSH exit-signal or clear diagnostic. This would make a CUDA SIGSEGV show up as SIGSEGV/139 instead of a generic exit 1.
   • Extend hardware-free tests from feat(gpu): derive sandbox access requirements from CDI specs #1606 with a fixture that includes nvidia-caps and per-file library mounts, then use this DGX host as real hardware validation before closing bug(gpu): cuInit segfaults in DGX OS 7.5 sandbox while plain CDI passes #2343.

   Scope Assessment

   • Root-cause fix complexity: High, already covered by feat(gpu): derive sandbox access requirements from CDI specs #1606 and currently state:in-progress.
   • Diagnostic fix complexity: Low to medium, likely localized to crates/openshell-supervisor-process/src/ssh.rs plus CLI/e2e expectations.
   • Confidence: Medium-high for the CDI-derived requirement class; low for the exact missing DGX resource until the selected CDI spec or trace is captured.
   • LSM sensitivity: Yes. Changes touch Landlock filesystem rules, /proc behavior, bind-mounted CDI specs, and supplemental groups. SELinux/AppArmor hosts may need explicit validation around Docker bind labels, procfs visibility, and group-mediated device access.

   Recommendation: keep this issue open and review-ready as the DGX validation target for #1606, with the SSH exec signal-reporting fix either split out or included as the diagnostic half of this issue. Do not apply state:agent-ready here unless maintainers want a separate implementation thread from #1606.
7. elezar commented on Jul 17, 2026

   

   elezar

   on Jul 17, 2026

   Member

   More actions

   local-validation

   I validated the reported CUDA init behavior on this local host and could not reproduce it here.

   Host/control details:

   • Host kernel/arch: Linux 6.17.0-1017-aws, x86_64
   • GPU through Docker CDI: NVIDIA L4, nvidia.com/gpu=0, nvidia.com/gpu=GPU-a6d5342f-d3d5-a14a-24d0-e92cad8530e8, nvidia.com/gpu=all
   • Docker CDI spec dirs: /etc/cdi, /run/cdi
   • Plain Docker CDI CUDA workload passed with OPENSHELL_GPU_WORKLOAD_SUCCESS cuda-basic.
   • mise run e2e:docker:gpu passed: 10/10 tests, including workloads::gpu_workload_manifest_runs_expected_workloads.

   Exact Python/ctypes probe results:

   # plain Docker CDI, OpenShell base image
   cuDriverGetVersion 0 13000
   cuInit(0)=0
   
   # OpenShell Docker-backed GPU sandbox initial command
   cuDriverGetVersion 0 13000
   cuInit(0)=0
   
   # OpenShell Docker-backed GPU sandbox via sandbox exec
   cuDriverGetVersion 0 13000
   cuInit(0)=0
   

   This confirms the failure is not reproducible on the local AWS x86_64 / NVIDIA L4 / driver API 13000 path. The issue still looks DGX OS 7.5 / GB300 / ARM64 specific, or at least specific to that kernel/driver/CDI platform shape.
8. senthilr-nv commented on Jul 17, 2026

   

   senthilr-nv

   on Jul 17, 2026

   Author

   More actions

   New root-cause evidence from NVIDIA/NemoClaw#7108 narrows this from the broader CDI hypotheses to sysfs visibility through Landlock.

   On a physical DGX OS 7.5 / GB300 Station, released OpenShell 0.0.85 passes cuInit(0), CUDA device count, and GB300 identification when the generated NemoClaw direct-GPU filesystem policy adds /sys to read_only. No patched or custom OpenShell binary is required.

   At the current PR head, the change is applied by buildDirectGpuPolicyYaml() only when the agent sandbox has direct GPU enabled; CPU-only sandboxes are unchanged:
   https://github.com/NVIDIA/NemoClaw/blob/58c11e9d687231baad2debb8e48dd1e6d9c281c8/src/lib/onboard/initial-policy.ts#L38-L66

   An equivalent GPU-only, read-only /sys rule in the OpenShell default GPU policy enrichment could therefore benefit all OpenShell GPU consumers, not just NemoClaw. We are also preparing an independent clean-host repetition of the final PR head.
9. ericksoa

   added a commit that references this issue on Jul 17, 2026

   fix(installer): enable stock DGX OS on Station GB300 (#7108)

   ...

   929389f
10. mohityadav8 commented on Jul 20, 2026

    

    mohityadav8

    on Jul 20, 2026

    More actions

    @elezar can i work on this
11. elezar commented on Jul 21, 2026

    

    elezar

    on Jul 21, 2026

    Member

    More actions

    Additional context from the merged NemoClaw PR NVIDIA/NemoClaw#7108: this now looks like a sysfs visibility gap in the OpenShell inner Landlock policy, not a CDI-spec parsing issue.

    The important nuance is that NVIDIA CDI specs do not generally enumerate the sysfs topology paths that CUDA/NVML may read during initialization. Docker CDI can still pass because the ordinary container view leaves enough of /sys readable. OpenShell then applies a stricter inner Landlock policy, so those implicit host sysfs reads become explicit sandbox policy requirements.

    The merged NemoClaw fix narrowed the workaround from broad read-only /sys to a Station GB300 direct-GPU allowlist:

    • exact read-only /sys/bus/pci/devices/<BDF> entries for discovered NVIDIA display-class PCI devices;
    • existing read-only /sys/devices/system/cpu, /sys/devices/system/memory, and /sys/devices/system/node;
    • existing read-only /sys/module/nvidia/initstate and /sys/module/nvidia_uvm/initstate.

    They explicitly avoid adding broad /sys, /sys/bus/pci/devices, PCI parent subtrees, or any sysfs write access.

    NemoClaw reports an A/B validation on physical DGX OS / GB300: without the sysfs exception, sandbox CUDA init failed; with the exact BDF plus CPU/memory/NUMA/module-init read-only paths, cuInit(0) passed, while unrelated sysfs paths such as /sys/fs/cgroup/cgroup.controllers and /sys/class/net/lo/address remained denied.

    This suggests the OpenShell-side fix should be a separate GPU runtime/sysfs enrichment, not a relaxation of CDI path validation and not an expectation that /sys will appear in CDI. A conservative first implementation could mirror the NemoClaw allowlist for selected NVIDIA GPU PCI BDFs and the small shared topology/module paths, then validate on this DGX OS 7.5 / GB300 reproducer.
12. github-actions commented on Aug 4, 2026

    

    github-actionsbot

    on Aug 4, 2026 – with GitHub Actions

    More actions

    This issue has had no activity for 14 days and is now marked stale. It may be closed in 7 days if there is no further activity. Comment or remove the state:stale label to keep it open.
13. github-actions

    added

    state:staleInactive item at risk of automatic closure.Inactive item at risk of automatic closure.

    on Aug 4, 2026