Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions architecture/compute-runtimes.md
Original file line number Diff line number Diff line change
Expand Up @@ -169,6 +169,18 @@ Resource requirements enter the driver layer through `SandboxSpec.resource_requi
can request a specific number of GPUs or the driver-specific default behaviour.
For all in-tree drivers, this is equivalent to selecting a single GPU.

For Docker GPU sandboxes, the driver treats CDI specs as runtime metadata for
both outer injection and inner sandbox policy. It selects opaque CDI device IDs,
passes them to Docker, mounts daemon-reported CDI spec directories into
supervisor-only paths, and bind-mounts a gateway-owned versioned CDI context
read-only before creating the container. The supervisor resolves that context
inside the sandbox and derives Landlock paths and supplemental groups from CDI
`containerEdits`. Host-side CDI spec paths are diagnostic only and are never
treated as sandbox policy paths.
Kubernetes must not infer CDI device IDs from the `nvidia.com/gpu` resource
request; it needs a node-local selected-device handoff before using the same
supervisor resolver.

VM runtime state paths are derived only from driver-validated sandbox IDs
matching `[A-Za-z0-9._-]{1,128}`. The gateway-owned VM driver socket uses a
private `run/` directory plus Unix peer UID/PID checks. Standalone
Expand Down
9 changes: 9 additions & 0 deletions crates/openshell-core/src/cdi.rs
Original file line number Diff line number Diff line change
Expand Up @@ -9,12 +9,21 @@ use serde::{Deserialize, Serialize};

pub const CDI_CONTEXT_VERSION: u32 = 1;

/// File name used for the serialized CDI context.
pub const CDI_CONTEXT_FILE_NAME: &str = "cdi-context.json";

/// Absolute supervisor path for the CDI context file mounted by a compute driver.
pub const CDI_CONTEXT_PATH: &str = "/run/openshell/supervisor/cdi-context.json";

/// Base supervisor path under which compute drivers mount CDI specification directories.
pub const CDI_SPEC_DIR_BASE: &str = "/run/openshell/supervisor/cdi-specs";

/// Return the supervisor path used for a CDI specification directory.
#[must_use]
pub fn cdi_spec_mount_path(index: usize) -> String {
format!("{CDI_SPEC_DIR_BASE}/{index}")
}

#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct CdiContext {
pub version: u32,
Expand Down
26 changes: 26 additions & 0 deletions crates/openshell-driver-docker/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,9 +86,35 @@ contract:
| `restart_policy = unless-stopped` | Keeps managed sandboxes resumable across daemon or gateway restarts. |
| `PidsLimit` | Enforces the sandbox PID budget at the Docker cgroup layer. Set `[openshell.drivers.docker].sandbox_pids_limit = 0` to inherit the Docker/runtime default. |
| CDI GPU request | Uses opaque `driver_config.cdi_devices` values when set; otherwise selects the requested count of NVIDIA CDI GPUs in round-robin order when daemon CDI support is detected. Docker daemon `/info` can permit `nvidia.com/gpu=all` as a WSL2 all-only compatibility fallback, where it counts as one selectable device. Exact CDI device lists must not contain duplicates and must match the effective GPU count. |
| CDI context mount | For GPU/CDI sandboxes only, creates a gateway-owned context file and bind-mounts it read-only at `/run/openshell/supervisor/cdi-context.json`; daemon-reported CDI spec directories are mounted read-only under `/run/openshell/supervisor/cdi-specs/<n>`. |

The agent child process does not retain these supervisor privileges.

## CDI GPU Metadata

Docker remains the source of truth for GPU injection. The driver selects opaque
CDI device IDs from `driver_config.cdi_devices` or the daemon's discovered CDI
inventory, then passes the same IDs to Docker with a CDI `DeviceRequest`.

When a GPU/CDI request is present, the driver also mounts the Docker
daemon-reported `Info.CDISpecDirs` into supervisor-only paths. Before container
creation, it writes a small versioned CDI context in gateway-owned state and
bind-mounts it read-only into the supervisor. The context uses container-side
spec paths for resolution and keeps host-side spec sources diagnostic-only. If
context or token creation fails, the driver removes any created state files; if
container creation or start fails, it also removes the container and state
files before reporting the failure.

The sandbox supervisor resolves the selected IDs from those mounted specs
before it launches agent processes. CDI device nodes become read-write
Landlock paths, mount destinations default to read-only paths, and
`additionalGids` become supplemental groups for the entrypoint and SSH child
processes. Writable CDI mount destinations are accepted only for exact
single-file paths already listed in the sandbox policy `read_write` list;
writable CDI directory mounts fail closed. Kubernetes, Podman, WSL2 hardware
validation, and Tegra/Jetson hardware validation are separate follow-up
targets.

## Driver Config Mounts

The gateway forwards the `docker` block from `--driver-config-json` to this
Expand Down
Loading
Loading