Add GPUCluster CRD and controller for DRA-based stack

[karthikvetrivel]

1. Overview

We introduce a new CRD named GPUCluster and a new controller for reconciling it. Like ClusterPolicy today, it is a singleton, cluster-scoped CRD that configures the operands needed to enable GPUs in Kubernetes. GPUCluster represents the new DRA-based software-enablement stack; it is an evolution of ClusterPolicy. Unlike ClusterPolicy, it does not manage the driver or a device plugin: the driver is either preinstalled on the host or managed by NVIDIADriver CRs, and GPUs are surfaced to workloads through DRA.

A GPUCluster may coexist with a ClusterPolicy in the same cluster, with every GPU node served by exactly one of the two stacks.

2. Testing

Unit tests cover the new API, operand rendering, controller status handling, and the node-labeling logic. I also tested this end-to-end on a 2-node heterogeneous cluster (A100 + T4, Kubernetes v1.34):

• GPUCluster-only install via Helm: all operands reconcile to ready. DRA allocation was validated for full GPUs, MIG, and adminAccess. The dra-driver-validator was verified against both host-installed and NVIDIADriver-managed drivers.
• Coexistence: ClusterPolicy and GPUCluster run in one cluster, with one node per plane. Each node runs only its plane's operands, with extended resources on the device-plugin node and ResourceSlices on the DRA node.
• Mode switching: flipping nvidia.com/gpu-operator.mode in either direction swaps the operand set. A shared NVIDIADriver-managed driver survives the switch without a reload.
• Label lifecycle: driver-manager pause values survive reconciles. A node's previous-plane labels are cleaned up, and unrecognized gpu.deploy.* labels are left untouched.

For compute domains, I created a ComputeDomain CR with numNodes: 2 and a channel ResourceClaimTemplate, then ran a 2-pod indexed Job with pod anti-affinity so one pod landed on each GB200 node. I gave each pod a resource claim against the channel template, and inside each container I listed /dev/nvidia-caps-imex-channels/ to confirm containerd injected channel0 via CDI. Note that compute domains functionality requires containerd 2.x+.

3. Misc Design Notes

The driver daemonset deployed by the NVIDIADriver CR is not gated on the deploy mode so that it is not evicted and re-scheduled when the user switches modes.

The DRA kubelet-plugin daemonset is gated on nvidia.com/gpu.deploy.dra-driver rather than the mode label. The kubelet plugin must outlive every pod holding a gpu.nvidia.com ResourceClaim. Unpreparing a claim requires the plugin, so if it unregisters before the claim holders are gone, those pods get stuck in a Terminating state. Gating the daemonset on the mode label would evict the plugin at the instant of a mode switch, before the workloads drained by that switch have terminated. Instead, the node-labeling controller defers removal of the gpu.deploy.dra-driver label until no pod on the node still holds a GPU claim (terminating pods and admin-access claims count, since unpreparing them also requires the plugin), so the kubelet plugin is always the last DRA component to leave the node.

4. Next Steps

PRs to land:

• Support NVIDIADriver reconciliation and driver upgrades without a ClusterPolicy karthikvetrivel/gpu-operator#39
• Add DRA support for GPU pod eviction during driver upgrades k8s-driver-manager#204

After we have a new validator daemonset, this PR can be marked as ready:

• Drain the DRA kubelet-plugin and validator across driver reloads k8s-driver-manager#203

---

Moved from #2513 (re-created with the head branch on NVIDIA/gpu-operator instead of a fork, to enable stacked PRs). The earlier review discussion — including the GPUClusterConfig → GPUCluster naming suggestion, since adopted — lives in #2513.

[coveralls]

coverage: 33.236% (+1.3%) from 31.955% — kv-gpuclusterconfig-crd into main

[cdesiniotis]

Not a blocker, but I am having trouble grokking this. My intuition is telling me we should only call AssignOwners() when drivers are managed by the NVIDIADriver controller. But here we are calling it even when ClusterPolicy is managing the driver daemonsets. What's the rationale here?

From the comment:

// When classicClusterPolicyDriver is true (a ClusterPolicy manages
// its own driver rather than delegating to NVIDIADriver CRs), device-plugin nodes get no owner:
// their driver comes from the ClusterPolicy DaemonSet, and assigning an owner would land a second
// driver DaemonSet on them.

If ClusterPolicy is managing the driver daemonsets, doesn't our NVIDIADriver controller's Reconcile() return early and not render any daemonsets in that case?

[karthikvetrivel]

The classicClusterPolicyDriver=true path only exists for the mixed cluster where ClusterPolicy coexists with GPUCluster. Here, the NVIDIADriver controller is active because DRA-mode nodes get their driver from NVIDIADriver CRs, so AssignOwners() must still run to label those nodes, and the flag just makes it skip (and clear) the owner label on device-plugin-mode nodes so the ClusterPolicy's own driver DaemonSet isn't joined by a second, NVIDIADriver-rendered one.

[cdesiniotis]

Here, the NVIDIADriver controller is active because DRA-mode nodes get their driver from NVIDIADriver CRs

This is the source of my confusion -- A) I was under the impression that the NVIDIADriver controller is never active whenever a ClusterPolicy-backed driver is active (since the ClusterPolicy-backed driver would target ALL gpu nodes), and B) I was under the impression that the DRA-mode did not strictly require that the NVIDIADriver controller was used for driver management (it is agnostic to how drivers are installed).

[tariq1890]

To add to what others have said here already - EXCELLENT work!

This looks to be in a good shape to merge. We can address the other outstanding comments in follow-ups