Mount vGPU licensing config as a directory for live updates#2677
Draft
abrarshivani wants to merge 1 commit into
Draft
Mount vGPU licensing config as a directory for live updates#2677abrarshivani wants to merge 1 commit into
abrarshivani wants to merge 1 commit into
Conversation
…ates The licensing Secret/ConfigMap is mounted into the driver container only via subPath mounts at /drivers/gridd.conf and /drivers/ClientConfigToken/client_configuration_token.tok. subPath mounts are resolved once when the container starts and kubelet explicitly excludes them from its atomic-symlink refresh, so an in-place update of the backing object (for example an NLS token rotated by ExternalSecrets/Vault) is never visible to a running nvidia-driver-daemonset pod. Recovering the new license today requires a `kubectl rollout restart`, which unloads the kernel module and evicts every GPU workload on the node. Add a third mount of the same licensing volume as a plain directory at /drivers/licensing-config with no subPath, so kubelet keeps its contents in sync while the pod runs. This gives the driver container entrypoint a live view of gridd.conf (and client_configuration_token.tok when NLS is enabled) that it can watch in order to restart nvidia-gridd in place instead of rolling the daemonset. The volume is reused rather than duplicated: the Items projection is identical, so a second volume would only add a redundant copy of the same projected keys and a second source of truth to keep in sync. Keys are still projected through Items so that extra keys carried by a user's Secret do not land in the mount. The two existing subPath mounts are retained unchanged. Every published driver image entrypoint copies from those exact paths, and removing them would break all of them. Applied to both the NVIDIADriver CR path (internal/state) and the ClusterPolicy path (controllers), which is still live whenever driver.useNvidiaDriverCRD is false (the default). Refs NVIDIA#2279
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Closes half of #2279.
When the Secret/ConfigMap referenced by
driver.licensingConfigis updated — an NLS token rotated by ExternalSecrets/Vault, say — runningnvidia-driver-daemonsetpods never see the new files. Recovering the new license today requires akubectl rollout restart, which unloads the kernel module and evicts every GPU workload on the node.The licensing volume is mounted only via
subPath:subPathmounts are resolved once when the container starts, and kubelet explicitly excludes them from its atomic-symlink refresh. An in-place update of the backing object is therefore never visible to a running pod.Change
Add a third mount of the same licensing volume as a plain directory at
/drivers/licensing-config, with nosubPath, so kubelet keeps its contents in sync while the pod runs.The two existing
subPathmounts are retained unchanged. Every published driver image entrypoint copies from those exact paths; removing them would break all of them. Keeping both means no driver-version detection is needed and no ordering constraint exists between this PR and its companion.Applied to both the NVIDIADriver CR path (
internal/state) and the ClusterPolicy path (controllers) — the latter is still the default wheneverdriver.useNvidiaDriverCRDis false.Why a new path rather than fixing the existing ones
/drivers/gridd.confis a single file inside the driver payload directory; you cannot mount a directory over a file path, nor over/driversitself. Making it live requires a new path regardless. Since mount liveness does nothing without the entrypoint watcher anyway, there is no value in preserving backward-compatible paths, and one uniform directory keeps the entrypoint to a single code branch.The new subdirectory collides with nothing under
/drivers: every consumer uses an exact filename (/drivers/nvidia.conf,/drivers/vgpuDriverCatalog.yaml,NVIDIA-Linux-*.run), andvgpu-utilfilters itsReadDiron^NVIDIA-Linux-x86_64-(.*)-grid.run.Why the volume is reused rather than duplicated
The
Itemsprojection would be byte-identical, so a second volume would only add a redundant copy of the same projected keys and a second source of truth to keep in sync — and that list already varies conditionally on NLS. One volume, three mounts, one place to edit.Keys are still projected through
Items, so extra keys carried by a user's Secret — common when one is synced from Vault — do not land in the mounted directory.Heads-up for upgraders
DRIVER_CONFIG_DIGESTchanges, so upgrading rolls the driver DaemonSet once for clusters using vGPU licensing. This is unavoidable for any pod-spec change. It is a one-time cost on upgrade, not on subsequent license rotations — which is precisely the rollout this feature exists to eliminate.Testing
go test ./internal/state/... ./controllers/...passes.TestDriverAdditionalConfigsVGPULicensing, covering both the Secret/NLS-enabled and ConfigMap/NLS-disabled cases: the directory mount carries nosubPath, the legacysubPathmounts survive, and theItemsprojection stays restricted to the known keys. Verified that it fails without the production change.getDriverAdditionalConfigswas previously reached only by the RHEL subscription-mounts test, so the licensing branch of the NVIDIADriver CR path had no direct coverage.Pre-existing gap noted, not addressed here
TestDriverVGPULicensing/...Secrethand-construct theiradditionalConfigsliteral rather than callinggetDriverAdditionalConfigs, so they are template-rendering tests and do not exercise this code. One visible symptom: the golden YAML shows noreadOnly: trueon any licensing mount, including the two pre-existing ones, even though the production code sets it. Worth a follow-up.Companion PR
This is one half of a two-repo change and does nothing on its own —
nvidia-griddreads its config only at process start and has no reload signal, so a live mount alone changes nothing. The driver-container half adds a watcher that bouncesnvidia-griddwhen the config actually changes: NVIDIA/gpu-driver-container#881Neither ordering breaks anything: this PR is ignored by images without the watcher, and the watcher falls back to the legacy paths without this PR.