[nvliyuan]

Summary

• Switch the Debian 12 path in spark-rapids/mig.sh from the CUDA 12.4 local repository's NVIDIA 550 driver to the NVIDIA online repository's pinned 580 driver branch.
• Install the headless nvidia-kernel-open-dkms and nvidia-driver-cuda packages while keeping the CUDA toolkit at 12.4.
• Preserve the existing Debian 10/11, Ubuntu, and Rocky installation paths.

Root cause

mig-on-dataproc-2.3-debian12 build #13 failed during node initialization. The retained make.log from the reproduction shows NVIDIA driver 550.54.15 calling the three-argument pci_resize_resource() API, while Debian 12 kernel headers 6.1.0-53 require four arguments. As a result, the NVIDIA DKMS module does not compile.

The regular Spark RAPIDS init action already avoids this incompatibility after #1400, but the copied driver installation logic in spark-rapids/mig.sh still used the old local installer.

Validation

• bash -n spark-rapids/mig.sh
• git diff --check
• Confirmed the Debian 12 NVIDIA repository currently contains nvidia-driver-pinning-580, nvidia-kernel-open-dkms, nvidia-driver-cuda, and cuda-toolkit-12-4.
• Confirmed the Debian 12 CUDA keyring URL returns HTTP 200.
• Create a 2.3-debian12 A100 cluster with spark-rapids/mig.sh and verify MIG configuration, nvidia-smi, and CUDA 12.4.

[gemini-code-assist]

Code Review

This pull request introduces a workaround for Debian 12 security kernels by overriding the NVIDIA driver version to 580.95.05 and implementing a repository-based installation path when USE_REPO_INSTALL is enabled. The feedback identifies a missing call to enable the contrib repository component in the new installation path, a redundant apt-get update execution, and an existing redirection bug in the clear_dkms_key function that creates an unwanted file named '2'.

[nvliyuan]

verified in local