Skip to content

Fix LD_LIBRARY_PATH inside startprefix for native GPU builds so that nvidia-smi works there #326

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
boegel merged 1 commit into from
Oct 16, 2026
Merged

Fix LD_LIBRARY_PATH inside startprefix for native GPU builds so that nvidia-smi works there #326

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions bot/build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -306,6 +306,9 @@ for ACCEL_OVERRIDE in "${EESSI_ACCELERATOR_TARGET_OVERRIDES[@]}"; do
if nvidia_gpu_available && nvidia_gpu_has_compute_capability "${nvidia_cc}" ; then
echo "bot/build.sh: GPU with the requested compute capability is available, using '--nvidia all'"
BUILD_STEP_ARGS_ACCEL+=("--nvidia" "all")
# Make sure that LD_LIBRARY_PATH gets set correctly in the startprefix environment
# See https://github.com/EESSI/software-layer/pull/1461#issuecomment-6194230190
export EESSI_PROPAGATE_INTO_PREFIX_LD_LIBRARY_PATH=/.singularity.d/libs/

@boegel boegel Oct 15, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@casparvl We is this suddenly required, and it wasn't before?

Or was nvidia-smi simply always broken in the build container, and this only surfaces now because EasyBuild's get_gpu_info function relies on nvidia-smi working correctly?

@laraPPr laraPPr Oct 16, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe I had also tested at UGent with the new easyblock but now I'm not sure anymore. It does look like nvidia-smi was already broken in the build container. In our utils we were already taking this into account. But doe not do the same mitagation.

software-layer-scripts/scripts/utils.sh

Lines 151 to 168 in 6327b3a

function nvidia_gpu_available {
if command_exists "nvidia-smi"; then
# We are careful here in case we are running in a container and LD_LIBARY_PATH has been wiped
LD_LIBRARY_PATH="/.singularity.d/libs:${LD_LIBRARY_PATH}" nvidia-smi --version
ec=$?
if [ ${ec} -eq 0 ]; then
echo "Command 'nvidia-smi' found."
return 0
else
echo "Warning: command 'nvidia-smi' found, but 'nvidia-smi --version' did not run succesfully."
echo "This script now assumes this is NOT a GPU node."
return 1
fi
else
echo "No 'nvidia-smi' found, no available GPU."
return 2
fi
}

else
echo "bot/build.sh: no GPU with the requested compute capability is available, using '--nvidia install'"
BUILD_STEP_ARGS_ACCEL+=("--nvidia" "install")
Expand Down
Loading