Skip to content

Fix(PW): avoid GPU crash in LDOS output - #7786

Open
CleverPhysician wants to merge 3 commits into
deepmodeling:developfrom
CleverPhysician:fix/pw-gpu-ldos-fft-dispatch
Open

Fix(PW): avoid GPU crash in LDOS output#7786
CleverPhysician wants to merge 3 commits into
deepmodeling:developfrom
CleverPhysician:fix/pw-gpu-ldos-fft-dispatch

Conversation

@CleverPhysician

@CleverPhysician CleverPhysician commented Aug 7, 2026

Copy link
Copy Markdown

Summary

Fix the PW-GPU segmentation fault during post-SCF LDOS output and enable the existing scf_out_ldos case in the GPU integration test list.

Root cause

The LDOS path consumed the synchronized host wavefunction but invoked the CPU-only PW_Basis_K::recip2real implementation on a CUDA-configured PW_Basis_K. The CUDA FFT backend implements the GPU 3D FFT interface, but not the CPU staged-FFT virtual methods used by recip2real. On Linux, the unimplemented weak virtual slot resolved to a null address, causing a host-side null dispatch after SCF convergence.

This is why the SCF loop completed normally and the failure appeared only when out_ldos triggered post-SCF processing.

Changes

  • Add a device-aware PW LDOS entry point while preserving the existing CPU-facing API.
  • Keep using the synchronized double-precision host wavefunction consumed by the original LDOS implementation.
  • For GPU output, allocate narrow-scope RAII reciprocal- and real-space device buffers, copy only the current band's npwk[ik] coefficients to the device, call the existing device-aware recip_to_real, and copy the nrxx real-space values back for the unchanged LDOS accumulation and cube output.
  • Keep the CPU path on the original recip2real implementation.
  • Enable the existing tests/01_PW/scf_out_ldos case in tests/01_PW/CASES_GPU.txt.
  • Stabilize the existing integration case with pw_seed 1 and scf_thr 1e-12, write the LDOS cube at precision 8, and regenerate its CPU reference while retaining the repository's original 1e-8 comparison threshold.

Verification

Environment: one MPI rank, OMP_NUM_THREADS=1, NVIDIA RTX 4090, CUDA 11.8, GCC 11.5, MPICH 4.3.2, RelWithDebInfo builds.

  • Reproduced the original GPU failure twice with exit code 139 after eight converged SCF iterations.
  • GDB located the null dispatch in PW_Basis_K::recip2real, called by ModuleIO::stm_mode_pw.
  • CPU and CUDA builds completed successfully after the change.
  • CPU scf_out_ldos: application exit code 0.
  • CUDA scf_out_ldos: application exit code 0; LDOS_2eV.cube generated with 729 finite, nonzero grid values.
  • With the stabilized case and precision-8 output, the same-environment CPU/GPU maximum absolute grid-point difference was 2.0e-9, below the unchanged 1e-8 threshold.
  • CPU and CUDA scf_out_ldos both passed the regenerated high-precision cube reference through the unmodified integration comparator.
  • The unmodified base CPU cube and patched CPU cube were byte-identical.
  • CUDA out_ldos=0 negative control exited with code 0 and did not generate an LDOS cube.
  • Compute Sanitizer memcheck exited with code 0 and reported ERROR SUMMARY: 0 errors.
  • Final Slurm validation job 207777 completed with exit code 0.
  • Stabilization and final validation jobs 208137 and 208140 completed with exit code 0.

The first CUDA CI run exposed that the precision-3 reference could not represent the existing 1e-8 comparison threshold. The final patch does not relax that threshold: it raises only this test's serialized cube precision, fixes the random seed, tightens SCF convergence to a stable common stopping point, and regenerates the corresponding CPU energy and cube references.

Behavior change

PW-GPU calculations requesting post-SCF LDOS output now use the supported GPU FFT interface and generate the requested LDOS output instead of crashing.

out_ldos=0 behavior and the SCF iteration path are unchanged.

INPUT/documentation impact

No user-facing INPUT behavior changes. Only the integration case selects an explicit random seed, tighter SCF threshold, and higher cube output precision. No keyword, default, parameter semantics, or production output definition changes.

Core-module impact

The change is limited to the post-SCF PW LDOS output boundary. It uses existing ABACUS device-memory and FFT abstractions and adds no dependency or global runtime state.

Limitations

The runtime validation covers nspin=1, double precision, one MPI rank, one CUDA GPU, and the existing stm_bias=2 integration case. Multi-rank GPU LDOS, other spin modes, single precision, ROCm, and line-LDOS modes were not runtime-tested. Wannier90 and SOC paths are not changed.

Fixes part of #7770

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant