Skip to content

NVHPC toolchain support#106

Closed
kvrigor wants to merge 18 commits into
masterfrom
env-nvhpc
Closed

NVHPC toolchain support#106
kvrigor wants to merge 18 commits into
masterfrom
env-nvhpc

Conversation

@kvrigor

@kvrigor kvrigor commented Nov 19, 2025 •
edited
Loading

Copy link
Copy Markdown
Member

This PR will add support for these toolchains:

  1. NVHPC+OpenMPI: env/jsc.2026.nvhpc.openmpi
Currently Loaded Modules:
  1) Stages/2026                   (S)  28) SQLite/3.50.1        (H)  55) PnetCDF/1.14.1
  2) GCCcore/14.3.0                (H)  29) libffi/3.5.1         (H)  56) CUDA/13              (g)
  3) zlib/1.3.1                    (H)  30) Python/3.13.5             57) UCX-settings/RC-CUDA
  4) binutils/2.44                 (H)  31) libidn2/2.3.8             58) FFTW/3.3.10
  5) numactl/2.0.19                     32) libiconv/1.18        (H)  59) FFTW.MPI/3.3.10
  6) nvidia-compilers/25.9-CUDA-13 (g)  33) libunistring/1.3     (H)  60) camp/2025.12.0       (g)
  7) XZ/5.8.1                      (H)  34) libpsl/0.21.5             61) fmt/11.2.0
  8) libxml2/2.14.3                (H)  35) cURL/8.14.1               62) Umpire/2025.09.0     (g)
  9) libpciaccess/0.18.1           (H)  36) gzip/1.14            (H)  63) Hypre/3.0.0
 10) hwloc/2.12.1                  (g)  37) lz4/1.10.0           (H)  64) libpng/1.6.50        (H)
 11) OpenSSL/3                          38) zstd/1.5.7           (H)  65) libde265/1.0.16
 12) libevent/2.1.12               (H)  39) libarchive/3.8.1          66) x265/4.1             (H)
 13) UCX/default                   (g)  40) CMake/4.0.3               67) PCRE2/10.45
 14) PMIx/5.0.8                         41) expat/2.7.1          (H)  68) util-linux/2.41      (H)
 15) NCCL/default-CUDA-13          (g)  42) gettext/0.25         (H)  69) GLib/2.85.3          (H)
 16) UCC/default                   (g)  43) Perl/5.40.2               70) jbigkit/2.1          (H)
 17) MPI-settings/UCX                   44) git/2.50.1                71) libdeflate/1.24      (H)
 18) OpenMPI/5.0.8                      45) libaec/1.1.4              72) LibTIFF/4.7.0        (H)
 19) OpenBLAS/0.3.30                    46) HDF5/1.14.6               73) Brotli/1.1.0         (H)
 20) imkl/2025.2.0                      47) Blosc/1.21.6         (H)  74) freetype/2.13.3      (H)
 21) FlexiBLAS/3.4.5                    48) NASM/2.16.03         (H)  75) fontconfig/2.17.0    (H)
 22) ScaLAPACK/2.2.2-fb                 49) libjpeg-turbo/3.1.1  (H)  76) xorg-macros/1.20.2   (H)
 23) bzip2/1.0.8                   (H)  50) Blosc2/2.19.0             77) X11/20250608
 24) ncurses/6.5                   (H)  51) HDF5-plugins/1.14.6       78) Gdk-Pixbuf/2.42.12   (H)
 25) libreadline/8.2               (H)  52) Szip/2.1.1           (H)  79) libheif/1.20.2
 26) libtommath/1.3.0                   53) netCDF/4.9.3              80) JasPer/4.2.8         (H)
 27) Tcl/9.0.1                          54) netCDF-Fortran/4.6.2      81) ecCodes/2.44.0

  Where:
   S:  Module is Sticky, requires --force to unload or purge
   g:  Built with GPU support
   H:             Hidden Module

=========================== COMPILER SETTINGS  =======================
   Machine: jurecadc on Stages/2026
   MPI lib: OpenMPI v5.0.8
         C: nvc 25.9-0 64-bit target on x86-64 Linux -tp znver2 
       C++: nvc++ 25.9-0 64-bit target on x86-64 Linux -tp znver2 
   Fortran: nvfortran 25.9-0 64-bit target on x86-64 Linux -tp znver2 
======================================================================
  1. NVHPC+ParaStationMPI: env/jsc.2026.nvhpc.psmpi
Currently Loaded Modules:
  1) Stages/2026                   (S)  29) SQLite/3.50.1        (H)  57) CUDA/13              (g)
  2) GCCcore/14.3.0                (H)  30) libffi/3.5.1         (H)  58) UCX-settings/RC-CUDA
  3) zlib/1.3.1                    (H)  31) Python/3.13.5             59) FFTW/3.3.10
  4) binutils/2.44                 (H)  32) libidn2/2.3.8             60) FFTW.MPI/3.3.10
  5) numactl/2.0.19                     33) libiconv/1.18        (H)  61) camp/2025.12.0       (g)
  6) nvidia-compilers/25.9-CUDA-13 (g)  34) libunistring/1.3     (H)  62) fmt/11.2.0
  7) UCX/default                   (g)  35) libpsl/0.21.5             63) Umpire/2025.09.0     (g)
  8) pscom/6-default               (H)  36) cURL/8.14.1               64) Hypre/3.0.0
  9) XZ/5.8.1                      (H)  37) gzip/1.14            (H)  65) libpng/1.6.50        (H)
 10) libxml2/2.14.3                (H)  38) lz4/1.10.0           (H)  66) libde265/1.0.16
 11) OpenSSL/3                          39) zstd/1.5.7           (H)  67) x265/4.1             (H)
 12) libevent/2.1.12               (H)  40) libarchive/3.8.1          68) PCRE2/10.45
 13) libpciaccess/0.18.1           (H)  41) CMake/4.0.3               69) util-linux/2.41      (H)
 14) hwloc/2.12.1                  (g)  42) expat/2.7.1          (H)  70) GLib/2.85.3          (H)
 15) PMIx/5.0.8                         43) gettext/0.25         (H)  71) jbigkit/2.1          (H)
 16) NCCL/default-CUDA-13          (g)  44) Perl/5.40.2               72) libdeflate/1.24      (H)
 17) UCC/default                   (g)  45) git/2.50.1                73) LibTIFF/4.7.0        (H)
 18) MPI-settings/UCX                   46) libaec/1.1.4              74) Brotli/1.1.0         (H)
 19) ParaStationMPI/5.13.0-1       (g)  47) HDF5/1.14.6               75) freetype/2.13.3      (H)
 20) OpenBLAS/0.3.30                    48) Blosc/1.21.6         (H)  76) fontconfig/2.17.0    (H)
 21) imkl/2025.2.0                      49) NASM/2.16.03         (H)  77) xorg-macros/1.20.2   (H)
 22) FlexiBLAS/3.4.5                    50) libjpeg-turbo/3.1.1  (H)  78) X11/20250608
 23) ScaLAPACK/2.2.2-fb                 51) Blosc2/2.19.0             79) Gdk-Pixbuf/2.42.12   (H)
 24) bzip2/1.0.8                   (H)  52) HDF5-plugins/1.14.6       80) libheif/1.20.2
 25) ncurses/6.5                   (H)  53) Szip/2.1.1           (H)  81) JasPer/4.2.8         (H)
 26) libreadline/8.2               (H)  54) netCDF/4.9.3              82) ecCodes/2.44.0
 27) libtommath/1.3.0                   55) netCDF-Fortran/4.6.2
 28) Tcl/9.0.1                          56) PnetCDF/1.14.1

  Where:
   S:  Module is Sticky, requires --force to unload or purge
   g:  Built with GPU support
   H:             Hidden Module

=========================== COMPILER SETTINGS  =======================
   Machine: jurecadc on Stages/2026
   MPI lib: ParaStationMPI 5.13.0-1
         C: nvc 25.9-0 64-bit target on x86-64 Linux -tp znver2 
       C++: nvc++ 25.9-0 64-bit target on x86-64 Linux -tp znver2 
   Fortran: nvfortran 25.9-0 64-bit target on x86-64 Linux -tp znver2 
======================================================================

TODO

  • Request JSC-support to add an NVHPC-built Hypre
  • Add NVHPC support to eCLM
  • Fix eCLM compile error when building in coupled mode
  • Fix ParFlowGPU build error

@kvrigor kvrigor changed the title Env files for NVHPC toolchain NVHPC toolchain support Nov 19, 2025
jjokella
jjokella reviewed Dec 10, 2025
View reviewed changes
Comment thread cmake/BuildParFlow.cmake
@jjokella jjokella mentioned this pull request Dec 10, 2025
Merged
@jjokella
Support NVHPC toolchain for PDAF (#116)
d338182 
* BuildPDAF: Support NVHPC toolchain for PDAF

* Also: Updating to newer version of PDAF (relies on HPSCTerrSys/pdaf#40)
@jjokella jjokella mentioned this pull request Dec 19, 2025
Merged
@kvrigor

kvrigor commented Jan 28, 2026

Copy link
Copy Markdown
Member Author

New error when building parflowgpu:

[ 20%] Building C object pfsimulator/amps/CMakeFiles/amps.dir/common/amps_invoice.c.o
"/p/project1/cslts/rigor1/TSMP2/models/parflow/pfsimulator/amps/common/amps_invoice.c", line 96: warning: variable "ptr" was set but never used [set_but_not_used]
    amps_InvoiceEntry *ptr, *new_entry;
                       ^

Remark: individual warnings can be suppressed with "--diag_suppress <warning-name>"

[ 20%] Building C object pfsimulator/amps/CMakeFiles/amps.dir/common/amps_io.c.o
[ 20%] Building C object pfsimulator/amps/CMakeFiles/amps.dir/common/amps_newhandle.c.o
[ 21%] Building C object pfsimulator/amps/CMakeFiles/amps.dir/common/amps_newpackage.c.o
[ 21%] Building C object pfsimulator/amps/CMakeFiles/amps.dir/common/amps_sfbcast.c.o
[ 21%] Building C object pfsimulator/amps/CMakeFiles/amps.dir/common/amps_sfclose.c.o
[ 21%] Building C object pfsimulator/amps/CMakeFiles/amps.dir/common/amps_sfopen.c.o
[ 22%] Building C object pfsimulator/amps/CMakeFiles/amps.dir/common/amps_wait.c.o
[ 22%] Building C object pfsimulator/amps/CMakeFiles/amps.dir/common/signal.c.o
[ 22%] Building C object pfsimulator/amps/CMakeFiles/amps.dir/common/amps_print.c.o
[ 22%] Building C object pfsimulator/amps/CMakeFiles/amps.dir/mpi1/amps_allreduce.c.o
"/p/project1/cslts/rigor1/TSMP2/models/parflow/pfsimulator/amps/mpi1/amps_allreduce.c", line 154: error: argument of type "int" is incompatible with parameter of type "struct cudaMemLocation"
          CUDA_ERRCHK(cudaMemPrefetchAsync(data, (size_t)len * element_size, cudaCpuDeviceId, 0));
          ^

"/p/project1/cslts/rigor1/TSMP2/models/parflow/pfsimulator/amps/mpi1/amps_allreduce.c", line 154: error: too few arguments in function call
          CUDA_ERRCHK(cudaMemPrefetchAsync(data, (size_t)len * element_size, cudaCpuDeviceId, 0));
          ^

"/p/project1/cslts/rigor1/TSMP2/models/parflow/pfsimulator/amps/mpi1/amps_allreduce.c", line 159: error: argument of type "int" is incompatible with parameter of type "struct cudaMemLocation"
            CUDA_ERRCHK(cudaMemPrefetchAsync(ptr_src, (size_t)element_size, cudaCpuDeviceId, 0));
            ^

"/p/project1/cslts/rigor1/TSMP2/models/parflow/pfsimulator/amps/mpi1/amps_allreduce.c", line 159: error: too few arguments in function call
            CUDA_ERRCHK(cudaMemPrefetchAsync(ptr_src, (size_t)element_size, cudaCpuDeviceId, 0));
            ^

4 errors detected in the compilation of "/p/project1/cslts/rigor1/TSMP2/models/parflow/pfsimulator/amps/mpi1/amps_allreduce.c".

@kvrigor

kvrigor commented Mar 17, 2026

Copy link
Copy Markdown
Member Author

./build_tsmp2.sh parflowgpu --no_update --clean_first --env env/jsc.2026.nvhpc.openmpi produces this error:

[ 19%] Building CUDA object pfsimulator/amps/CMakeFiles/amps.dir/mpi1/amps_gpupacking.cu.o
cd TSMP2/bld/JURECADC_ParFlowGPU/ParFlow/src/ParFlow-build/pfsimulator/amps && /p/software/default/stages/2026/software/CUDA/13/bin/nvcc -forward-unknown-to-host-compiler -DCAMP_HAVE_CUDA -DFMT_HEADER_ONLY=1 --options-file CMakeFiles/amps.dir/includes_CUDA.rsp --expt-extended-lambda -I/p/software/default/stages/2026/software/OpenMPI/5.0.8-nvidia-compilers-25.9-CUDA-13/include -O3 -DNDEBUG "--generate-code=arch=compute_80,code=[compute_80,sm_80]" -Xcompiler=-mp -MD -MT pfsimulator/amps/CMakeFiles/amps.dir/mpi1/amps_gpupacking.cu.o -MF CMakeFiles/amps.dir/mpi1/amps_gpupacking.cu.o.d -x cu -c /p/project1/cslts/rigor1/TSMP2/models/parflow/pfsimulator/amps/mpi1/amps_gpupacking.cu -o CMakeFiles/amps.dir/mpi1/amps_gpupacking.cu.o
gcc: error: unrecognized command-line option ‘-mp’; did you mean ‘-MP’?
make[5]: *** [pfsimulator/amps/CMakeFiles/amps.dir/build.make:570: pfsimulator/amps/CMakeFiles/amps.dir/mpi1/amps_gpupacking.cu.o] Error 1

Too many gotchas with nvidia-compilers/25.9-CUDA-13 when building ParFlowGPU, so for now I'll drop this PR in favor of #122.

@kvrigor kvrigor closed this Mar 17, 2026
@github-project-automation github-project-automation Bot moved this from New features to Issues in TSMP2: Towards Stages/2026 support Mar 17, 2026
@kvrigor kvrigor deleted the env-nvhpc branch June 17, 2026 07:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jjokella jjokella jjokella left review comments

Assignees

No one assigned

Labels

None yet

Projects

TSMP2: Towards Stages/2026 support
Status: Issues

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@kvrigor @jjokella @AGonzalezNicolas