CRTM seg faults when built and run with spack-stack/1.9.1

[RussTreadon-NOAA]

As documented in GDASApp issue #1283, ctests which execute the CRTM abort with a segmentation fault.

Below is a the traceback from a Hera run of test_gdasapp_atm_jjob_var_run. This tests assimilates amsu_n19 observations.

0: Loading MW water emissivity coefficients: /scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/spack-stack/sorc/gdas.cd/build/gdas/test/atm/global-workflow/testrun/RUND\
IRS/gdas_test/gdasatmanl_18/crtm/FASTEM6.MWwater.EmisCoeff.bin
0:
0:    Using  1 OpenMP threads =  1 for profiles and  1 for channels.
0: [h5c54:1670078:0:1670078] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x7ffc1dfa3008)
0: ==== backtrace (tid:1670078) ====
0:  0 0x0000000000053519 ucs_debug_print_backtrace()  ???:0
0:  1 0x0000000000012990 __funlockfile()  :0
0:  2 0x00000000000a7614 _f90_dope_vector_init()  ???:0
0:  3 0x0000000000417092 crtm_k_matrix_modulecrtm_k_matrix_mp_profile_solution_()  /scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/spack-stack/sorc/gdas.cd/bundle/crtm/src/CRTM_K_Matrix_Module.f90:932
0: =================================
srun: error: h5c54: task 0: Segmentation fault (core dumped)

Line 932 of CRTM_K_Matrix_Module.f90 is the Thread_Loop line below

        ! THREAD LOOP
        ! ------------
!$OMP PARALLEL DO NUM_THREADS(n_channel_threads)                        &
!$OMP    FIRSTPRIVATE(ln, r_cloudy)                                     &
!$OMP    PRIVATE(Message, ChannelIndex, n_Full_Streams, PVar, AAvar,    &
!$OMP          start_ch, end_ch, Wavenumber, Status_FWD, Status_K,      &
!$OMP          transmittance, transmittance_K, transmittance_clear,     &
!$OMP          transmittance_clear_K, l, mth_Azi)
        Thread_Loop: DO nt = 1, n_channel_threads
          start_ch = (nt - 1) * chunk_ch + 1
          IF ( nt == n_channel_threads) THEN
            end_ch = n_sensor_channels
          ELSE
            end_ch = start_ch + chunk_ch - 1
          END IF
          ln = (start_ch - 1) - n_inactive_channels(nt)

          ! -------------
          ! CHANNEL LOOP
          ! -------------
          Channel_Loop: DO l = start_ch, end_ch

It has been confirmed that test_gdasapp_atm_jjob_var_run runs with OMP_NUMTHREADS=1. --cpus-per-task=1 is specified on the srun command.

This issue is opened to document this problem and its resolution.

[RussTreadon-NOAA]

Attention @emilyhcliu and @ADCollard

[RussTreadon-NOAA]

As a test comment out the !$OMP directives before the Thread_Loop line. Also comment out the closing !$OMP END PARALLEL DO directive

@@ -923,12 +923,12 @@ CONTAINS
         ! ------------
         ! THREAD LOOP
         ! ------------
-!$OMP PARALLEL DO NUM_THREADS(n_channel_threads)                        &
-!$OMP    FIRSTPRIVATE(ln, r_cloudy)                                     &
-!$OMP    PRIVATE(Message, ChannelIndex, n_Full_Streams, PVar, AAvar,    &
-!$OMP          start_ch, end_ch, Wavenumber, Status_FWD, Status_K,      &
-!$OMP          transmittance, transmittance_K, transmittance_clear,     &
-!$OMP          transmittance_clear_K, l, mth_Azi)
+!!!$OMP PARALLEL DO NUM_THREADS(n_channel_threads)                        &
+!!!$OMP    FIRSTPRIVATE(ln, r_cloudy)                                     &
+!!!$OMP    PRIVATE(Message, ChannelIndex, n_Full_Streams, PVar, AAvar,    &
+!!!$OMP          start_ch, end_ch, Wavenumber, Status_FWD, Status_K,      &
+!!!$OMP          transmittance, transmittance_K, transmittance_clear,     &
+!!!$OMP          transmittance_clear_K, l, mth_Azi)
         Thread_Loop: DO nt = 1, n_channel_threads
           start_ch = (nt - 1) * chunk_ch + 1
           IF ( nt == n_channel_threads) THEN
@@ -1682,7 +1682,7 @@ CONTAINS

           END DO Channel_Loop
         END DO Thread_Loop
-!$OMP END PARALLEL DO
+!!!$OMP END PARALLEL DO
         IF ( Error_Status == FAILURE ) RETURN
         ln = ln + n_sensor_channels - n_inactive_channels(n_channel_threads)

Rebuild GDASApp and rerun test_gdasapp_atm_jjob_var_run on Hera. Job ran past previous failure point but was eventually killed by the system with an OOM error

0: OOPS_TRACE[0] Increment<MODEL>::serialize starting
0: slurmstepd: error: Detected 1 oom_kill event in StepId=10465141.0. Some of the step tasks have been OOM Killed.
srun: error: h2c54: task 2: Out Of Memory
srun: Terminating StepId=10465141.0
0: slurmstepd: error: *** STEP 10465141.0 ON h2c54 CANCELLED AT 2025-05-07T13:30:11 ***

Specify 96Gb memory limit for the job. Rerun test_gdasapp_atm_jjob_var_run. Job ran to completion.

Test project /scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/spack-stack/sorc/gdas.cd/build
    Start 2176: test_gdasapp_atm_jjob_var_run
1/1 Test #2176: test_gdasapp_atm_jjob_var_run ....   Passed  618.83 sec

100% tests passed, 0 tests failed out of 1

We currently use spack-stack/1.6.0 to build GDASApp develop on Hera. spack-stack/1.6.0 loads

intel/2022.1.2
stack-intel/2021.5.0
impi/2022.1.2
stack-intel-oneapi-mpi/2021.5.1

spack-stack/1.9.1 loads

intel/2024.2.1
stack-oneapi/2024.2.1
impi/2024.2.1
stack-intel-oneapi-mpi/2021.13

Some questions:

• Has the newer intel compiler with spack-stack/1.9.1 uncovered a threading bug in src/CRTM_K_Matrix_Module.f90?
• Do we need to change compiler options when using the new intel compiler?
• Does the newer compiler require changes to how the job environment is configured (e.g., set certain environment variables, add/remove/change srun options)?

@emilyhcliu , @ADCollard , and @danholdaway : Where do you think we should go from here?

[RussTreadon-NOAA]

GDASApp develop currently points at crtm @ bb7adbf. The work reported in this issue is being done in feature/spack-stack191. This branch points at the same crtm hash.

crtm @ bb7adbf is CRTM tag v2.4.1-jedi.1

After talking with @ADCollard , I will try using a different CRTM tag to build with spack-stack/1.9.1.

[RussTreadon-NOAA]

@ADCollard , I found CRTMv3 PR #207. I am attempting to build GDASApp with your CRTMv3 branch feature/gsi_validation_netcdf.

Where can I find CRTMv3 coefficients?

[RussTreadon-NOAA]

@ADCollard , I found CRTMv3 PR #207. I am attempting to build GDASApp with your CRTMv3 branch feature/gsi_validation_netcdf.

Where can I find CRTMv3 coefficients?

Saw and executed ./Get_CRTM_Binary_Files.sh from sorc/crtm. fix_REL-3.1.1.2.tgz was downloaded.

[RussTreadon-NOAA]

Attempt to use CRTMv3

Attempts to build and run feature/spack-stack191 using CRTMv3 branch feature/gsi_validation_netcdf have not been successful. The variational run seg faults with coefficient errors. Updating GDASApp to CRTMv3 can / should be the subject of a separate issue.

Attempt to use spack-stack/1.9.1 crtm/v2.4.1-jedi

Comment out

  ecbuild_bundle( PROJECT crtm SOURCE "../sorc/crtm" )

from gdas.cd/bundle/CMakeLists.txt and add

load("crtm/v2.4.1-jedi")

to gdas.cd/modulefiles/GDAS/hera.intel.lua

Build GDASApp and run test_gdasapp_setup_atm_jjob_cycled_exp followed by test_gdasapp_atm_jjob_var_init and test_gdasapp_atm_jjob_var_run. gdas.x failed with the following error when executing the CRTM

0:
0:    Using  1 OpenMP threads =  1 for profiles and  1 for channels.
0: [h34m26:3320071:0:3320071] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x7ffc67359008)
0: ==== backtrace (tid:3320071) ====
0:  0 0x0000000000053519 ucs_debug_print_backtrace()  ???:0
0:  1 0x0000000000012990 __funlockfile()  :0
0:  2 0x00000000000a7614 _f90_dope_vector_init()  ???:0
0:  3 0x00000000002a214d crtm_k_matrix_modulecrtm_k_matrix_mp_profile_solution_()  CRTM_K_Matrix_Module.f90:0
0: =================================
srun: error: h34m26: task 0: Segmentation fault (core dumped)

Additional attempts?
spack-stack/1.9.1 has following crtm versions available

        crtm/v2.4-jedi.2
        crtm/v2.4.1-jedi
        crtm/2.4.0.1
        crtm/3.1.1-build1

I could build GDASApp with the other versions and see what happens. My expectation is that the code will fail. I think we may be dealing with issues between the CRTM code and the fortran compiler being used.

spack-stack/1.9.1 uses

-- Fortran -- Intel 2021.1.0.20240703
--     compiler   : /apps/oneapi/mpi/2021.13/bin/mpiifort

The spack-stack/1.6.0 build uses

-- Fortran -- Intel 2021.5.0.20211109
--     compiler   : /apps/oneapi/mpi/2021.5.1/bin/mpiifort

I could try loading intel/2021.5 in the spack-stack/1.9.1 build but this may trigger other problems.

@danholdaway , @ADCollard, @emilyhcliu: Do you recommend we invest time trying to get crtm/2.4.x working with spack-stack/1.9.1 or focus on crtm/3.1.1? Do we have DA staff available to work on this?

FYI, I was able to build and run GSI develop using spack-stack/1.9.1. The GSI spack-stack/1.9.1 build used crtm/2.4.0.1.

[danholdaway]

Would it be worth trying the runs using the ifx compilers?

export FC=mpiifx

[emilyhcliu]

@RussTreadon-NOAA I am sorry that I have not been following this issue you opened about CRTM and spack-stack 1.9.1

You mentioned the spack-stack1.9.1 works with the following CRTM versions.

        crtm/v2.4-jedi.2
        crtm/v2.4.1-jedi
        crtm/2.4.0.1
        crtm/3.1.1-build1

I tested crtm/3.1.1 version in GSI, and the scientific performance for crtm 3.1.1 (not exactly crtm 3.1.1-build1) does not change much from crtm 2.4.1-jedi.1

Using crtm3.1.1 version, we can benefit from the improved surface emissivity for IR sensors and coefficients for new sensors (Tropics, tomorow-io, ..etc).

The significant scientific change in CRTM 3.1.1 is the account for the polarization RTM for short-wave channels, which we do not use in the operational systems.

So, I suggest we do not spend time fixing problems in crtm v2.4.x version with spack-stack 1.9.1. Let's just jump into crtm3.1.1. We can check the results of the phase 2 for radiance data running with CRTM 3.1.1 in GDASApp again. It is always good to revisit to make sure things will work anyway.

--- just my two cents ---

[RussTreadon-NOAA]

Thank you @emilyhcliu for your reply. I agree. We should move forward.

It was my impression that we faced challenges using CRTM v3 in the GSI. Good to hear that these issues have been resolved. Will we move GFS v17 GSI to CRTMv3?

[RussTreadon-NOAA]

Would it be worth trying the runs using the ifx compilers?

export FC=mpiifx

Good suggestion. The working copy of the spack-stack/1.9.1 GDASApp hera.intel.lua has been updated and the build is underway.

[ADCollard]

Thank you @emilyhcliu for your reply. I agree. We should move forward.

It was my impression that we faced challenges using CRTM v3 in the GSI. Good to hear that these issues have been resolved. Will we move GFS v17 GSI to CRTMv3?

@RussTreadon-NOAA (@emilyhcliu correct me if I am wrong), I believe Emily's tests used the binary files. My tests are using the netcdf files and I keep getting seg faults.

[emilyhcliu]

Thank you @emilyhcliu for your reply. I agree. We should move forward.
It was my impression that we faced challenges using CRTM v3 in the GSI. Good to hear that these issues have been resolved. Will we move GFS v17 GSI to CRTMv3?

@RussTreadon-NOAA (@emilyhcliu correct me if I am wrong), I believe Emily's tests used the binary files. My tests are using the netcdf files and I keep getting seg faults.

@ADCollard You are correct! I have been testing CRTM3.1.1 versions with binary files.
We can use CRTM v3.1.1 with binary coefficients for now and move to netCDF later when the segfaults are fixed.

[RussTreadon-NOAA]

Would it be worth trying the runs using the ifx compilers?
export FC=mpiifx

Good suggestion. The working copy of the spack-stack/1.9.1 GDASApp hera.intel.lua has been updated and the build is underway.

The build failed with error #5633: **Internal compiler error: segmentation violation signal raised** while compiling gdas.cd/bundle/soca/external/mom6/MOM6/src/tracer/MARBL_tracers.F90. Below is the traceback.

[ 60%] Building Fortran object soca/external/mom6/CMakeFiles/mom6.dir/MOM6/src/tracer/MARBL_tracers.F90.o
cd /scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/spack-stack/sorc/gdas.cd/build/soca/external/mom6 && /apps/oneapi/mpi/2021.13/bin/mpiifx -DNDEBUG -DSPMD -Duse_libMPI -Duse_netCDF -I/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/spack-stack/sorc/gdas.cd/build/soca/external/mom6/mod -I/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/spack-stack/sorc/gdas.cd/bundle/soca/external/mom6/MOM6/config_src/memory/dynamic_nonsymmetric -I/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/spack-stack/sorc/gdas.cd/bundle/soca/external/mom6/MOM6/src/framework -I/contrib/spack-stack/spack-stack-1.9.1/envs/ue-oneapi-2024.2.1/install/oneapi/2024.2.1/netcdf-c-4.9.2-dpxwqeq/include -I/contrib/spack-stack/spack-stack-1.9.1/envs/ue-oneapi-2024.2.1/install/oneapi/2024.2.1/netcdf-fortran-4.6.1-re63v7j/include -I/contrib/spack-stack/spack-stack-1.9.1/envs/ue-oneapi-2024.2.1/install/oneapi/2024.2.1/netcdf-cxx4-4.3.1-vneiycr/include -I/contrib/spack-stack/spack-stack-1.9.1/envs/ue-oneapi-2024.2.1/install/oneapi/2024.2.1/fms-2024.02-sj5z5c2/include_r8 -qopenmp-stubs -g -traceback -i4 -r8 -fno-alias -auto -safe-cray-ptr -ftz -assume byterecl -sox  -O2 -g -module mod -fPIC -fiopenmp -c /scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/spack-stack/sorc/gdas.cd/bundle/soca/external/mom6/MOM6/src/tracer/MARBL_tracers.F90 -o CMakeFiles/mom6.dir/MOM6/src/tracer/MARBL_tracers.F90.o
ifx: command line warning #10148: option '-sox' not supported
ifx: command line warning #10121: overriding '-qopenmp-stubs' with '-fiopenmp'
2 warnings generated.
5418ib2c9G/ifxzG9qrY.i90: error #5633: **Internal compiler error: segmentation violation signal raised** Please report this error along with the circumstances in which it occurred in a Software Problem Report.  Note: File and line given may not be explicit cause of this error.
compilation aborted for /scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/spack-stack/sorc/gdas.cd/bundle/soca/external/mom6/MOM6/src/tracer/MARBL_tracers.F90 (code 3)
make[2]: *** [soca/external/mom6/CMakeFiles/mom6.dir/build.make:2675: soca/external/mom6/CMakeFiles/mom6.dir/MOM6/src/tracer/MARBL_tracers.F90.o] Error 3
make[2]: Leaving directory '/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/spack-stack/sorc/gdas.cd/build'
make[1]: *** [CMakeFiles/Makefile2:27543: soca/external/mom6/CMakeFiles/mom6.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....

We need to get all JEDI components to build with ifx before we can see if building CRTMv3 with ifx gets us past the problems in CRTM_K_Matrix_Module.f90