SIS2 in specified-ice mode in AMIP does not reproduce across ice_layout change

[nikizadehgfdl]

When we use SIS2 in AMIP runs in specified-ice mode , the experiments do not produce the same answers across different ice layouts. Here are the summary of AM4 tests I have run (different checksums signify different answers and different restart files for ice and atm):

core layout     checksum
384  6,64       __424C70A5536B7D1C_C2E57F5A1BDAC6A4_14CFCF3FC03837EB
        
384  8,48       __42FCE772E9B1A753_D1EA63314E674157_6F0C745CC5B90E85

384  32,12      __432CA0640A7C557D_F665C984763D4D4D_D354D9CA41B0D7D1
384  64,6       __432CA0640A7C557D_F665C984763D4D4D_D354D9CA41B0D7D1
432  72,6       __432CA0640A7C557D_F665C984763D4D4D_D354D9CA41B0D7D1

384  128,3      __4344CB54390A818E_D767E9E50BC4ECAA_F5011D93C14370D8
216  72,3       __4344CB54390A818E_D767E9E50BC4ECAA_F5011D93C14370D8

This happens in AM5 too which is planing to use SIS2 as the spec-ice component ( in order to have the same executable for amips and cms).

The SIS2 overrides to use for spec-ice mode are:

#override SPECIFIED_ICE = True
#override USE_SLAB_ICE = True
#override USE_SLAB_ICE_OPTICS = True
#override ICE_ALBEDO = 0.90                   !set to ice_sis max_ice_alb
#override SLAB_OPTICS_MIN_ICE_ALBEDO = 0.60   !set to ice_sis min_ice_alb   
#override SNOW_ALBEDO = 0.80
#override ALBEDO_T_MELT_RANGE = 10.0
#override APPLY_MASKS_AFTER_RESTART = False
#override FATAL_UNUSED_PARAMS = False
#override ICE_STATS_INTERVAL = 30
#override MIN_H_FOR_TEMP_CALC = 0.1

[nikizadehgfdl]

When I run with DEBUG=True, the first instance of diff in stdout of two diverging ice_layout runs is:

vimdiff 
/lustre/f2/scratch/Niki.Zadeh/fre/FMS2023.03-alpha2_mom6_20230828/c96L33_am4p0_SIS2/ncrc5.intel23-repro/stdout/run/c96L33_am4p0_SIS2_1x0m2d_216x1a.o134568335 
/lustre/f2/scratch/Niki.Zadeh/fre/FMS2023.03-alpha2_mom6_20230828/c96L33_am4p0_SIS2/ncrc5.intel23-repro/stdout/run/c96L33_am4p0_SIS2_1x0m2d_216x1a1.o134568336


<   End do_update_ice_model_fast Ice%albedo_vis_dir    5.8124289438911423E+05     52626167                   
>   End do_update_ice_model_fast Ice%albedo_vis_dir    5.8121910074618098E+05     52626816                   

I tracked this down to be due to the else statement on line 1277 of ice_model.F90:

  elseif (coszen_changed) then
    call set_ocean_albedo_from_coszen(Ice, G, Rad%coszen_nextrad)

Apparently for SPECIFIED_ICE, the variable coszen_changed is not updated as frequently as it should.

The following change seems to fix the layout non-repro issue for spec-ice model:

-  elseif (coszen_changed) then
+ elseif (coszen_changed .or. Ice%sCS%specified_ice) then
     call set_ocean_albedo_from_coszen(Ice, G, Rad%coszen_nextrad)

[nikizadehgfdl]

@marshallward @Hallberg-NOAA I found another model that does not reproduce across ice_layout changes for the same reason of infrequent coszen update. This time it is a lowres ESM4.2 fully coupled model and my "fix" for amip does not affect or fix it.

Again, changing coszen_changed from .false. to .true. at line 1255 of ice_model.F90 fixes the issue.
So, it seems like the logic of flipping this switch at line 1261 has to be enhanced if we want to do less frequent coszen updates.

In this esm4.2 model MOM6 DT = 2700.0, DT_THERM = 10800.0, coupler dt_cpld=7200, dt_atmos=1800.
I also tried with DT_THERM=7200 and got the same non-reproducing issue.

[uramirez8707]

SPEAR is also not reproducing across ice_layouts changes.