E3SM Unified 1.13.0 testing #802

forsyth2 · 2026-04-01T15:47:35Z

forsyth2
Apr 1, 2026
Maintainer

I'm starting the testing log for E3SM Unified 1.13.0 (as a follow-up to the 1.12.0 log) rather than using the regular (weekly log).

However, in the beginning threads we'll still be using dev environments. The idea is that these will set the expected results (on Chrysalis, Perlmutter, Compy) to use during the Unified 1.13.0 testing period.

forsyth2 · 2026-04-01T15:50:35Z

forsyth2
Apr 1, 2026
Maintainer Author

2026-03-31 Chrysalis

We're testing the main branch here.

What changes are we testing?

The latest expected results are based on the (2026-03-27 test) run.

Package	Changes since 3/27 test
e3sm_diags	E3SM-Project/e3sm_diags#1042
mpas_analysis	MPAS-Dev/MPAS-Analysis#1139, MPAS-Dev/MPAS-Analysis#1141
zppy-interfaces	None
zppy	#788, #801

Testing process

Set up environments

lcrc_conda # Activate conda
lcrc-quota # Have enough space

# Set up e3sm_diags env
cd ~/ez/e3sm_diags
git status
# On branch main
# nothing to commit, working tree clean
git fetch upstream main
git reset --hard upstream/main
git log --oneline | head -n 1
# b1e98504 Fix Python 3.14 stall due to repeated climo opens without close (#1042)
# Good, matches https://github.com/E3SM-Project/e3sm_diags/commits/main
rm -rf build
conda clean --all --y
conda env create -f conda-env/dev.yml -n test-diags-main-20260331
conda activate test-diags-main-20260331
python -m pip install .

# Set up mpas_analysis env for the first time
cd ~/ez/MPAS-Analysis/
git status
# On branch develop
# nothing to commit, working tree clean
git fetch upstream develop
git reset --hard upstream/develop
git log --oneline | head -n 1
# 0eb9ef491 Merge branch 'xylar/fix-non-editable-install' into develop
# Good, matches https://github.com/MPAS-Dev/MPAS-Analysis/commits/develop/
rm -rf build
conda clean --all --y
head -n 2 dev-spec.txt 
# This file may be used to create an environment using:
# $ conda create --name <env> --file <this file>
conda create --name test-mpas-analysis-develop-20260331 --file dev-spec.txt
conda activate test-mpas-analysis-develop-20260331
python -m pip install .

# LIVVKit
# We have no instructions to create our own livvkit environment
# We will simply use the provided one:
# source /lcrc/group/e3sm/livvkit/software/load_e3sm_unified_1.12.1_lex.sh

# Set up zppy-interfaces env
cd ~/ez/zppy-interfaces
git status
# On branch main
# nothing to commit, working tree clean
git fetch upstream main
git reset --hard upstream/main
git log --oneline | head -n 1
# 02638f4 Add Python 3.14 support (#44)
# Good, matches https://github.com/E3SM-Project/zppy-interfaces/commits/main
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n test-zi-main-20260331
conda activate test-zi-main-20260331
python -m pip install .
pytest tests/unit/global_time_series/test_*.py # 10 passed in 21.89s
pytest tests/unit/pcmdi_diags/test_*.py # 7 passed in 8.88s

zppy run

# zppy itself #################################################################
cd ~/ez/zppy
git status
# On branch fix-legacy-310-tests
# nothing to commit, working tree clean
git fetch upstream main
git checkout -b test-zppy-main-20260331 upstream/main
git log --oneline | head -n 1
# 0ad4774a Fix legacy 3.1.0 www paths (#801)
# Good, matches https://github.com/E3SM-Project/zppy/commits/main
git log --oneline | head -n 3
# 0ad4774a Fix legacy 3.1.0 www paths (#801)
# 30c2cee0 Add mpas_analysis_subsection parameter to global_time_series (#788)
# 016d220b Merge pull request #783 from mkstratos/add-livvkit
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n test-zppy-main-20260331
conda activate test-zppy-main-20260331
pre-commit run --all-files
python -m pip install .
pytest tests/test_*.py # 44 passed in 0.89s

# Edit tests/integration/utils.py:

TEST_SPECIFICS: Dict[str, Any] = {
    # These are custom environment_commands for specific tasks.
    # Never set these to "", because they will print the line
    # `environment_commands = ""` for the corresponding task,
    # thus overriding the value set higher up in the cfg.
    # That is, there will be no environment set.
    # (`environment_commands = ""` only redirects to Unified
    # if specified under the [default] task)
    "diags_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-diags-main-20260331",
    "mpas_analysis_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-mpas-analysis-develop-20260331",
    "global_time_series_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-zi-main-20260331",
    "livvkit_environment_commands": "source /lcrc/group/e3sm/livvkit/software/load_e3sm_unified_1.12.1_lex.sh",
    "pcmdi_diags_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-zi-main-20260331",
    # This is the environment setup for other tasks.
    # Leave as "" to use the latest Unified environment.
    "environment_commands": "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh",
    # For a complete test, run the set of latest cfgs and at least one set of legacy cfgs
    "cfgs_to_run": [
        "weekly_bundles",
        "weekly_comprehensive_v2",
        "weekly_comprehensive_v3",
        "weekly_legacy_3.1.0_bundles",
        "weekly_legacy_3.1.0_comprehensive_v2",
        "weekly_legacy_3.1.0_comprehensive_v3",
        "weekly_legacy_3.0.0_bundles",
        "weekly_legacy_3.0.0_comprehensive_v2",
        "weekly_legacy_3.0.0_comprehensive_v3",
    ],
    "tasks_to_run": [
        "e3sm_diags",
        "mpas_analysis",
        "global_time_series",
        "ilamb",
        "livvkit",
        "pcmdi_diags",
    ],
    "unique_id": "zppy_main_branch_test_20260331",
}

git diff # Diff looks good
python tests/integration/utils.py
# CFG FILES HAVE BEEN GENERATED FROM TEMPLATES WITH THESE SETTINGS:
# UNIQUE_ID=zppy_main_branch_test_20260331
# diags_environment_commands=source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-diags-main-20260331
# mpas_analysis_environment_commands=source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-mpas-analysis-develop-20260331
# global_time_series_environment_commands=source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-zi-main-20260331
# livvkit_environment_commands=source /lcrc/group/e3sm/livvkit/software/load_e3sm_unified_1.12.1_lex.sh
# pcmdi_diags_environment_commands=source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-zi-main-20260331
# environment_commands=source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh
# Reminder: `environment_commands=''` => the latest E3SM Unified environment will be used

# For reference:
alias sq
# alias sq='sqa -u ac.forsyth2'
alias sqa
# alias sqa='squeue -o "%8u %.7a %.4D %.9P %7i %.2t %.10r %.10M %.10l %j" --sort=P,-t,-p'
sq
# No jobs currently queued
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg

zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_comprehensive_v3_chrysalis.cfg

zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v3_chrysalis.cfg

sq | wc -l # Tue 3/31 15:54 => 231 - header row = 230 jobs
sq | wc -l # Tue 3/31 20:52 => 20 - header row = 19 jobs
sq
# ac.forsy    e3sm    4   compute 1209408  R       None    4:58:21    7:00:00 bundle1
# ac.forsy    e3sm    4   compute 1209328  R       None    4:58:51    7:00:00 bundle1
# ac.forsy    e3sm    4   compute 1209239  R       None    4:59:51    7:00:00 bundle1
# ac.forsy    e3sm    1   compute 1209433  R       None    4:34:21    5:00:00 e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983
# ac.forsy    e3sm    1   compute 1209434  R       None    4:34:51    5:00:00 e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1980-1981_vs_1980-1981
# ac.forsy    e3sm    1   compute 1209353  R       None    4:36:51    5:00:00 e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983
# ac.forsy    e3sm    1   compute 1209354  R       None    4:37:21    5:00:00 e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1980-1981_vs_1980-1981
# ac.forsy    e3sm    1   compute 1209463  R       None    4:37:21    5:00:00 e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1987-1988
# ac.forsy    e3sm    1   compute 1209264  R       None    4:38:51    5:00:00 e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983
# ac.forsy    e3sm    1   compute 1209265  R       None    4:38:51    5:00:00 e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1980-1981_vs_1980-1981
# ac.forsy    e3sm    1   compute 1209389  R       None    4:39:51    5:00:00 e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1987-1988
# ac.forsy    e3sm    1   compute 1209308  R       None    4:41:21    5:00:00 e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1987-1988_vs_1985-1986
# ac.forsy    e3sm    1   compute 1209307  R       None    4:41:51    5:00:00 e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1987-1988
# ac.forsy    e3sm    1   compute 1209465  R       None    4:43:21    5:00:00 e3sm_diags_lnd_monthly_mvm_lnd_model_vs_model_1987-1988_vs_1985-1986
# ac.forsy    e3sm    1   compute 1209435  R       None    4:45:21    5:00:00 e3sm_diags_lnd_monthly_mvm_lnd_model_vs_model_1982-1983_vs_1980-1981
# ac.forsy    e3sm    1   compute 1209391  R       None    4:47:51    5:00:00 e3sm_diags_lnd_monthly_mvm_lnd_model_vs_model_1987-1988_vs_1985-1986
# ac.forsy    e3sm    1   compute 1209355  R       None    4:49:51    5:00:00 e3sm_diags_lnd_monthly_mvm_lnd_model_vs_model_1982-1983_vs_1980-1981
# ac.forsy    e3sm    1   compute 1209309  R       None    4:54:21    5:00:00 e3sm_diags_lnd_monthly_mvm_lnd_model_vs_model_1987-1988_vs_1985-1986
# ac.forsy    e3sm    1   compute 1209266  R       None    4:56:21    5:00:00 e3sm_diags_lnd_monthly_mvm_lnd_model_vs_model_1982-1983_vs_1980-1981

# Something seems wrong; why are all these tasks taking 4+ hours?

# Picking up 4/1 ##############################################################
sq | wc -l # Wed 4/1 08:04 => 1 - header row = 0 jobs

Bundles part 2

# Check on bundles status
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/zppy_main_branch_test_20260331/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# bundle1.status:RUNNING 1209239
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1985-1986.status:RUNNING 1209239
tail bundle1.o1209239
# 2026-03-31 18:04:35,013 [INFO]: dataset_xr.py(_get_dataset_with_derived_climo_var:870) >> Getting dataset with derivation function
# 2026-03-31 18:04:35,019 [INFO]: dataset_xr.py(_get_dataset_with_derived_climo_var:877) >> Successfully derived variable 'PRECT' for test climatology dataset.
# 2026-03-31 18:04:35,167 [INFO]: dataset_xr.py(_get_dataset_with_derived_climo_var:863) >> Deriving the ref climatology variable using the source variables: ('sat_gauge_precip',)
# 2026-03-31 18:04:35,175 [INFO]: dataset_xr.py(_get_dataset_with_derived_climo_var:870) >> Getting dataset with derivation function
# 2026-03-31 18:04:35,177 [INFO]: dataset_xr.py(_get_dataset_with_derived_climo_var:877) >> Successfully derived variable 'PRECT' for ref climatology dataset.
# srun: Job step aborted: Waiting up to 92 seconds for job step to finish.
# slurmstepd: error: *** STEP 1209239.358 ON chr-0208 CANCELLED AT 2026-04-01T00:52:58 DUE TO TIME LIMIT ***
# slurmstepd: error: *** JOB 1209239 ON chr-0208 CANCELLED AT 2026-04-01T00:52:58 DUE TO TIME LIMIT ***
# srun: got SIGCONT
# srun: forcing job termination

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_bundles_output/zppy_main_branch_test_20260331/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# bundle1.status:RUNNING 1209328
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1985-1986.status:RUNNING 1209328
tail bundle1.o1209328
# 2026-03-31 18:04:30,373 [INFO]: dataset_xr.py(_get_dataset_with_derived_climo_var:877) >> Successfully derived variable 'U' for ref climatology dataset.
# 2026-03-31 18:04:31,916 [INFO]: polar_driver.py(_run_diags_3d:209) >> Selected pressure level(s): [850.0]
# 2026-03-31 18:04:32,804 [INFO]: regrid.py(subset_and_align_datasets:70) >> Selected region: polar_S
# 2026-03-31 18:04:38,285 [INFO]: io.py(_save_data_metrics_and_plots:167) >> Metrics saved in model_vs_obs_1985-1986/polar/ERA5/ERA5-U-850-DJF-polar_S.json
# 2026-03-31 18:04:42,601 [INFO]: utils.py(_save_main_plot:122) >> Plot saved in: model_vs_obs_1985-1986/polar/ERA5/ERA5-U-850-DJF-polar_S.png
# slurmstepd: error: *** STEP 1209328.358 ON chr-0081 CANCELLED AT 2026-04-01T00:53:58 DUE TO TIME LIMIT ***
# slurmstepd: error: *** JOB 1209328 ON chr-0081 CANCELLED AT 2026-04-01T00:53:58 DUE TO TIME LIMIT ***
# srun: Job step aborted: Waiting up to 92 seconds for job step to finish.
# srun: got SIGCONT
# srun: forcing job termination

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/zppy_main_branch_test_20260331/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# bundle1.status:RUNNING 1209408
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1985-1986.status:RUNNING 1209408
tail bundle1.o1209408 
# 2026-03-31 18:06:33,748 [INFO]: dataset_xr.py(_get_dataset_with_derived_climo_var:870) >> Getting dataset with derivation function
# 2026-03-31 18:06:33,750 [INFO]: dataset_xr.py(_get_dataset_with_derived_climo_var:877) >> Successfully derived variable 'PRECT' for ref climatology dataset.
# 2026-03-31 18:06:35,225 [INFO]: regrid.py(subset_and_align_datasets:70) >> Selected region: polar_N
# 2026-03-31 18:06:37,094 [INFO]: io.py(_save_data_metrics_and_plots:167) >> Metrics saved in model_vs_obs_1985-1986/polar/GPCP_v3.2/GPCP_v3.2-PRECT-ANN-polar_N.json
# 2026-03-31 18:06:40,883 [INFO]: utils.py(_save_main_plot:122) >> Plot saved in: model_vs_obs_1985-1986/polar/GPCP_v3.2/GPCP_v3.2-PRECT-ANN-polar_N.png
# srun: Job step aborted: Waiting up to 92 seconds for job step to finish.
# slurmstepd: error: *** STEP 1209408.358 ON chr-0087 CANCELLED AT 2026-04-01T00:54:28 DUE TO TIME LIMIT ***
# slurmstepd: error: *** JOB 1209408 ON chr-0087 CANCELLED AT 2026-04-01T00:54:28 DUE TO TIME LIMIT ***
# srun: got SIGCONT
# srun: forcing job termination

# Can't proceed with running bundles part 2.

Review finished runs

### v2  ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/zppy_main_branch_test_20260331/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.status:RUNNING 1209264
# e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1980-1981_vs_1980-1981.status:RUNNING 1209265
# e3sm_diags_lnd_monthly_mvm_lnd_model_vs_model_1982-1983_vs_1980-1981.status:RUNNING 1209266
tail e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.o1209264 
# 2026-03-31 19:12:35,703 [INFO]: dataset_xr.py(_get_dataset_with_derived_climo_var:870) >> Getting dataset with derivation function
# 2026-03-31 19:12:35,710 [INFO]: dataset_xr.py(_get_dataset_with_derived_climo_var:877) >> Successfully derived variable 'TAUXY' for test climatology dataset.
# 2026-03-31 19:12:35,784 [INFO]: dataset_xr.py(_get_dataset_with_derived_climo_var:863) >> Deriving the ref climatology variable using the source variables: ('tauu', 'tauv')
# 2026-03-31 19:12:35,813 [INFO]: dataset_xr.py(_get_dataset_with_derived_climo_var:870) >> Getting dataset with derivation function
# 2026-03-31 19:12:35,822 [INFO]: dataset_xr.py(_get_dataset_with_derived_climo_var:877) >> Successfully derived variable 'TAUXY' for ref climatology dataset.
# srun: Job step aborted: Waiting up to 92 seconds for job step to finish.
# slurmstepd: error: *** STEP 1209264.0 ON chr-0096 CANCELLED AT 2026-03-31T23:13:57 DUE TO TIME LIMIT ***
# slurmstepd: error: *** JOB 1209264 ON chr-0096 CANCELLED AT 2026-03-31T23:13:57 DUE TO TIME LIMIT ***
# srun: got SIGCONT
# srun: forcing job termination
tail e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1980-1981_vs_1980-1981.o1209265 
# 2026-03-31 18:28:23,355 [INFO]: polar_driver.py(run_diag:41) >> Variable: FLDS
# 2026-03-31 18:28:27,492 [INFO]: regrid.py(subset_and_align_datasets:70) >> Selected region: polar_S
# 2026-03-31 18:28:27,697 [INFO]: io.py(_save_data_metrics_and_plots:167) >> Metrics saved in model_vs_model_1980-1981/polar/model_vs_model/v2.LR.historical_0201-FLDS-DJF-polar_S.json
# 2026-03-31 18:28:28,745 [INFO]: utils.py(_save_main_plot:122) >> Plot saved in: model_vs_model_1980-1981/polar/model_vs_model/v2.LR.historical_0201-FLDS-DJF-polar_S.png
# 2026-03-31 18:28:28,745 [INFO]: polar_driver.py(run_diag:41) >> Variable: FLDS
# slurmstepd: error: *** STEP 1209265.0 ON chr-0253 CANCELLED AT 2026-03-31T23:13:57 DUE TO TIME LIMIT ***
# srun: Job step aborted: Waiting up to 92 seconds for job step to finish.
# slurmstepd: error: *** JOB 1209265 ON chr-0253 CANCELLED AT 2026-03-31T23:13:57 DUE TO TIME LIMIT ***
# srun: got SIGCONT
# srun: forcing job termination
tail e3sm_diags_lnd_monthly_mvm_lnd_model_vs_model_1982-1983_vs_1980-1981.o1209266 
# 2026-03-31 18:01:16,035 [INFO]: dataset_xr.py(_get_land_sea_mask:1627) >> No land sea mask datasets were found for the given season.
# 2026-03-31 18:01:16,036 [INFO]: dataset_xr.py(_get_default_land_sea_mask_dataset:1674) >> Using default land sea mask located at `/gpfs/fs1/home/ac.forsyth2/miniforge3/envs/test-diags-main-20260331/share/e3sm_diags/acme_ne30_ocean_land_mask.nc`.
# 2026-03-31 18:01:16,224 [INFO]: dataset_xr.py(_get_dataset_with_derived_climo_var:863) >> Deriving the ref climatology variable using the source variables: ('FIRA_U',)
# 2026-03-31 18:01:16,279 [INFO]: dataset_xr.py(_get_dataset_with_derived_climo_var:870) >> Getting dataset with derivation function
# 2026-03-31 18:01:16,281 [INFO]: dataset_xr.py(_get_dataset_with_derived_climo_var:877) >> Successfully derived variable 'FIRA_U' for ref climatology dataset.
# srun: Job step aborted: Waiting up to 92 seconds for job step to finish.
# slurmstepd: error: *** STEP 1209266.0 ON chr-0121 CANCELLED AT 2026-03-31T22:56:27 DUE TO TIME LIMIT ***
# slurmstepd: error: *** JOB 1209266 ON chr-0121 CANCELLED AT 2026-03-31T22:56:27 DUE TO TIME LIMIT ***
# srun: got SIGCONT
# srun: forcing job termination

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_comprehensive_v2_output/zppy_main_branch_test_20260331/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.status:RUNNING 1209353
# e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1980-1981_vs_1980-1981.status:RUNNING 1209354
# e3sm_diags_lnd_monthly_mvm_lnd_model_vs_model_1982-1983_vs_1980-1981.status:RUNNING 1209355
grep -l -n "DUE TO TIME LIMIT" *.o*
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.o1209353
# e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1980-1981_vs_1980-1981.o1209354
# e3sm_diags_lnd_monthly_mvm_lnd_model_vs_model_1982-1983_vs_1980-1981.o1209355

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/zppy_main_branch_test_20260331/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.status:RUNNING 1209433
# e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1980-1981_vs_1980-1981.status:RUNNING 1209434
# e3sm_diags_lnd_monthly_mvm_lnd_model_vs_model_1982-1983_vs_1980-1981.status:RUNNING 1209435
grep -l -n "DUE TO TIME LIMIT" *.o*
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.o1209433
# e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1980-1981_vs_1980-1981.o1209434
# e3sm_diags_lnd_monthly_mvm_lnd_model_vs_model_1982-1983_vs_1980-1981.o1209435

### v3 ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/zppy_main_branch_test_20260331/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1987-1988.status:RUNNING 1209307
# e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1987-1988_vs_1985-1986.status:RUNNING 1209308
# e3sm_diags_lnd_monthly_mvm_lnd_model_vs_model_1987-1988_vs_1985-1986.status:RUNNING 1209309
grep -l -n "DUE TO TIME LIMIT" *.o*
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1987-1988.o1209307
# e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1987-1988_vs_1985-1986.o1209308
# e3sm_diags_lnd_monthly_mvm_lnd_model_vs_model_1987-1988_vs_1985-1986.o1209309

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_comprehensive_v3_output/zppy_main_branch_test_20260331/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1987-1988.status:RUNNING 1209389
# e3sm_diags_lnd_monthly_mvm_lnd_model_vs_model_1987-1988_vs_1985-1986.status:RUNNING 1209391
grep -l -n "DUE TO TIME LIMIT" *.o*
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1987-1988.o1209389
# e3sm_diags_lnd_monthly_mvm_lnd_model_vs_model_1987-1988_vs_1985-1986.o1209391

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/zppy_main_branch_test_20260331/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1987-1988.status:RUNNING 1209463
# e3sm_diags_lnd_monthly_mvm_lnd_model_vs_model_1987-1988_vs_1985-1986.status:RUNNING 1209465
grep -l -n "DUE TO TIME LIMIT" *.o*
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1987-1988.o1209463
# e3sm_diags_lnd_monthly_mvm_lnd_model_vs_model_1987-1988_vs_1985-1986.o1209465

post/scripts directory status: @tomvothecoder @chengzhuzhang 19 e3sm_diags tasks hit the time limit (ranging from 5 hours to 7 hours). Are we sure E3SM-Project/e3sm_diags#1042 truly fixed the stalling issue? Can this be resolved this week so we can get release candidates in on time (and before Chrysalis goes down)? Thanks.

(Note my test doesn't include the latest e3sm_diags commit (E3SM-Project/e3sm_diags#1031), but that looks like a feature, not a bug fix.)

Results analysis

I gave my time limit errors and cd e3sm_diags; git diff HEAD^ HEAD e3sm_diags/driver/utils/dataset_xr.py | cat to ChatGPT, which suggest the following:

ChatGPT response

Short answer: the “fix” removed a resource leak, but it also destroyed all the implicit caching/reuse that was making things fast — so now you’re re-opening and re-reading NetCDF data over and over, which blows up runtime and hits the wall clock limit.

What changed (the critical behavior shift)

The diff aggressively closes datasets everywhere:

ds_open.close() in _get_climo_dataset
ds_open.close() in time series loader
Closing both ds_subset and ds inside _subset_vars_and_load
Converting attrs + closing immediately
Returning “detached” datasets explicitly

This is consistent with the intent noted in the docstring:

“detached from the file-backed backend”

So the design goal was:
👉 avoid file-handle leaks / GC issues (esp. Python 3.14)

Why this caused the slowdown

1. You killed xarray/dask’s lazy reuse

Previously:

open_mfdataset() stayed alive
Variables were lazily loaded
Multiple operations reused the same file handles + dask graph

Now:

Every call:
- opens files
- loads a subset
- closes immediately
Next operation:
- re-opens the same files again

👉 Net effect: massive repeated I/O

2. `_subset_vars_and_load()` now forces full eager loads repeatedly

ds_subset.load(scheduler="sync")

Combined with:

finally:
    dataset.close()

This means:

Every variable access becomes:
1. open files
2. fully read into memory (sync, single-threaded)
3. close files
No sharing across variables or diagnostics

👉 Before: amortized I/O
👉 Now: O(N variables × full dataset reads)

3. Derived variables amplify the problem

From your logs:

Deriving ... using ('tauu', 'tauv')

Each derivation:

triggers _subset_vars_and_load
which:
- loads source vars
- closes datasets

Then the next variable:

reopens everything again

👉 Derived vars = multiplicative I/O explosion

4. You likely broke backend caching semantics

Even though .load() is called, xarray objects can still:

retain references to backend encodings
rely on shared file managers

By force-closing both ds_subset and the parent ds, you:

invalidate shared state
prevent any reuse across datasets

Why this looks like “stalling”

It’s not actually stuck — it’s just:

doing way more disk I/O
serial (scheduler="sync")
repeatedly reopening large multi-file datasets

So runtime >2× is expected; in practice it can be 5–20× slower, which matches hitting SLURM limits.

The core mistake

You fixed:

“datasets not being closed”

…but introduced:

“datasets never staying open long enough to be reused”

This is a classic HPC tradeoff:

Correctness fix (no leaks) vs
Performance regression (no reuse)

What likely went wrong in design

Two key overcorrections:

❌ Closing in every layer

_get_climo_dataset
_subset_vars_and_load
time series loader

👉 This stacks and guarantees zero reuse

❌ Closing both `ds_subset` and `ds`

for dataset in (ds_subset, ds):
    dataset.close()

This is especially aggressive:

ds_subset may share backing stores with ds
closing both guarantees teardown of everything

What to fix (high-impact)

1. Do NOT close in `_subset_vars_and_load`

This is the hottest path. Removing the close here alone will likely recover most performance.

2. Only close at the outermost lifecycle

Good pattern:

Open once
Do all ops
Close once

Bad pattern (current):

Open → load → close → repeat

3. Keep `open_mfdataset()` alive longer

This is critical for:

file manager reuse
dask graph reuse
chunk caching

4. Optional: remove `scheduler="sync"`

This forces single-threaded reads — bad for HPC.

TL;DR

The diff fixed file-handle leaks by aggressively closing datasets, but in doing so it:

eliminated xarray/dask reuse
forced repeated full data loads
multiplied I/O per variable/derivation

👉 Result: jobs now run >2× slower and hit time limits.

If you want, I can point to the minimal safe rollback (i.e., exactly which close() calls to remove without reintroducing the original leak).

15 replies

xylar Apr 2, 2026
Collaborator

I'd caution using ChatGPT to do source code analysis and instead use Copilot or some other repo-aware AI software dev tool where it has the ability to reference the source code.

Yes, I agree.

forsyth2 Apr 2, 2026
Maintainer Author

I'm not sure; I feel like I haven't had much success with Copilot explaining how the code is structured. For instance, with zstash, I feel like I've had much more success asking web-interface AI clients about logical flows in single files. Maybe that's just a function of needing a better agent instructions file though. (I also didn't want to have Copilot open a PR/post on e3sm_diags just to get some preliminary ideas).

chengzhuzhang Apr 2, 2026
Maintainer

I also didn't want to have Copilot open a PR/post on e3sm_diags just to get some preliminary ideas

Just want to recommend using Copilot CLI instead of web-interface AI in this case, no PR/post will be auto triggered using CLI.

forsyth2 Apr 2, 2026
Maintainer Author

Oh I haven't tried the CLI yet; thanks for the info!

xylar Apr 2, 2026
Collaborator

In VS code with a paid copilot account, it works wonders!

forsyth2 · 2026-04-09T20:57:05Z

forsyth2
Apr 9, 2026
Maintainer Author

2026-04-08 Compy

What expected results are currently on Compy?

Setup steps

As of 2026-03-31:

ls /compyfs/www/zppy_test_resources/
# expected_bash_files                       image_list_expected_legacy_3.0.0_bundles.txt
# expected_bundles                          image_list_expected_legacy_3.0.0_comprehensive_v2.txt
# expected_comprehensive_v2                 image_list_expected_legacy_3.0.0_comprehensive_v3.txt
# expected_comprehensive_v3                 test_campaign_cryosphere_expected_files
# expected_legacy_3.0.0_bundles             test_campaign_cryosphere_override_expected_files
# expected_legacy_3.0.0_comprehensive_v2    test_campaign_high_res_v1_expected_files
# expected_legacy_3.0.0_comprehensive_v3    test_campaign_none_expected_files
# image_list_expected_bundles.txt           test_campaign_water_cycle_expected_files
# image_list_expected_comprehensive_v2.txt  test_campaign_water_cycle_override_expected_files
# image_list_expected_comprehensive_v3.txt  test_defaults_expected_files
ls /compyfs/www/fors729/zppy_test_resources_previous
# expected_results_for_unified_1.12.0  expected_results_until_20250429_from_unified_1.11.1
ls -lt /compyfs/www/zppy_test_resources/
# Last updates were Nov. 18
ls -lt /compyfs/www/fors729/zppy_test_resources_previous/expected_results_for_unified_1.12.0
# Last updates were Dec. 1

Note #758 and #759 were merged between 11/17 and 12/1. Let's update the expected results we test against to be the Unified 1.12.0 official results

cd /compyfs/www/
# Swap out the expected results:

# Unlike on Perlmutter, we don't have write-access to this level of the directory tree on Compy.
# That means we can't simply run:
# mv zppy_test_resources/ zppy_test_resources_previous/expected_results_until_20260331
# Because then zppy_test_resources would be gone

# Backup the current contents somewhere writable under the existing tree
mkdir -p fors729/zppy_test_resources_previous/expected_results_until_20260331

# Move everything out of zppy_test_resources/, including hidden files
# From ChatGPT:
# * = "all visible files"
# dotglob = "actually, include hidden ones too"
# nullglob = "and don’t break if there are none"
shopt -s dotglob nullglob
mv zppy_test_resources/* fors729/zppy_test_resources_previous/expected_results_until_20260331/
shopt -u dotglob nullglob

# Copy the replacement expected results into the existing directory
cp -a fors729/zppy_test_resources_previous/expected_results_for_unified_1.12.0/. zppy_test_resources/
# Took over FOUR HOURS to complete
ls -lt zppy_test_resources
# Good, last update is Dec. 1

The latest expected results have thus been updated to be the E3SM Unified 1.12.0 official results.

What changes are we testing?

Package	Changes since last test
e3sm_diags	Anything merged sinced E3SM Unified 1.12.0
mpas_analysis	Anything merged sinced E3SM Unified 1.12.0
zppy-interfaces	Anything merged sinced E3SM Unified 1.12.0
zppy	Anything merged sinced E3SM Unified 1.12.0

Testing process

Starting actual testing on 2026-04-08
Branch: zppy main branch + commit from https://github.com/E3SM-Project/zppy/pull/805/commits
Environments: zppy -- dev environment; everything else -- E3SM Unified 1.13.0rc4

Set up environments

We can mostly skip this step, since all the packages will be tested using the E3SM Unified RC.

cd ~/ez/zppy-interfaces
git status
# # On branch official_results_unified_1.12.0_compy
# nothing to commit, working directory clean

# REMINDER: On Compy, you must NOT specify the branch, 
# otherwise it won't actually fetch the latest commits!!
git fetch upstream 
git checkout main
git reset --hard upstream/main
git log --oneline | head -n 1
# 6b0758f Bump to 0.2.1rc1 (#48)
# Good, matches https://github.com/E3SM-Project/zppy-interfaces/commits/main
source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc4_compy.sh

# Initial permission failure, but now fixed

# Check zppy-interfaces tests
pytest tests/unit/global_time_series/test_*.py # 10 passed, 2 warnings in 14.35s
# SyntaxWarning: invalid escape sequence '\w'
pytest tests/unit/pcmdi_diags/test_*.py # 7 passed in 9.52s

zppy run

# zppy itself #################################################################
cd ~/ez/zppy
git status
# On branch official_results_unified_1.12.0_compy
git add -A
git commit -m "Testing" --no-verify

git fetch upstream # Remember, do NOT specify the branch on Compy!
git checkout -b test-zppy-main-plus-unified-1-13rc4 upstream/main
git log --oneline | head -n 1
# 00cd1bd Only use climo_subsections if it is non-empty (#804)
# Good, matches https://github.com/E3SM-Project/zppy/commits/main

# Add the commit from unmerged PR 
# https://github.com/E3SM-Project/zppy/pull/805/commits
# We've already fetched all branches from upstream, 
# BUT this commit is from a fork
git remote -v
git remote add mkstratos git@github.com:mkstratos/zppy.git
git remote -v
git fetch mkstratos # Remember, do NOT specify the branch on Compy!
git cherry-pick 5c61e1e91c19200210ed26f400df0139354899f1
git log --oneline | head -n 3
# 97be160 Add energy balance timeseries analysis to livvkit
# 00cd1bd Only use climo_subsections if it is non-empty (#804)
# 0ad4774 Fix legacy 3.1.0 www paths (#801)

lfs quota -hu fors729 /compyfs # Have enough space on /compyfs
# Not sure how to check space on /qfs
# (Currently in /qfs/people/fors729/ez/zppy)

compy_conda # Function to activate conda
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n test-zppy-main-plus-unified-1-13rc4
conda activate test-zppy-main-plus-unified-1-13rc4
pre-commit run --all-files # Confirm pre-commit still passes
python -m pip install .
conda list | grep python # Check Python version
# python                    3.14.4

# Run zppy unit tests
pytest tests/test_*.py # 44 passed in 0.85s

# Edit tests/integration/utils.py:

TEST_SPECIFICS: Dict[str, Any] = {
    # These are custom environment_commands for specific tasks.
    # Never set these to "", because they will print the line
    # `environment_commands = ""` for the corresponding task,
    # thus overriding the value set higher up in the cfg.
    # That is, there will be no environment set.
    # (`environment_commands = ""` only redirects to Unified
    # if specified under the [default] task)
    "diags_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc4_compy.sh",
    "mpas_analysis_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc4_compy.sh",
    "global_time_series_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc4_compy.sh",
    "livvkit_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc4_compy.sh",
    "pcmdi_diags_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc4_compy.sh",
    # This is the environment setup for other tasks.
    # Leave as "" to use the latest Unified environment.
    "environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc4_compy.sh",
    # For a complete test, run the set of latest cfgs and at least one set of legacy cfgs
    "cfgs_to_run": [
        "weekly_bundles",
        "weekly_comprehensive_v2",
        "weekly_comprehensive_v3",
        "weekly_legacy_3.1.0_bundles",
        "weekly_legacy_3.1.0_comprehensive_v2",
        "weekly_legacy_3.1.0_comprehensive_v3",
        "weekly_legacy_3.0.0_bundles",
        "weekly_legacy_3.0.0_comprehensive_v2",
        "weekly_legacy_3.0.0_comprehensive_v3",
    ],
    "tasks_to_run": [
        "e3sm_diags",
        "mpas_analysis",
        "global_time_series",
        "ilamb",
        "livvkit",
        "pcmdi_diags",
    ],
    "unique_id": "zppy_main_branch_plus_unified_1_13rc4",
}

git diff # Diff looks good
python tests/integration/utils.py
# CFG FILES HAVE BEEN GENERATED FROM TEMPLATES WITH THESE SETTINGS:
# UNIQUE_ID=zppy_main_branch_plus_unified_1_13rc4
# diags_environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc4_compy.sh
# mpas_analysis_environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc4_compy.sh
# global_time_series_environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc4_compy.sh
# livvkit_environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc4_compy.sh
# pcmdi_diags_environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc4_compy.sh
# environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc4_compy.sh
# Reminder: `environment_commands=''` => the latest E3SM Unified environment will be used

# For reference:
alias sq
# alias sq='sqa -u fors729'
alias sqa
# alias sqa='squeue -o "%8u %.7a %.4D %.9P %7i %.2t %.10r %.10M %.10l %.8Q %j" --sort=P,-t,-p'
sq
# No jobs currently queued

# Launch jobs; start with the most important one, comprehensive_v3
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_compy.cfg
zppy -c tests/integration/generated/test_weekly_bundles_compy.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_compy.cfg

zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_bundles_compy.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_comprehensive_v2_compy.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_comprehensive_v3_compy.cfg

zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_compy.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v2_compy.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v3_compy.cfg

sq | wc -l # Wed 4/8 17:22 => 224 - header row = 223 jobs
sq | wc -l # Wed 4/8 17:41 => 171 - header row = 170 jobs

# Picking up 2026-04-09
sq | wc -l # Thu 4/9 09:37 => 1 - header row = 0 jobs

Bundles part 2

# Check on bundles status
cd /compyfs/fors729/zppy_weekly_bundles_output/zppy_main_branch_plus_unified_1_13rc4/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors
cd /compyfs/fors729/zppy_weekly_legacy_3.1.0_bundles_output/zppy_main_branch_plus_unified_1_13rc4/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors
cd /compyfs/fors729/zppy_weekly_legacy_3.0.0_bundles_output/zppy_main_branch_plus_unified_1_13rc4/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors

# Now, run bundles part 2 for the working cfgs.
cd ~/ez/zppy
git status
# On branch test-zppy-main-plus-unified-1-13rc4
# Good, correct branch
# Need to re-activate conda env
compy_conda
conda activate test-zppy-main-plus-unified-1-13rc4

zppy -c tests/integration/generated/test_weekly_bundles_compy.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_bundles_compy.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_compy.cfg
sq | wc -l # Thu 4/9 09:43 => 7 - header row = 6 jobs

Review finished runs

### v2  ###
cd /compyfs/fors729/zppy_weekly_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc4/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# climo_atm_monthly_180x360_aave_1980-1981.status:RUNNING 750973
# climo_atm_monthly_180x360_aave_1982-1983.status:RUNNING 750974
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.status:WAITING 750997
# e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1980-1981_vs_1980-1981.status:WAITING 750998
# e3sm_to_cmip_land_monthly_1980-1981-0002.status:WAITING 750993
# e3sm_to_cmip_land_monthly_1982-1983-0002.status:WAITING 750994
# global_time_series_1980-1990.status:WAITING 751002
# ilamb_1980-1981.status:WAITING 751003
# ilamb_1982-1983.status:WAITING 751004
# mpas_analysis_ts_1980-1984_climo_1980-1984.status:ERROR (1)
# mpas_analysis_ts_1980-1990_climo_1985-1990.status:WAITING 751001
# ts_land_monthly_1980-1981-0002.status:RUNNING 750989
cd /compyfs/fors729/zppy_weekly_legacy_3.1.0_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc4/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# climo_atm_monthly_180x360_aave_1980-1981.status:RUNNING 751008
# climo_atm_monthly_180x360_aave_1982-1983.status:RUNNING 751009
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.status:WAITING 751032
# e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1980-1981_vs_1980-1981.status:WAITING 751033
# e3sm_to_cmip_land_monthly_1980-1981-0002.status:RUNNING 751028
# global_time_series_1980-1990.status:WAITING 751037
# ilamb_1980-1981.status:WAITING 751038
# ilamb_1982-1983.status:WAITING 751039
# mpas_analysis_ts_1980-1984_climo_1980-1984.status:ERROR (1)
# mpas_analysis_ts_1980-1990_climo_1985-1990.status:WAITING 751036
# tc_analysis_1980-1981.status:RUNNING 751030
cd /compyfs/fors729/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc4/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.status:WAITING 751112
# e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1980-1981_vs_1980-1981.status:WAITING 751113
# e3sm_to_cmip_atm_monthly_180x360_aave_1980-1981-0002.status:RUNNING 751106
# e3sm_to_cmip_atm_monthly_180x360_aave_1982-1983-0002.status:RUNNING 751107
# e3sm_to_cmip_land_monthly_1980-1981-0002.status:WAITING 751108
# e3sm_to_cmip_land_monthly_1982-1983-0002.status:WAITING 751109
# global_time_series_1980-1990.status:WAITING 751117
# ilamb_1980-1981.status:WAITING 751118
# ilamb_1982-1983.status:WAITING 751119
# mpas_analysis_ts_1980-1984_climo_1980-1984.status:ERROR (1)
# mpas_analysis_ts_1980-1990_climo_1985-1990.status:WAITING 751116
# tc_analysis_1982-1983.status:RUNNING 751111
# ts_atm_monthly_glb_1980-1984-0005.status:RUNNING 751100
# ts_atm_monthly_glb_1985-1989-0005.status:RUNNING 751101
# ts_land_monthly_1980-1981-0002.status:RUNNING 751104
# ts_land_monthly_1982-1983-0002.status:RUNNING 751105
# ts_rof_monthly_1980-1981-0002.status:RUNNING 751098
# ts_rof_monthly_1982-1983-0002.status:RUNNING 751099

### v3 ###
cd /compyfs/fors729/zppy_weekly_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc4/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# climo_land_monthly_180x360_traave_1985-1994.status:ERROR (3)
# climo_land_monthly_climo_era5_1985-1994.status:ERROR (3)
# climo_land_monthly_climo_merra2_1985-1994.status:ERROR (3)
# climo_land_monthly_climo_racmo_ais_1985-1994.status:ERROR (3)
# climo_land_monthly_climo_racmo_gis_1985-1994.status:ERROR (3)
# global_time_series_classic_original_8_no_ocn_1985-1995.status:RUNNING 750964
# global_time_series_classic_pdf_both_1985-1995.status:WAITING 750960
# global_time_series_classic_pdf_original_1985-1995.status:WAITING 750961
# global_time_series_viewer_both_1985-1995.status:WAITING 750957
# global_time_series_viewer_component_1985-1995.status:RUNNING 750959
# global_time_series_viewer_original_1985-1995.status:WAITING 750958
# ilamb_1985-1988.status:ERROR (3)
# livvkit_1985-1994.status:WAITING 750966
# mpas_analysis_mvm_ts_1985-1995_climo_1990-1995_vs_ref_ts_1985-1989_climo_1985-1989.status:WAITING 750956
# mpas_analysis_reference_ts_1985-1989_climo_1985-1989.status:ERROR (1)
# mpas_analysis_test_ts_1985-1995_climo_1990-1995.status:WAITING 750955
# pcmdi_diags_synthetic_plots_model_vs_obs.status:WAITING 750970
# pcmdi_diags_variability_modes_cpl_model_vs_obs_1985-1994.status:ERROR (14)
cd /compyfs/fors729/zppy_weekly_legacy_3.1.0_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc4/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# climo_atm_monthly_180x360_aave_1985-1986.status:RUNNING 751040
# climo_atm_monthly_diurnal_8xdaily_180x360_aave_1985-1986.status:RUNNING 751042
# climo_atm_monthly_diurnal_8xdaily_180x360_aave_1987-1988.status:RUNNING 751043
# climo_land_monthly_climo_1985-1986.status:RUNNING 751044
# climo_land_monthly_climo_1987-1988.status:RUNNING 751045
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1987-1988.status:WAITING 751068
# e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1987-1988_vs_1985-1986.status:WAITING 751069
# e3sm_diags_lnd_monthly_mvm_lnd_model_vs_model_1987-1988_vs_1985-1986.status:WAITING 751070
# e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.status:WAITING 751061
# e3sm_to_cmip_atm_monthly_180x360_aave_1987-1988-0002.status:WAITING 751062
# e3sm_to_cmip_atm_monthly_180x360_aave_1989-1990-0002.status:WAITING 751063
# e3sm_to_cmip_atm_monthly_180x360_aave_1991-1992-0002.status:WAITING 751064
# e3sm_to_cmip_atm_monthly_180x360_aave_1993-1994-0002.status:WAITING 751065
# global_time_series_all_lnd_var_viewer_1985-1995.status:WAITING 751079
# global_time_series_classic_original_8_no_ocn_1985-1995.status:WAITING 751080
# global_time_series_classic_pdf_both_1985-1995.status:WAITING 751076
# global_time_series_classic_pdf_component_1985-1995.status:WAITING 751078
# global_time_series_classic_pdf_original_1985-1995.status:WAITING 751077
# global_time_series_viewer_both_1985-1995.status:WAITING 751073
# global_time_series_viewer_component_1985-1995.status:WAITING 751075
# global_time_series_viewer_original_1985-1995.status:WAITING 751074
# ilamb_1985-1988.status:WAITING 751081
# mpas_analysis_ts_1985-1989_climo_1985-1989.status:ERROR (1)
# mpas_analysis_ts_1985-1995_climo_1990-1995.status:WAITING 751072
# pcmdi_diags_mean_climate_model_vs_obs_1985-1994.status:WAITING 751082
# pcmdi_diags_synthetic_plots_model_vs_obs.status:WAITING 751085
# pcmdi_diags_variability_modes_atm_model_vs_obs_1985-1994.status:WAITING 751084
# pcmdi_diags_variability_modes_cpl_model_vs_obs_1985-1994.status:WAITING 751083
# ts_atm_monthly_180x360_aave_1985-1986-0002.status:RUNNING 751046
# ts_atm_monthly_180x360_aave_1987-1988-0002.status:RUNNING 751047
# ts_atm_monthly_180x360_aave_1989-1990-0002.status:RUNNING 751048
# ts_atm_monthly_glb_1985-1989-0005.status:RUNNING 751055
# ts_lnd_monthly_glb_1985-1989-0005.status:RUNNING 751057
# ts_lnd_monthly_glb_1990-1994-0005.status:RUNNING 751058
# ts_rof_monthly_1985-1986-0002.status:RUNNING 751053
# ts_rof_monthly_1987-1988-0002.status:RUNNING 751054
cd /compyfs/fors729/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc4/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# climo_atm_monthly_180x360_aave_1985-1986.status:RUNNING 751120
# climo_land_monthly_climo_1987-1988.status:RUNNING 751125
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1987-1988.status:WAITING 751142
# e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1987-1988_vs_1985-1986.status:WAITING 751143
# e3sm_diags_lnd_monthly_mvm_lnd_model_vs_model_1987-1988_vs_1985-1986.status:WAITING 751144
# e3sm_to_cmip_atm_monthly_180x360_aave_1987-1988-0002.status:RUNNING 751139
# e3sm_to_cmip_land_monthly_1985-1986-0002.status:WAITING 751140
# e3sm_to_cmip_land_monthly_1987-1988-0002.status:WAITING 751141
# global_time_series_1985-1995.status:WAITING 751147
# ilamb_1985-1988.status:WAITING 751148
# mpas_analysis_ts_1985-1989_climo_1985-1989.status:ERROR (1)
# mpas_analysis_ts_1985-1995_climo_1990-1995.status:WAITING 751146
# ts_atm_monthly_glb_1985-1989-0005.status:RUNNING 751132
# ts_atm_monthly_glb_1990-1994-0005.status:RUNNING 751133
# ts_land_monthly_1985-1986-0002.status:RUNNING 751136
# ts_land_monthly_1987-1988-0002.status:RUNNING 751137
# ts_lnd_monthly_glb_1990-1994-0005.status:RUNNING 751135
# ts_rof_monthly_1985-1986-0002.status:RUNNING 751130
# ts_rof_monthly_1987-1988-0002.status:RUNNING 751131

### bundles ###
sq | wc -l # Thu 4/9 09:51 => 1 - header row = 0 jobs
cd /compyfs/fors729/zppy_weekly_bundles_output/zppy_main_branch_plus_unified_1_13rc4/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# ilamb_1985-1986.status:ERROR (3)
cd /compyfs/fors729/zppy_weekly_legacy_3.1.0_bundles_output/zppy_main_branch_plus_unified_1_13rc4/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# ilamb_1985-1986.status:ERROR (3)
cd /compyfs/fors729/zppy_weekly_legacy_3.0.0_bundles_output/zppy_main_branch_plus_unified_1_13rc4/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# ilamb_1985-1986.status:ERROR (3)

post/scripts directory status: Many many errors -- will review in Results Analysis section below

Python tests

cd ~/ez/zppy
git status
# On branch test-zppy-main-plus-unified-1-13rc4
# Good, correct branch
# Env: test-zppy-main-plus-unified-1-13rc4
# Good, correct env
ls tests/integration/test_*.py

# Errors look to be expected based on code changes during this development period:
pytest tests/integration/test_bash_generation.py 
# 1 failed in 3.74s
pytest tests/integration/test_campaign.py
# 6 failed in 10.97s
pytest tests/integration/test_defaults.py
# 1 failed in 5.61s

# Passing
pytest tests/integration/test_last_year.py
# 1 passed in 0.91s

# Tests that rely on the `zppy` runs we just did:
pytest tests/integration/test_bundles.py 
# 2 passed in 0.85s
# Even though ILAMB had issues, it looks like this test is fine.

# Not going to run `tests/integration/test_images.py` because we had so many output directory failures.

Results analysis

The Python tests that don't rely on zppy runs had failures that were to be expected based on changes over the development period since E3SM Unified 1.12.0. So, I think those are fine.

The larger issue is the errors showing up in the the output directories (post/scripts).

Comprehensive v3 test

Let's start with the non-legacy v3 run, which has the most up-to-date config file.

Debugging

cd /compyfs/fors729/zppy_weekly_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc4/v3.LR.historical_0051/post/scripts

grep -v "OK" *status | wc -l # 18
grep "RUNNING" *status | wc -l # 2
grep "WAITING" *status | wc -l # 8
grep "ERROR" *status | wc -l # 8

## Review RUNNING jobs ########################################################
# Usually this happens if jobs hit the time limit.

grep "RUNNING" *status 
# global_time_series_classic_original_8_no_ocn_1985-1995.status:RUNNING 750964
# global_time_series_viewer_component_1985-1995.status:RUNNING 750959
cat global_time_series_classic_original_8_no_ocn_1985-1995.o750964 
# verifying deployed e3sm-unified version...
cat global_time_series_viewer_component_1985-1995.o750959 
# verifying deployed e3sm-unified version...

# So, what happened? There's certainly no time limit being hit.
# These didn't get very far at all.

emacs global_time_series_classic_original_8_no_ocn_1985-1995.bash
# set -e
# source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc4_compy.sh
# set +e
# So, it seems like this failed immediately.
source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc4_compy.sh
# Produces non-zero exit code so it works fine, if a bit slow.
# BUT, it prints more:
# verifying deployed e3sm-unified version...
#   Verified version 1.13.0rc4.
# loading login pixi env...
#    pixi env loaded.
# loading e3sm-unified environment variables...
#    e3sm-unified environment variables loaded.

# So, that means the Unified sourcing command isn't completing.

# The `zi-global-time-series` call should probably have a `if [ $? != 0 ]` error check after it.

## Review WAITING jobs ########################################################
# No jobs were left in the queue (sq), so why do so many say waiting?
# Did Compy automatically remove jobs waiting on a dependency that could never be resolved?

grep "WAITING" *status
# global_time_series_classic_pdf_both_1985-1995.status:WAITING 750960
# global_time_series_classic_pdf_original_1985-1995.status:WAITING 750961
# global_time_series_viewer_both_1985-1995.status:WAITING 750957
# global_time_series_viewer_original_1985-1995.status:WAITING 750958
# livvkit_1985-1994.status:WAITING 750966
# mpas_analysis_mvm_ts_1985-1995_climo_1990-1995_vs_ref_ts_1985-1989_climo_1985-1989.status:WAITING 750956
# mpas_analysis_test_ts_1985-1995_climo_1990-1995.status:WAITING 750955
# pcmdi_diags_synthetic_plots_model_vs_obs.status:WAITING 750970

## Review ERROR jobs ##########################################################

grep "ERROR" *status 
# climo_land_monthly_180x360_traave_1985-1994.status:ERROR (3)
# climo_land_monthly_climo_era5_1985-1994.status:ERROR (3)
# climo_land_monthly_climo_merra2_1985-1994.status:ERROR (3)
# climo_land_monthly_climo_racmo_ais_1985-1994.status:ERROR (3)
# climo_land_monthly_climo_racmo_gis_1985-1994.status:ERROR (3)
# ilamb_1985-1988.status:ERROR (3)
# mpas_analysis_reference_ts_1985-1989_climo_1985-1989.status:ERROR (1)
# pcmdi_diags_variability_modes_cpl_model_vs_obs_1985-1994.status:ERROR (14)

cat climo_land_monthly_180x360_traave_1985-1994.o750922 
# verifying deployed e3sm-unified version...
#   Verified version 1.13.0rc4.
# loading compute pixi env...
#    pixi env loaded.
# activating spack env...
#    spack env activated.
# loading e3sm-unified environment variables...
#    e3sm-unified environment variables loaded.
# ncclimo: ERROR Unable to find specified regrid map /lcrc/group/e3sm/ac.forsyth2/maps/map_r05_to_cmip6_180x360_traave.20240901.nc
# ncclimo: HINT Supply the full path-name for the regridding map

# Ah, it's looking for a LCRC grid path: /lcrc/group/e3sm/ac.forsyth2/maps/map_r05_to_cmip6_180x360_traave.20240901.nc

# Is, this what's happening elsewhere?
grep lcrc *.o*
# climo_land_monthly_180x360_traave_1985-1994.o750922:ncclimo: ERROR Unable to find specified regrid map /lcrc/group/e3sm/ac.forsyth2/maps/map_r05_to_cmip6_180x360_traave.20240901.nc
# climo_land_monthly_climo_era5_1985-1994.o750927:ncclimo: ERROR Unable to find specified regrid map /lcrc/group/e3sm/public_html/diagnostics/maps/map_r05_to_era5_s2n_721x1440.traave.20240124.nc
# climo_land_monthly_climo_merra2_1985-1994.o750926:ncclimo: ERROR Unable to find specified regrid map /lcrc/group/e3sm/public_html/diagnostics/maps/map_r05_to_merra2_traave.20250124.nc
# climo_land_monthly_climo_racmo_ais_1985-1994.o750925:ncclimo: ERROR Unable to find specified regrid map /lcrc/group/e3sm/public_html/diagnostics/maps/map_r05_to_racmo_ais_591x726.nc
# climo_land_monthly_climo_racmo_gis_1985-1994.o750924:ncclimo: ERROR Unable to find specified regrid map /lcrc/group/e3sm/ac.forsyth2/maps/map_r05_to_racmo_gis_566x438_traave.20240801.nc

# So, it explains the 5 climo failures only.

emacs ilamb_1985-1988.o750965 
# FileNotFoundError: [Errno 2] No such file or directory: '/compyfs/diagnostics/ilamb_data/DATA/twsa/mrro/Dai/basins_0.5x0.5.nc'
# srun: error: n0001: task 0: Exited with exit code 1

# So, ilamb is looking on the right machine, but the data isn't there.

ls /compyfs/diagnostics/ilamb_data/DATA/twsa/
# GRACE

# So, no `mrro` subdir

grep -in error mpas_analysis_reference_ts_1985-1989_climo_1985-1989.o750954 
# 94:WARNING: There was an error initializing an OpenFabrics device.
# 111:WARNING: There was an error initializing an OpenFabrics device.
# 117:WARNING: There was an error initializing an OpenFabrics device.
# 123:WARNING: There was an error initializing an OpenFabrics device.
# 151:WARNING: There was an error initializing an OpenFabrics device.
# 157:WARNING: There was an error initializing an OpenFabrics device.
# 182: ERROR: Problem on processor            1 . Please see the PET*.RegridWeightGen.Log files for a traceback.
# 185:with errorcode 1.
# 191: ERROR: Problem on processor            5 . Please see the PET*.RegridWeightGen.Log files for a traceback.
# 194:with errorcode 1.
# 200: ERROR: Problem on processor            3 . Please see the PET*.RegridWeightGen.Log files for a traceback.
# 203:with errorcode 1.
# 209: ERROR: Problem on processor            2 . Please see the PET*.RegridWeightGen.Log files for a traceback.
# 210: ERROR: Problem on processor            4 . Please see the PET*.RegridWeightGen.Log files for a traceback.
# 213:with errorcode 1.
# 221:with errorcode 1.
# 228: ERROR: Problem on processor            0 . Please see the PET*.RegridWeightGen.Log files for a traceback.
# 231:with errorcode 1.
# 237:slurmstepd: error: *** STEP 750954.0 ON n0059 CANCELLED AT 2026-04-08T17:32:41 ***
# 238:srun: error: n0059: tasks 0-5: Exited with exit code 1
# 274:    raise CalledProcessError(retcode, process.args,
# 276:subprocess.CalledProcessError: Command '['srun', '--mpi=pmi2', '-n', '6', 'ESMF_RegridWeightGen', '--source', '/compyfs/fors729/zppy_weekly_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc4/v3.LR.historical_0051/post/analysis/mpas_analysis/mvo/ts_1985-1989_climo_1985-1989/mapping/tmpd7fkgvz7/src_mesh.nc', '--destination', '/compyfs/fors729/zppy_weekly_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc4/v3.LR.historical_0051/post/analysis/mpas_analysis/mvo/ts_1985-1989_climo_1985-1989/mapping/tmpd7fkgvz7/dst_mesh.nc', '--weight', '/compyfs/fors729/zppy_weekly_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc4/v3.LR.historical_0051/post/scripts/../analysis/mpas_analysis/mvo/ts_1985-1989_climo_1985-1989/mapping/map_IcoswISC30E3r5_to_WOA_transects_5km_bilinear.nc', '--method', 'bilinear', '--netcdf4', '--src_regional', '--dst_regional', '--ignore_unmapped']' returned non-zero exit status 1.
# 284:ERROR in task refYearMpasClimatologyOcean.  See log file /compyfs/fors729/zppy_weekly_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc4/v3.LR.historical_0051/post/scripts/../analysis/mpas_analysis/mvo/ts_1985-1989_climo_1985-1989/logs/refYearMpasClimatologyOcean.log for details
# 707:There were errors in task refYearMpasClimatologyOcean
grep -in error /compyfs/fors729/zppy_weekly_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc4/v3.LR.historical_0051/post/scripts/../analysis/mpas_analysis/mvo/ts_1985-1989_climo_1985-1989/logs/refYearMpasClimatologyOcean.log
# 7:ncclimo: ERROR Number of input files = 60 differs from number of months = 12 computed from calendar inputs
# 23:    raise subprocess.CalledProcessError(process.returncode,
# 25:subprocess.CalledProcessError: Command 'ncclimo --no_stdin -4 --clm_md=mth -a sdd -P mpaso -p bck -j 12 -v [...]

# This appears to match up with the error reported in https://github.com/MPAS-Dev/MPAS-Analysis/issues/1145

emacs pcmdi_diags_variability_modes_cpl_model_vs_obs_1985-1994.o750968 
# rsync: [sender] link_stat "/compyfs/fors729/zppy_weekly_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc4/v3.LR.historical_0051/post/scripts/tmp.p\
# cmdi_diags_variability_modes_cpl_model_vs_obs_1985-1994.750968.EGYm/model_vs_obs" failed: No such file or directory (2)
# rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1338) [sender=3.4.1]
grep -in error pcmdi_diags_variability_modes_cpl_model_vs_obs_1985-1994.o750968 
# 78:2026-04-08 17:39:51,216 [ERROR]: utils.py(run_parallel_jobs:67) >> ERROR: Process failed: 'variability_modes_driver.py -p parameterfile.py --variability_mode PDO --eofn_mod 1 --eofn_obs 1 --varOBS ts --osyear 1985 --oeyear 1994 --reference_data_name HadISST2 --reference_data_path ts_ref/obs.historical.HadISST2.00.Amon.ts.198501-199412.nc --case_id v20260408'
# 79:Error: /qfs/projects/ops/rh7/apps/E3SM/conda_envs/e3smu_1_13_0rc4/compy/pixi/.pixi/envs/default/lib/python3.13/site-packages/pcmdi_metrics/utils/land_sea_mask.py:216: UserWarning: landfrac is not provided thus generated using the 'create_land_sea_mask' function
# 106:  File "/qfs/projects/ops/rh7/apps/E3SM/conda_envs/e3smu_1_13_0rc4/compy/pixi/.pixi/envs/default/lib/python3.13/site-packages/numpy/linalg/_linalg.py", line 157, in _raise_linalgerror_lstsq
# 107:    raise LinAlgError("SVD did not converge in Linear Least Squares")
# 108:numpy.linalg.LinAlgError: SVD did not converge in Linear Least Squares
# 129:rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1338) [sender=3.4.1]

# Not sure what would have caused this...

Comprehensive v2 test

Debugging

cd /compyfs/fors729/zppy_weekly_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc4/v2.LR.historical_0201/post/scripts
grep -v "OK" *status | wc -l # 12
grep "RUNNING" *status | wc -l # 3
grep "WAITING" *status | wc -l # 8
grep "ERROR" *status | wc -l # 1

## Review RUNNING jobs ########################################################
# Usually this happens if jobs hit the time limit.

grep "RUNNING" *status
# climo_atm_monthly_180x360_aave_1980-1981.status:RUNNING 750973
# climo_atm_monthly_180x360_aave_1982-1983.status:RUNNING 750974
# ts_land_monthly_1980-1981-0002.status:RUNNING 750989

cat climo_atm_monthly_180x360_aave_1980-1981.o750973 
# verifying deployed e3sm-unified version...
cat climo_atm_monthly_180x360_aave_1982-1983.o750974 
# verifying deployed e3sm-unified version...
cat ts_land_monthly_1980-1981-0002.o750989
# verifying deployed e3sm-unified version...

# Interestingly this is the same RUNNING error we got for v3, but now on different tasks entirely...

## Review WAITING jobs ########################################################
# No jobs were left in the queue (sq), so why do so many say waiting?
# Did Compy automatically remove jobs waiting on a dependency that could never be resolved?

grep "WAITING" *status
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.status:WAITING 750997
# e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1980-1981_vs_1980-1981.status:WAITING 750998
# e3sm_to_cmip_land_monthly_1980-1981-0002.status:WAITING 750993
# e3sm_to_cmip_land_monthly_1982-1983-0002.status:WAITING 750994
# global_time_series_1980-1990.status:WAITING 751002
# ilamb_1980-1981.status:WAITING 751003
# ilamb_1982-1983.status:WAITING 751004
# mpas_analysis_ts_1980-1990_climo_1985-1990.status:WAITING 751001

## Review ERROR jobs ##########################################################

grep "ERROR" *status 
# mpas_analysis_ts_1980-1984_climo_1980-1984.status:ERROR (1)

grep -in error mpas_analysis_ts_1980-1984_climo_1980-1984.o751000
# 114:WARNING: There was an error initializing an OpenFabrics device.
# 120:WARNING: There was an error initializing an OpenFabrics device.
# 126:WARNING: There was an error initializing an OpenFabrics device.
# 132:WARNING: There was an error initializing an OpenFabrics device.
# 138:WARNING: There was an error initializing an OpenFabrics device.
# 155:WARNING: There was an error initializing an OpenFabrics device.
# 180: ERROR: Problem on processor            3 . Please see the PET*.RegridWeightGen.Log files for a traceback.
# 183:with errorcode 1.
# 190:slurmstepd: error: *** STEP 751000.0 ON n0062 CANCELLED AT 2026-04-08T17:28:06 ***
# 191:srun: error: n0062: tasks 0-2,4-5: Killed
# 192:srun: error: n0062: task 3: Exited with exit code 1
# 227:    raise CalledProcessError(retcode, process.args,
# 229:subprocess.CalledProcessError: Command '['srun', '--mpi=pmi2', '-n', '6', 'ESMF_RegridWeightGen', '--source', '/compyfs/fors729/zppy_weekly_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc4/v2.LR.historical_0201/post/analysis/mpas_analysis/mvo/ts_1980-1984_climo_1980-1984/mapping/tmp6_cbz_6f/src_mesh.nc', '--destination', '/compyfs/fors729/zppy_weekly_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc4/v2.LR.historical_0201/post/analysis/mpas_analysis/mvo/ts_1980-1984_climo_1980-1984/mapping/tmp6_cbz_6f/dst_mesh.nc', '--weight', '/compyfs/fors729/zppy_weekly_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc4/v2.LR.historical_0201/post/scripts/../analysis/mpas_analysis/mvo/ts_1980-1984_climo_1980-1984/mapping/map_EC30to60E2r2_to_AntShip_transects_5km_bilinear.nc', '--method', 'bilinear', '--netcdf4', '--src_regional', '--dst_regional', '--ignore_unmapped']' returned non-zero exit status 137.
# 268:WARNING: There was an error initializing an OpenFabrics device.
# 274:WARNING: There was an error initializing an OpenFabrics device.
# 280:WARNING: There was an error initializing an OpenFabrics device.
# 297:WARNING: There was an error initializing an OpenFabrics device.
# 325:WARNING: There was an error initializing an OpenFabrics device.
# 331:WARNING: There was an error initializing an OpenFabrics device.
# 356: ERROR: Problem on processor            3 . Please see the PET*.RegridWeightGen.Log files for a traceback.
# 359:with errorcode 1.
# 366:slurmstepd: error: *** STEP 751000.1 ON n0062 CANCELLED AT 2026-04-08T17:29:23 ***
# 367:srun: error: n0062: tasks 0-2,4-5: Killed
# 368:srun: error: n0062: task 3: Exited with exit code 1
# 404:    raise CalledProcessError(retcode, process.args,
# 406:subprocess.CalledProcessError: Command '['srun', '--mpi=pmi2', '-n', '6', 'ESMF_RegridWeightGen', '--source', '/compyfs/fors729/zppy_weekly_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc4/v2.LR.historical_0201/post/analysis/mpas_analysis/mvo/ts_1980-1984_climo_1980-1984/mapping/tmpqycas985/src_mesh.nc', '--destination', '/compyfs/fors729/zppy_weekly_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc4/v2.LR.historical_0201/post/analysis/mpas_analysis/mvo/ts_1980-1984_climo_1980-1984/mapping/tmpqycas985/dst_mesh.nc', '--weight', '/compyfs/fors729/zppy_weekly_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc4/v2.LR.historical_0201/post/scripts/../analysis/mpas_analysis/mvo/ts_1980-1984_climo_1980-1984/mapping/map_EC30to60E2r2_to_WOA_transects_5km_bilinear.nc', '--method', 'bilinear', '--netcdf4', '--src_regional', '--dst_regional', '--ignore_unmapped']' returned non-zero exit status 137.
# 414:ERROR in task refYearMpasClimatologyOcean.  See log file /compyfs/fors729/zppy_weekly_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc4/v2.LR.historical_0201/post/scripts/../analysis/mpas_analysis/mvo/ts_1980-1984_climo_1980-1984/logs/refYearMpasClimatologyOcean.log for details
# 821:There were errors in task refYearMpasClimatologyOcean
grep -in error /compyfs/fors729/zppy_weekly_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc4/v2.LR.historical_0201/post/scripts/../analysis/mpas_analysis/mvo/ts_1980-1984_climo_1980-1984/logs/refYearMpasClimatologyOcean.log
# 7:ncclimo: ERROR Number of input files = 60 differs from number of months = 12 computed from calendar inputs
# 23:    raise subprocess.CalledProcessError(process.returncode,
# 25:subprocess.CalledProcessError: Command 'ncclimo --no_stdin -4 --clm_md=mth -a sdd -P mpaso -p bck -j 12 -v timeMonthly_avg_activeTracers_temperature,timeMonthly_avg_layerThickness --seasons=ANN -s 1980 -e 1980 -o [...]

# This appears to match up with the error reported in https://github.com/MPAS-Dev/MPAS-Analysis/issues/1145

Comprehensive bundles test

Debugging

cd /compyfs/fors729/zppy_weekly_bundles_output/zppy_main_branch_plus_unified_1_13rc4/v3.LR.historical_0051/post/scripts
grep -v "OK" *status | wc -l # 1
grep "ERROR" *status | wc -l # 1

## Review ERROR jobs ##########################################################

grep "ERROR" *status 
# ilamb_1985-1986.status:ERROR (3)

emacs ilamb_1985-1986.o751210
# FileNotFoundError: [Errno 2] No such file or directory: '/compyfs/diagnostics/ilamb_data/DATA/twsa/mrro/Dai/basins_0.5x0.5.nc'
# srun: error: n0423: task 0: Exited with exit code 1

# Same error that we encountered before

Legacy 3.1.0 tests

Debugging

### v2  ###
cd /compyfs/fors729/zppy_weekly_legacy_3.1.0_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc4/v2.LR.historical_0201/post/scripts
grep -v "OK" *status | wc -l # 11
grep "RUNNING" *status | wc -l # 4
grep "WAITING" *status | wc -l # 6 -- can't really do anything about these
grep "ERROR" *status | wc -l # 1

grep "RUNNING" *status
# climo_atm_monthly_180x360_aave_1980-1981.status:RUNNING 751008
# climo_atm_monthly_180x360_aave_1982-1983.status:RUNNING 751009
# e3sm_to_cmip_land_monthly_1980-1981-0002.status:RUNNING 751028
# tc_analysis_1980-1981.status:RUNNING 751030
wc -l climo_atm_monthly_180x360_aave_1980-1981.o751008 # 1 line
wc -l climo_atm_monthly_180x360_aave_1982-1983.o751009 # 1 line
wc -l e3sm_to_cmip_land_monthly_1980-1981-0002.o751028 # 1 line
wc -l tc_analysis_1980-1981.o751030 # 1 line
grep "verifying deployed e3sm-unified version..." climo_atm_monthly_180x360_aave_1980-1981.o751008 climo_atm_monthly_180x360_aave_1982-1983.o751009 e3sm_to_cmip_land_monthly_1980-1981-0002.o751028 tc_analysis_1980-1981.o751030
# climo_atm_monthly_180x360_aave_1980-1981.o751008:verifying deployed e3sm-unified version...
# climo_atm_monthly_180x360_aave_1982-1983.o751009:verifying deployed e3sm-unified version...
# e3sm_to_cmip_land_monthly_1980-1981-0002.o751028:verifying deployed e3sm-unified version...
# tc_analysis_1980-1981.o751030:verifying deployed e3sm-unified version...
# So, same errors we saw earlier.

grep "ERROR" *status
# mpas_analysis_ts_1980-1984_climo_1980-1984.status:ERROR (1)
# We're expecting an MPAS-Analysis error

### v3 ###
cd /compyfs/fors729/zppy_weekly_legacy_3.1.0_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc4/v3.LR.historical_0051/post/scripts

grep -v "OK" *status | wc -l # 36
grep "RUNNING" *status | wc -l # 13
grep "WAITING" *status | wc -l # 22 -- can't really do anything about these
grep "ERROR" *status | wc -l # 1

grep "RUNNING" *status
# climo_atm_monthly_180x360_aave_1985-1986.status:RUNNING 751040
# climo_atm_monthly_diurnal_8xdaily_180x360_aave_1985-1986.status:RUNNING 751042
# climo_atm_monthly_diurnal_8xdaily_180x360_aave_1987-1988.status:RUNNING 751043
# climo_land_monthly_climo_1985-1986.status:RUNNING 751044
# climo_land_monthly_climo_1987-1988.status:RUNNING 751045
# ts_atm_monthly_180x360_aave_1985-1986-0002.status:RUNNING 751046
# ts_atm_monthly_180x360_aave_1987-1988-0002.status:RUNNING 751047
# ts_atm_monthly_180x360_aave_1989-1990-0002.status:RUNNING 751048
# ts_atm_monthly_glb_1985-1989-0005.status:RUNNING 751055
# ts_lnd_monthly_glb_1985-1989-0005.status:RUNNING 751057
# ts_lnd_monthly_glb_1990-1994-0005.status:RUNNING 751058
# ts_rof_monthly_1985-1986-0002.status:RUNNING 751053
# ts_rof_monthly_1987-1988-0002.status:RUNNING 751054

# Check if the corresponding output has file has just 1 line:
wc -l climo_atm_monthly_180x360_aave_1985-1986.o751040
# 1 climo_atm_monthly_180x360_aave_1985-1986.o751040

# Check if that 1 line is what we've seen in other output files
grep -n "verifying deployed e3sm-unified version..." climo_atm_monthly_180x360_aave_1985-1986.o751040
# 1:verifying deployed e3sm-unified version...

# Make /qfs/people/fors729/ez/utils/helper_functions.bash.
# Add this AI-generated function:
# check_stuck_jobs() {
#   local f jobid base ofile
#   for f in *status; do
#     grep -q "RUNNING" "$f" || continue
#     jobid=$(awk '/RUNNING/{print $2}' "$f")
#     base="${f%.status}"
#     ofile="${base}.o${jobid}"

#     if [[ -f "$ofile" ]] &&
#        [[ $(wc -l < "$ofile") -eq 1 ]] &&
#        grep -qx "verifying deployed e3sm-unified version..." "$ofile"; then
#       echo "STUCK? $f -> $ofile"
#     fi
#   done
# }
chmod 755 /qfs/people/fors729/ez/utils/helper_functions.bash
source /qfs/people/fors729/ez/utils/helper_functions.bash
check_stuck_jobs
# STUCK? climo_atm_monthly_180x360_aave_1985-1986.status -> climo_atm_monthly_180x360_aave_1985-1986.o751040
# STUCK? climo_atm_monthly_diurnal_8xdaily_180x360_aave_1985-1986.status -> climo_atm_monthly_diurnal_8xdaily_180x360_aave_1985-1986.o751042
# STUCK? climo_atm_monthly_diurnal_8xdaily_180x360_aave_1987-1988.status -> climo_atm_monthly_diurnal_8xdaily_180x360_aave_1987-1988.o751043
# STUCK? climo_land_monthly_climo_1985-1986.status -> climo_land_monthly_climo_1985-1986.o751044
# STUCK? climo_land_monthly_climo_1987-1988.status -> climo_land_monthly_climo_1987-1988.o751045
# STUCK? ts_atm_monthly_180x360_aave_1985-1986-0002.status -> ts_atm_monthly_180x360_aave_1985-1986-0002.o751046
# STUCK? ts_atm_monthly_180x360_aave_1987-1988-0002.status -> ts_atm_monthly_180x360_aave_1987-1988-0002.o751047
# STUCK? ts_atm_monthly_180x360_aave_1989-1990-0002.status -> ts_atm_monthly_180x360_aave_1989-1990-0002.o751048
# STUCK? ts_atm_monthly_glb_1985-1989-0005.status -> ts_atm_monthly_glb_1985-1989-0005.o751055
# STUCK? ts_lnd_monthly_glb_1985-1989-0005.status -> ts_lnd_monthly_glb_1985-1989-0005.o751057
# STUCK? ts_lnd_monthly_glb_1990-1994-0005.status -> ts_lnd_monthly_glb_1990-1994-0005.o751058
# STUCK? ts_rof_monthly_1985-1986-0002.status -> ts_rof_monthly_1985-1986-0002.o751053
# STUCK? ts_rof_monthly_1987-1988-0002.status -> ts_rof_monthly_1987-1988-0002.o751054

# Yes, all 13 run into the same issue.

grep "ERROR" *status 
# mpas_analysis_ts_1985-1989_climo_1985-1989.status:ERROR (1)
# We're expecting an MPAS-Analysis error


### bundles ###
cd /compyfs/fors729/zppy_weekly_legacy_3.1.0_bundles_output/zppy_main_branch_plus_unified_1_13rc4/v3.LR.historical_0051/post/scripts

grep -v "OK" *status | wc -l # 1
grep "ERROR" *status | wc -l # 1

grep "ERROR" *status 
# ilamb_1985-1986.status:ERROR (3)
emacs ilamb_1985-1986.o751212
# FileNotFoundError: [Errno 2] No such file or directory: '/compyfs/diagnostics/ilamb_data/DATA/twsa/mrro/Dai/basins_0.5x0.5.nc'
# srun: error: n0427: task 0: Exited with exit code 1

# This is the same error we saw before.

Legacy 3.0.0 tests

Debugging

### v2  ###
cd /compyfs/fors729/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc4/v2.LR.historical_0201/post/scripts

grep -v "OK" *status | wc -l # 18
grep "RUNNING" *status | wc -l # 9
grep "WAITING" *status | wc -l # 8 -- can't really do anything about these
grep "ERROR" *status | wc -l # 1

check_stuck_jobs | wc -l # 9 # Matches RUNNING, as expected
grep "ERROR" *status
# mpas_analysis_ts_1980-1984_climo_1980-1984.status:ERROR (1)
# We're expecting an MPAS-Analysis error

### v3 ###
cd /compyfs/fors729/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc4/v3.LR.historical_0051/post/scripts

grep -v "OK" *status | wc -l # 19
grep "RUNNING" *status | wc -l # 10
grep "WAITING" *status | wc -l # 8 -- can't really do anything about these
grep "ERROR" *status | wc -l # 1

check_stuck_jobs | wc -l # 10 # Matches RUNNING, as expected
grep "ERROR" *status
# mpas_analysis_ts_1985-1989_climo_1985-1989.status:ERROR (1)
# We're expecting an MPAS-Analysis error

### bundles ###
cd /compyfs/fors729/zppy_weekly_legacy_3.0.0_bundles_output/zppy_main_branch_plus_unified_1_13rc4/v3.LR.historical_0051/post/scripts
grep -v "OK" *status | wc -l # 1
grep "ERROR" *status 
# ilamb_1985-1986.status:ERROR (3)
emacs ilamb_1985-1986.o751214
# FileNotFoundError: [Errno 2] No such file or directory: '/compyfs/diagnostics/ilamb_data/DATA/twsa/mrro/Dai/basins_0.5x0.5.nc'
# srun: error: n0429: task 0: Exited with exit code 1

# This is the same error we saw before.

Fixes to make

Me:

The zi-global-time-series call should probably have a if [ $? != 0 ] error check after it
Fix hard-coded LCRC paths
Figure out why "SVD did not converge in Linear Least Squares" in pcmdi_diags. I'm not sure if this is a zppy-interfaces issue or something actually inside pcmdi_diags. See /compyfs/fors729/zppy_weekly_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc4/v3.LR.historical_0051/post/scripts/pcmdi_diags_variability_modes_cpl_model_vs_obs_1985-1994.o750968. This might be a question for @zhangshixuan1987

@xylar:

Resolve Bad file list passed to ncclimo MPAS-Dev/MPAS-Analysis#1145
Resolve intermittent verifying deployed e3sm-unified version... failures occuring on various jobs from different tasks. I'm pretty sure this is a E3SM-Unified 1.13.0 issue but it could be a zppy issue. See for example cat /compyfs/fors729/zppy_weekly_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc4/v2.LR.historical_0201/post/scripts/climo_atm_monthly_180x360_aave_1980-1981.o750973 gives just one line, verifying deployed e3sm-unified version..., which is not the complete printout of a succesful Unified launch.

@xylar or @chengzhuzhang

Add mrro/Dai/basins_0.5x0.5.nc to /compyfs/diagnostics/ilamb_data/DATA/twsa/. I guess it needs to be synced to Compy?

20 replies

chengzhuzhang Apr 10, 2026
Maintainer

It does seem that there was a bug in ilamb(rubisco-sfa/ILAMB#125), this was fixed on ilamb main branch. The bug was introduced in ilamb 2.7.3 release. I think what we can do is to ask ilamb to have a new release, or revert to use an old version of ilamb? or fix the .cfg file in zppy to remove the twsa entry, (this is sloppy but fast and under our control) it looks like we not yet have a twsa equivalent variable supported in e3sm_to_cmip.

chengzhuzhang Apr 10, 2026
Maintainer

Actually, we can probably revert to use 2.7.2, going over 2.7.3 changes, it's mostly a minor release without new feature added.

forsyth2 Apr 10, 2026
Maintainer Author

@xylar ^ I updated https://e3sm.atlassian.net/wiki/spaces/DOC/pages/129732419/Packages+in+the+E3SM+Unified+conda+environment accordingly.

xylar Apr 10, 2026
Collaborator

Noted.

forsyth2 Apr 10, 2026
Maintainer Author

@zhangshixuan1987 I've made a new issue (#807) for the pcmdi_diags bug. Please respond on that page instead. Thanks!

forsyth2 · 2026-04-14T17:55:05Z

forsyth2
Apr 14, 2026
Maintainer Author

2026-04-13 Compy E3SM Unified 1.13.0rc5

What expected results are currently on Compy?

Setup steps

ls /compyfs/www/fors729/zppy_test_resources_previous
# expected_results_for_unified_1.12.0                  expected_results_until_20260331
# expected_results_until_20250429_from_unified_1.11.1
ls -lt /compyfs/www/fors729/zppy_test_resources_previous/expected_results_for_unified_1.12.0
# Last updates were Dec. 1
ls -lt /compyfs/www/zppy_test_resources/
# Good, last updates were also Dec. 1

The latest expected results are still the E3SM Unified 1.12.0 official results.

What changes are we testing?

Package	Changes since last test
e3sm_diags	Anything merged sinced E3SM Unified 1.12.0 (new this time: Unified rc5 instead of rc4)
mpas_analysis	Anything merged sinced E3SM Unified 1.12.0 (new this time: Unified rc5 instead of rc4)
zppy-interfaces	Anything merged sinced E3SM Unified 1.12.0 (new this time: Unified rc5 instead of rc4)
zppy	Anything merged sinced E3SM Unified 1.12.0 + #805 (new this time: #806 )

Testing process

Starting actual testing on 2026-04-13
Branch: zppy main branch + commits from https://github.com/E3SM-Project/zppy/pull/805/commits & #806
Environments: zppy -- dev environment; everything else -- E3SM Unified 1.13.0rc5

Set up environments

We can mostly skip this step, since all the packages will be tested using the E3SM Unified RC.

cd ~/ez/zppy-interfaces
git status
# # On branch main
# nothing to commit, working directory clean

# REMINDER: On Compy, you must NOT specify the branch, 
# otherwise it won't actually fetch the latest commits!!
git fetch upstream 
git checkout main
git reset --hard upstream/main
git log --oneline | head -n 1
# 6b0758f Bump to 0.2.1rc1 (#48)
# Good, matches https://github.com/E3SM-Project/zppy-interfaces/commits/main
source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc5_compy.sh

# Check zppy-interfaces tests
pytest tests/unit/global_time_series/test_*.py # 10 passed, 2 warnings in 12.64s
# SyntaxWarning: invalid escape sequence '\w'
pytest tests/unit/pcmdi_diags/test_*.py # 7 passed in 9.82s

zppy run

# zppy itself #################################################################
cd ~/ez/zppy
git status
# On branch test-zppy-main-plus-unified-1-13rc4
git add -A
git commit -m "Testing" --no-verify

git fetch upstream # Remember, do NOT specify the branch on Compy!
git checkout -b test-zppy-main-plus-unified-1-13rc5 upstream/main
git log --oneline | head -n 1
# 00cd1bd Only use climo_subsections if it is non-empty (#804)
# Good, matches https://github.com/E3SM-Project/zppy/commits/main

# Add commits from unmerged PRs

# 1) https://github.com/E3SM-Project/zppy/pull/805/commits
# We've already fetched all branches from upstream, 
# BUT this branch is from a fork
git fetch mkstratos # Remember, do NOT specify the branch on Compy!
git log --oneline | head -n 3
# 00cd1bd Only use climo_subsections if it is non-empty (#804)
# 0ad4774 Fix legacy 3.1.0 www paths (#801)
# 30c2cee Add mpas_analysis_subsection parameter to global_time_series (#788)
# There have been 3 new commits tohttps://github.com/E3SM-Project/zppy/pull/805/commits
git cherry-pick 5c61e1e91c19200210ed26f400df0139354899f1
git cherry-pick 901e4d34a203b7e25eb61ec80ea32ace7b13a093
# error: Commit 901e4d34a203b7e25eb61ec80ea32ace7b13a093 is a merge but no -m option was given.
# fatal: cherry-pick failed
# (I think this is ok, since this is just merging `main`)
git cherry-pick ddb4fdc1eda0078f11bbe56e8eceea84411ed7ed
git cherry-pick 447ee80af36ac27e45eadcaf657d07a927cc116f
git log --oneline | head -n 6
# 0aee8c2 Fix for start/end years that aren't 4 digits
# a611d65 Fix livvkit climo_subsections for correct default Mis-read default in fix #804, actual default is list with an empty string, so it also evaluates to "Truth-y", so it now checks against that default
# f807614 Add energy balance timeseries analysis to livvkit
# 00cd1bd Only use climo_subsections if it is non-empty (#804)
# 0ad4774 Fix legacy 3.1.0 www paths (#801)
# 30c2cee Add mpas_analysis_subsection parameter to global_time_series (#788)

# 2) https://github.com/E3SM-Project/zppy/pull/806/commits
git cherry-pick 7d5f8afd21b8ff0eb9cf1e9ef556645c65d299e7
git cherry-pick a8eee94d724979fc344e3946d088f81931aed68d
git cherry-pick 2ca288b46732ed986c736aab8d729412176a290d
git cherry-pick d74d8da7c708c7d34620b966ef447764d5bd32d7
git log --oneline | head -n 10
# 69354e0 Set standard mapping file paths
# 1ab97b8 Fix hard-coded mapping file LCRC paths
# eac10f7 Fix hard-coded non-mapping file LCRC paths
# 3bf4f33 Fix error codes
# 0aee8c2 Fix for start/end years that aren't 4 digits
# a611d65 Fix livvkit climo_subsections for correct default Mis-read default in fix #804, actual default is list with an empty string, so it also evaluates to "Truth-y", so it now checks against that default
# f807614 Add energy balance timeseries analysis to livvkit
# 00cd1bd Only use climo_subsections if it is non-empty (#804)
# 0ad4774 Fix legacy 3.1.0 www paths (#801)
# 30c2cee Add mpas_analysis_subsection parameter to global_time_series (#788)


lfs quota -hu fors729 /compyfs # Have enough space on /compyfs
# Not sure how to check space on /qfs
# (Currently in /qfs/people/fors729/ez/zppy)

compy_conda # Function to activate conda
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n test-zppy-main-plus-unified-1-13rc5
conda activate test-zppy-main-plus-unified-1-13rc5
pre-commit run --all-files # Confirm pre-commit still passes
python -m pip install .
conda list | grep python # Check Python version
# python                    3.14.4

# Run zppy unit tests
pytest tests/test_*.py # 44 passed in 0.43s

# Edit tests/integration/utils.py:

TEST_SPECIFICS: Dict[str, Any] = {
    # These are custom environment_commands for specific tasks.
    # Never set these to "", because they will print the line
    # `environment_commands = ""` for the corresponding task,
    # thus overriding the value set higher up in the cfg.
    # That is, there will be no environment set.
    # (`environment_commands = ""` only redirects to Unified
    # if specified under the [default] task)
    "diags_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc5_compy.sh",
    "mpas_analysis_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc5_compy.sh",
    "global_time_series_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc5_compy.sh",
    "livvkit_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc5_compy.sh",
    "pcmdi_diags_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc5_compy.sh",
    # This is the environment setup for other tasks.
    # Leave as "" to use the latest Unified environment.
    "environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc5_compy.sh",
    # For a complete test, run the set of latest cfgs and at least one set of legacy cfgs
    "cfgs_to_run": [
        "weekly_bundles",
        "weekly_comprehensive_v2",
        "weekly_comprehensive_v3",
        "weekly_legacy_3.1.0_bundles",
        "weekly_legacy_3.1.0_comprehensive_v2",
        "weekly_legacy_3.1.0_comprehensive_v3",
        "weekly_legacy_3.0.0_bundles",
        "weekly_legacy_3.0.0_comprehensive_v2",
        "weekly_legacy_3.0.0_comprehensive_v3",
    ],
    "tasks_to_run": [
        "e3sm_diags",
        "mpas_analysis",
        "global_time_series",
        "ilamb",
        "livvkit",
        "pcmdi_diags",
    ],
    "unique_id": "zppy_main_branch_plus_unified_1_13rc5",
}

git diff # Diff looks good
python tests/integration/utils.py
# CFG FILES HAVE BEEN GENERATED FROM TEMPLATES WITH THESE SETTINGS:
# UNIQUE_ID=zppy_main_branch_plus_unified_1_13rc5
# diags_environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc5_compy.sh
# mpas_analysis_environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc5_compy.sh
# global_time_series_environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc5_compy.sh
# livvkit_environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc5_compy.sh
# pcmdi_diags_environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc5_compy.sh
# environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc5_compy.sh
# Reminder: `environment_commands=''` => the latest E3SM Unified environment will be used

# For reference:
alias sq
# alias sq='sqa -u fors729'
alias sqa
# alias sqa='squeue -o "%8u %.7a %.4D %.9P %7i %.2t %.10r %.10M %.10l %.8Q %j" --sort=P,-t,-p'
sq
# No jobs currently queued

# Launch jobs; start with the most important one, comprehensive_v3
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_compy.cfg
zppy -c tests/integration/generated/test_weekly_bundles_compy.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_compy.cfg

zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_bundles_compy.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_comprehensive_v2_compy.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_comprehensive_v3_compy.cfg

zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_compy.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v2_compy.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v3_compy.cfg

sq | wc -l # Mon 4/13 16:33 => 207 - header row = 206 jobs
sq | wc -l # Mon 4/13 18:02 => 21 - header row = 20 jobs

# Picking up Tuesday 2026-04-14
sq | wc -l # Tue 4/14 09:17 => 1 - header row = 0 jobs

Bundles part 2

# Check on bundles status
cd /compyfs/fors729/zppy_weekly_bundles_output/zppy_main_branch_plus_unified_1_13rc5/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# bundle1.status:ERROR (1)
# e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.status:ERROR (2)
# e3sm_to_cmip_atm_monthly_180x360_aave_1987-1988-0002.status:ERROR (2)
cd /compyfs/fors729/zppy_weekly_legacy_3.1.0_bundles_output/zppy_main_branch_plus_unified_1_13rc5/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# bundle1.status:ERROR (1)
# e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.status:ERROR (2)
# e3sm_to_cmip_atm_monthly_180x360_aave_1987-1988-0002.status:ERROR (2)
cd /compyfs/fors729/zppy_weekly_legacy_3.0.0_bundles_output/zppy_main_branch_plus_unified_1_13rc5/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# bundle1.status:ERROR (1)
# e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.status:ERROR (2)
# e3sm_to_cmip_atm_monthly_180x360_aave_1987-1988-0002.status:ERROR (2)

# Now, run bundles part 2 for the working cfgs.
cd ~/ez/zppy
git status
# On branch test-zppy-main-plus-unified-1-13rc5
# Good, correct branch
# Need to re-activate conda env
compy_conda
conda activate test-zppy-main-plus-unified-1-13rc5

# None of the bundle cfgs were working though, so won't run any of these:
# zppy -c tests/integration/generated/test_weekly_bundles_compy.cfg
# zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_bundles_compy.cfg
# zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_compy.cfg

Review finished runs

### v2  ###
cd /compyfs/fors729/zppy_weekly_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc5/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.status:RUNNING 752318
# e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1980-1981_vs_1980-1981.status:WAITING 752319
# e3sm_to_cmip_atm_monthly_180x360_aave_1980-1981-0002.status:ERROR (2)
# e3sm_to_cmip_atm_monthly_180x360_aave_1982-1983-0002.status:ERROR (2)
# ilamb_1980-1981.status:WAITING 752324
# ilamb_1982-1983.status:WAITING 752325
# tc_analysis_1980-1981.status:RUNNING 752316
cd /compyfs/fors729/zppy_weekly_legacy_3.1.0_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc5/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.status:RUNNING 752352
# e3sm_to_cmip_atm_monthly_180x360_aave_1980-1981-0002.status:ERROR (2)
# e3sm_to_cmip_atm_monthly_180x360_aave_1982-1983-0002.status:ERROR (2)
# ilamb_1980-1981.status:WAITING 752358
# ilamb_1982-1983.status:WAITING 752359
cd /compyfs/fors729/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc5/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.status:RUNNING 752432
# e3sm_to_cmip_atm_monthly_180x360_aave_1980-1981-0002.status:ERROR (2)
# e3sm_to_cmip_atm_monthly_180x360_aave_1982-1983-0002.status:ERROR (2)
# ilamb_1980-1981.status:WAITING 752438
# ilamb_1982-1983.status:WAITING 752439


### v3 ###
cd /compyfs/fors729/zppy_weekly_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc5/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.status:ERROR (2)
# e3sm_to_cmip_atm_monthly_180x360_aave_1987-1988-0002.status:ERROR (2)
# e3sm_to_cmip_atm_monthly_180x360_aave_1989-1990-0002.status:ERROR (2)
# e3sm_to_cmip_atm_monthly_180x360_aave_1991-1992-0002.status:ERROR (2)
# e3sm_to_cmip_atm_monthly_180x360_aave_1993-1994-0002.status:ERROR (2)
# ilamb_1985-1988.status:WAITING 752286
# pcmdi_diags_mean_climate_model_vs_obs_1985-1994.status:WAITING 752288
# pcmdi_diags_synthetic_plots_model_vs_obs.status:WAITING 752291
# pcmdi_diags_variability_modes_atm_model_vs_obs_1985-1994.status:WAITING 752290
# pcmdi_diags_variability_modes_cpl_model_vs_obs_1985-1994.status:WAITING 752289
cd /compyfs/fors729/zppy_weekly_legacy_3.1.0_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc5/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.status:ERROR (2)
# e3sm_to_cmip_atm_monthly_180x360_aave_1987-1988-0002.status:ERROR (2)
# e3sm_to_cmip_atm_monthly_180x360_aave_1989-1990-0002.status:ERROR (2)
# e3sm_to_cmip_atm_monthly_180x360_aave_1991-1992-0002.status:ERROR (2)
# e3sm_to_cmip_atm_monthly_180x360_aave_1993-1994-0002.status:ERROR (2)
# e3sm_to_cmip_land_monthly_1985-1986-0002.status:ERROR (2)
# ilamb_1985-1988.status:WAITING 752401
# pcmdi_diags_mean_climate_model_vs_obs_1985-1994.status:WAITING 752402
# pcmdi_diags_synthetic_plots_model_vs_obs.status:WAITING 752405
# pcmdi_diags_variability_modes_atm_model_vs_obs_1985-1994.status:WAITING 752404
# pcmdi_diags_variability_modes_cpl_model_vs_obs_1985-1994.status:WAITING 752403
cd /compyfs/fors729/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc5/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.status:ERROR (2)
# e3sm_to_cmip_atm_monthly_180x360_aave_1987-1988-0002.status:ERROR (2)
# e3sm_to_cmip_land_monthly_1985-1986-0002.status:ERROR (2)
# ilamb_1985-1988.status:WAITING 752468

### bundles ###
# We didn't rerun, so there's nothing new to check.

post/scripts directory status: Many many errors -- will review in Results Analysis section below

Python tests: We'll skip this because of the post/scripts directory errors.

Results analysis

Comprehensive v3 test

Let's start with the non-legacy v3 run, which has the most up-to-date config file.

Debugging

cd /compyfs/fors729/zppy_weekly_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc5/v3.LR.historical_0051/post/scripts

grep -v "OK" *status | wc -l # 10
grep "WAITING" *status | wc -l # 5
grep "RUNNING" *status | wc -l # 0
grep "ERROR" *status | wc -l # 5

## Review WAITING jobs ########################################################
# No jobs were left in the queue (sq), so why do so many say waiting?
# Did Compy automatically remove jobs waiting on a dependency that could never be resolved?

grep "WAITING" *status
# ilamb_1985-1988.status:WAITING 752286
# pcmdi_diags_mean_climate_model_vs_obs_1985-1994.status:WAITING 752288
# pcmdi_diags_synthetic_plots_model_vs_obs.status:WAITING 752291
# pcmdi_diags_variability_modes_atm_model_vs_obs_1985-1994.status:WAITING 752290
# pcmdi_diags_variability_modes_cpl_model_vs_obs_1985-1994.status:WAITING 752289

## Review ERROR jobs ##########################################################

grep "ERROR" *status 
# e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.status:ERROR (2)
# e3sm_to_cmip_atm_monthly_180x360_aave_1987-1988-0002.status:ERROR (2)
# e3sm_to_cmip_atm_monthly_180x360_aave_1989-1990-0002.status:ERROR (2)
# e3sm_to_cmip_atm_monthly_180x360_aave_1991-1992-0002.status:ERROR (2)
# e3sm_to_cmip_atm_monthly_180x360_aave_1993-1994-0002.status:ERROR (2)
grep -n error e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.o752265 
# 7:/qfs/projects/ops/rh7/apps/E3SM/conda_envs/e3smu_1_13_0rc5/compy/spack/var/spack/environments/e3sm_unified_gnu_openmpi/.spack-env/view/bin/ncremap: line 2327: % 1: syntax error: operand expected (error token is "% 1")
# 10:/qfs/projects/ops/rh7/apps/E3SM/conda_envs/e3smu_1_13_0rc5/compy/spack/var/spack/environments/e3sm_unified_gnu_openmpi/.spack-env/view/bin/ncremap: line 2327: % 1: syntax error: operand expected (error token is "% 1")
# 167:srun: error: n0003: task 0: Exited with exit code 1

# Is the ncremap syntax error a problem in zppy, e3sm_to_cmip, or ncremap itself?
grep -n ncremap e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.bash
# 174:        ncremap -p mpi --vrt_ntp=log --vrt_xtr=mss_val --vrt_out='/compyfs/diagnostics/e3sm_to_cmip_data/maps/vrt_remap_plev19.nc' ${file} ${file}.plev

grep -ln "syntax error: operand expected" e3sm_to_cmip_atm_monthly_180x360_aave_*.o*
# e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.o752265
# e3sm_to_cmip_atm_monthly_180x360_aave_1989-1990-0002.o752267
# e3sm_to_cmip_atm_monthly_180x360_aave_1991-1992-0002.o752268
# e3sm_to_cmip_atm_monthly_180x360_aave_1993-1994-0002.o752269

# That's only 4, what's the missing file?
grep -n error e3sm_to_cmip_atm_monthly_180x360_aave_1987-1988-0002.o752266 
# 469:srun: error: n0004: task 0: Exited with exit code 1
grep -n fail e3sm_to_cmip_atm_monthly_180x360_aave_1987-1988-0002.o752266
# 90:_pickle.PicklingError: Can't pickle <class '_cmor.CMORError'>: import of module '_cmor' failed
# [Repeats many times]
# 466:_pickle.PicklingError: Can't pickle <class '_cmor.CMORError'>: import of module '_cmor' failed
# 468:2026-04-13 14:33:46.404691 [ERROR]: util.py(exit_failure:86) >> Exiting with failure code (1).
grep -ln "Can't pickle" e3sm_to_cmip_atm_monthly_180x360_aave_*.o*
# e3sm_to_cmip_atm_monthly_180x360_aave_1987-1988-0002.o752266

# Note the ncremap error occurs very early on in the output log.
tail -n 4 e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.o752265 
# 2026-04-13 14:33:45.722101 [INFO]: handler.py(_cmor_write_with_time:680) >> rlut: Writing variable to file...
#   0%|          | 0/16 [00:09<?, ?it/s]
# 2026-04-13 14:33:51.684032 [ERROR]: util.py(exit_failure:86) >> Exiting with failure code (1).
# srun: error: n0003: task 0: Exited with exit code 1

# These all fail on rlut:
tail -n 4 e3sm_to_cmip_atm_monthly_180x360_aave_1989-1990-0002.o752267
tail -n 4 e3sm_to_cmip_atm_monthly_180x360_aave_1991-1992-0002.o752268
tail -n 4 e3sm_to_cmip_atm_monthly_180x360_aave_1993-1994-0002.o752269

# This one is definitely failing on `import of module '_cmor' failed` though:
tail -n 4 e3sm_to_cmip_atm_monthly_180x360_aave_1987-1988-0002.o752266 

# The ERROR status for ALL of the relevant jobs is 2
# That is the status of the e3sm_to_cmip call.
# If the ncremap call failed, the ERROR status would have been 1.
# So, this is a problem inside e3sm_to_cmip or perhaps even inside NCO/ncremap itself.

Bugs found:

ncremap: line 2327: % 1: syntax error: operand expected (error token is "% 1") is appearing in output logs for e3sm_to_cmip. I've determined it's not a zppy bug, but rather an e3sm_to_cmip bug, or perhaps even a bug in NCO/ncremap itself. However, e3sm_to_cmip seems to be able to move on after this.
The larger e3sm_to_cmip issue is that rlut: Writing variable to file... is failing on 4 out of 5 of the erroring jobs.
The fifth failure comes from Can't pickle <class '_cmor.CMORError'>: import of module '_cmor' failed inside the e3sm_to_cmip call.

Comprehensive v2 test

Debugging

cd /compyfs/fors729/zppy_weekly_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc5/v2.LR.historical_0201/post/scripts
grep -v "OK" *status | wc -l # 7
grep "WAITING" *status | wc -l # 3
grep "RUNNING" *status | wc -l # 2
grep "ERROR" *status | wc -l # 2

## Review WAITING jobs ########################################################
# No jobs were left in the queue (sq), so why do so many say waiting?
# Did Compy automatically remove jobs waiting on a dependency that could never be resolved?

grep "WAITING" *status
# e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1980-1981_vs_1980-1981.status:WAITING 752319
# ilamb_1980-1981.status:WAITING 752324
# ilamb_1982-1983.status:WAITING 752325

## Review RUNNING jobs ########################################################
# Usually this happens if jobs hit the time limit.

grep "RUNNING" *status
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.status:RUNNING 752318
# tc_analysis_1980-1981.status:RUNNING 752316

grep -n error e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.o752318 
# 181:WARNING: There was an error initializing an OpenFabrics device.
# [Repeats many times]
# 3820:WARNING: There was an error initializing an OpenFabrics device.
# 41519:slurmstepd: error: *** JOB 752318 ON n0060 CANCELLED AT 2026-04-13T17:56:12 DUE TO TIME LIMIT ***
# 41521:slurmstepd: error: *** STEP 752318.0 ON n0060 CANCELLED AT 2026-04-13T17:56:12 DUE TO TIME LIMIT ***

grep -n error tc_analysis_1980-1981.o752316 
# 111:*** An error occurred in MPI_Init
# 115:[n0004.local:438291] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!

# The job still says RUNNING, so we didn't hit one of the expected ERROR statuses
emacs tc_analysis_1980-1981.bash

emacs tc_analysis_1980-1981.o752316
# A call to mkdir was unable to create the desired directory:
#
#   Directory: /tmp/ompi.n0004.267857
#   Error:     No space left on device

lfs quota -hu fors729 /compyfs
# I'm not even at half my space limit though

emacs tc_analysis_1980-1981.o752316
# This seems to have been the last successful operation:

# Converting mesh to connectivity format
# ..Total calculated area: 1.256637061435934e+01 sr
# ..Done
# Writing connectivity file.. Done

emacs tc_analysis_1980-1981.bash
# Presumably that's from this section:

# Generate connectivity files (.dat)
# GenerateConnectivityFile --in_mesh ${result_dir}outCSne$res.g --out_type $out_type --out_connect ${result_dir}connect_CSne${res}_v2.dat

## Review ERROR jobs ##########################################################

grep "ERROR" *status 
# e3sm_to_cmip_atm_monthly_180x360_aave_1980-1981-0002.status:ERROR (2)
# e3sm_to_cmip_atm_monthly_180x360_aave_1982-1983-0002.status:ERROR (2)

# Same error we saw before?
tail -n 4 e3sm_to_cmip_atm_monthly_180x360_aave_1980-1981-0002.o752312
tail -n 4 e3sm_to_cmip_atm_monthly_180x360_aave_1982-1983-0002.o752313
# Yes, in both cases rlut fails

New bugs found:

v2 (non-legacy) e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983 needs a longer time limit.
tc_analysis.bash template needs more ERROR status blocks
It appears it's GenerateConnectivityFile that's failing in tc_analysis.bash due to an MPI error that's due to an out-of-space error, even though I have enough space available.

Comprehensive bundles test

Debugging

cd /compyfs/fors729/zppy_weekly_bundles_output/zppy_main_branch_plus_unified_1_13rc5/v3.LR.historical_0051/post/scripts
grep -v "OK" *status | wc -l # 3
grep "WAITING" *status | wc -l # 0
grep "RUNNING" *status | wc -l # 0
grep "ERROR" *status | wc -l # 3

## Review ERROR jobs ##########################################################

grep "ERROR" *status 
# bundle1.status:ERROR (1)
# e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.status:ERROR (2)
# e3sm_to_cmip_atm_monthly_180x360_aave_1987-1988-0002.status:ERROR (2)

# Same error we saw before?
grep -A 4 "rlut: Writing variable to file..." bundle1.o752293
# 2026-04-13 14:50:52.987504 [INFO]: handler.py(_cmor_write_with_time:680) >> rlut: Writing variable to file...
#   0%|          | 0/15 [00:08<?, ?it/s]
# 2026-04-13 14:50:58.906523 [ERROR]: util.py(exit_failure:86) >> Exiting with failure code (1).
# srun: error: n0061: task 0: Exited with exit code 1
# === e3sm_to_cmip_atm_monthly_180x360_aave_1987-1988-0002.bash ===
# --
# 2026-04-13 14:51:17.635451 [INFO]: handler.py(_cmor_write_with_time:680) >> rlut: Writing variable to file...
# 2026-04-13 14:51:17.649767 [INFO]: handler.py(_cmor_write_with_time:677) >> rtmt: time span 50005.0 - 50735.0
# 2026-04-13 14:51:17.649881 [INFO]: handler.py(_cmor_write_with_time:680) >> rtmt: Writing variable to file...
#   0%|          | 0/15 [00:07<?, ?it/s]
# 2026-04-13 14:51:23.557208 [ERROR]: util.py(exit_failure:86) >> Exiting with failure code (1).

# Yes, that seems to account for both

No new bugs found.

Legacy 3.1.0 tests

Debugging

### v2  ###
cd /compyfs/fors729/zppy_weekly_legacy_3.1.0_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc5/v2.LR.historical_0201/post/scripts
grep -v "OK" *status | wc -l # 5
grep "WAITING" *status | wc -l # 2 -- can't really do anything about these
grep "RUNNING" *status | wc -l # 1
grep "ERROR" *status | wc -l # 2

grep "RUNNING" *status
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.status:RUNNING 752352
grep -n error e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.o*
# Hit time limit

grep "ERROR" *status
# e3sm_to_cmip_atm_monthly_180x360_aave_1980-1981-0002.status:ERROR (2)
# e3sm_to_cmip_atm_monthly_180x360_aave_1982-1983-0002.status:ERROR (2)
grep -A 4 "rlut: Writing variable to file..." e3sm_to_cmip_atm_monthly_180x360_aave_1980-1981-0002.o752346
grep -A 4 "rlut: Writing variable to file..." e3sm_to_cmip_atm_monthly_180x360_aave_1982-1983-0002.o752347
# Yes, it's the rlut failure in both cases.

### v3 ###
cd /compyfs/fors729/zppy_weekly_legacy_3.1.0_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc5/v3.LR.historical_0051/post/scripts

grep -v "OK" *status | wc -l # 11
grep "WAITING" *status | wc -l # 5 -- can't really do anything about these
grep "RUNNING" *status | wc -l # 0
grep "ERROR" *status | wc -l # 6

grep "ERROR" *status
# e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.status:ERROR (2)
# e3sm_to_cmip_atm_monthly_180x360_aave_1987-1988-0002.status:ERROR (2)
# e3sm_to_cmip_atm_monthly_180x360_aave_1989-1990-0002.status:ERROR (2)
# e3sm_to_cmip_atm_monthly_180x360_aave_1991-1992-0002.status:ERROR (2)
# e3sm_to_cmip_atm_monthly_180x360_aave_1993-1994-0002.status:ERROR (2)
# e3sm_to_cmip_land_monthly_1985-1986-0002.status:ERROR (2)
grep -l "rlut: Writing variable to file..." e3sm_to_cmip*.o*
# e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.o752381
# e3sm_to_cmip_atm_monthly_180x360_aave_1989-1990-0002.o752383
# e3sm_to_cmip_atm_monthly_180x360_aave_1991-1992-0002.o752384
# e3sm_to_cmip_atm_monthly_180x360_aave_1993-1994-0002.o752385
grep -A 4 "rlut: Writing variable to file..." e3sm_to_cmip*.o*
# Yes, it's that error in all cases.
grep -l "Can't pickle" e3sm_to_cmip*.o*
# e3sm_to_cmip_atm_monthly_180x360_aave_1987-1988-0002.o752382
# e3sm_to_cmip_land_monthly_1985-1986-0002.o752386

# We've seen these errors before.

### bundles ###
cd /compyfs/fors729/zppy_weekly_legacy_3.1.0_bundles_output/zppy_main_branch_plus_unified_1_13rc5/v3.LR.historical_0051/post/scripts

grep -v "OK" *status | wc -l # 3
grep "WAITING" *status | wc -l # 0 -- can't really do anything about these
grep "RUNNING" *status | wc -l # 0
grep "ERROR" *status | wc -l # 3

grep "ERROR" *status 
# bundle1.status:ERROR (1)
# e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.status:ERROR (2)
# e3sm_to_cmip_atm_monthly_180x360_aave_1987-1988-0002.status:ERROR (2)
grep -A 4 "rlut: Writing variable to file..." bundle1.o*
# Yes, it's the rlut failure in both cases.

New bugs found:

v2 (legacy 3.1.0) e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983 needs a longer time limit

Legacy 3.0.0 tests

Debugging

### v2  ###
cd /compyfs/fors729/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc5/v2.LR.historical_0201/post/scripts

grep -v "OK" *status | wc -l # 5
grep "WAITING" *status | wc -l # 2 -- can't really do anything about these
grep "RUNNING" *status | wc -l # 1
grep "ERROR" *status | wc -l # 2

grep "RUNNING" *status
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.status:RUNNING 752432
tail e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.o*
# Hit time limit

grep "ERROR" *status
# e3sm_to_cmip_atm_monthly_180x360_aave_1980-1981-0002.status:ERROR (2)
# e3sm_to_cmip_atm_monthly_180x360_aave_1982-1983-0002.status:ERROR (2)
grep -l "rlut: Writing variable to file..." e3sm_to_cmip*.o*
# e3sm_to_cmip_atm_monthly_180x360_aave_1980-1981-0002.o752426
# e3sm_to_cmip_atm_monthly_180x360_aave_1982-1983-0002.o752427
grep -A 4 "rlut: Writing variable to file..." e3sm_to_cmip*.o*
# Yes, it's that error in both cases.

# Same errors as before

### v3 ###
cd /compyfs/fors729/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc5/v3.LR.historical_0051/post/scripts

grep -v "OK" *status | wc -l # 4
grep "WAITING" *status | wc -l # 1 -- can't really do anything about these
grep "RUNNING" *status | wc -l # 0
grep "ERROR" *status | wc -l # 3

grep "ERROR" *status
# e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.status:ERROR (2)
# e3sm_to_cmip_atm_monthly_180x360_aave_1987-1988-0002.status:ERROR (2)
# e3sm_to_cmip_land_monthly_1985-1986-0002.status:ERROR (2)
grep -l "rlut: Writing variable to file..." e3sm_to_cmip*.o*
# e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.o752458
# e3sm_to_cmip_atm_monthly_180x360_aave_1987-1988-0002.o752459
grep -A 4 "rlut: Writing variable to file..." e3sm_to_cmip*.o*
# Yes, it's that error in both cases.

### bundles ###
cd /compyfs/fors729/zppy_weekly_legacy_3.0.0_bundles_output/zppy_main_branch_plus_unified_1_13rc5/v3.LR.historical_0051/post/scripts
grep -v "OK" *status | wc -l # 3
grep "WAITING" *status | wc -l # 0 -- can't really do anything about these
grep "RUNNING" *status | wc -l # 0
grep "ERROR" *status | wc -l # 3

grep "ERROR" *status 
bundle1.status:ERROR (1)
# e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.status:ERROR (2)
# e3sm_to_cmip_atm_monthly_180x360_aave_1987-1988-0002.status:ERROR (2)
grep -l "rlut: Writing variable to file..." bundle1.o*
# bundle1.o752407
grep -A 4 "rlut: Writing variable to file..." bundle1.o*
# Yes, it's that error in both cases

New bugs found:

v2 (legacy 3.0.0) e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983 needs a longer time limit

Fixes to make

@chengzhuzhang (and/or @TonyB9000, @czender):

ncremap: line 2327: % 1: syntax error: operand expected (error token is "% 1") is appearing in output logs for e3sm_to_cmip. I've determined it's not a zppy bug, but rather an e3sm_to_cmip bug, or perhaps even a bug in NCO/ncremap itself. However, e3sm_to_cmip seems to be able to move on after this.
The larger e3sm_to_cmip issue is that rlut: Writing variable to file... is failing on 4 out of 5 of the erroring jobs.
The fifth failure comes from Can't pickle <class '_cmor.CMORError'>: import of module '_cmor' failed inside the e3sm_to_cmip call.

@chengzhuzhang

It appears it's GenerateConnectivityFile that's failing in tc_analysis.bash due to an MPI error that's due to an out-of-space error, even though I have enough space available.

For me:

It might be useful to add a log line of what package version we're using in each bash script. Xylar noted that we should be using pixi list instead of conda list, i.e. pixi list package
v2 (non-legacy, legacy 3.1.0, legacy 3.0.0) e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983 needs a longer time limit.
tc_analysis.bash template needs more ERROR status blocks

1 reply

forsyth2 Apr 14, 2026
Maintainer Author

I've added 3 commits to https://github.com/E3SM-Project/zppy/pull/806/commits to address those last 3 action items.

TonyB9000 · 2026-04-14T19:13:25Z

TonyB9000
Apr 14, 2026
Collaborator

@forsyth2 Regarding "e3sm_to_cmip ... rlut: Writing variable to file... is failing", I note that there are occasions when a variable (e.g. rlut) exists in multiple tables (e.g. Amon.rlut, day.rlut), the handler returned can be less than predictable. Just somewhere to look.

5 replies

chengzhuzhang Apr 14, 2026
Maintainer

@forsyth2 thank you for the results! There are some changes in e3sm_to_cmip/zppy that can potentially cause the problems listed here. Though I don't have access to Compy and won't examine the log files, I will trouble shooting when a NERSC test is ready...Appreciate it!

forsyth2 Apr 14, 2026
Maintainer Author

@chengzhuzhang I haven't started a NERSC test yet. I wasn't planning to do so until the Compy test was working. It takes so long to get nodes on Perlmutter I determined it would make sense to be almost sure we would have a working version before running there.

Frankly, I think it'd be faster for me to simply Globus transfer some of the log files to Perlmutter so you can look at them there. Let me try that.

forsyth2 Apr 14, 2026
Maintainer Author

Actually I'm doing the performance profiling (for zstash) now and I don't want to mess up anything with the Globus transfers, so I'll do that once that's done.

chengzhuzhang Apr 14, 2026
Maintainer

Thank you, Ryan. If you can have a cope of the log files on PM, that would be great!

forsyth2 Apr 15, 2026
Maintainer Author

Copying Compy log files

Collect relevant files

On Compy:

cd /compyfs/fors729/
mkdir e3sm_unified_1.13.0rc5_test_logs
cd e3sm_unified_1.13.0rc5_test_logs
mkdir provenance
mkdir rlut_failures
mkdir cmor_pickle_failures
mkdir mpi_failures

# non-legacy v3 ###############################################################
cd /compyfs/fors729/zppy_weekly_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc5/v3.LR.historical_0051/post/scripts
cp provenance.20260413_213105_290068.cfg /compyfs/fors729/e3sm_unified_1.13.0rc5_test_logs/provenance/non_legacy_v3_provenance.cfg

cp e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.o752265 /compyfs/fors729/e3sm_unified_1.13.0rc5_test_logs/rlut_failures/non_legacy_v3_e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.o752265
cp e3sm_to_cmip_atm_monthly_180x360_aave_1989-1990-0002.o752267 /compyfs/fors729/e3sm_unified_1.13.0rc5_test_logs/rlut_failures/non_legacy_v3_e3sm_to_cmip_atm_monthly_180x360_aave_1989-1990-0002.o752267
cp e3sm_to_cmip_atm_monthly_180x360_aave_1991-1992-0002.o752268 /compyfs/fors729/e3sm_unified_1.13.0rc5_test_logs/rlut_failures/non_legacy_v3_e3sm_to_cmip_atm_monthly_180x360_aave_1991-1992-0002.o752268
cp e3sm_to_cmip_atm_monthly_180x360_aave_1993-1994-0002.o752269 /compyfs/fors729/e3sm_unified_1.13.0rc5_test_logs/rlut_failures/non_legacy_v3_e3sm_to_cmip_atm_monthly_180x360_aave_1993-1994-0002.o752269 

cp e3sm_to_cmip_atm_monthly_180x360_aave_1987-1988-0002.o752266 /compyfs/fors729/e3sm_unified_1.13.0rc5_test_logs/cmor_pickle_failures/non_legacy_v3_e3sm_to_cmip_atm_monthly_180x360_aave_1987-1988-0002.o752266

# non-legacy v2 ###############################################################
cd /compyfs/fors729/zppy_weekly_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc5/v2.LR.historical_0201/post/scripts
cp provenance.20260413_213131_478801.cfg /compyfs/fors729/e3sm_unified_1.13.0rc5_test_logs/provenance/non_legacy_v2_provenance.cfg

cp tc_analysis_1980-1981.o752316  /compyfs/fors729/e3sm_unified_1.13.0rc5_test_logs/mpi_failures/non_legacy_v2_tc_analysis_1980-1981.o752316

cp e3sm_to_cmip_atm_monthly_180x360_aave_1980-1981-0002.o752312 /compyfs/fors729/e3sm_unified_1.13.0rc5_test_logs/rlut_failures/non_legacy_v2_e3sm_to_cmip_atm_monthly_180x360_aave_1980-1981-0002.o752312
cp e3sm_to_cmip_atm_monthly_180x360_aave_1982-1983-0002.o752313 /compyfs/fors729/e3sm_unified_1.13.0rc5_test_logs/rlut_failures/non_legacy_v2_e3sm_to_cmip_atm_monthly_180x360_aave_1982-1983-0002.o752313

# non-legacy bundles ##########################################################
cd /compyfs/fors729/zppy_weekly_bundles_output/zppy_main_branch_plus_unified_1_13rc5/v3.LR.historical_0051/post/scripts
cp provenance.20260413_213125_142912.cfg /compyfs/fors729/e3sm_unified_1.13.0rc5_test_logs/provenance/non_legacy_bundles_provenance.cfg

cp bundle1.o752293 /compyfs/fors729/e3sm_unified_1.13.0rc5_test_logs/rlut_failures/non_legacy_bundles_bundle1.o752293

# legacy 3.1.0 v3 #############################################################
cd /compyfs/fors729/zppy_weekly_legacy_3.1.0_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc5/v3.LR.historical_0051/post/scripts
cp provenance.20260413_213229_246026.cfg /compyfs/fors729/e3sm_unified_1.13.0rc5_test_logs/provenance/legacy3.1.0_v3_provenance.cfg

cp e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.o752381 /compyfs/fors729/e3sm_unified_1.13.0rc5_test_logs/rlut_failures/legacy3.1.0_v3_e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.o752381
cp e3sm_to_cmip_atm_monthly_180x360_aave_1989-1990-0002.o752383 /compyfs/fors729/e3sm_unified_1.13.0rc5_test_logs/rlut_failures/legacy3.1.0_v3_e3sm_to_cmip_atm_monthly_180x360_aave_1989-1990-0002.o752383
cp e3sm_to_cmip_atm_monthly_180x360_aave_1991-1992-0002.o752384 /compyfs/fors729/e3sm_unified_1.13.0rc5_test_logs/rlut_failures/legacy3.1.0_v3_e3sm_to_cmip_atm_monthly_180x360_aave_1991-1992-0002.o752384
cp e3sm_to_cmip_atm_monthly_180x360_aave_1993-1994-0002.o752385 /compyfs/fors729/e3sm_unified_1.13.0rc5_test_logs/rlut_failures/legacy3.1.0_v3_e3sm_to_cmip_atm_monthly_180x360_aave_1993-1994-0002.o752385

cp e3sm_to_cmip_atm_monthly_180x360_aave_1987-1988-0002.o752382 /compyfs/fors729/e3sm_unified_1.13.0rc5_test_logs/cmor_pickle_failures/legacy3.1.0_v3_e3sm_to_cmip_atm_monthly_180x360_aave_1987-1988-0002.o752382
cp e3sm_to_cmip_land_monthly_1985-1986-0002.o752386 /compyfs/fors729/e3sm_unified_1.13.0rc5_test_logs/cmor_pickle_failures/legacy3.1.0_v3_e3sm_to_cmip_land_monthly_1985-1986-0002.o752386

# legacy 3.1.0 v2 #############################################################
cd /compyfs/fors729/zppy_weekly_legacy_3.1.0_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc5/v2.LR.historical_0201/post/scripts
cp provenance.20260413_213204_112097.cfg /compyfs/fors729/e3sm_unified_1.13.0rc5_test_logs/provenance/legacy3.1.0_v2_provenance.cfg

cp e3sm_to_cmip_atm_monthly_180x360_aave_1980-1981-0002.o752346 /compyfs/fors729/e3sm_unified_1.13.0rc5_test_logs/rlut_failures/legacy3.1.0_v2_e3sm_to_cmip_atm_monthly_180x360_aave_1980-1981-0002.o752346
cp e3sm_to_cmip_atm_monthly_180x360_aave_1982-1983-0002.o752347 /compyfs/fors729/e3sm_unified_1.13.0rc5_test_logs/rlut_failures/legacy3.1.0_v2_e3sm_to_cmip_atm_monthly_180x360_aave_1982-1983-0002.o752347

# legacy 3.1.0 bundles ########################################################
cd /compyfs/fors729/zppy_weekly_legacy_3.1.0_bundles_output/zppy_main_branch_plus_unified_1_13rc5/v3.LR.historical_0051/post/scripts
cp provenance.20260413_213148_352497.cfg /compyfs/fors729/e3sm_unified_1.13.0rc5_test_logs/provenance/legacy3.1.0_bundles_provenance.cfg

cp bundle1.o752327 /compyfs/fors729/e3sm_unified_1.13.0rc5_test_logs/rlut_failures/legacy3.1.0_bundles_bundle1.o752327

# legacy 3.0.0 v3 #############################################################
cd /compyfs/fors729/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc5/v3.LR.historical_0051/post/scripts
cp provenance.20260413_213333_927541.cfg /compyfs/fors729/e3sm_unified_1.13.0rc5_test_logs/provenance/legacy3.0.0_v3_provenance.cfg

cp e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.o752458 /compyfs/fors729/e3sm_unified_1.13.0rc5_test_logs/rlut_failures/legacy3.0.0_v3_e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.o752458 
cp e3sm_to_cmip_atm_monthly_180x360_aave_1987-1988-0002.o752459 /compyfs/fors729/e3sm_unified_1.13.0rc5_test_logs/rlut_failures/legacy3.0.0_v3_e3sm_to_cmip_atm_monthly_180x360_aave_1987-1988-0002.o752459

# Looks like we missed one error earlier
grep -l "Can't pickle " e3sm_to_cmip*.o*
# e3sm_to_cmip_land_monthly_1985-1986-0002.o752460
cp e3sm_to_cmip_land_monthly_1985-1986-0002.o752460 /compyfs/fors729/e3sm_unified_1.13.0rc5_test_logs/cmor_pickle_failures/legacy3.0.0_v3_e3sm_to_cmip_land_monthly_1985-1986-0002.o752460

# legacy 3.0.0 v2 #############################################################
cd /compyfs/fors729/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc5/v2.LR.historical_0201/post/scripts
cp provenance.20260413_213312_125725.cfg /compyfs/fors729/e3sm_unified_1.13.0rc5_test_logs/provenance/legacy3.0.0_v2_provenance.cfg

cp e3sm_to_cmip_atm_monthly_180x360_aave_1980-1981-0002.o752426 /compyfs/fors729/e3sm_unified_1.13.0rc5_test_logs/rlut_failures/legacy3.0.0_v2_e3sm_to_cmip_atm_monthly_180x360_aave_1980-1981-0002.o752426
cp e3sm_to_cmip_atm_monthly_180x360_aave_1982-1983-0002.o752427 /compyfs/fors729/e3sm_unified_1.13.0rc5_test_logs/rlut_failures/legacy3.0.0_v2_3sm_to_cmip_atm_monthly_180x360_aave_1982-1983-0002.o752427

# legacy 3.0.0 bundles ########################################################
cd /compyfs/fors729/zppy_weekly_legacy_3.0.0_bundles_output/zppy_main_branch_plus_unified_1_13rc5/v3.LR.historical_0051/post/scripts
cp provenance.20260413_213304_416982.cfg /compyfs/fors729/e3sm_unified_1.13.0rc5_test_logs/provenance/legacy3.0.0_bundles_provenance.cfg

cp bundle1.o752407 /compyfs/fors729/e3sm_unified_1.13.0rc5_test_logs/rlut_failures/legacy3.0.0_bundles_bundle1.o752407

Now we have a directory with everything relevant in it:

cd /compyfs/fors729/e3sm_unified_1.13.0rc5_test_logs/
ls rlut_failures/
# legacy3.0.0_bundles_bundle1.o752407                                          legacy3.1.0_v3_e3sm_to_cmip_atm_monthly_180x360_aave_1991-1992-0002.o752384
# legacy3.0.0_v2_3sm_to_cmip_atm_monthly_180x360_aave_1982-1983-0002.o752427   legacy3.1.0_v3_e3sm_to_cmip_atm_monthly_180x360_aave_1993-1994-0002.o752385
# legacy3.0.0_v2_e3sm_to_cmip_atm_monthly_180x360_aave_1980-1981-0002.o752426  non_legacy_bundles_bundle1.o752293
# legacy3.0.0_v3_e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.o752458  non_legacy_v2_e3sm_to_cmip_atm_monthly_180x360_aave_1980-1981-0002.o752312
# legacy3.0.0_v3_e3sm_to_cmip_atm_monthly_180x360_aave_1987-1988-0002.o752459  non_legacy_v2_e3sm_to_cmip_atm_monthly_180x360_aave_1982-1983-0002.o752313
# legacy3.1.0_v2_e3sm_to_cmip_atm_monthly_180x360_aave_1980-1981-0002.o752346  non_legacy_v3_e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.o752265
# legacy3.1.0_v2_e3sm_to_cmip_atm_monthly_180x360_aave_1982-1983-0002.o752347  non_legacy_v3_e3sm_to_cmip_atm_monthly_180x360_aave_1989-1990-0002.o752267
# legacy3.1.0_v3_e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.o752381  non_legacy_v3_e3sm_to_cmip_atm_monthly_180x360_aave_1991-1992-0002.o752268
# legacy3.1.0_v3_e3sm_to_cmip_atm_monthly_180x360_aave_1989-1990-0002.o752383  non_legacy_v3_e3sm_to_cmip_atm_monthly_180x360_aave_1993-1994-0002.o752269
ls cmor_pickle_failures/
# legacy3.0.0_v3_e3sm_to_cmip_land_monthly_1985-1986-0002.o752460              legacy3.1.0_v3_e3sm_to_cmip_land_monthly_1985-1986-0002.o752386
# legacy3.1.0_v3_e3sm_to_cmip_atm_monthly_180x360_aave_1987-1988-0002.o752382  non_legacy_v3_e3sm_to_cmip_atm_monthly_180x360_aave_1987-1988-0002.o752266
ls mpi_failures/
# non_legacy_v2_tc_analysis_1980-1981.o752316
ls provenance/
# legacy3.0.0_bundles_provenance.cfg  legacy3.0.0_v3_provenance.cfg       legacy3.1.0_v2_provenance.cfg  non_legacy_bundles_provenance.cfg  non_legacy_v3_provenance.cfg
# legacy3.0.0_v2_provenance.cfg       legacy3.1.0_bundles_provenance.cfg  legacy3.1.0_v3_provenance.cfg  non_legacy_v2_provenance.cfg
du -sh . # 1.1M

Now, Globus transfer this directory

@chengzhuzhang I've copied output files, sorted on 3 axes (failure type, non_legacy/3.1.0/3.0.0, and v2/v3/bundles). I've also included provenance cfg files.

Perlmutter:

ls /global/homes/f/forsyth/copied_from_compy/e3sm_unified_1.13.0rc5_test_logs/
# cmor_pickle_failures  mpi_failures  provenance  rlut_failures

LCRC:

ls /home/ac.forsyth2/copied_from_compy/e3sm_unified_1.13.0rc5_test_logs
# cmor_pickle_failures  mpi_failures  provenance  rlut_failures

TonyB9000 · 2026-04-15T18:08:48Z

TonyB9000
Apr 15, 2026
Collaborator

@forsyth2 @chengzhuzhang I am perusing Ryan's logs
(/home/ac.forsyth2/copied_from_compy/e3sm_unified_1.13.0rc5_test_logs/rlut_failures)
A few points:

(1) There are 4 variables (huss, pr, rlut, tas) that have multiple frequencies, and appear in multiple tables.

    huss:
        Amon.huss   QREFHT
        day.huss    QREFHT

    pr:
        3hr.pr      (no table specified, falls under "sub-monthly" with "day"?)
        Amon.pr     PRECC, PRECL
        day.pr      PRECT

    rlut:
        Amon.rlut   FSNTOA, FSNT, FLNT
        day.rlut    FLUT

    tas:
        Amon.tas    TREFHT
        day.tas     TREFHT

Most concerning to me is that in the e3sm_to_cmip "handlers.py", the rlut specifications (yaml) read

    - name: rlut
      units: W m-2
      raw_variables: [FLUT]
      table: CMIP6_day.json
      unit_conversion: null
      formula: FSNTOA - FSNT + FLNT
      positive: up
      levels: null
    - name: rlut
      units: W m-2
      raw_variables: [FSNTOA, FSNT, FLNT]
      table: CMIP6_Amon.json
      unit_conversion: null
      formula: FSNTOA - FSNT + FLNT
      positive: up
      levels: null

I don't see how that first handler can be correct.

I worry that the mechanism in e3sm_to_cmip for selecting the appropriate handler (by variable name) is broken, returning the first one encountered when provided multiple variables to process. That is why, in "dsm_generate_CMIP6", I process variable-at-a-time, and call "e3sm_to_cmip --info" on each variable, receiving (in such cases) multiple handlers and then selecting the handler appropriate to the frequency.

(2) There is no variable "wa". I think that was supposed to be the Atmos 3D variable "wap".

(3) I see no errors in these files related to "rlut". Note that the "bundles" do not seem to include rlut at all.

9 replies

forsyth2 Apr 15, 2026
Maintainer Author

leaves 14 jobs launched from the other 7 config files unaccounted for.

See here. The issue is that wa is actually hard-coded outside the test cfg, inside e3sm_to_cmip.py. Thanks Jill!

forsyth2 Apr 15, 2026
Maintainer Author

That is, I think that ncremap call in the bash script is a red herring.

Actually, I need to qualify this. I noticed I also included:

Note the ncremap error occurs very early on in the output log.

So, yes, it does seem like it's that ncremap line in the bash script that is producing a syntax error, BUT it doesn't appear to be crashing the whole program. The code keeps going. I'm wondering if that syntax error is causing downstream effects though. That would explain complications in interpreting the actual failures. I.e., makes it look like it failed on rlut when the true problem is we weren't getting accurate results from ncremap.

chengzhuzhang Apr 15, 2026
Maintainer

@forsyth2 I think you can verify on compy, if these U,V,T,Q,RELHUM,OMEGA,Z3 are being vertically regrided then cmorized:
e.g. under
/compyfs/fors729/zppy_weekly_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc5/v2.LR.historical_0201/post/atm

forsyth2 Apr 15, 2026
Maintainer Author

ls -R /compyfs/fors729/zppy_weekly_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc5/v2.LR.historical_0201/post/atm > non_legacy_v2_ls_results.txt

# Has results:
grep -n "^U_" non_legacy_v2_ls_results.txt 
# 533:U_198001_198112.nc
# 534:U_198201_198312.nc
# 558:U_198001_198112.nc
# 582:U_198201_198312.nc
# First 2 are in /compyfs/fors729/zppy_weekly_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc5/v2.LR.historical_0201/post/atm/180x360_aave/ts/monthly/2yr
# 3rd is in /compyfs/fors729/zppy_weekly_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc5/v2.LR.historical_0201/post/atm/180x360_aave/ts/monthly/2yr/1980_1981
# 4th is in /compyfs/fors729/zppy_weekly_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc5/v2.LR.historical_0201/post/atm/180x360_aave/ts/monthly/2yr/1982_1983


# No results: 
grep "^V_" non_legacy_v2_ls_results.txt
grep "^T_" non_legacy_v2_ls_results.txt
grep "^Q_" non_legacy_v2_ls_results.txt
grep "^RELHUM_" non_legacy_v2_ls_results.txt
grep "^OMEGA_" non_legacy_v2_ls_results.txt 
grep "^Z3_" non_legacy_v2_ls_results.txt

TonyB9000 Apr 15, 2026
Collaborator

@forsyth2 FWIW, I never ran into this problem, as I always supply explicit variable names on my e3sm_to_cmip command lines.

czender · 2026-04-15T20:56:47Z

czender
Apr 15, 2026

@chengzhuzhang Thank you folks for pointing out the ncremap problem to me. I have not yet had the time to look into the ncremap error due to having too many meetings yesterday and today. I will fix it with (yet another) NCO release ASAP, probably tomorrow or Friday.

1 reply

forsyth2 Apr 15, 2026
Maintainer Author

Thank you @czender!

chengzhuzhang · 2026-04-15T23:37:31Z

chengzhuzhang
Apr 15, 2026
Maintainer

@forsyth2 e3sm_to_cmip has a fix on main branch (E3SM-Project/e3sm_to_cmip#344), which is more tolerant to missing data and supposedly can get the downstream packages (ilamb, pmp) to run. I think you have a workflow to test main branch? If it is not too much trouble, could you rerun to try continue this test ( just one related failed test is sufficient)? If this is not a good idea, we can make a tag or rc for next round of test.

3 replies

forsyth2 Apr 15, 2026
Maintainer Author

more tolerant to missing data

@chengzhuzhang As in I could run it today even though the ncremap issue hasn't been resolved yet?

I think you have a workflow to test main branch?

Yes, I can set environment_commands to a dev environment for e3sm_to_cmip. But now that I'm looking at the test setup, it looks like we haven't been doing that, so I need to update the testing code to funnel that environment into the various test cfgs. (Really, I suppose we should be testing with a dev environment for e3sm_to_cmip during the development period, just as we do for e3sm_diags, zppy-interfaces, and recently mpas_analysis).

forsyth2 Apr 16, 2026
Maintainer Author

@chengzhuzhang I tried to launch a limited test run (see here) but ran into issues. I'll have to look into it more tomorrow.

Unless @xylar needs the next e3sm_to_cmip RC ASAP, I'd recommend waiting to make that until I can debug these issues and Charlie can look into the ncremap issue.

xylar Apr 16, 2026
Collaborator

@forsyth2, @chengzhuzhang, do what you need to do to debug. The next Unified RC will mostly be for you because I'm not aware of any other outstanding issues besides what you both are already aware of.

forsyth2 · 2026-04-16T00:04:16Z

forsyth2
Apr 16, 2026
Maintainer Author

Also, just a general note to all: I was trying to keep one thread per test (i.e., 2026-03-31 Chrysalis test, 2026-04-08 Compy test), so that comments would be associated with a particular test run.

When new threads are started (as I'm doing with this comment), it can clutter the space a bit. Not a huge deal, but it makes it harder to know what comments go with which test. An important thing to note is that on GitHub Discussions, the top-level comments are not in a set-in-stone order. They can be re-sequenced by "Oldest", "Newest", or "Top", which could cause these unassociated comments to end up isolated from the main thread they're referring to.

0 replies

forsyth2 · 2026-04-16T01:22:40Z

forsyth2
Apr 16, 2026
Maintainer Author

2026-04-15 Compy E3SM Unified 1.13.0rc5 + fixes, limited test

We're only going to run the non-legacy v3/v2 tests here, just to see if we're on the right track with our bug fixes so far. (Skipping all legacy tests, and all bundles tests).

This will also be using a dev environment for not only zppy, but also e3sm_to_cmip.

What expected results are currently on Compy?

The latest expected results are still the E3SM Unified 1.12.0 official results.

What changes are we testing?

Package	Changes since 2026-04-13 Compy E3SM Unified 1.13.0rc5 test
e3sm_to_cmip	#344
e3sm_diags	N/A
mpas_analysis	N/A
zppy-interfaces	N/A
zppy	post-rc5 labeled commits on #806

Testing process

Starting actual testing on 2026-04-15
Branch: zppy main branch + commits from https://github.com/E3SM-Project/zppy/pull/805/commits & #806
Environments: zppy, e3sm_to_cmip -- dev environment; everything else -- E3SM Unified 1.13.0rc5

Set up environments

We can mostly skip this step, since almost all the packages will be tested using the E3SM Unified RC.

cd ~/ez/e3sm_to_cmip/
git status
# # On branch master
# nothing to commit, working directory clean

# REMINDER: On Compy, you must NOT specify the branch, 
# otherwise it won't actually fetch the latest commits!!
git fetch upstream
git reset --hard upstream/master
git log --oneline | head -n 2
# 7a78612 Fix bug: parallel mode incorrectly stops on handler failure when --on-var-failure=ignore (#344)
# 2efb904 bump to 1.14.0rc2 (#343)
# Good, matches https://github.com/E3SM-Project/e3sm_to_cmip/commits/master/
compy_conda # Bash function to activate conda
rm -rf build
conda clean --all --y
conda env create -f conda-env/dev.yml -n e3sm_to_cmip_20260415
conda activate e3sm_to_cmip_20260415
python -m pip install .

# zppy-interfaces
# We've already tested zppy-interfaces using Unified rc5

zppy run

# zppy itself #################################################################
cd ~/ez/zppy
git status
# On branch test-zppy-main-plus-unified-1-13rc5
git add -A
git commit -m "Testing" --no-verify

git fetch upstream # Remember, do NOT specify the branch on Compy!
git checkout -b test-zppy-main-plus-unified-1-13rc5-with-fixes upstream/main
git log --oneline | head -n 1
# 00cd1bd Only use climo_subsections if it is non-empty (#804)
# Good, matches https://github.com/E3SM-Project/zppy/commits/main

# Add commits from unmerged PRs:

# 1) https://github.com/E3SM-Project/zppy/pull/805/commits
# No new commits, so we don't need this:
# git fetch mkstratos
git cherry-pick 5c61e1e91c19200210ed26f400df0139354899f1
# git cherry-pick 901e4d34a203b7e25eb61ec80ea32ace7b13a093 # Skip because it's a merge commit
git cherry-pick ddb4fdc1eda0078f11bbe56e8eceea84411ed7ed
git cherry-pick 447ee80af36ac27e45eadcaf657d07a927cc116f
git log --oneline | head -n 4
# d0ab469 Fix for start/end years that aren't 4 digits
# 6621a9f Fix livvkit climo_subsections for correct default Mis-read default in fix #804, actual default is list with an empty string, so it also evaluates to "Truth-y", so it now checks against that default
# de6f988 Add energy balance timeseries analysis to livvkit
# 00cd1bd Only use climo_subsections if it is non-empty (#804)

# 2) https://github.com/E3SM-Project/zppy/pull/806/commits
# There are new commits (the ones labeled post-rc5 fix),
# but we've already fetched upstream

# 4 original:
git cherry-pick 7d5f8afd21b8ff0eb9cf1e9ef556645c65d299e7
git cherry-pick a8eee94d724979fc344e3946d088f81931aed68d
git cherry-pick 2ca288b46732ed986c736aab8d729412176a290d
git cherry-pick d74d8da7c708c7d34620b966ef447764d5bd32d7

# 5 new:
git cherry-pick 1e5e622ca6079a3d0ee44bec8aac1bfbb47f851b
git cherry-pick 6549125f5e2cd50d768d8b52d162a96b387ce039
git cherry-pick 14e25c02087656b1d43966cca2d9da77fd537084
git cherry-pick f85ee7645a17b47e9c6c98b3bcd5cfd28c32857a
git cherry-pick fd6398cfb05667a26c495e83d7bc37f731cfe31f

git log --oneline | head -n 13
# 2d0b453 post-rc5 fix: Add e3sm_to_cmip_environment_commands
# a8f5794 post-rc5 fix: Replace wa with wap
# 361ab90 post-rc5 fix: increase diags walltime
# 2549d12 post-rc5 fix: add more status logging to tc_analysis
# 752a997 post-rc5 fix: print package versions
# 5efdc0e Set standard mapping file paths
# 2f154ee Fix hard-coded mapping file LCRC paths
# 685a8fe Fix hard-coded non-mapping file LCRC paths
# e755034 Fix error codes
# d0ab469 Fix for start/end years that aren't 4 digits
# 6621a9f Fix livvkit climo_subsections for correct default Mis-read default in fix #804, actual default is list with an empty string, so it also evaluates to "Truth-y", so it now checks against that default
# de6f988 Add energy balance timeseries analysis to livvkit
# 00cd1bd Only use climo_subsections if it is non-empty (#804)


lfs quota -hu fors729 /compyfs # Have enough space on /compyfs
# Not sure how to check space on /qfs
# (Currently in /qfs/people/fors729/ez/zppy)
# Cancelled, took too long

compy_conda # Bash function to activate conda
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n test-zppy-main-plus-unified-1-13rc5-with-fixes
conda activate test-zppy-main-plus-unified-1-13rc5-with-fixes
pre-commit run --all-files # Confirm pre-commit still passes
python -m pip install .
conda list | grep python # Check Python version
# python                    3.14.4

# Run zppy unit tests
pytest tests/test_*.py # 44 passed in 0.47s

# Edit tests/integration/utils.py:

TEST_SPECIFICS: Dict[str, Any] = {
    # These are custom environment_commands for specific tasks.
    # Never set these to "", because they will print the line
    # `environment_commands = ""` for the corresponding task,
    # thus overriding the value set higher up in the cfg.
    # That is, there will be no environment set.
    # (`environment_commands = ""` only redirects to Unified
    # if specified under the [default] task)
    "e3sm_to_cmip_environment_commands": "source /qfs/people/fors729/miniforge3/etc/profile.d/conda.sh; conda activate e3sm_to_cmip_20260415",
    "diags_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc5_compy.sh",
    "mpas_analysis_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc5_compy.sh",
    "global_time_series_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc5_compy.sh",
    "livvkit_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc5_compy.sh",
    "pcmdi_diags_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc5_compy.sh",
    # This is the environment setup for other tasks.
    # Leave as "" to use the latest Unified environment.
    "environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc5_compy.sh",
    # For a complete test, run the set of latest cfgs and at least one set of legacy cfgs
    "cfgs_to_run": [
        # "weekly_bundles",
        "weekly_comprehensive_v2",
        "weekly_comprehensive_v3",
        # "weekly_legacy_3.1.0_bundles",
        # "weekly_legacy_3.1.0_comprehensive_v2",
        # "weekly_legacy_3.1.0_comprehensive_v3",
        # "weekly_legacy_3.0.0_bundles",
        # "weekly_legacy_3.0.0_comprehensive_v2",
        # "weekly_legacy_3.0.0_comprehensive_v3",
    ],
    "tasks_to_run": [
        "e3sm_diags",
        "mpas_analysis",
        "global_time_series",
        "ilamb",
        "livvkit",
        "pcmdi_diags",
    ],
    "unique_id": "zppy_main_branch_plus_unified_1_13rc5_with_fixes",
}

git diff # Diff looks good
python tests/integration/utils.py
# CFG FILES HAVE BEEN GENERATED FROM TEMPLATES WITH THESE SETTINGS:
# UNIQUE_ID=zppy_main_branch_plus_unified_1_13rc5_with_fixes
# e3sm_to_cmip_environment_commands=source /qfs/people/fors729/miniforge3/etc/profile.d/conda.sh; conda activate e3sm_to_cmip_20260415
# diags_environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc5_compy.sh
# mpas_analysis_environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc5_compy.sh
# global_time_series_environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc5_compy.sh
# livvkit_environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc5_compy.sh
# pcmdi_diags_environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc5_compy.sh
# environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc5_compy.sh
# Reminder: `environment_commands=''` => the latest E3SM Unified environment will be used

# For reference:
alias sq
# alias sq='sqa -u fors729'
alias sqa
# alias sqa='squeue -o "%8u %.7a %.4D %.9P %7i %.2t %.10r %.10M %.10l %.8Q %j" --sort=P,-t,-p'
sq
# No jobs currently queued

# Launch jobs; remember we're doing a limited run:
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_compy.cfg
# Hangs after launching 1 job

# On a new tab:
compy_conda
conda activate test-zppy-main-plus-unified-1-13rc5-with-fixes
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_compy.cfg
# Traceback (most recent call last):
#   File "/qfs/people/fors729/miniforge3/envs/test-zppy-main-plus-unified-1-13rc5-with-fixes/bin/zppy", line 6, in <module>
#     sys.exit(main())
#              ~~~~^^
#   File "/qfs/people/fors729/miniforge3/envs/test-zppy-main-plus-unified-1-13rc5-with-fixes/lib/python3.14/site-packages/zppy/__main__.py", line 55, in main
#     user_config, plugins = _handle_plugins(user_config, default_config, args)
#                            ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#   File "/qfs/people/fors729/miniforge3/envs/test-zppy-main-plus-unified-1-13rc5-with-fixes/lib/python3.14/site-packages/zppy/__main__.py", line 118, in _handle_plugins
#     if "plugins" in user_config["default"].keys():
#                     ~~~~~~~~~~~^^^^^^^^^^^
#   File "/qfs/people/fors729/miniforge3/envs/test-zppy-main-plus-unified-1-13rc5-with-fixes/lib/python3.14/site-packages/configobj/__init__.py", line 549, in __getitem__
#     val = dict.__getitem__(self, key)
# KeyError: 'default'

# Not sure what would be causing this issue...

Unfortunately, I couldn't get either the v3 test or the v2 test to launch. v3 is hanging (which I assume to be a Compy issue considering commands have generally been slow). v2 is giving a KeyError whose source I can't quickly figure out.

1 reply

forsyth2 Apr 16, 2026
Maintainer Author

Very strange, rerunning today these errors don't occur:

compy_conda
conda activate test-zppy-main-plus-unified-1-13rc5-with-fixes
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_compy.cfg 
# Starts launching jobs, no key errors 
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_compy.cfg
# Starts launching jobs, without hanging

I want to include some more updates (i.e., using the new ncremap) though, so I'll launch a fresh test later today.

TonyB9000 · 2026-04-16T23:08:33Z

TonyB9000
Apr 16, 2026
Collaborator

@forsyth2 Out every 100 [ncclimo +] e3sm_to_cmip jobs I run , about 2-3 fail, quite randomly. I suspect some transient slurm I/O issue. When I rerun the very same jobs, they complete successfully. If it were possible to separate these kinds of errors from "real" (non-transient) errors, a system of "automatic try again" would be nice.

1 reply

forsyth2 Apr 17, 2026
Maintainer Author

zppy can be re-launched, automatically running jobs that failed previously. That's not automatic though. This also seems like an issue that should be addressed inside e3sm_to_cmip.

TonyB9000 · 2026-04-17T16:49:42Z

TonyB9000
Apr 17, 2026
Collaborator

@forsyth2 The "dsm_generate_CMIP6" script must sometimes call "ncclimo THEN e3sm_to_cmip", and sometimes (MPAS) just e3sm_to_cmip. Since these transient failures can occur due to Slurm (I believe) it can plague both NCO and E2C stuff. So I guess I really need to embed the "automatic" at an upper level, or else have such detailed error info to make a better determination.

If e2c fails because ncclimo lost some output files, re-running e2c won't do any good.

Because dsm_generate_CMIP6 always runs each variable individually (slurm-parallel by decade), it runs e2c per-decade in serial mode. If it were REALLY smart, and knew that a decade had failed, it could re-run ONLY that decade, rather than 200 years, etc.

2 replies

forsyth2 Apr 17, 2026
Maintainer Author

@TonyB9000 Should this be on a page for DSM? (Does the DSM have a GitHub repo?) Is there a lesson for zppy testing?

TonyB9000 Apr 17, 2026
Collaborator

There is, and it should. I'll see about adding it to a feature request. I don't know about a "lesson for zppy testing".

forsyth2 · 2026-04-17T20:08:36Z

forsyth2
Apr 17, 2026
Maintainer Author

2026-04-15 Perlmutter E3SM Unified 1.13.0rc5 + fixes, extremely limited test

We're only concerned here with checking if e3sm_to_cmip.bash is working correctly, since there have been upstream fixes in both e3sm_to_cmip and ncremap.

Set up environments

cd ~/ez/e3sm_to_cmip/
# No such file or directory
cd ~/ez

# Go to https://github.com/E3SM-Project/e3sm_to_cmip
git clone git@github.com:E3SM-Project/e3sm_to_cmip.git
cd e3sm_to_cmip/
git remote -v
git remote add upstream git@github.com:E3SM-Project/e3sm_to_cmip.git
git remote -v
git status
# On branch master
# nothing to commit, working tree clean
git log --oneline | head -n 2
# 7a78612 Fix bug: parallel mode incorrectly stops on handler failure when --on-var-failure=ignore (#344)
# 2efb904 bump to 1.14.0rc2 (#343)
# Good, matches https://github.com/E3SM-Project/e3sm_to_cmip/commits/master/
nersc_conda # Bash function to activate conda
rm -rf build
conda clean --all --y
conda env create -f conda-env/dev.yml -n e3sm_to_cmip_20260416
conda activate e3sm_to_cmip_20260416
python -m pip install .

zppy run

# zppy itself #################################################################
cd ~/ez/zppy
git status
# On branch issue-809-nco-path
# nothing to commit, working tree clean

# This branch has most of what we need, but not everything...
# It has the commits for #806 and #809, but not #805
# We need to add in https://github.com/E3SM-Project/zppy/pull/805/commits

git checkout -b unified-rc5-20260416-fixes
git fetch mkstratos
git cherry-pick 5c61e1e91c19200210ed26f400df0139354899f1
# Ignore merge commit:
# git cherry-pick 901e4d34a203b7e25eb61ec80ea32ace7b13a093
git cherry-pick ddb4fdc1eda0078f11bbe56e8eceea84411ed7ed
git cherry-pick 447ee80af36ac27e45eadcaf657d07a927cc116f
git log --oneline | head -n 17

# From #805:
# aea4a74 Fix for start/end years that aren't 4 digits
# 854a9fa Fix livvkit climo_subsections for correct default Mis-read default in fix #804, actual default is list with an empty string, so it also evaluates to "Truth-y", so it now checks against that default
# 907cbbf Add energy balance timeseries analysis to livvkit

# From #809:
# 699701c Allow integration tests to change nco_path
# 72fc774 Add nco_path support for other NCO commands
# b9d48fd Deprecate ncclimo_cmd
# 26a5985 Add nco_path parameter

# From #806:
# fd6398c post-rc5 fix: Add e3sm_to_cmip_environment_commands
# f85ee76 post-rc5 fix: Replace wa with wap
# 14e25c0 post-rc5 fix: increase diags walltime
# 6549125 post-rc5 fix: add more status logging to tc_analysis
# 1e5e622 post-rc5 fix: print package versions
# d74d8da Set standard mapping file paths
# 2ca288b Fix hard-coded mapping file LCRC paths
# a8eee94 Fix hard-coded non-mapping file LCRC paths
# 7d5f8af Fix error codes

# From https://github.com/E3SM-Project/zppy/commits/main/:
# 00cd1bd Only use climo_subsections if it is non-empty (#804)

showquota # Have enough space on home & pscratch

nersc_conda # Bash function to activate conda
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n test-zppy-main-unifiedrc5-20260416-fixes
conda activate test-zppy-main-unifiedrc5-20260416-fixes 
pre-commit run --all-files # Confirm pre-commit still passes
python -m pip install .
conda list | grep python # Check Python version
# python                    3.14.4

# Run zppy unit tests
pytest tests/test_*.py # 44 passed in 0.20s

# Edit tests/integration/utils.py:

TEST_SPECIFICS: Dict[str, Any] = {
    # This is the NCO path.
    # Keep as "" to use the production-version NCO commands.
    # Set to a specific path to use development-version NCO commands.
    "nco_path": "/global/cfs/cdirs/e3sm/zender/bin_perlmutter/",
    # These are custom environment_commands for specific tasks.
    # Never set these to "", because they will print the line
    # `environment_commands = ""` for the corresponding task,
    # thus overriding the value set higher up in the cfg.
    # That is, there will be no environment set.
    # (`environment_commands = ""` only redirects to Unified
    # if specified under the [default] task)
    "e3sm_to_cmip_environment_commands": "source /global/homes/f/forsyth/miniforge3/etc/profile.d/conda.sh; conda activate e3sm_to_cmip_20260416",
    "diags_environment_commands": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc5_pm-cpu.sh",
    "mpas_analysis_environment_commands": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc5_pm-cpu.sh",
    "global_time_series_environment_commands": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc5_pm-cpu.sh",
    "livvkit_environment_commands": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc5_pm-cpu.sh",
    "pcmdi_diags_environment_commands": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc5_pm-cpu.sh",
    # This is the environment setup for other tasks.
    # Leave as "" to use the latest Unified environment.
    "environment_commands": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc5_pm-cpu.sh",
    # For a complete test, run the set of latest cfgs and at least one set of legacy cfgs
    "cfgs_to_run": [
        # "weekly_bundles",
        # "weekly_comprehensive_v2",
        "weekly_comprehensive_v3",
        # "weekly_legacy_3.1.0_bundles",
        # "weekly_legacy_3.1.0_comprehensive_v2",
        # "weekly_legacy_3.1.0_comprehensive_v3",
        # "weekly_legacy_3.0.0_bundles",
        # "weekly_legacy_3.0.0_comprehensive_v2",
        # "weekly_legacy_3.0.0_comprehensive_v3",
    ],
    "tasks_to_run": [
        # "e3sm_diags",
        # "mpas_analysis",
        "global_time_series",
        "ilamb",
        "livvkit",
        "pcmdi_diags",
    ],
    "unique_id": "test_zppy_main_unifiedrc5_20260416_fixes ",
}

I also want to test livvkit with the changes of #805, so let's include livvkit too.

git diff # Diff looks good
python tests/integration/utils.py
# CFG FILES HAVE BEEN GENERATED FROM TEMPLATES WITH THESE SETTINGS:
# UNIQUE_ID=test_zppy_main_unifiedrc5_20260416_fixes 
# nco_path=/global/cfs/cdirs/e3sm/zender/bin_perlmutter/
# Reminder: `nco_path=''` => the production-version NCO commands will be used
# e3sm_to_cmip_environment_commands=source /global/homes/f/forsyth/miniforge3/etc/profile.d/conda.sh; conda activate e3sm_to_cmip_20260416
# diags_environment_commands=source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc5_pm-cpu.sh
# mpas_analysis_environment_commands=source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc5_pm-cpu.sh
# global_time_series_environment_commands=source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc5_pm-cpu.sh
# livvkit_environment_commands=source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc5_pm-cpu.sh
# pcmdi_diags_environment_commands=source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc5_pm-cpu.sh
# environment_commands=source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc5_pm-cpu.sh
# Reminder: `environment_commands=''` => the latest E3SM Unified environment will be used

# For reference:
alias sq
# alias sq='sqa -u forsyth'
alias sqa
# alias sqa='squeue -o "%8u %.7a %.4D %.9P %8i %.2t %.10r %.10M %.10l %j" --sort=P,-t,-p'
sq
# No jobs currently queued

# Launch jobs; remember we're doing a limited run, so it's only this line:
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_pm-cpu.cfg
# climo_atm_monthly_180x360_aave_1985-1986
# ...
# RuntimeError: Problem submitting script /global/cfs/cdirs/e3sm/forsyth/zppy_weekly_comprehensive_v3_output/test_zppy_main_unifiedrc5_20260416_fixes /v3.LR.historical_0051/post/scripts/climo_atm_monthly_180x360_aave_1985-1986.bash

# UNIQUE_ID had a trailing space!!
# Edit that:
# "unique_id": "test_zppy_main_unifiedrc5_20260416_fixes_try2",
python tests/integration/utils.py
sq # No jobs currently queued
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_pm-cpu.cfg
sq | wc -l # Thu 4/16 17:18 => 47 - header row = 46 jobs
sq | wc -l # Thu 4/16 17:56 => 47 - header row = 46 jobs (no change)

# We have some changes to make, given
# https://github.com/E3SM-Project/zppy/pull/810#issuecomment-4264499498
# So let's cancel and try again.
# It also seems like we might want to launch even fewer jobs...
scancel -u forsyth

git add -A
git commit -m "Testing"
git checkout issue-809-nco-path
# Add new commit on PR 810

git checkout unified-rc5-20260416-fixes
git fetch upstream issue-809-nco-path
git rebase upstream/issue-809-nco-path
git log --oneline | head -n 18
# The new commit is present:
# fabbb57 Address comments

# Still in this env: test-zppy-main-unifiedrc5-20260416-fixes
python -m pip install .
# Edit tests/integration/utils.py
# "unique_id": "test_zppy_main_unifiedrc5_20260416_fixes_try3",
# We want to run even fewer jobs, so set ILAMB as the only task
# That will also launch the e3sm_to_cmip jobs
# Plus we can see if the reverted ILAMB version is working.
# "tasks_to_run": [
#     # "e3sm_diags",
#     # "mpas_analysis",
#     # "global_time_series",
#     "ilamb",
#     # "livvkit",
#     # "pcmdi_diags",
# ],
python tests/integration/utils.py
sq # No jobs currently queued
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_pm-cpu.cfg
sq | wc -l # Thu 4/16 18:17 => 20 - header row = 19 jobs

# Picking up 2026-04-17 #######################################################
sq | wc -l  # Fri 4/17 8:18 => 14 - header row = 13 jobs (6 out of 19 jobs have run)
sq | wc -l  # Fri 4/17 12:41 => 2 - header row = 1 jobs (18 out of 19 jobs have run)
sq # Only ILAMB is waiting to run; it is not waiting because of a missing dependency.

Review results

cd /global/cfs/cdirs/e3sm/forsyth/zppy_weekly_comprehensive_v3_output/test_zppy_main_unifiedrc5_20260416_fixes_try3/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# ilamb_1985-1988.status:WAITING 51671059
# Nothing failed! Only ILAMB is still waiting

# e3sm_to_cmip status #########################################################
grep -in error e3sm_to_cmip*.o*
# There are error lines, but evidently none of these actually caused the program to crash.

# Package version printer #####################################################
emacs e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.o51671029
# source /global/homes/f/forsyth/miniforge3/etc/profile.d/conda.sh
# pixi list python:
# /var/spool/slurmd/job51671029/slurm_script: line 67: pixi: command not found
# pixi list e3sm_to_cmip:
# /var/spool/slurmd/job51671029/slurm_script: line 69: pixi: command not found

# Strange, it's looking for pixi despite the command containing conda:
emacs e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.bash
# if echo source /global/homes/f/forsyth/miniforge3/etc/profile.d/conda.sh; conda activate e3sm_to_cmip_20260416 | grep -q "conda"; then
#     pkg_manager="conda"
# else
#     pkg_manager="pixi"
# fi

# zppy/templates/e3sm_to_cmip.bash has:
# # Detect whether to use pixi or conda based on environment_commands
# if echo {{ environment_commands }} | grep -q "conda"; then
#     pkg_manager="conda"
# else
#     pkg_manager="pixi"
# fi

# It looks like we need to encase the whole of environment_commands in quotes:
# if echo "{{ environment_commands }}" | grep -q "conda"; then

I ran an extremely limited subset of tests, but it looks like e3sm_to_cmip has been fixed by @chengzhuzhang's E3SM-Project/e3sm_to_cmip#344 and @czender's ncremap fix. Thank you both!

Action items for me:

Fix package management version printer

3 replies

chengzhuzhang Apr 17, 2026
Maintainer

Great news to hear before the weekend! Thanks @forsyth2 .

forsyth2 Apr 17, 2026
Maintainer Author

Fix package management version printer

Resolved with this commit

forsyth2 Apr 17, 2026
Maintainer Author

I think I'll go ahead and merge #806 and #810 so that they're in zppy's main. I'm already at 10+5=15 commits between the two, which I have to keep cherry-picking or rebasing onto testing branches.

(While we haven't run a truly complete test yet, I think I've tested these commits enough at this point to include them on main. As it is, #806 was intended to be fixes for rc4, but is now fixes for both rc4 and rc5. I think it would be cleaner to open a fresh PR for any issues that show up in Unified rc6.)

#805 on the other hand I'm going to wait to merge, since I haven't run LIVVkit to completion yet to be sure that doesn't break anything.

forsyth2 · 2026-04-21T00:55:22Z

forsyth2
Apr 21, 2026
Maintainer Author

2026-04-20 Compy E3SM Unified 1.13.0rc6

What expected results are currently on Compy?

The latest expected results are still the E3SM Unified 1.12.0 official results.

What changes are we testing?

Package	Changes since last test
NCO	Unified rc6 instead of rc5
e3sm_to_cmip	Unified rc6 instead of rc5
e3sm_diags	Unified rc6 instead of rc5
mpas_analysis	Unified rc6 instead of rc5
zppy-interfaces	Unified rc6 instead of rc5
zppy	#806, #810 + unmerged commits from #805

Testing process

Starting actual testing on 2026-04-20
Branch: zppy main branch + commits from https://github.com/E3SM-Project/zppy/pull/805/commits
Environments: zppy -- dev environment; everything else -- E3SM Unified 1.13.0rc6

Set up environments

We can mostly skip this step, since all the packages will be tested using the E3SM Unified RC.

cd ~/ez/zppy-interfaces
git status
# # On branch main
# nothing to commit, working directory clean

# REMINDER: On Compy, you must NOT specify the branch, 
# otherwise it won't actually fetch the latest commits!!
git fetch upstream 
git checkout main
git reset --hard upstream/main
git log --oneline | head -n 1
# 6b0758f Bump to 0.2.1rc1 (#48)
# Good, matches https://github.com/E3SM-Project/zppy-interfaces/commits/main
source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc6_compy.sh

# Check zppy-interfaces tests
pytest tests/unit/global_time_series/test_*.py # 10 passed, 2 warnings in 13.89s
# SyntaxWarning: invalid escape sequence '\w'
# SyntaxWarning: invalid escape sequence '\s'
pytest tests/unit/pcmdi_diags/test_*.py # 7 passed in 9.58s

zppy run

# zppy itself #################################################################
cd ~/ez/zppy
git status
# # On branch test-zppy-main-plus-unified-1-13rc5-with-fixes
# nothing to commit, working directory clean

git fetch upstream # Remember, do NOT specify the branch on Compy!
git checkout -b test-zppy-main-plus-unified-1-13rc6 upstream/main
git log --oneline | head -n 1
# 2186207 Merge pull request #810 from E3SM-Project/issue-809-nco-path
# Good, matches https://github.com/E3SM-Project/zppy/commits/main

# Add commits from unmerged PRs

# 1) https://github.com/E3SM-Project/zppy/pull/805/commits
# We've already fetched all branches from upstream, 
# BUT this branch is from a fork
git fetch mkstratos # Remember, do NOT specify the branch on Compy!
git log --oneline | head -n 3
# 2186207 Merge pull request #810 from E3SM-Project/issue-809-nco-path
# 92ef7f8 Address comments
# c74f0d5 Allow integration tests to change nco_path
# There have been no new commits to https://github.com/E3SM-Project/zppy/pull/805/commits
# We have the three commits to chery pick
git cherry-pick 5c61e1e91c19200210ed26f400df0139354899f1
# git cherry-pick 901e4d34a203b7e25eb61ec80ea32ace7b13a093 # Merge commit
git cherry-pick ddb4fdc1eda0078f11bbe56e8eceea84411ed7ed
git cherry-pick 447ee80af36ac27e45eadcaf657d07a927cc116f
git log --oneline | head -n 6
# 3ccbfbf Fix for start/end years that aren't 4 digits
# 7af605a Fix livvkit climo_subsections for correct default Mis-read default in fix #804, actual default is list with an empty string, so it also evaluates to "Truth-y", so it now checks against that default
# ddbfe70 Add energy balance timeseries analysis to livvkit
# 2186207 Merge pull request #810 from E3SM-Project/issue-809-nco-path
# 92ef7f8 Address comments
# c74f0d5 Allow integration tests to change nco_path

lfs quota -hu fors729 /compyfs # Have enough space on /compyfs
# Not sure how to check space on /qfs
# (Currently in /qfs/people/fors729/ez/zppy)

compy_conda # Function to activate conda
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n test-zppy-main-plus-unified-1-13rc6
conda activate test-zppy-main-plus-unified-1-13rc6
pre-commit run --all-files # Confirm pre-commit still passes
python -m pip install .
conda list | grep python # Check Python version
# python                    3.14.4

# Run zppy unit tests
pytest tests/test_*.py # 44 passed in 0.57s

# Edit tests/integration/utils.py:
# e3sm_to_cmip and NCO updates are now in Unified rc6, 
# so we don't need to specify dev environments for either of those.

TEST_SPECIFICS: Dict[str, Any] = {
    # This is the NCO path.
    # Keep as "" to use the production-version NCO commands.
    # Set to a specific path to use development-version NCO commands.
    "nco_path": "",
    # These are custom environment_commands for specific tasks.
    # Never set these to "", because they will print the line
    # `environment_commands = ""` for the corresponding task,
    # thus overriding the value set higher up in the cfg.
    # That is, there will be no environment set.
    # (`environment_commands = ""` only redirects to Unified
    # if specified under the [default] task)
    "e3sm_to_cmip_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc6_compy.sh",
    "diags_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc6_compy.sh",
    "mpas_analysis_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc6_compy.sh",
    "global_time_series_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc6_compy.sh",
    "livvkit_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc6_compy.sh",
    "pcmdi_diags_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc6_compy.sh",
    # This is the environment setup for other tasks.
    # Leave as "" to use the latest Unified environment.
    "environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc6_compy.sh",
    # For a complete test, run the set of latest cfgs and at least one set of legacy cfgs
    "cfgs_to_run": [
        "weekly_bundles",
        "weekly_comprehensive_v2",
        "weekly_comprehensive_v3",
        # "weekly_legacy_3.1.0_bundles",
        # "weekly_legacy_3.1.0_comprehensive_v2",
        # "weekly_legacy_3.1.0_comprehensive_v3",
        # "weekly_legacy_3.0.0_bundles",
        # "weekly_legacy_3.0.0_comprehensive_v2",
        # "weekly_legacy_3.0.0_comprehensive_v3",
    ],
    "tasks_to_run": [
        "e3sm_diags",
        "mpas_analysis",
        "global_time_series",
        "ilamb",
        "livvkit",
        "pcmdi_diags",
    ],
    "unique_id": "zppy_main_branch_plus_unified_1_13rc6",
}

git diff # Diff looks good
python tests/integration/utils.py
# CFG FILES HAVE BEEN GENERATED FROM TEMPLATES WITH THESE SETTINGS:
# UNIQUE_ID=zppy_main_branch_plus_unified_1_13rc6
# nco_path=
# Reminder: `nco_path=''` => the production-version NCO commands will be used
# e3sm_to_cmip_environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc6_compy.sh
# diags_environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc6_compy.sh
# mpas_analysis_environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc6_compy.sh
# global_time_series_environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc6_compy.sh
# livvkit_environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc6_compy.sh
# pcmdi_diags_environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc6_compy.sh
# environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc6_compy.sh
# Reminder: `environment_commands=''` => the latest E3SM Unified environment will be used

# For reference:
alias sq
# alias sq='sqa -u fors729'
alias sqa
# alias sqa='squeue -o "%8u %.7a %.4D %.9P %7i %.2t %.10r %.10M %.10l %.8Q %j" --sort=P,-t,-p'
sq
# No jobs currently queued

# Launch jobs; start with the most important one, comprehensive_v3
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_compy.cfg
zppy -c tests/integration/generated/test_weekly_bundles_compy.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_compy.cfg

# Only running the non-legacy tests on Compy

sq | wc -l # Mon 4/20 15:16 => 85 - header row = 84 jobs
sq | wc -l # Mon 4/20 15:35 => 22 - header row = 21 jobs
sq | wc -l # Mon 4/20 16:50 => 8 - header row = 7 jobs
sq
# USER     ACCOUNT NODE PARTITION JOBID   ST     REASON       TIME TIME_LIMIT PRIORITY NAME
# fors729     e3sm    8     short 753790  PD Dependency       0:00    2:00:00      233 ilamb_1985-1988
# fors729     e3sm    1     short 753792  PD Dependency       0:00    2:00:00      233 pcmdi_diags_mean_climate_model_vs_obs_1985-1994
# fors729     e3sm    1     short 753793  PD Dependency       0:00    2:00:00      233 pcmdi_diags_variability_modes_cpl_model_vs_obs_1985-1994
# fors729     e3sm    1     short 753794  PD Dependency       0:00    2:00:00      233 pcmdi_diags_variability_modes_atm_model_vs_obs_1985-1994
# fors729     e3sm    1     short 753795  PD Dependency       0:00    2:00:00      233 pcmdi_diags_synthetic_plots_model_vs_obs
# fors729     e3sm    8     short 753828  PD Dependency       0:00    2:00:00      233 ilamb_1980-1981
# fors729     e3sm    8     short 753829  PD Dependency       0:00    2:00:00      233 ilamb_1982-1983

# They're all waiting for dependencies!

# HERE

Bundles part 2

# Check on bundles status
cd /compyfs/fors729/zppy_weekly_bundles_output/zppy_main_branch_plus_unified_1_13rc6/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# bundle1.status:ERROR (1)
# e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.status:ERROR (2)
# e3sm_to_cmip_atm_monthly_180x360_aave_1987-1988-0002.status:ERROR (2)

grep -in error bundle1.o*
# Too many to list
grep -n "e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002" bundle1.o*

grep -n "e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002" bundle1.o*
# 909:=== e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.bash ===
# 917:./e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.bash: line 45: conda: command not found
# 919:./e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.bash: line 76: conda: command not found
emacs bundle1.o753797

# 2026-04-20 15:20:24.491632 [INFO]: handler.py(_cmor_write_with_time:680) >> rlut: Writing variable to file...
# ^M  0%|          | 0/15 [00:08<?, ?it/s]
# 2026-04-20 15:20:30.930397 [ERROR]: util.py(exit_failure:86) >> Exiting with failure code (1).
# srun: error: n0286: task 0: Exited with exit code 1


# srun: Warning: can't run 1 processes on 4 nodes, setting nnodes to 1
# /qfs/projects/ops/rh7/apps/E3SM/conda_envs/e3smu_1_13_0rc6/compy/pixi/.pixi/envs/default/lib/python3.13/site-packages/output_viewer/utils.py:14: SyntaxWarning: invalid escape sequence '\w'
#   value = str(re.sub('[^\w\s-]', '', str(value)).strip().lower())
# /qfs/projects/ops/rh7/apps/E3SM/conda_envs/e3smu_1_13_0rc6/compy/pixi/.pixi/envs/default/lib/python3.13/site-packages/output_viewer/utils.py:15: SyntaxWarning: invalid escape sequence '\s'
#   return re.sub('[-\s]+', '-', str(value))
# Traceback (most recent call last):
#   File "/qfs/projects/ops/rh7/apps/E3SM/conda_envs/e3smu_1_13_0rc6/compy/pixi/.pixi/envs/default/lib/python3.13/site-packages/e3sm_diags/e3sm_diags_driver.py", line 101, in save_provenance
#     paths["env_yml_path"] = _save_env_yml(prov_dir)
#                             ~~~~~~~~~~~~~^^^^^^^^^^
#   File "/qfs/projects/ops/rh7/apps/E3SM/conda_envs/e3smu_1_13_0rc6/compy/pixi/.pixi/envs/default/lib/python3.13/site-packages/e3sm_diags/e3sm_diags_driver.py", line 133, in _save_env_yml
#     p = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
#   File "/qfs/projects/ops/rh7/apps/E3SM/conda_envs/e3smu_1_13_0rc6/compy/pixi/.pixi/envs/default/lib/python3.13/subprocess.py", line 1039, in __init__
#     self._execute_child(args, executable, preexec_fn, close_fds,
#     ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#                         pass_fds, cwd, env,
#                         ^^^^^^^^^^^^^^^^^^^
#     ...<5 lines>...
#                         gid, gids, uid, umask,
#                         ^^^^^^^^^^^^^^^^^^^^^^
#                         start_new_session, process_group)
#                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#   File "/qfs/projects/ops/rh7/apps/E3SM/conda_envs/e3smu_1_13_0rc6/compy/pixi/.pixi/envs/default/lib/python3.13/subprocess.py", line 1991, in _execute_child
#     raise child_exception_type(errno_num, err_msg, err_filename)
# FileNotFoundError: [Errno 2] No such file or directory: 'conda'
# 2026-04-20 15:21:04,238 [INFO]: e3sm_diags_driver.py(_log_diagnostic_run_info:525) >>

# Because of the errors, we won't run bundles part 2

Review finished runs

### v2  ###
cd /compyfs/fors729/zppy_weekly_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc6/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# e3sm_to_cmip_atm_monthly_180x360_aave_1980-1981-0002.status:ERROR (2)
# e3sm_to_cmip_atm_monthly_180x360_aave_1982-1983-0002.status:ERROR (2)
# ilamb_1980-1981.status:WAITING 753828
# ilamb_1982-1983.status:WAITING 753829

grep -in error e3sm_to_cmip*.o* # Not particularly helpful

emacs e3sm_to_cmip_atm_monthly_180x360_aave_1980-1981-0002.o753816 
# 2026-04-20 15:20:28.292379 [INFO]: handler.py(_cmor_write_with_time:680) >> rlut: Writing variable to file...
# 2026-04-20 15:20:28.292767 [INFO]: handler.py(_cmor_write_with_time:677) >> rtmt: time span 47450.0 - 48180.0
# 2026-04-20 15:20:28.292926 [INFO]: handler.py(_cmor_write_with_time:680) >> rtmt: Writing variable to file...
# ^M  0%|          | 0/15 [00:08<?, ?it/s]
# 2026-04-20 15:20:34.269504 [ERROR]: util.py(exit_failure:86) >> Exiting with failure code (1).
# srun: error: n0010: task 0: Exited with exit code 1

emacs e3sm_to_cmip_atm_monthly_180x360_aave_1982-1983-0002.o753817
# 2026-04-20 15:20:38.753266 [INFO]: handler.py(_cmor_write_with_time:680) >> rlut: Writing variable to file...
# ^M  0%|          | 0/15 [00:08<?, ?it/s]
# 2026-04-20 15:20:44.607406 [ERROR]: util.py(exit_failure:86) >> Exiting with failure code (1).
# srun: error: n0002: task 0: Exited with exit code 1

# ILAMB is stuck waiting because of e3sm_to_cmip


### v3 ###
cd /compyfs/fors729/zppy_weekly_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc6/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.status:ERROR (2)
# e3sm_to_cmip_atm_monthly_180x360_aave_1987-1988-0002.status:ERROR (2)
# e3sm_to_cmip_atm_monthly_180x360_aave_1989-1990-0002.status:ERROR (2)
# e3sm_to_cmip_atm_monthly_180x360_aave_1991-1992-0002.status:ERROR (2)
# e3sm_to_cmip_atm_monthly_180x360_aave_1993-1994-0002.status:ERROR (2)
# ilamb_1985-1988.status:WAITING 753790
# pcmdi_diags_mean_climate_model_vs_obs_1985-1994.status:WAITING 753792
# pcmdi_diags_synthetic_plots_model_vs_obs.status:WAITING 753795
# pcmdi_diags_variability_modes_atm_model_vs_obs_1985-1994.status:WAITING 753794
# pcmdi_diags_variability_modes_cpl_model_vs_obs_1985-1994.status:WAITING 753793

grep -B 3 "Exited with exit code 1" e3sm_to_cmip*.o*
# e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.o753769-2026-04-20 15:18:24.685218 [INFO]: handler.py(_cmor_write_with_time:680) >> rlut: Writing variable to file...
#   0%|          | 0/16 [00:09<?, ?it/s]1985-1986-0002.o753769-
# e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.o753769-2026-04-20 15:18:30.579353 [ERROR]: util.py(exit_failure:86) >> Exiting with failure code (1).
# e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.o753769:srun: error: n0024: task 0: Exited with exit code 1
# --
# e3sm_to_cmip_atm_monthly_180x360_aave_1987-1988-0002.o753770-2026-04-20 15:18:42.628737 [INFO]: handler.py(_cmor_write_with_time:680) >> rlut: Writing variable to file...
#   0%|          | 0/16 [00:08<?, ?it/s]1987-1988-0002.o753770-
# e3sm_to_cmip_atm_monthly_180x360_aave_1987-1988-0002.o753770-2026-04-20 15:18:48.552595 [ERROR]: util.py(exit_failure:86) >> Exiting with failure code (1).
# e3sm_to_cmip_atm_monthly_180x360_aave_1987-1988-0002.o753770:srun: error: n0017: task 0: Exited with exit code 1
# --
# e3sm_to_cmip_atm_monthly_180x360_aave_1989-1990-0002.o753771-2026-04-20 15:18:39.327297 [INFO]: handler.py(_cmor_write_with_time:680) >> rlut: Writing variable to file...
#   0%|          | 0/16 [00:08<?, ?it/s]1989-1990-0002.o753771-
# e3sm_to_cmip_atm_monthly_180x360_aave_1989-1990-0002.o753771-2026-04-20 15:18:45.331850 [ERROR]: util.py(exit_failure:86) >> Exiting with failure code (1).
# e3sm_to_cmip_atm_monthly_180x360_aave_1989-1990-0002.o753771:srun: error: n0029: task 0: Exited with exit code 1
# --
# e3sm_to_cmip_atm_monthly_180x360_aave_1991-1992-0002.o753772-2026-04-20 15:18:46.850343 [INFO]: handler.py(_cmor_write_with_time:680) >> rtmt: Writing variable to file...
#   0%|          | 0/16 [00:08<?, ?it/s]1991-1992-0002.o753772-
# e3sm_to_cmip_atm_monthly_180x360_aave_1991-1992-0002.o753772-2026-04-20 15:18:52.760193 [ERROR]: util.py(exit_failure:86) >> Exiting with failure code (1).
# e3sm_to_cmip_atm_monthly_180x360_aave_1991-1992-0002.o753772:srun: error: n0021: task 0: Exited with exit code 1
# --
# e3sm_to_cmip_atm_monthly_180x360_aave_1993-1994-0002.o753773-2026-04-20 15:18:55.883270 [INFO]: handler.py(_cmor_write_with_time:680) >> rlut: Writing variable to file...
#   0%|          | 0/16 [00:08<?, ?it/s]1993-1994-0002.o753773-
# e3sm_to_cmip_atm_monthly_180x360_aave_1993-1994-0002.o753773-2026-04-20 15:19:01.855836 [ERROR]: util.py(exit_failure:86) >> Exiting with failure code (1).
# e3sm_to_cmip_atm_monthly_180x360_aave_1993-1994-0002.o753773:srun: error: n0001: task 0: Exited with exit code 1

# ILAMB, pcmdi_diags are stuck waiting because of e3sm_to_cmip

### bundles ###
# We didn't rerun, so there's nothing new to check.

post/scripts directory status: Many many errors -- will review in Results Analysis section below

Python tests: We'll skip this because of the post/scripts directory errors.

Results analysis

LIVVkit review

@mkstratos Review of #805: The LIVVkit viewer page shows no results.

cd /compyfs/fors729/zppy_weekly_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc6/v3.LR.historical_0051/post/scripts
cat livvkit_1985-1994.status
# OK
# So, it didn't error out
grep -in error livvkit_1985-1994.o753791 
# 1278:    raise ValueError('Unable to discover machine from host name')
# 1279:ValueError: Unable to discover machine from host name
emacs livvkit_1985-1994.o753791 

# Traceback (most recent call last):
#   File "/qfs/projects/ops/rh7/apps/E3SM/conda_envs/e3smu_1_13_0rc6/compy/pixi/.pixi/envs/default/bin/lex-cfg", line 10, in <module>
#     sys.exit(main())
#              ~~~~^^
#   File "/qfs/projects/ops/rh7/apps/E3SM/conda_envs/e3smu_1_13_0rc6/compy/pixi/.pixi/envs/default/lib/python3.13/site-packages/livvext/generate_cfg.py", line 159, in main
#     mach_info = mache.MachineInfo()
#   File "/qfs/projects/ops/rh7/apps/E3SM/conda_envs/e3smu_1_13_0rc6/compy/pixi/.pixi/envs/default/lib/python3.13/site-packages/mache/machine_info.py", line 90, in __init__
#     raise ValueError('Unable to discover machine from host name')
# ValueError: Unable to discover machine from host name

@xylar ^ I'm thinking this is an issue on the LIVVkit side, but tagging you in case it's on the Mache side.

e3sm_to_cmip

@chengzhuzhang There were e3sm_to_cmip failures, again failing around rlut, but sometimes other variables. I know that had been resolved in our dev environment testing. Did we actually get those changes into E3SM Unified rc6? I'm not seeing an e3sm_to_cmip rc3 on the feedstock page, nor on the commits page.

Where to find output files to debug

I'm not going to be able to do further testing/debugging until Thursday afternoon at the earliest. In the meantime, I tried copying over results from this test run to NERSC and LCRC, for others to review while I'm out, but I can't seem to get the Compy endpoint working on Globus.

Copying process

cd /compyfs/fors729
mkdir e3sm_unified_1.13.0rc6_test_logs
cd e3sm_unified_1.13.0rc6_test_logs

cp -R /compyfs/fors729/zppy_weekly_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc6/v3.LR.historical_0051/post/scripts v3_results
cp -R /compyfs/fors729/zppy_weekly_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc6/v2.LR.historical_0201/post/scripts v2_results
cp -R /compyfs/fors729/zppy_weekly_bundles_output/zppy_main_branch_plus_unified_1_13rc6/v3.LR.historical_0051/post/scripts bundles_results

globus.org: I can't seem to get the connection working.

10 replies

xylar Apr 21, 2026
Collaborator

I'm adding this in mache:
E3SM-Project/mache#414

forsyth2 Apr 23, 2026
Maintainer Author

e3sm_to_cmip haven't had a release candidate after the one line fix, so the same errors are expected

@chengzhuzhang Ok, but we do need that in the Unified RC so I can do a true Unified test rather than a dev environment test. I see Xylar just made Unified rc7, but it looks like e3sm_to_cmip still isn't included: no rc3 on feedstock commits or e3sm_to_cmip commits.

I guess we can check if the ncremap errors are gone in this test.

cd /compyfs/fors729/zppy_weekly_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc6/v3.LR.historical_0051/post/scripts
grep -in ncremap *.o* # No results

cd /compyfs/fors729/zppy_weekly_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc6/v2.LR.historical_0201/post/scripts
grep -in ncremap *.o* # No results

Yes, it looks like the ncremap issue is indeed resolved. Thanks @czender!

I ran into permission issue to review the logs, could you fix when you have a chance.

Sorry about that. I just ran chmod -R 755 e3sm_unified_1.13.0rc6_test_logs, changing ls-lt results from drwxr-x---+ to drwxr-xr-x+, so it should work now.

a few lines later into the config generation and it will fail anyway due to livvext not being set up for compy. I can work on a fix for that part and hopefully fix the discovery part too. Do you want that pushed to the #805 PR?

Thanks @mkstratos. It sounds like Xylar has the discovery part fixed in E3SM-Project/mache#414, but yes we should also fix the livvskit setup if that's not covered there as well. Since #805 is specifically about adding "energy balance timeseries analysis" I think it would make more sense to make a distinct PR focused on Compy setup.

I'm adding this in mache

Thanks @xylar!

mkstratos Apr 23, 2026

@forsyth2 I completely forgot that it's not a PR to zppy to fix the potential setup issue on compy, it was a LIVVext PR! That's done and released, and LIVVext v1.1.2 should be available on conda-forge now.

forsyth2 Apr 23, 2026
Maintainer Author

Thanks @mkstratos, ^ @xylar FYI for the next Unified rc

xylar Apr 23, 2026
Collaborator

Noted!

czender · 2026-04-21T14:48:51Z

czender
Apr 21, 2026

FWIW, NCO has long used n[0123456789][0123456789][0123456789][0123456789] for compy and it seems to work.

1 reply

xylar Apr 21, 2026
Collaborator

Okay, I'll add it.

forsyth2 · 2026-04-23T14:41:17Z

forsyth2
Apr 23, 2026
Maintainer Author

2026-04-20 Perlmutter E3SM Unified 1.13.0rc6

Per my testing strategy above:

Chrysalis: run tests on all 9 cfgs

Compy: run all 3 non-legacy test cfgs

Perlmutter: run just the non-legacy v3 test cfg

I only ran the non-legacy v3 test cfg on Perlmutter.

What expected results are currently on Compy?

The latest expected results are still the E3SM Unified 1.12.0 official results.

What changes are we testing?

Package	Changes since last test
NCO	Unified rc6 instead of rc5
e3sm_to_cmip	Unified rc6 instead of rc5
e3sm_diags	Unified rc6 instead of rc5
mpas_analysis	Unified rc6 instead of rc5
zppy-interfaces	Unified rc6 instead of rc5
zppy	#806, #810 + unmerged commits from #805

Testing process

Starting actual testing on 2026-04-20
Branch: zppy main branch + commits from https://github.com/E3SM-Project/zppy/pull/805/commits
Environments: zppy -- dev environment; everything else -- E3SM Unified 1.13.0rc6

Set up environments

We can mostly skip this step, since all the packages will be tested using the E3SM Unified RC.

cd ~/ez/zppy-interfaces
git status
# On branch main
# nothing to commit, working tree clean

git fetch upstream main
git checkout main
git reset --hard upstream/main
git log --oneline | head -n 1
# 6b0758f Bump to 0.2.1rc1 (#48)
# Good, matches https://github.com/E3SM-Project/zppy-interfaces/commits/main
source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc6_pm-cpu.sh

# Check zppy-interfaces tests
pytest tests/unit/global_time_series/test_*.py # 10 passed, 2 warnings in 11.56s
# SyntaxWarning: invalid escape sequence '\w'
# SyntaxWarning: invalid escape sequence '\s'
pytest tests/unit/pcmdi_diags/test_*.py # 7 passed in 9.18s

zppy run

# zppy itself #################################################################
cd ~/ez/zppy
git status
# On branch main
# nothing to commit, working tree clean

git fetch upstream main
git checkout -b test-zppy-main-plus-unified-1-13rc6 upstream/main
git log --oneline | head -n 1
# 2186207 Merge pull request #810 from E3SM-Project/issue-809-nco-path
# Good, matches https://github.com/E3SM-Project/zppy/commits/main

# Add commits from unmerged PRs

# 1) https://github.com/E3SM-Project/zppy/pull/805/commits
# We've already fetched all branches from upstream, 
# BUT this branch is from a fork
git fetch mkstratos livv-energy-timeseries
git log --oneline | head -n 3
# 2186207 Merge pull request #810 from E3SM-Project/issue-809-nco-path
# 92ef7f8 Address comments
# c74f0d5 Allow integration tests to change nco_path
# There have been no new commits to https://github.com/E3SM-Project/zppy/pull/805/commits
# We have the three commits to chery pick
git cherry-pick 5c61e1e91c19200210ed26f400df0139354899f1
# git cherry-pick 901e4d34a203b7e25eb61ec80ea32ace7b13a093 # Merge commit
git cherry-pick ddb4fdc1eda0078f11bbe56e8eceea84411ed7ed
git cherry-pick 447ee80af36ac27e45eadcaf657d07a927cc116f
git log --oneline | head -n 6
# 0283ea8 Fix for start/end years that aren't 4 digits
# f4f6d08 Fix livvkit climo_subsections for correct default Mis-read default in fix #804, actual default is list with an empty string, so it also evaluates to "Truth-y", so it now checks against that default
# cca0dfd Add energy balance timeseries analysis to livvkit
# 2186207 Merge pull request #810 from E3SM-Project/issue-809-nco-path
# 92ef7f8 Address comments
# c74f0d5 Allow integration tests to change nco_path

showquota # Have enough space on home and pscratch
# Not sure how to check space on /qfs
# (Currently in /global/homes/f/forsyth/ez/zppy)

nersc_conda # Function to activate conda
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n test-zppy-main-plus-unified-1-13rc6
conda activate test-zppy-main-plus-unified-1-13rc6
pre-commit run --all-files # Confirm pre-commit still passes
python -m pip install .
conda list | grep python # Check Python version
# python                    3.14.4

# Run zppy unit tests
pytest tests/test_*.py # 44 passed in 0.42s

# Edit tests/integration/utils.py:
# e3sm_to_cmip and NCO updates are now in Unified rc6, 
# so we don't need to specify dev environments for either of those.

TEST_SPECIFICS: Dict[str, Any] = {
    # This is the NCO path.
    # Keep as "" to use the production-version NCO commands.
    # Set to a specific path to use development-version NCO commands.
    "nco_path": "",
    # These are custom environment_commands for specific tasks.
    # Never set these to "", because they will print the line
    # `environment_commands = ""` for the corresponding task,
    # thus overriding the value set higher up in the cfg.
    # That is, there will be no environment set.
    # (`environment_commands = ""` only redirects to Unified
    # if specified under the [default] task)
    "e3sm_to_cmip_environment_commands": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc6_pm-cpu.sh",
    "diags_environment_commands": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc6_pm-cpu.sh",
    "mpas_analysis_environment_commands": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc6_pm-cpu.sh",
    "global_time_series_environment_commands": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc6_pm-cpu.sh",
    "livvkit_environment_commands": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc6_pm-cpu.sh",
    "pcmdi_diags_environment_commands": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc6_pm-cpu.sh",
    # This is the environment setup for other tasks.
    # Leave as "" to use the latest Unified environment.
    "environment_commands": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc6_pm-cpu.sh",
    # For a complete test, run the set of latest cfgs and at least one set of legacy cfgs
    "cfgs_to_run": [
        # "weekly_bundles",
        # "weekly_comprehensive_v2",
        "weekly_comprehensive_v3",
        # "weekly_legacy_3.1.0_bundles",
        # "weekly_legacy_3.1.0_comprehensive_v2",
        # "weekly_legacy_3.1.0_comprehensive_v3",
        # "weekly_legacy_3.0.0_bundles",
        # "weekly_legacy_3.0.0_comprehensive_v2",
        # "weekly_legacy_3.0.0_comprehensive_v3",
    ],
    "tasks_to_run": [
        "e3sm_diags",
        "mpas_analysis",
        "global_time_series",
        "ilamb",
        "livvkit",
        "pcmdi_diags",
    ],
    "unique_id": "zppy_main_branch_plus_unified_1_13rc6",
}

git diff # Diff looks good
python tests/integration/utils.py
# CFG FILES HAVE BEEN GENERATED FROM TEMPLATES WITH THESE SETTINGS:
# UNIQUE_ID=zppy_main_branch_plus_unified_1_13rc6
# nco_path=
# Reminder: `nco_path=''` => the production-version NCO commands will be used
# e3sm_to_cmip_environment_commands=source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc6_pm-cpu.sh
# diags_environment_commands=source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc6_pm-cpu.sh
# mpas_analysis_environment_commands=source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc6_pm-cpu.sh
# global_time_series_environment_commands=source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc6_pm-cpu.sh
# livvkit_environment_commands=source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc6_pm-cpu.sh
# pcmdi_diags_environment_commands=source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc6_pm-cpu.sh
# environment_commands=source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc6_pm-cpu.sh
# Reminder: `environment_commands=''` => the latest E3SM Unified environment will be used


# For reference:
alias sq
# alias sq='sqa -u fors729'
alias sqa
# alias sqa='squeue -o "%8u %.7a %.4D %.9P %7i %.2t %.10r %.10M %.10l %.8Q %j" --sort=P,-t,-p'
sq
# No jobs currently queued

# Launch jobs; only running one cfg on Perlmutter:
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_pm-cpu.cfg

sq | wc -l # Mon 4/20 15:16 => 56 - header row = 55 jobs
sq | wc -l # Mon 4/20 16:50 => 56 - header row = 55 jobs

# Picking up Thu 2026-04-23 ###################################################
sq | wc -l # Thu 4/23 07:25 => 6 - header row = 5 jobs
sq # All jobs are waiting on unfulfilled dependencies
scancel -u forsyth

Review finished run

cd /global/cfs/cdirs/e3sm/forsyth/zppy_weekly_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc6/v3.LR.historical_0051/post/scripts/
grep -v "OK" *status
# e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.status:ERROR (2)
# e3sm_to_cmip_atm_monthly_180x360_aave_1987-1988-0002.status:ERROR (2)
# e3sm_to_cmip_atm_monthly_180x360_aave_1989-1990-0002.status:ERROR (2)
# e3sm_to_cmip_atm_monthly_180x360_aave_1991-1992-0002.status:ERROR (2)
# e3sm_to_cmip_atm_monthly_180x360_aave_1993-1994-0002.status:ERROR (2)
# ilamb_1985-1988.status:WAITING 51820057
# livvkit_1985-1994.status:RUNNING 51820058
# pcmdi_diags_mean_climate_model_vs_obs_1985-1994.status:WAITING 51820059
# pcmdi_diags_synthetic_plots_model_vs_obs.status:WAITING 51820063
# pcmdi_diags_variability_modes_atm_model_vs_obs_1985-1994.status:WAITING 51820062
# pcmdi_diags_variability_modes_cpl_model_vs_obs_1985-1994.status:WAITING 51820060

grep -B 3 "Exited with exit code 1" e3sm_to_cmip*.o*
# e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.o51820027-2026-04-21 04:29:42.630530 [INFO]: handler.py(_cmor_write_with_time:680) >> rlut: Writing variable to file...
#   0%|          | 0/16 [00:07<?, ?it/s]1985-1986-0002.o51820027-
# e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.o51820027-2026-04-21 04:29:46.900894 [ERROR]: util.py(exit_failure:86) >> Exiting with failure code (1).
# e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.o51820027:srun: error: nid007057: task 0: Exited with exit code 1
# --
# e3sm_to_cmip_atm_monthly_180x360_aave_1987-1988-0002.o51820029-2026-04-21 04:34:14.220647 [INFO]: handler.py(_cmor_write_with_time:680) >> rlut: Writing variable to file...
#   0%|          | 0/16 [00:06<?, ?it/s]1987-1988-0002.o51820029-
# e3sm_to_cmip_atm_monthly_180x360_aave_1987-1988-0002.o51820029-2026-04-21 04:34:18.506941 [ERROR]: util.py(exit_failure:86) >> Exiting with failure code (1).
# e3sm_to_cmip_atm_monthly_180x360_aave_1987-1988-0002.o51820029:srun: error: nid005824: task 0: Exited with exit code 1
# --
# e3sm_to_cmip_atm_monthly_180x360_aave_1989-1990-0002.o51820030-2026-04-21 05:49:46.151589 [INFO]: handler.py(_cmor_write_with_time:680) >> rlut: Writing variable to file...
#   0%|          | 0/16 [00:07<?, ?it/s]1989-1990-0002.o51820030-
# e3sm_to_cmip_atm_monthly_180x360_aave_1989-1990-0002.o51820030-2026-04-21 05:49:50.403670 [ERROR]: util.py(exit_failure:86) >> Exiting with failure code (1).
# e3sm_to_cmip_atm_monthly_180x360_aave_1989-1990-0002.o51820030:srun: error: nid006176: task 0: Exited with exit code 1
# --
# e3sm_to_cmip_atm_monthly_180x360_aave_1991-1992-0002.o51820031-2026-04-21 05:49:46.357342 [INFO]: handler.py(_cmor_write_with_time:680) >> rlut: Writing variable to file...
#   0%|          | 0/16 [00:07<?, ?it/s]1991-1992-0002.o51820031-
# e3sm_to_cmip_atm_monthly_180x360_aave_1991-1992-0002.o51820031-2026-04-21 05:49:50.674935 [ERROR]: util.py(exit_failure:86) >> Exiting with failure code (1).
# e3sm_to_cmip_atm_monthly_180x360_aave_1991-1992-0002.o51820031:srun: error: nid004845: task 0: Exited with exit code 1
# --
# e3sm_to_cmip_atm_monthly_180x360_aave_1993-1994-0002.o51820033-2026-04-21 06:54:53.809645 [INFO]: handler.py(_cmor_write_with_time:680) >> rlut: Writing variable to file...
#   0%|          | 0/16 [00:07<?, ?it/s]1993-1994-0002.o51820033-
# e3sm_to_cmip_atm_monthly_180x360_aave_1993-1994-0002.o51820033-2026-04-21 06:54:58.106544 [ERROR]: util.py(exit_failure:86) >> Exiting with failure code (1).
# e3sm_to_cmip_atm_monthly_180x360_aave_1993-1994-0002.o51820033:srun: error: nid007100: task 0: Exited with exit code 1

# That matches Compy

tail -n 1 livvkit_1985-1994.o51820058
# [2026-04-22T07:47:38.367] error: *** JOB 51820058 ON nid006391 CANCELLED AT 2026-04-22T07:47:38 DUE TO TIME LIMIT ***

Results Analysis

Results largely match what we're seeing on Compy, so it doesn't look like there's some major Perlmutter-specific issue popping up. LIVVkit did hit a time limit though, so I'll have to increase its walltime on the next run.

0 replies

forsyth2 · 2026-05-05T17:58:10Z

forsyth2
May 5, 2026
Maintainer Author

2026-05-04 Compy E3SM Unified 1.13.0rc8

Chrysalis: run tests on all 9 cfgs
Compy: run all 3 non-legacy test cfgs
Perlmutter: run just the non-legacy v3 test cfg

What expected results are currently on Compy?

The latest expected results are still the E3SM Unified 1.12.0 official results.

What changes are we testing?

Package	Changes since last test
NCO	Unified rc8 instead of rc7
e3sm_to_cmip	Unified rc8 instead of rc7
e3sm_diags	Unified rc8 instead of rc7
mpas_analysis	Unified rc8 instead of rc7
zppy-interfaces	Unified rc8 instead of rc7
zppy	NEW Using `zppy v3.2.0rc1` in Unified rc8, instead of a dev environment

Testing process

Starting actual testing on 2026-05-04
Branch: zppy main branch
Environments: zppy & packages for all tasks -- E3SM Unified 1.13.0rc8

Set up environments

We can mostly skip this step, since all the packages will be tested using the E3SM Unified RC.

cd ~/ez/zppy-interfaces
git status
# # On branch main
# nothing to commit, working directory clean

# REMINDER: On Compy, you must NOT specify the branch, 
# otherwise it won't actually fetch the latest commits!!
git fetch upstream 
git checkout main
git reset --hard upstream/main
git log --oneline | head -n 1
# 6b0758f Bump to 0.2.1rc1 (#48)
# Good, matches https://github.com/E3SM-Project/zppy-interfaces/commits/main

bash # Run bash so we're in an isolated subshell
source ~/.bash_profile
source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc8_compy.sh
# Check zppy-interfaces tests
pytest tests/unit/global_time_series/test_*.py # 10 passed, 2 warnings in 16.56s
# SyntaxWarning: invalid escape sequence '\w'
# SyntaxWarning: invalid escape sequence '\s'
pytest tests/unit/pcmdi_diags/test_*.py # 7 passed in 9.50s

# Stay in the bash subshell

zppy run

# zppy itself #################################################################
cd ~/ez/zppy
git status
# # On branch zppy-debug-pcmdi-20260428
git add -A
git commit -m "Testing" --no-verify # Clean up uncommitted changes

git fetch upstream # Remember, do NOT specify the branch on Compy!
git checkout -b test-zppy-main-plus-unified-1-13rc8 upstream/main
git log --oneline | head -n 2
# 57d82a3 Bump to 3.2.0rc1 (#813)
# 6588842 Merge pull request #812 from E3SM-Project/fixes-of-rc7
# Good, matches https://github.com/E3SM-Project/zppy/commits/main

lfs quota -hu fors729 /compyfs # Have enough space on /compyfs
# Not sure how to check space on /qfs
# (Currently in /qfs/people/fors729/ez/zppy)

# WE ARE ALREADY IN UNIFIED RC8
# DO NOT MAKE A DEV ENV

pixi list python # Check Python version
# python                    3.13.13

# Run zppy unit tests
pytest tests/test_*.py # 44 passed in 1.37s

# Edit tests/integration/utils.py:

TEST_SPECIFICS: Dict[str, Any] = {
    # This is the NCO path.
    # Keep as "" to use the production-version NCO commands.
    # Set to a specific path to use development-version NCO commands.
    "nco_path": "",
    # These are custom environment_commands for specific tasks.
    # Never set these to "", because they will print the line
    # `environment_commands = ""` for the corresponding task,
    # thus overriding the value set higher up in the cfg.
    # That is, there will be no environment set.
    # (`environment_commands = ""` only redirects to Unified
    # if specified under the [default] task)
    "e3sm_to_cmip_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc8_compy.sh",
    "diags_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc8_compy.sh",
    "mpas_analysis_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc8_compy.sh",
    "global_time_series_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc8_compy.sh",
    "livvkit_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc8_compy.sh",
    "pcmdi_diags_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc8_compy.sh",
    # This is the environment setup for other tasks.
    # Leave as "" to use the latest Unified environment.
    "environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc8_compy.sh",
    # For a complete test, run the set of latest cfgs and at least one set of legacy cfgs
    "cfgs_to_run": [
        "weekly_bundles",
        "weekly_comprehensive_v2",
        "weekly_comprehensive_v3",
        # "weekly_legacy_3.1.0_bundles",
        # "weekly_legacy_3.1.0_comprehensive_v2",
        # "weekly_legacy_3.1.0_comprehensive_v3",
        # "weekly_legacy_3.0.0_bundles",
        # "weekly_legacy_3.0.0_comprehensive_v2",
        # "weekly_legacy_3.0.0_comprehensive_v3",
    ],
    "tasks_to_run": [
        "e3sm_diags",
        "mpas_analysis",
        "global_time_series",
        "ilamb",
        "livvkit",
        "pcmdi_diags",
    ],
    "unique_id": "zppy_main_branch_plus_unified_1_13rc8",
}

git diff # Diff looks good
python tests/integration/utils.py
# CFG FILES HAVE BEEN GENERATED FROM TEMPLATES WITH THESE SETTINGS:
# UNIQUE_ID=zppy_main_branch_plus_unified_1_13rc8
# nco_path=
# Reminder: `nco_path=''` => the production-version NCO commands will be used
# e3sm_to_cmip_environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc8_compy.sh
# diags_environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc8_compy.sh
# mpas_analysis_environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc8_compy.sh
# global_time_series_environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc8_compy.sh
# livvkit_environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc8_compy.sh
# pcmdi_diags_environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc8_compy.sh
# environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc8_compy.sh
# Reminder: `environment_commands=''` => the latest E3SM Unified environment will be used

# For reference:
alias sq
# Perlmutter: alias sq='sqa -u forsyth'
# Compy:      alias sq='sqa -u fors729'
alias sqa
# Perlmutter: alias sqa='squeue -o "%8u %.7a %.4D %.9P %8i %.2t %.10r %.10M %.10l %j" --sort=P,-t,-p'
# Compy:      alias sqa='squeue -o "%8u %.7a %.4D %.9P %7i %.2t %.10r %.10M %.10l %.8Q %j" --sort=P,-t,-p'
sq
# No jobs currently queued

# Launch jobs; start with the most important one, comprehensive_v3
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_compy.cfg
zppy -c tests/integration/generated/test_weekly_bundles_compy.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_compy.cfg

# Only running the non-legacy tests on Compy

sq | wc -l # Mon 5/04 15:17 => 86 - header row = 85 jobs

Bundles part 2

# Picking up 2026-05-05

# Re-establish Unified env
bash
source ~/.bash_profile
source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc8_compy.sh

# Check on bundles status
cd /compyfs/fors729/zppy_weekly_bundles_output/zppy_main_branch_plus_unified_1_13rc8/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors

# Now, run bundles part 2
cd ~/ez/zppy
git status
# Good, correct branch: test-zppy-main-plus-unified-1-13rc8
# Good, correct env: Unified rc8

zppy -c tests/integration/generated/test_weekly_bundles_compy.cfg
sq | wc -l # Tue 5/05 09:15 => 3 - header row = 2 jobs

Review finished runs

### v2  ###
cd /compyfs/fors729/zppy_weekly_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc8/v2.LR.historical_0201/post/scripts
grep -v "OK" *status # Good, no errors

### v3 ###
cd /compyfs/fors729/zppy_weekly_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc8/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# pcmdi_diags_synthetic_plots_model_vs_obs.status:ERROR (8)

# Debug:
grep -in error pcmdi_diags_synthetic_plots_model_vs_obs.o*
# 87:2026-05-04 15:54:18,087 [ERROR]: utils.py(find_latest_file_list:48) >> No files found in /compyfs/www/fors729/zppy_weekly_comprehensive_v3_www/zppy_main_branch_plus_unified_1_13rc8/v3.LR.historical_0051/pcmdi_diags/model_vs_obs/metrics_data/mean_climate that match pattern: *.v*.json
# 88:2026-05-04 15:54:18,087 [ERROR]: synthetic_metrics_plotter.py(generate:170) >> Failed to handle metric=mean_climate: No synthetic mean climate metrics found for model: v3.LR.historical_0051
# 98:    raise FileNotFoundError(
# 101:FileNotFoundError: No synthetic mean climate metrics found for model: v3.LR.historical_0051
# 107:2026-05-04 15:54:21,416 [ERROR]: utils.py(find_latest_file_list:48) >> No files found in /compyfs/www/fors729/zppy_weekly_comprehensive_v3_www/zppy_main_branch_plus_unified_1_13rc8/v3.LR.historical_0051/pcmdi_diags/model_vs_obs/metrics_data/variability_modes/*/* that match pattern: var_mode_*.json
# 108:2026-05-04 15:54:21,416 [ERROR]: synthetic_metrics_plotter.py(generate:170) >> Failed to handle metric=variability_modes: No Synthetic MoVs Metrics Data For v3.LR.historical_0051, Aborting.
# 117:    raise FileNotFoundError(
# 120:FileNotFoundError: No Synthetic MoVs Metrics Data For v3.LR.historical_0051, Aborting.
# 129:    raise RuntimeError("No synthetic metrics plots could be generated.")
# 130:RuntimeError: No synthetic metrics plots could be generated.

# This is going to need a deeper dive...

### bundles ###
sq | wc -l # Tue 5/5 => 1 - header row = 0 jobs
cd /compyfs/fors729/zppy_weekly_bundles_output/zppy_main_branch_plus_unified_1_13rc8/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, still no errors after running bundles part 2

post/scripts directory status: pcmdi_diags failed to complete, but it did make it all the way to the synthetic plots; I need to debug the cause.

Python tests

cd ~/ez/zppy
git status
# Good, correct branch: test-zppy-main-plus-unified-1-13rc8
# Good, correct env: Unified rc8
ls tests/integration/test_*.py

# Errors look to be expected based on code changes during this development period:
pytest tests/integration/test_bash_generation.py # 1 failed in 6.33s
pytest tests/integration/test_campaign.py # 6 failed in 11.95s
pytest tests/integration/test_defaults.py # 1 failed in 2.14s
pytest tests/integration/test_bundles.py 
# (Relies on the `zppy` runs we just did)
# 1 failed, 1 passed in 3.15s

# Passing
pytest tests/integration/test_last_year.py
# 1 passed in 1.41s

# We'll now proceed with the image checker test.
# On Chrysalis, we were running `salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm`
# Compute node image tests were introduced in https://github.com/E3SM-Project/zppy/pull/752.
# Referring to notes from the week that merged, it looks like we want to use this line to get a Compy compute note:
# salloc --nodes=1 --partition=short --time=00:30:00 --account=e3sm
salloc --nodes=1 --partition=short --time=01:00:00 --account=e3sm
# Re-establish Unified env
bash
source ~/.bash_profile
source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc8_compy.sh
pytest tests/integration/test_images.py # 1 failed in 1718.01s (0:28:38)
cat test_images_summary.md

Output:

----------------------------------------------------------------------------- Captured stdout call -----------------------------------------------------------------------------
Preparing weekly cfg tests
Preparing legacy 3.1.0 cfg tests
Preparing legacy 3.0.0 cfg tests
Running 3 tests in parallel
Individual test logs will be written to test_<name>.log files
✓ Completed: bundles (3 tasks) (log: test_bundles.log)
✓ Completed: comprehensive_v2 (4 tasks) (log: test_comprehensive_v2.log)
✓ Completed: comprehensive_v3 (5 tasks) (log: test_comprehensive_v3.log)
Copy the output of test_images_summary.md to a Pull Request comment

Test Summary:
Test                                                    Total    Correct     Status
----------------------------------------------------------------------------------
bundles_e3sm_diags                                       1692       1692     ✓ PASS
bundles_global_time_series                                  3          3     ✓ PASS
bundles_ilamb                                             388        388     ✓ PASS
comprehensive_v2_e3sm_diags                              3691       3666     ✗ FAIL
comprehensive_v2_mpas_analysis                            856          0     ✗ FAIL
comprehensive_v2_global_time_series                        12         12     ✓ PASS
comprehensive_v2_ilamb                                    776        776     ✓ PASS
comprehensive_v3_e3sm_diags                              5210       5185     ✗ FAIL
comprehensive_v3_mpas_analysis                            856          0     ✗ FAIL
comprehensive_v3_global_time_series                      1404       1404     ✓ PASS
comprehensive_v3_ilamb                                    837        837     ✓ PASS
comprehensive_v3_pcmdi_diags                              617        335     ✗ FAIL
----------------------------------------------------------------------------------

⚠ Some tests had mismatched or missing images. Check individual log files for details.
=========================================================================== short test summary info ============================================================================
FAILED tests/integration/test_images.py::test_images - assert 3691 == 3666
======================================================================== 1 failed in 1718.01s (0:28:38) ========================================================================

Complete summary table:

Test name	Total images	Correct images	Missing images	Mismatched images
bundles_e3sm_diags	1692	1692	0	0
bundles_global_time_series	3	3	0	0
bundles_ilamb	388	388	0	0
comprehensive_v2_e3sm_diags	3691	3666	0	25 (list, grid)
comprehensive_v2_mpas_analysis	856	0	856 (list)	0
comprehensive_v2_global_time_series	12	12	0	0
comprehensive_v2_ilamb	776	776	0	0
comprehensive_v3_e3sm_diags	5210	5185	0	25 (list, grid)
comprehensive_v3_mpas_analysis	856	0	856 (list)	0
comprehensive_v3_global_time_series	1404	1404	0	0
comprehensive_v3_ilamb	837	837	0	0
comprehensive_v3_pcmdi_diags	617	335	182 (list)	100 (list, grid)

Summary table -- only failing image-check tests

Test name	Total images	Correct images	Missing images	Mismatched images
comprehensive_v2_e3sm_diags	3691	3666	0	25 (list, grid)
comprehensive_v2_mpas_analysis	856	0	856 (list)	0
comprehensive_v3_e3sm_diags	5210	5185	0	25 (list, grid)
comprehensive_v3_mpas_analysis	856	0	856 (list)	0
comprehensive_v3_pcmdi_diags	617	335	182 (list)	100 (list, grid)

We haven't updated expected results since the last test (2026-04-27 Compy E3SM Unified 1.13.0rc7 retest with new fixes in zppy). If we account for expected results that need to be updated, the above table collapses to:

Test name	Total images	Correct images	Missing images	Mismatched images
comprehensive_v3_pcmdi_diags	617	335	182 (list)	100 (list, grid)

The pcmdi_diags results are the only new issues. pcmdi_diags did make it further on this run than that run.

Test name	Total images	Correct images	Missing images	Mismatched images
pcmdi_diags (this test)	617	335	182	100
pcmdi_diags (previous test)	617	486	125	6
Diff	0	-151	+57	+94

Results Analysis

Action items for me:

Debug pcmdi_diags failure in the post/scripts directory
Debug pcmdi_diags failures in the image checking tests

16 replies

forsyth2 May 6, 2026
Maintainer Author

@chengzhuzhang Are we still treating pcmdi_diags as a "beta release"? In the official release notes for E3SM Unified 1.12.0 we put:

Please note we are considering this to be a "beta release" of pcmdi_diags. Please let the developers know if you encounter any issues

Since we still don't have ENSO support in pcmdi_diags, I think it makes sense to continue this as another "beta release". In that case, it's probably fine if we don't have the robustness improvements of #815 for this release.

xylar May 6, 2026
Collaborator

I'm fine either way. Chrysalis still not being available means we're on hold. But I agree with @forsyth2 that we want to test #814 via new zppy and unified RCs first.

forsyth2 May 6, 2026
Maintainer Author

@xylar That makes sense, I'll make a zppy rc2 today so we can get a Unified rc9 out. And after that & time permitting, we can look at including Shixuan's pcmdi_diags robustness improvements.

chengzhuzhang May 6, 2026
Maintainer

@forsyth2 I would still consider pcmdi_diags a beta feature until the ENSO set is merged, and expect we’ll continue improving it as @zhangshixuan1987 introduces further changes and refinements. Since Shixuan is both the primary user and tester, I’m inclined to also include #815. I’ve done a brief review and the updates appear minor, though they will still need testing.

forsyth2 May 6, 2026
Maintainer Author

Thanks @chengzhuzhang. After we test Unified rc9, I'll try to get a test of #815 going as well.

forsyth2 · 2026-05-05T21:05:04Z

forsyth2
May 5, 2026
Maintainer Author

2026-05-04 Perlmutter E3SM Unified 1.13.0rc8

Chrysalis: run tests on all 9 cfgs
Compy: run all 3 non-legacy test cfgs
Perlmutter: run just the non-legacy v3 test cfg

What expected results are currently on Compy?

The latest expected results are still the E3SM Unified 1.12.0 official results.

What changes are we testing?

Package	Changes since last test
NCO	Unified rc8 instead of rc7
e3sm_to_cmip	Unified rc8 instead of rc7
e3sm_diags	Unified rc8 instead of rc7
mpas_analysis	Unified rc8 instead of rc7
zppy-interfaces	Unified rc8 instead of rc7
zppy	NEW Using `zppy v3.2.0rc1` in Unified rc8, instead of a dev environment

Testing process

Starting actual testing on 2026-05-04
Branch: zppy main branch
Environments: zppy & packages for all tasks -- E3SM Unified 1.13.0rc8

Set up environments

We can mostly skip this step, since all the packages will be tested using the E3SM Unified RC.

cd ~/ez/zppy-interfaces
git status
# # On branch main
# nothing to commit, working directory clean
git fetch upstream main
git checkout main
git reset --hard upstream/main
git log --oneline | head -n 1
# 6b0758f Bump to 0.2.1rc1 (#48)
# Good, matches https://github.com/E3SM-Project/zppy-interfaces/commits/main
source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc8_pm-cpu.sh
bash # Run bash so we're in an isolated subshell
source ~/.bash_profile.ext
source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc8_compy.sh
# Check zppy-interfaces tests
pytest tests/unit/global_time_series/test_*.py # 10 passed, 2 warnings in 8.45s
# SyntaxWarning: invalid escape sequence '\w'
# SyntaxWarning: invalid escape sequence '\s'
pytest tests/unit/pcmdi_diags/test_*.py # 7 passed in 4.56s

# Stay in the bash subshell

zppy run

# zppy itself #################################################################
cd ~/ez/zppy
git status
# On branch main
# nothing to commit, working tree clean

git fetch upstream main
git checkout -b test-zppy-main-plus-unified-1-13rc8 upstream/main
git log --oneline | head -n 2
# 57d82a3 Bump to 3.2.0rc1 (#813)
# 6588842 Merge pull request #812 from E3SM-Project/fixes-of-rc7
# Good, matches https://github.com/E3SM-Project/zppy/commits/main

showquota # Have enough space on home and pscratch
# Not sure how to check space on /qfs
# (Currently in /global/homes/f/forsyth/ez/zppy)

# WE ARE ALREADY IN UNIFIED RC8
# DO NOT MAKE A DEV ENV

pixi list python # Check Python version
# python                    3.13.13

# Run zppy unit tests
pytest tests/test_*.py # 44 passed in 0.40s

# Edit tests/integration/utils.py:

TEST_SPECIFICS: Dict[str, Any] = {
    # This is the NCO path.
    # Keep as "" to use the production-version NCO commands.
    # Set to a specific path to use development-version NCO commands.
    "nco_path": "",
    # These are custom environment_commands for specific tasks.
    # Never set these to "", because they will print the line
    # `environment_commands = ""` for the corresponding task,
    # thus overriding the value set higher up in the cfg.
    # That is, there will be no environment set.
    # (`environment_commands = ""` only redirects to Unified
    # if specified under the [default] task)
    "e3sm_to_cmip_environment_commands": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc8_pm-cpu.sh",
    "diags_environment_commands": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc8_pm-cpu.sh",
    "mpas_analysis_environment_commands": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc8_pm-cpu.sh",
    "global_time_series_environment_commands": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc8_pm-cpu.sh",
    "livvkit_environment_commands": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc8_pm-cpu.sh",
    "pcmdi_diags_environment_commands": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc8_pm-cpu.sh",
    # This is the environment setup for other tasks.
    # Leave as "" to use the latest Unified environment.
    "environment_commands": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc8_pm-cpu.sh",
    # For a complete test, run the set of latest cfgs and at least one set of legacy cfgs
    "cfgs_to_run": [
        # "weekly_bundles",
        # "weekly_comprehensive_v2",
        "weekly_comprehensive_v3",
        # "weekly_legacy_3.1.0_bundles",
        # "weekly_legacy_3.1.0_comprehensive_v2",
        # "weekly_legacy_3.1.0_comprehensive_v3",
        # "weekly_legacy_3.0.0_bundles",
        # "weekly_legacy_3.0.0_comprehensive_v2",
        # "weekly_legacy_3.0.0_comprehensive_v3",
    ],
    "tasks_to_run": [
        "e3sm_diags",
        "mpas_analysis",
        "global_time_series",
        "ilamb",
        "livvkit",
        "pcmdi_diags",
    ],
    "unique_id": "zppy_main_branch_plus_unified_1_13rc8",
}

git diff # Diff looks good
python tests/integration/utils.py
# CFG FILES HAVE BEEN GENERATED FROM TEMPLATES WITH THESE SETTINGS:
# UNIQUE_ID=zppy_main_branch_plus_unified_1_13rc8
# nco_path=
# Reminder: `nco_path=''` => the production-version NCO commands will be used
# e3sm_to_cmip_environment_commands=source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc8_pm-cpu.sh
# diags_environment_commands=source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc8_pm-cpu.sh
# mpas_analysis_environment_commands=source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc8_pm-cpu.sh
# global_time_series_environment_commands=source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc8_pm-cpu.sh
# livvkit_environment_commands=source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc8_pm-cpu.sh
# pcmdi_diags_environment_commands=source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc8_pm-cpu.sh
# environment_commands=source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc8_pm-cpu.sh
# Reminder: `environment_commands=''` => the latest E3SM Unified environment will be used

# For reference:
alias sq
# Perlmutter: alias sq='sqa -u forsyth'
# Compy:      alias sq='sqa -u fors729'
alias sqa
# Perlmutter: alias sqa='squeue -o "%8u %.7a %.4D %.9P %8i %.2t %.10r %.10M %.10l %j" --sort=P,-t,-p'
# Compy:      alias sqa='squeue -o "%8u %.7a %.4D %.9P %7i %.2t %.10r %.10M %.10l %.8Q %j" --sort=P,-t,-p'
sq
# No jobs currently queued

# This is the only cfg we're running on Perlmutter:
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_pm-cpu.cfg

sq | wc -l # Mon 5/04 15:27 => 56 - header row = 55 jobs

Review finished runs

# Picking up 2026-05-05

# Re-establish Unified env
bash
source ~/.bash_profile.ext
source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc8_pm-cpu.sh

### v3 ###
cd /global/cfs/cdirs/e3sm/forsyth/zppy_weekly_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc8/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# pcmdi_diags_synthetic_plots_model_vs_obs.status:ERROR (8)

# Debug:
grep -in error pcmdi_diags_synthetic_plots_model_vs_obs.o*
# 91:2026-05-05 10:24:06,047 [ERROR]: utils.py(find_latest_file_list:48) >> No files found in /global/cfs/cdirs/e3sm/www/forsyth/zppy_weekly_comprehensive_v3_www/zppy_main_branch_plus_unified_1_13rc8/v3.LR.historical_0051/pcmdi_diags/model_vs_obs/metrics_data/mean_climate that match pattern: *.v*.json
# 92:2026-05-05 10:24:06,047 [ERROR]: synthetic_metrics_plotter.py(generate:170) >> Failed to handle metric=mean_climate: No synthetic mean climate metrics found for model: v3.LR.historical_0051
# 102:    raise FileNotFoundError(
# 105:FileNotFoundError: No synthetic mean climate metrics found for model: v3.LR.historical_0051
# 111:2026-05-05 10:24:07,285 [ERROR]: utils.py(find_latest_file_list:48) >> No files found in /global/cfs/cdirs/e3sm/www/forsyth/zppy_weekly_comprehensive_v3_www/zppy_main_branch_plus_unified_1_13rc8/v3.LR.historical_0051/pcmdi_diags/model_vs_obs/metrics_data/variability_modes/*/* that match pattern: var_mode_*.json
# 112:2026-05-05 10:24:07,285 [ERROR]: synthetic_metrics_plotter.py(generate:170) >> Failed to handle metric=variability_modes: No Synthetic MoVs Metrics Data For v3.LR.historical_0051, Aborting.
# 121:    raise FileNotFoundError(
# 124:FileNotFoundError: No Synthetic MoVs Metrics Data For v3.LR.historical_0051, Aborting.
# 133:    raise RuntimeError("No synthetic metrics plots could be generated.")
# 134:RuntimeError: No synthetic metrics plots could be generated.

# This is going to need a deeper dive...

post/scripts directory status: pcmdi_diags failed to complete, but it did make it all the way to the synthetic plots; I need to debug the cause. (NOTE: SAME BEHAVIOR AS ON COMPY)

Python tests

cd ~/ez/zppy
git status
# Good, correct branch: test-zppy-main-plus-unified-1-13rc8
# Good, correct env: Unified rc8
ls tests/integration/test_*.py

# Errors look to be expected based on code changes during this development period:
pytest tests/integration/test_bash_generation.py # 1 failed in 1.36s
pytest tests/integration/test_campaign.py # 6 failed in 3.68s
pytest tests/integration/test_defaults.py # 1 failed in 0.73s

# Skipping because we didn't run the bundles cfg:
# pytest tests/integration/test_bundles.py 

# Passing
pytest tests/integration/test_last_year.py
# 1 passed in 0.52s

# On Chrysalis, we were running `salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm`
# Compute node image tests were introduced in https://github.com/E3SM-Project/zppy/pull/752.
# Referring to notes from the week that merged, it looks like we want to use this line to get a Perlmutter compute note:
salloc --nodes=1 --qos=interactive --time=01:00:00 --constraint=cpu --account=e3sm
# Re-establish Unified env
bash
source ~/.bash_profile.ext
source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc8_pm-cpu.sh
pytest tests/integration/test_images.py # 1 failed in 2461.11s (0:41:01)
cat test_images_summary.md

Output:

----------------------------------------------------------------------------- Captured stdout call -----------------------------------------------------------------------------
Preparing weekly cfg tests
Preparing legacy 3.1.0 cfg tests
Preparing legacy 3.0.0 cfg tests
Running 1 tests in parallel
Individual test logs will be written to test_<name>.log files
✓ Completed: comprehensive_v3 (5 tasks) (log: test_comprehensive_v3.log)
Copy the output of test_images_summary.md to a Pull Request comment

Test Summary:
Test                                                    Total    Correct     Status
----------------------------------------------------------------------------------
comprehensive_v3_e3sm_diags                              5210       5185     ✗ FAIL
comprehensive_v3_mpas_analysis                            856          0     ✗ FAIL
comprehensive_v3_global_time_series                      1404       1404     ✓ PASS
comprehensive_v3_ilamb                                    837        837     ✓ PASS
comprehensive_v3_pcmdi_diags                              617        387     ✗ FAIL
----------------------------------------------------------------------------------

⚠ Some tests had mismatched or missing images. Check individual log files for details.
=========================================================================== short test summary info ============================================================================
FAILED tests/integration/test_images.py::test_images - assert 5210 == 5185
======================================================================== 1 failed in 2461.11s (0:41:01) ========================================================================

Complete summary table:

Test name	Total images	Correct images	Missing images	Mismatched images
comprehensive_v3_e3sm_diags	5210	5185	0	25 (list, grid)
comprehensive_v3_mpas_analysis	856	0	856 (list)	0
comprehensive_v3_global_time_series	1404	1404	0	0
comprehensive_v3_ilamb	837	837	0	0
comprehensive_v3_pcmdi_diags	617	387	97 (list)	133 (list, grid)

Summary table -- only failing image-check tests

Test name	Total images	Correct images	Missing images	Mismatched images
comprehensive_v3_e3sm_diags	5210	5185	0	25 (list, grid)
comprehensive_v3_mpas_analysis	856	0	856 (list)	0
comprehensive_v3_pcmdi_diags	617	387	97 (list)	133 (list, grid)

We haven't updated expected results recently on Perlmutter. (See similar errors from the last Compy test (2026-04-27 Compy E3SM Unified 1.13.0rc7 retest with new fixes in zppy)). If we account for expected results that need to be updated, the above table collapses to:

Test name	Total images	Correct images	Missing images	Mismatched images
comprehensive_v3_pcmdi_diags	617	387	97 (list)	133 (list, grid)

The pcmdi_diags results are the only new issues:

Test name	Total images	Correct images	Missing images	Mismatched images
pcmdi_diags (this test, Perlmutter's Unified rc8)	617	387	97	133
pcmdi_diags (Compy's Unified rc8)	617	335	182	100
Diff	0	+52	-85	+33

Interestingly, compared to Compy, we have 85 fewer missing images. It appears 52 of those went to correct images here, whereas 33 went to mismatched images. (That's of course on the assumption no images present on Compy's run were missing on Perlmutter's run).

Results Analysis

Action items for me:

Debug pcmdi_diags failure in the post/scripts directory
Debug pcmdi_diags failures in the image checking tests AND why they differ from Compy's image check test

0 replies

forsyth2 · 2026-05-06T22:49:34Z

forsyth2
May 6, 2026
Maintainer Author

2026-05-04 Compy E3SM Unified 1.13.0rc9

Chrysalis: run tests on all 9 cfgs
Compy: run all 3 non-legacy test cfgs
Perlmutter: run just the non-legacy v3 test cfg

What expected results are currently on Compy?

The latest expected results are still the E3SM Unified 1.12.0 official results.

What changes are we testing?

Package	Changes since last test
NCO	Unified rc9 instead of rc8
e3sm_to_cmip	Unified rc9 instead of rc8
e3sm_diags	Unified rc9 instead of rc8
mpas_analysis	Unified rc9 instead of rc8
zppy-interfaces	Unified rc9 instead of rc8
zppy	Unified rc9 (zppy v3.2.0rc2) instead of rc8 (zppy v3.2.0rc1)

Testing process

Starting actual testing on 2026-05-06
Branch: zppy main branch
Environments: zppy & packages for all tasks -- E3SM Unified 1.13.0rc9

Set up environments

We can mostly skip this step, since all the packages will be tested using the E3SM Unified RC.

cd ~/ez/zppy-interfaces
git status
# # On branch main
# nothing to commit, working directory clean

# REMINDER: On Compy, you must NOT specify the branch, 
# otherwise it won't actually fetch the latest commits!!
git fetch upstream 
git checkout main
git reset --hard upstream/main
git log --oneline | head -n 1
# 6b0758f Bump to 0.2.1rc1 (#48)
# Good, matches https://github.com/E3SM-Project/zppy-interfaces/commits/main

bash # Run bash so we're in an isolated subshell
source ~/.bash_profile
source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc9_compy.sh
# Check zppy-interfaces tests
pytest tests/unit/global_time_series/test_*.py # 10 passed, 2 warnings in 11.67s
# SyntaxWarning: invalid escape sequence '\w'
# SyntaxWarning: invalid escape sequence '\s'
pytest tests/unit/pcmdi_diags/test_*.py # 7 passed in 8.74s

# Stay in the bash subshell

zppy run

# zppy itself #################################################################
cd ~/ez/zppy
git status
# # On branch fixes-of-rc8-try2
git add -A
git commit -m "Testing" --no-verify # Clean up uncommitted changes

git fetch upstream # Remember, do NOT specify the branch on Compy!
git checkout -b test-zppy-main-plus-unified-1-13rc9 upstream/main
git log --oneline | head -n 2
# f368ab7 Bump to 3.2.0rc2 (#816)
# f335f0e Merge pull request #814 from E3SM-Project/fixes-of-rc8
# Good, matches https://github.com/E3SM-Project/zppy/commits/main

lfs quota -hu fors729 /compyfs # Have enough space on /compyfs
# Not sure how to check space on /qfs
# (Currently in /qfs/people/fors729/ez/zppy)

# WE ARE ALREADY IN UNIFIED RC8
# DO NOT MAKE A DEV ENV

pixi list python # Check Python version
# python                    3.13.13

# Run zppy unit tests
pytest tests/test_*.py # 44 passed in 0.84s

# Edit tests/integration/utils.py:

TEST_SPECIFICS: Dict[str, Any] = {
    # This is the NCO path.
    # Keep as "" to use the production-version NCO commands.
    # Set to a specific path to use development-version NCO commands.
    "nco_path": "",
    # These are custom environment_commands for specific tasks.
    # Never set these to "", because they will print the line
    # `environment_commands = ""` for the corresponding task,
    # thus overriding the value set higher up in the cfg.
    # That is, there will be no environment set.
    # (`environment_commands = ""` only redirects to Unified
    # if specified under the [default] task)
    "e3sm_to_cmip_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc9_compy.sh",
    "diags_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc9_compy.sh",
    "mpas_analysis_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc9_compy.sh",
    "global_time_series_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc9_compy.sh",
    "livvkit_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc9_compy.sh",
    "pcmdi_diags_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc9_compy.sh",
    # This is the environment setup for other tasks.
    # Leave as "" to use the latest Unified environment.
    "environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc9_compy.sh",
    # For a complete test, run the set of latest cfgs and at least one set of legacy cfgs
    "cfgs_to_run": [
        "weekly_bundles",
        "weekly_comprehensive_v2",
        "weekly_comprehensive_v3",
        # "weekly_legacy_3.1.0_bundles",
        # "weekly_legacy_3.1.0_comprehensive_v2",
        # "weekly_legacy_3.1.0_comprehensive_v3",
        # "weekly_legacy_3.0.0_bundles",
        # "weekly_legacy_3.0.0_comprehensive_v2",
        # "weekly_legacy_3.0.0_comprehensive_v3",
    ],
    "tasks_to_run": [
        "e3sm_diags",
        "mpas_analysis",
        "global_time_series",
        "ilamb",
        "livvkit",
        "pcmdi_diags",
    ],
    "unique_id": "zppy_main_branch_plus_unified_1_13rc9",
}

git diff # Diff looks good
python tests/integration/utils.py
# CFG FILES HAVE BEEN GENERATED FROM TEMPLATES WITH THESE SETTINGS:
# UNIQUE_ID=zppy_main_branch_plus_unified_1_13rc9
# nco_path=
# Reminder: `nco_path=''` => the production-version NCO commands will be used
# e3sm_to_cmip_environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc9_compy.sh
# diags_environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc9_compy.sh
# mpas_analysis_environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc9_compy.sh
# global_time_series_environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc9_compy.sh
# livvkit_environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc9_compy.sh
# pcmdi_diags_environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc9_compy.sh
# environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc9_compy.sh
# Reminder: `environment_commands=''` => the latest E3SM Unified environment will be used

# For reference:
alias sq
# Perlmutter: alias sq='sqa -u forsyth'
# Compy:      alias sq='sqa -u fors729'
alias sqa
# Perlmutter: alias sqa='squeue -o "%8u %.7a %.4D %.9P %8i %.2t %.10r %.10M %.10l %j" --sort=P,-t,-p'
# Compy:      alias sqa='squeue -o "%8u %.7a %.4D %.9P %7i %.2t %.10r %.10M %.10l %.8Q %j" --sort=P,-t,-p'
sq
# No jobs currently queued

# Launch jobs; start with the most important one, comprehensive_v3
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_compy.cfg
zppy -c tests/integration/generated/test_weekly_bundles_compy.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_compy.cfg

sq | wc -l # Wed 05/06 13:19 => 90 - header row = 89 jobs
sq | wc -l # Wed 05/06 15:19 => 1 - header row = 0 jobs

Bundles part 2

# Check on bundles status
cd /compyfs/fors729/zppy_weekly_bundles_output/zppy_main_branch_plus_unified_1_13rc9/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors

# Now, run bundles part 2
cd ~/ez/zppy
git status
# Good, correct branch: test-zppy-main-plus-unified-1-13rc9
# Good, correct env: Unified rc9

zppy -c tests/integration/generated/test_weekly_bundles_compy.cfg
sq | wc -l # Wed 05/06 15:19 => 3 - header row = 2 jobs

Review finished runs

### v2  ###
cd /compyfs/fors729/zppy_weekly_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc9/v2.LR.historical_0201/post/scripts
grep -v "OK" *status # Good, no errors

### v3 ###
cd /compyfs/fors729/zppy_weekly_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc9/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# pcmdi_diags_synthetic_plots_model_vs_obs.status:ERROR (12)

# See "Debugging" section below

### bundles ###
sq | wc -l # Wed 05/06 15:42 => 1 - header row = 0 jobs
cd /compyfs/fors729/zppy_weekly_bundles_output/zppy_main_branch_plus_unified_1_13rc9/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, still no errors after running bundles part 2

post/scripts directory status: pcmdi_diags failed to complete, which is strange because the test of #814 worked.

Python tests: skipping because we only changed pcmdi_diags code and that is not yet completing successfully.

Debugging

cd /compyfs/fors729/zppy_weekly_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc9/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# pcmdi_diags_synthetic_plots_model_vs_obs.status:ERROR (12)

# Debug:
grep -B 20 -n "ERROR (12)" pcmdi_diags_synthetic_plots_model_vs_obs.bash
# 911-
# 912-
# 913-#add command for enso viewer
# 914-enso_keys="--enso_viewer False --cmip_enso_dir placeholder_dir --cmip_enso_set cmip6.historical.v20210620 --enso_vars psl,pr,prsn,ts,tas,tauu,tauv,hflx,hfss,rlds,rsds,rlus,rlut,rsdt --enso_years 1985-1994"
# 915-
# 916-
# 917-command="zi-pcmdi-synthetic-plots --synthetic_sets portrait,parcoord --figure_format png --www ${www} --results_dir ${results_dir} --case v3.LR.historical_0051 --model_name e3sm.historical.v3-LR.0051 --model_tableID Amon --web_dir=${web_dir} --pcmdi_webtitle E3SM-PMP-Diagnostics --pcmdi_version v3.8.2 --run_type ${run_type} --pcmdi_external_prefix /compyfs/diagnostics --pcmdi_viewer_template pcmdi_data/viewer --save_all_data True ${clim_keys} ${movs_keys} ${enso_keys} --debug ${debug,,}"
# 918-
# 919-
# 920-
# 921-# Run diagnostics
# 922-set -e
# 923-source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc9_compy.sh
# 924-set +e
# 925-
# 926-echo "Running a zi-pcmdi command: ${command}"
# 927-echo "The current directory is: $PWD" # This will be of the form .../post/scripts/tmpDir
# 928-time ${command}
# 929-if [ $? != 0 ]; then
# 930-  cd /compyfs/fors729/zppy_weekly_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc9/v3.LR.historical_0051/post/scripts
# 931:  echo 'ERROR (12)' > pcmdi_diags_synthetic_plots_model_vs_obs.status

grep -in error pcmdi_diags_synthetic_plots_model_vs_obs.o*
# 88:2026-05-06 13:53:31,241 [ERROR]: utils.py(find_latest_file_list:48) >> No files found in /compyfs/www/fors729/zppy_weekly_comprehensive_v3_www/zppy_main_branch_plus_unified_1_13rc9/v3.LR.historical_0051/pcmdi_diags/model_vs_obs/metrics_data/mean_climate that match pattern: *.v*.json
# 89:2026-05-06 13:53:31,241 [ERROR]: synthetic_metrics_plotter.py(generate:170) >> Failed to handle metric=mean_climate: No synthetic mean climate metrics found for model: v3.LR.historical_0051
# 99:    raise FileNotFoundError(
# 102:FileNotFoundError: No synthetic mean climate metrics found for model: v3.LR.historical_0051
# 108:2026-05-06 13:53:33,597 [ERROR]: utils.py(find_latest_file_list:48) >> No files found in /compyfs/www/fors729/zppy_weekly_comprehensive_v3_www/zppy_main_branch_plus_unified_1_13rc9/v3.LR.historical_0051/pcmdi_diags/model_vs_obs/metrics_data/variability_modes/*/* that match pattern: var_mode_*.json
# 109:2026-05-06 13:53:33,597 [ERROR]: synthetic_metrics_plotter.py(generate:170) >> Failed to handle metric=variability_modes: No Synthetic MoVs Metrics Data For v3.LR.historical_0051, Aborting.
# 118:    raise FileNotFoundError(
# 121:FileNotFoundError: No Synthetic MoVs Metrics Data For v3.LR.historical_0051, Aborting.
# 130:    raise RuntimeError("No synthetic metrics plots could be generated.")
# 131:RuntimeError: No synthetic metrics plots could be generated.

grep -l -i error pcmdi_diags_*.o*
# pcmdi_diags_mean_climate_model_vs_obs_1985-1994.o756449
# pcmdi_diags_synthetic_plots_model_vs_obs.o756452
grep -i error pcmdi_diags_mean_climate_model_vs_obs_1985-1994.o756449 | wc -l # 172 matches
grep -i error pcmdi_diags_synthetic_plots_model_vs_obs.o756452 | wc -l # 10 matches

tail -n 50 pcmdi_diags_mean_climate_model_vs_obs_1985-1994.o756449
# FileNotFoundError: [Errno 2] No such file or directory: 'conda'
# Land mask estimated using regionmask method.
# Land mask estimated using regionmask method.
# Land mask estimated using regionmask method.
# Execution failed: Subprocess failed: mean_climate_driver.py -p parameterfile.py --vars pr -r default --case_id v20260506
# successfully finish all jobs....

# This is still appearing. Does pcmdi_metrics need to be updated to use pixi? Dev environments use conda, but the Unified env is using pixi now.

tail -n 50 pcmdi_diags_synthetic_plots_model_vs_obs.o756452
# 2026-05-06 13:53:33,597 [ERROR]: utils.py(find_latest_file_list:48) >> No files found in /compyfs/www/fors729/zppy_weekly_comprehensive_v3_www/zppy_main_branch_plus_unified_1_13rc9/v3.LR.historical_0051/pcmdi_diags/model_vs_obs/metrics_data/variability_modes/*/* that match pattern: var_mode_*.json
# 2026-05-06 13:53:33,597 [ERROR]: synthetic_metrics_plotter.py(generate:170) >> Failed to handle metric=variability_modes: No Synthetic MoVs Metrics Data For v3.LR.historical_0051, Aborting.
# Traceback (most recent call last):
#   File "/qfs/projects/ops/rh7/apps/E3SM/conda_envs/e3smu_1_13_0rc9/compy/pixi/.pixi/envs/default/lib/python3.13/site-packages/zppy_interfaces/pcmdi_diags/synthetic_plots/synthetic_metrics_plotter.py", line 167, in generate
#     handler(metric)
#     ~~~~~~~^^^^^^^^
#   File "/qfs/projects/ops/rh7/apps/E3SM/conda_envs/e3smu_1_13_0rc9/compy/pixi/.pixi/envs/default/lib/python3.13/site-packages/zppy_interfaces/pcmdi_diags/synthetic_plots/synthetic_metrics_plotter.py", line 252, in _handle_variability_modes
#     merge_lib, mode_season_list = reader.collect_metrics()
#                                   ~~~~~~~~~~~~~~~~~~~~~~^^
#   File "/qfs/projects/ops/rh7/apps/E3SM/conda_envs/e3smu_1_13_0rc9/compy/pixi/.pixi/envs/default/lib/python3.13/site-packages/zppy_interfaces/pcmdi_diags/synthetic_plots/movs_metrics_reader.py", line 55, in collect_metrics
#     raise FileNotFoundError(
#         f"No Synthetic MoVs Metrics Data For {model_name}, Aborting."
#     )
# FileNotFoundError: No Synthetic MoVs Metrics Data For v3.LR.historical_0051, Aborting.
# Traceback (most recent call last):
#   File "/qfs/projects/ops/rh7/apps/E3SM/conda_envs/e3smu_1_13_0rc9/compy/pixi/.pixi/envs/default/bin/zi-pcmdi-synthetic-plots", line 8, in <module>
#     sys.exit(main())
#              ~~~~^^
#   File "/qfs/projects/ops/rh7/apps/E3SM/conda_envs/e3smu_1_13_0rc9/compy/pixi/.pixi/envs/default/lib/python3.13/site-packages/zppy_interfaces/pcmdi_diags/pcmdi_synthetic_plots.py", line 147, in main
#     plotter.generate()
#     ~~~~~~~~~~~~~~~~^^
#   File "/qfs/projects/ops/rh7/apps/E3SM/conda_envs/e3smu_1_13_0rc9/compy/pixi/.pixi/envs/default/lib/python3.13/site-packages/zppy_interfaces/pcmdi_diags/synthetic_plots/synthetic_metrics_plotter.py", line 174, in generate
#     raise RuntimeError("No synthetic metrics plots could be generated.")
# RuntimeError: No synthetic metrics plots could be generated.

# Check if conda is available in the unified env
which conda  # likely missing or points to pixi shim
# /usr/bin/which: no conda in (/qfs/projects/ops/rh7/apps/E3SM/conda_envs/e3smu_1_13_0rc9/compy/pixi_login/.pixi/envs/default/bin:/share/apps/modules/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/ganglia/bin:/opt/ganglia/sbin:/opt/pdsh/bin:/opt/rocks/bin:/opt/rocks/sbin:/qfs/people/fors729/.local/bin:/qfs/people/fors729/bin:/qfs/people/fors729/.local/bin:/qfs/people/fors729/bin)

# Check if pcmdi_metrics calls conda internally
grep -r "conda" ~/ez/pcmdi_metrics --include="*.py" | grep -v "#"
# /qfs/people/fors729/ez/pcmdi_metrics/pcmdi_metrics/graphics/parallel_coordinate_plot/parallel_coordinate_plot_lib.py:        Secondary list of model names for alternate grouping.
# /qfs/people/fors729/ez/pcmdi_metrics/pcmdi_metrics/io/base.py:def _determine_conda():
# /qfs/people/fors729/ez/pcmdi_metrics/pcmdi_metrics/io/base.py:    Attempt to determine the path to the conda executable
# /qfs/people/fors729/ez/pcmdi_metrics/pcmdi_metrics/io/base.py:    override_conda_exe = os.environ.get("PCMDI_CONDA_EXE")
# /qfs/people/fors729/ez/pcmdi_metrics/pcmdi_metrics/io/base.py:    if override_conda_exe:
# /qfs/people/fors729/ez/pcmdi_metrics/pcmdi_metrics/io/base.py:        return override_conda_exe
# /qfs/people/fors729/ez/pcmdi_metrics/pcmdi_metrics/io/base.py:    conda_python_exe = os.environ.get("CONDA_PYTHON_EXE", "")
# /qfs/people/fors729/ez/pcmdi_metrics/pcmdi_metrics/io/base.py:    if conda_python_exe != "":
# /qfs/people/fors729/ez/pcmdi_metrics/pcmdi_metrics/io/base.py:        conda_exe = os.path.join(os.path.dirname(conda_python_exe), "conda")
# /qfs/people/fors729/ez/pcmdi_metrics/pcmdi_metrics/io/base.py:        conda_exe = "conda"
# /qfs/people/fors729/ez/pcmdi_metrics/pcmdi_metrics/io/base.py:    return conda_exe
# /qfs/people/fors729/ez/pcmdi_metrics/pcmdi_metrics/io/base.py:CONDA = _determine_conda()
# /qfs/people/fors729/ez/pcmdi_metrics/pcmdi_metrics/io/base.py:    prov["conda"] = OrderedDict()
# /qfs/people/fors729/ez/pcmdi_metrics/pcmdi_metrics/io/base.py:        "Version": "conda version ",
# /qfs/people/fors729/ez/pcmdi_metrics/pcmdi_metrics/io/base.py:        "IsPrivate": "conda is private ",
# /qfs/people/fors729/ez/pcmdi_metrics/pcmdi_metrics/io/base.py:        "envVersion": "conda-env version ",
# /qfs/people/fors729/ez/pcmdi_metrics/pcmdi_metrics/io/base.py:        "buildVersion": "conda-build version ",
# /qfs/people/fors729/ez/pcmdi_metrics/pcmdi_metrics/io/base.py:    populate_prov(prov["conda"], CONDA + " info", pairs, sep=":", index=-1)
# /qfs/people/fors729/ez/pcmdi_metrics/pcmdi_metrics/io/base.py:    prov["conda"]["yaml"] = o.decode("utf-8")
# /qfs/people/fors729/ez/pcmdi_metrics/pcmdi_metrics/io/base.py:            if "conda" in out_dict["provenance"]:
# /qfs/people/fors729/ez/pcmdi_metrics/pcmdi_metrics/io/base.py:                if "yaml" in out_dict["provenance"]["conda"]:
# /qfs/people/fors729/ez/pcmdi_metrics/pcmdi_metrics/io/base.py:                        out_dict["YAML"] = out_dict["provenance"]["conda"]["yaml"]
# /qfs/people/fors729/ez/pcmdi_metrics/pcmdi_metrics/io/base.py:                    del out_dict["provenance"]["conda"]["yaml"]
# /qfs/people/fors729/ez/pcmdi_metrics/pcmdi_metrics/variability_mode/eof_classification/eof_classification.py:            "Install it with: pip install scikit-learn (or conda install scikit-learn). "
# /qfs/people/fors729/ez/pcmdi_metrics/tests/test_resources.py:def test_conda_env(tmpdir):
# /qfs/people/fors729/ez/pcmdi_metrics/tests/test_resources.py:    conda_prefix = os.path.join(tmpdir, "conda")
# /qfs/people/fors729/ez/pcmdi_metrics/tests/test_resources.py:    pmp_share_path = os.path.join(conda_prefix, "share", "pmp")
# /qfs/people/fors729/ez/pcmdi_metrics/tests/test_resources.py:    with mock.patch.dict(os.environ, {"CONDA_PREFIX": conda_prefix}):
# /qfs/people/fors729/ez/pcmdi_metrics/tests/test_resources.py:    assert path == os.path.join(tmpdir, "conda", "share", "pmp")
# /qfs/people/fors729/ez/pcmdi_metrics/tests/test_resources.py:def test_conda_env_no_exist(distribution, getcwd, tmpdir):
# /qfs/people/fors729/ez/pcmdi_metrics/tests/test_resources.py:    conda_prefix = os.path.join(tmpdir, "conda")
# /qfs/people/fors729/ez/pcmdi_metrics/tests/test_resources.py:    with mock.patch.dict(os.environ, {"CONDA_PREFIX": conda_prefix}):

# Look at the full mean_climate log around the conda error
grep -B 40 -A 5 "FileNotFoundError.*conda" pcmdi_diags_mean_climate_model_vs_obs_1985-1994.o756449
# WARNING: There was an error initializing an OpenFabrics device.

#   Local host:   n0001
#   Local device: hfi1_0
# --------------------------------------------------------------------------
# Traceback (most recent call last):
#   File "/qfs/projects/ops/rh7/apps/E3SM/conda_envs/e3smu_1_13_0rc9/compy/pixi/.pixi/envs/default/bin/mean_climate_driver.py", line 530, in <module>
#     mean_climate_metrics_to_json(
#     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
#         metrics_output_path, json_filename, result_dict, cmec_flag=cmec, debug=debug
#         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#     )
#     ^
#   File "/qfs/projects/ops/rh7/apps/E3SM/conda_envs/e3smu_1_13_0rc9/compy/pixi/.pixi/envs/default/lib/python3.13/site-packages/pcmdi_metrics/mean_climate/lib/mean_climate_metrics_to_json.py", line 50, in mean_climate_metrics_to_json
#     JSON.write(
#     ~~~~~~~~~~^
#         json_dict,
#         ^^^^^^^^^^
#     ...<11 lines>...
#         sort_keys=False,
#         ^^^^^^^^^^^^^^^^
#     )
#     ^
#   File "/qfs/projects/ops/rh7/apps/E3SM/conda_envs/e3smu_1_13_0rc9/compy/pixi/.pixi/envs/default/lib/python3.13/site-packages/pcmdi_metrics/io/base.py", line 321, in write
#     out_dict = OrderedDict({"provenance": generateProvenance()})
#                                           ~~~~~~~~~~~~~~~~~~^^
#   File "/qfs/projects/ops/rh7/apps/E3SM/conda_envs/e3smu_1_13_0rc9/compy/pixi/.pixi/envs/default/lib/python3.13/site-packages/pcmdi_metrics/io/base.py", line 169, in generateProvenance
#     p = Popen(shlex.split(CONDA + " env export"), stdout=PIPE, stderr=PIPE)
#   File "/qfs/projects/ops/rh7/apps/E3SM/conda_envs/e3smu_1_13_0rc9/compy/pixi/.pixi/envs/default/lib/python3.13/subprocess.py", line 1039, in __init__
#     self._execute_child(args, executable, preexec_fn, close_fds,
#     ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#                         pass_fds, cwd, env,
#                         ^^^^^^^^^^^^^^^^^^^
#     ...<5 lines>...
#                         gid, gids, uid, umask,
#                         ^^^^^^^^^^^^^^^^^^^^^^
#                         start_new_session, process_group)
#                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#   File "/qfs/projects/ops/rh7/apps/E3SM/conda_envs/e3smu_1_13_0rc9/compy/pixi/.pixi/envs/default/lib/python3.13/subprocess.py", line 1991, in _execute_child
#     raise child_exception_type(errno_num, err_msg, err_filename)
# FileNotFoundError: [Errno 2] No such file or directory: 'conda'
# Land mask estimated using regionmask method.
# Land mask estimated using regionmask method.
# Land mask estimated using regionmask method.
# Execution failed: Subprocess failed: mean_climate_driver.py -p parameterfile.py --vars pr -r default --case_id v20260506
# successfully finish all jobs....

I did see the conda issue on the Unified rc8 results, but AI interepreted that more as a red herring. Here though, it seems we have proof it is in fact causing problems, especially considering the dev environment tests worked for #814.

AI-generated report:

generateProvenance() crashes with FileNotFoundError in pixi-based environments

When running pcmdi_metrics inside an environment that uses pixi rather than conda (e.g. E3SM Unified ≥1.13), mean_climate_driver.py crashes at JSON write time because generateProvenance() calls conda env export via subprocess.Popen, and conda is not on the PATH in pixi environments. The metrics are computed successfully but the output JSON is never written, silently breaking downstream tasks.

Interestingly, populate_prov() already guards its Popen call with a broad try/except, but the conda env export call just below it does not — so this is an easy fix.

Suggested fix (pcmdi_metrics/io/base.py):

-    # Actual environement used
-    p = Popen(shlex.split(CONDA + " env export"), stdout=PIPE, stderr=PIPE)
-    o, e = p.communicate()
-    prov["conda"]["yaml"] = o.decode("utf-8")
+    # Actual environement used
+    try:
+        p = Popen(shlex.split(CONDA + " env export"), stdout=PIPE, stderr=PIPE)
+        o, e = p.communicate()
+        prov["conda"]["yaml"] = o.decode("utf-8")
+    except FileNotFoundError:
+        prov["conda"]["yaml"] = f"conda not available (executable not found: {CONDA})"

Workaround until a fix is released: set PCMDI_CONDA_EXE to a no-op script that exits 0. This suppresses the error at the cost of empty provenance metadata:

echo -e '#!/bin/bash\nexit 0' > /tmp/fake_conda && chmod +x /tmp/fake_conda
export PCMDI_CONDA_EXE=/tmp/fake_conda

Traceback:

File ".../pcmdi_metrics/io/base.py", line 169, in generateProvenance
    p = Popen(shlex.split(CONDA + " env export"), stdout=PIPE, stderr=PIPE)
FileNotFoundError: [Errno 2] No such file or directory: 'conda'

4 replies

forsyth2 May 6, 2026
Maintainer Author

@zhangshixuan1987 @xylar It appears that there are in fact some conda/pixi compatibility issues with pcmdi_metrics. I had noticed these during rc8, but they didn't seem to be a root cause. Here though, they seem to be the major issue, especially considering the dev environment test worked fine.

xylar May 7, 2026
Collaborator

That's a problem to fix in pcmdi_metrics. I'll try to help.

xylar May 7, 2026
Collaborator

I believe this should be fixed in PCMDI/pcmdi_metrics#1374.

forsyth2 May 7, 2026
Maintainer Author

Thanks @xylar!

forsyth2 · 2026-05-08T17:48:22Z

forsyth2
May 8, 2026
Maintainer Author

2026-05-04 Perlmutter E3SM Unified 1.13.0rc9 (zppy v3.2.0rc2)

Note: E3SM Unified 1.13.0rc10 is now available, but this test was launched earlier in the week, when rc9 was the most recent.

Chrysalis: run tests on all 9 cfgs
Compy: run all 3 non-legacy test cfgs
Perlmutter: run just the non-legacy v3 test cfg

What expected results are currently on Compy?

The latest expected results are still the E3SM Unified 1.12.0 official results.

What changes are we testing?

Package	Changes since last test
NCO	Unified rc9 instead of rc8
e3sm_to_cmip	Unified rc9 instead of rc8
e3sm_diags	Unified rc9 instead of rc8
mpas_analysis	Unified rc9 instead of rc8
zppy-interfaces	Unified rc9 instead of rc8
zppy	Unified rc9 (zppy v3.2.0rc2) instead of rc8 (zppy v3.2.0rc1)

Testing process

Starting actual testing on 2026-05-06
Branch: zppy main branch
Environments: zppy & packages for all tasks -- E3SM Unified 1.13.0rc9 (zppy v3.2.0rc2)

Set up environments

We can mostly skip this step, since all the packages will be tested using the E3SM Unified RC.

cd ~/ez/zppy-interfaces
git status
# # On branch main
# nothing to commit, working directory clean
git fetch upstream main
git checkout main
git reset --hard upstream/main
git log --oneline | head -n 1
# 6b0758f Bump to 0.2.1rc1 (#48)
# Good, matches https://github.com/E3SM-Project/zppy-interfaces/commits/main
bash # Run bash so we're in an isolated subshell
source ~/.bash_profile.ext
source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc9_pm-cpu.sh
# Check zppy-interfaces tests
pytest tests/unit/global_time_series/test_*.py # 10 passed, 2 warnings in 10.90s
# SyntaxWarning: invalid escape sequence '\w'
# SyntaxWarning: invalid escape sequence '\s'
pytest tests/unit/pcmdi_diags/test_*.py # 7 passed in 4.64s

# Stay in the bash subshell

zppy run

# zppy itself #################################################################
cd ~/ez/zppy
git status
# On branch main
# nothing to commit, working tree clean

git fetch upstream main
git checkout -b test-zppy-main-plus-unified-1-13rc9 upstream/main
git log --oneline | head -n 2
# f368ab71 Bump to 3.2.0rc2 (#816)
# f335f0e3 Merge pull request #814 from E3SM-Project/fixes-of-rc8
# Good, matches https://github.com/E3SM-Project/zppy/commits/main

showquota # Have enough space on home and pscratch
# Not sure how to check space on /qfs
# (Currently in /global/homes/f/forsyth/ez/zppy)

# WE ARE ALREADY IN UNIFIED RC8
# DO NOT MAKE A DEV ENV

pixi list python # Check Python version
# python                    3.13.13

# Run zppy unit tests
pytest tests/test_*.py # 44 passed in 0.25s

# Edit tests/integration/utils.py:

TEST_SPECIFICS: Dict[str, Any] = {
    # This is the NCO path.
    # Keep as "" to use the production-version NCO commands.
    # Set to a specific path to use development-version NCO commands.
    "nco_path": "",
    # These are custom environment_commands for specific tasks.
    # Never set these to "", because they will print the line
    # `environment_commands = ""` for the corresponding task,
    # thus overriding the value set higher up in the cfg.
    # That is, there will be no environment set.
    # (`environment_commands = ""` only redirects to Unified
    # if specified under the [default] task)
    "e3sm_to_cmip_environment_commands": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc9_pm-cpu.sh",
    "diags_environment_commands": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc9_pm-cpu.sh",
    "mpas_analysis_environment_commands": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc9_pm-cpu.sh",
    "global_time_series_environment_commands": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc9_pm-cpu.sh",
    "livvkit_environment_commands": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc9_pm-cpu.sh",
    "pcmdi_diags_environment_commands": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc9_pm-cpu.sh",
    # This is the environment setup for other tasks.
    # Leave as "" to use the latest Unified environment.
    "environment_commands": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc9_pm-cpu.sh",
    # For a complete test, run the set of latest cfgs and at least one set of legacy cfgs
    "cfgs_to_run": [
        # "weekly_bundles",
        # "weekly_comprehensive_v2",
        "weekly_comprehensive_v3",
        # "weekly_legacy_3.1.0_bundles",
        # "weekly_legacy_3.1.0_comprehensive_v2",
        # "weekly_legacy_3.1.0_comprehensive_v3",
        # "weekly_legacy_3.0.0_bundles",
        # "weekly_legacy_3.0.0_comprehensive_v2",
        # "weekly_legacy_3.0.0_comprehensive_v3",
    ],
    "tasks_to_run": [
        "e3sm_diags",
        "mpas_analysis",
        "global_time_series",
        "ilamb",
        "livvkit",
        "pcmdi_diags",
    ],
    "unique_id": "zppy_main_branch_plus_unified_1_13rc9",
}

git diff # Diff looks good
python tests/integration/utils.py
# CFG FILES HAVE BEEN GENERATED FROM TEMPLATES WITH THESE SETTINGS:
# UNIQUE_ID=zppy_main_branch_plus_unified_1_13rc9
# nco_path=
# Reminder: `nco_path=''` => the production-version NCO commands will be used
# e3sm_to_cmip_environment_commands=source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc9_pm-cpu.sh
# diags_environment_commands=source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc9_pm-cpu.sh
# mpas_analysis_environment_commands=source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc9_pm-cpu.sh
# global_time_series_environment_commands=source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc9_pm-cpu.sh
# livvkit_environment_commands=source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc9_pm-cpu.sh
# pcmdi_diags_environment_commands=source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc9_pm-cpu.sh
# environment_commands=source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc9_pm-cpu.sh
# Reminder: `environment_commands=''` => the latest E3SM Unified environment will be used

# For reference:
alias sq
# Perlmutter: alias sq='sqa -u forsyth'
# Compy:      alias sq='sqa -u fors729'
alias sqa
# Perlmutter: alias sqa='squeue -o "%8u %.7a %.4D %.9P %8i %.2t %.10r %.10M %.10l %j" --sort=P,-t,-p'
# Compy:      alias sqa='squeue -o "%8u %.7a %.4D %.9P %7i %.2t %.10r %.10M %.10l %.8Q %j" --sort=P,-t,-p'
sq
# No jobs currently queued

# This is the only cfg we're running on Perlmutter:
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_pm-cpu.cfg

sq | wc -l # Wed 05/06 13:02 => 56 - header row = 55 jobs
# HERE

Review finished runs

# Picking up 2026-05-08
sq | wc -l # Fri 05/08 10:39 => 1 - header row = 0 jobs

# Re-establish Unified env
bash
source ~/.bash_profile.ext
source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc9_pm-cpu.sh

### v3 ###
cd /global/cfs/cdirs/e3sm/forsyth/zppy_weekly_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc9/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# pcmdi_diags_synthetic_plots_model_vs_obs.status:ERROR (12)

# Let's check that this matches the error we saw on Compy
grep -l -i error pcmdi_diags_*.o*
# pcmdi_diags_mean_climate_model_vs_obs_1985-1994.o52577264
# pcmdi_diags_synthetic_plots_model_vs_obs.o52577271
grep -l "FileNotFoundError: No Synthetic MoVs Metrics Data For v3.LR.historical_0051" pcmdi_diags_*.o*
# pcmdi_diags_synthetic_plots_model_vs_obs.o52577271
grep -l "FileNotFoundError.*conda" pcmdi_diags_*.o*
# pcmdi_diags_mean_climate_model_vs_obs_1985-1994.o52577264

# Yes, this matches up.

post/scripts directory status: pcmdi_diags failed to complete, which matches up with what we saw on the Compy test, so no new issues to report.

Python tests: skipping because we only changed pcmdi_diags code and that is not yet completing successfully.

0 replies

zhangshixuan1987 · 2026-05-08T18:31:56Z

zhangshixuan1987
May 8, 2026
Collaborator

@forsyth2 @xylar Hi Ryan and Xylar, I was not aware of the issues reported here. Is there anything I can help with?

2 replies

forsyth2 May 8, 2026
Maintainer Author

Hi @zhangshixuan1987, I think @xylar has resolved the issue with PCMDI/pcmdi_metrics#1374. I'm currently running tests now on E3SM Unified 1.13.0rc10 to confirm.

zhangshixuan1987 May 8, 2026
Collaborator

@forsyth2 : Great! please feel free to let me know if any actions on my side is needed or helpful! Thanks!

forsyth2 · 2026-05-09T01:08:09Z

forsyth2
May 9, 2026
Maintainer Author

2026-05-08 Compy E3SM Unified 1.13.0rc10 (zppy v3.2.0rc2)

Chrysalis: run tests on all 9 cfgs
Compy: run all 3 non-legacy test cfgs
Perlmutter: run just the non-legacy v3 test cfg

What expected results are currently on Compy?

The latest expected results are still the E3SM Unified 1.12.0 official results.

What changes are we testing?

Package	Changes since last test
NCO	Unified rc10 instead of rc9
e3sm_to_cmip	Unified rc10 instead of rc9
e3sm_diags	Unified rc10 instead of rc9
mpas_analysis	Unified rc10 instead of rc9
zppy-interfaces	Unified rc10 instead of rc9
zppy	Unified rc10 instead of rc9 (in both RCs: zppy v3.2.0rc2)

Testing process

Starting actual testing on 2026-05-08
Branch: zppy main branch
Environments: zppy & packages for all tasks -- E3SM Unified 1.13.0rc10

Set up environments

We can mostly skip this step, since all the packages will be tested using the E3SM Unified RC.

cd ~/ez/zppy-interfaces
git status
# # On branch main
# nothing to commit, working directory clean

# REMINDER: On Compy, you must NOT specify the branch, 
# otherwise it won't actually fetch the latest commits!!
git fetch upstream 
git checkout main
git reset --hard upstream/main
git log --oneline | head -n 1
# 6b0758f Bump to 0.2.1rc1 (#48)
# Good, matches https://github.com/E3SM-Project/zppy-interfaces/commits/main

bash # Run bash so we're in an isolated subshell
source ~/.bash_profile
source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc10_compy.sh
# Check zppy-interfaces tests
pytest tests/unit/global_time_series/test_*.py # 10 passed, 2 warnings in 14.60s
# SyntaxWarning: invalid escape sequence '\w'
# SyntaxWarning: invalid escape sequence '\s'
pytest tests/unit/pcmdi_diags/test_*.py # 7 passed in 8.84s

# Stay in the bash subshell

zppy run

# zppy itself #################################################################
cd ~/ez/zppy
git status
# # On branch test-zppy-main-plus-unified-1-13rc9
git add -A
git commit -m "Testing" --no-verify # Clean up uncommitted changes

git fetch upstream # Remember, do NOT specify the branch on Compy!
git checkout -b test-zppy-main-plus-unified-1-13rc10 upstream/main
git log --oneline | head -n 2
# f368ab7 Bump to 3.2.0rc2 (#816)
# f335f0e Merge pull request #814 from E3SM-Project/fixes-of-rc8
# Good, matches https://github.com/E3SM-Project/zppy/commits/main

lfs quota -hu fors729 /compyfs # Have enough space on /compyfs
# Not sure how to check space on /qfs
# (Currently in /qfs/people/fors729/ez/zppy)

# WE ARE ALREADY IN UNIFIED RC10
# DO NOT MAKE A DEV ENV

pixi list python # Check Python version
# python                    3.13.13

# Run zppy unit tests
pytest tests/test_*.py # 44 passed in 0.53s

# Edit tests/integration/utils.py:

TEST_SPECIFICS: Dict[str, Any] = {
    # This is the NCO path.
    # Keep as "" to use the production-version NCO commands.
    # Set to a specific path to use development-version NCO commands.
    "nco_path": "",
    # These are custom environment_commands for specific tasks.
    # Never set these to "", because they will print the line
    # `environment_commands = ""` for the corresponding task,
    # thus overriding the value set higher up in the cfg.
    # That is, there will be no environment set.
    # (`environment_commands = ""` only redirects to Unified
    # if specified under the [default] task)
    "e3sm_to_cmip_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc10_compy.sh",
    "diags_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc10_compy.sh",
    "mpas_analysis_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc10_compy.sh",
    "global_time_series_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc10_compy.sh",
    "livvkit_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc10_compy.sh",
    "pcmdi_diags_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc10_compy.sh",
    # This is the environment setup for other tasks.
    # Leave as "" to use the latest Unified environment.
    "environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc10_compy.sh",
    # For a complete test, run the set of latest cfgs and at least one set of legacy cfgs
    "cfgs_to_run": [
        "weekly_bundles",
        "weekly_comprehensive_v2",
        "weekly_comprehensive_v3",
        # "weekly_legacy_3.1.0_bundles",
        # "weekly_legacy_3.1.0_comprehensive_v2",
        # "weekly_legacy_3.1.0_comprehensive_v3",
        # "weekly_legacy_3.0.0_bundles",
        # "weekly_legacy_3.0.0_comprehensive_v2",
        # "weekly_legacy_3.0.0_comprehensive_v3",
    ],
    "tasks_to_run": [
        "e3sm_diags",
        "mpas_analysis",
        "global_time_series",
        "ilamb",
        "livvkit",
        "pcmdi_diags",
    ],
    "unique_id": "zppy_main_branch_plus_unified_1_13rc10",
}

git diff # Diff looks good
python tests/integration/utils.py
# CFG FILES HAVE BEEN GENERATED FROM TEMPLATES WITH THESE SETTINGS:
# UNIQUE_ID=zppy_main_branch_plus_unified_1_13rc10
# nco_path=
# Reminder: `nco_path=''` => the production-version NCO commands will be used
# e3sm_to_cmip_environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc10_compy.sh
# diags_environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc10_compy.sh
# mpas_analysis_environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc10_compy.sh
# global_time_series_environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc10_compy.sh
# livvkit_environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc10_compy.sh
# pcmdi_diags_environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc10_compy.sh
# environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc10_compy.sh
# Reminder: `environment_commands=''` => the latest E3SM Unified environment will be used

# For reference:
alias sq
# Perlmutter: alias sq='sqa -u forsyth'
# Compy:      alias sq='sqa -u fors729'
alias sqa
# Perlmutter: alias sqa='squeue -o "%8u %.7a %.4D %.9P %8i %.2t %.10r %.10M %.10l %j" --sort=P,-t,-p'
# Compy:      alias sqa='squeue -o "%8u %.7a %.4D %.9P %7i %.2t %.10r %.10M %.10l %.8Q %j" --sort=P,-t,-p'
sq
# No jobs currently queued

# Launch jobs; start with the most important one, comprehensive_v3
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_compy.cfg
zppy -c tests/integration/generated/test_weekly_bundles_compy.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_compy.cfg

sq | wc -l # Fri 05/08 => 59 - header row = 58 jobs
sq | wc -l # Fri 05/08 11:06 => 44 - header row = 43 jobs
sq | wc -l # Fri 05/08 12:31 => 1 - header row = 0 jobs

Bundles part 2

# Check on bundles status
cd /compyfs/fors729/zppy_weekly_bundles_output/zppy_main_branch_plus_unified_1_13rc10/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors

# Now, run bundles part 2
cd ~/ez/zppy
git status
# Good, correct branch: test-zppy-main-plus-unified-1-13rc10
# Good, correct env: Unified rc10

zppy -c tests/integration/generated/test_weekly_bundles_compy.cfg
sq | wc -l # Fri 05/08 12:34 => 3 - header row = 2 jobs

Review finished runs

### v2  ###
cd /compyfs/fors729/zppy_weekly_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10/v2.LR.historical_0201/post/scripts
grep -v "OK" *status # Good, no errors

### v3 ###
cd /compyfs/fors729/zppy_weekly_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc10/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors

### bundles ###
sq | wc -l # Fri 05/08 12:35 => 3 - header row = 2 jobs
sq | wc -l # Fri 05/08 13:31 => 1 - header row = 0 jobs
cd /compyfs/fors729/zppy_weekly_bundles_output/zppy_main_branch_plus_unified_1_13rc10/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, still no errors after running bundles part 2

post/scripts directory status: NO ERRORS

Python tests

cd ~/ez/zppy
git status
# Good, correct branch: test-zppy-main-plus-unified-1-13rc10
# Good, correct env: Unified rc10
ls tests/integration/test_*.py

# Errors look to be expected based on code changes during this development period:
pytest tests/integration/test_bash_generation.py # 1 failed in 6.76s
pytest tests/integration/test_campaign.py # 6 failed in 11.95s
pytest tests/integration/test_defaults.py # 1 failed in 1.80s
pytest tests/integration/test_bundles.py 
# (Relies on the `zppy` runs we just did)
# 1 failed, 1 passed in 0.68s

# Passing
pytest tests/integration/test_last_year.py
# 1 passed in 1.42s

# We'll now proceed with the image checker test.
# On Chrysalis, we were running `salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm`
# Compute node image tests were introduced in https://github.com/E3SM-Project/zppy/pull/752.
# Referring to notes from the week that merged, it looks like we want to use this line to get a Compy compute note:
# salloc --nodes=1 --partition=short --time=00:30:00 --account=e3sm
salloc --nodes=1 --partition=short --time=01:00:00 --account=e3sm
# Re-establish Unified env
bash
source ~/.bash_profile
source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc10_compy.sh
pytest tests/integration/test_images.py # 1 failed in 3175.89s (0:52:55)
cat test_images_summary.md

Output:

------------------------------------------------------- Captured stdout call -------------------------------------------------------
Preparing weekly cfg tests
Preparing legacy 3.1.0 cfg tests
Preparing legacy 3.0.0 cfg tests
Running 3 tests in parallel
Individual test logs will be written to test_<name>.log files
✓ Completed: bundles (3 tasks) (log: test_bundles.log)
✓ Completed: comprehensive_v2 (4 tasks) (log: test_comprehensive_v2.log)
✓ Completed: comprehensive_v3 (5 tasks) (log: test_comprehensive_v3.log)
Copy the output of test_images_summary.md to a Pull Request comment

Test Summary:
Test                                                    Total    Correct     Status
----------------------------------------------------------------------------------
bundles_e3sm_diags                                       1692       1692     ✓ PASS
bundles_global_time_series                                  3          3     ✓ PASS
bundles_ilamb                                             388        388     ✓ PASS
comprehensive_v2_e3sm_diags                              3691       3666     ✗ FAIL
comprehensive_v2_mpas_analysis                            856          0     ✗ FAIL
comprehensive_v2_global_time_series                        12         12     ✓ PASS
comprehensive_v2_ilamb                                    776        776     ✓ PASS
comprehensive_v3_e3sm_diags                              5210       5185     ✗ FAIL
comprehensive_v3_mpas_analysis                            856          0     ✗ FAIL
comprehensive_v3_global_time_series                      1404       1404     ✓ PASS
comprehensive_v3_ilamb                                    837        837     ✓ PASS
comprehensive_v3_pcmdi_diags                              617        400     ✗ FAIL
----------------------------------------------------------------------------------

⚠ Some tests had mismatched or missing images. Check individual log files for details.
===================================================== short test summary info ======================================================
FAILED tests/integration/test_images.py::test_images - assert 3691 == 3666
================================================== 1 failed in 3175.89s (0:52:55) ==================================================

Complete summary table:

Test name	Total images	Correct images	Missing images	Mismatched images
bundles_e3sm_diags	1692	1692	0	0
bundles_global_time_series	3	3	0	0
bundles_ilamb	388	388	0	0
comprehensive_v2_e3sm_diags	3691	3666	0	25 (list, grid)
comprehensive_v2_mpas_analysis	856	0	856 (list)	0
comprehensive_v2_global_time_series	12	12	0	0
comprehensive_v2_ilamb	776	776	0	0
comprehensive_v3_e3sm_diags	5210	5185	0	25 (list, grid)
comprehensive_v3_mpas_analysis	856	0	856 (list)	0
comprehensive_v3_global_time_series	1404	1404	0	0
comprehensive_v3_ilamb	837	837	0	0
comprehensive_v3_pcmdi_diags	617	400	0	217 (list, grid)

Summary table -- only failing image-check tests

Test name	Total images	Correct images	Missing images	Mismatched images
comprehensive_v2_e3sm_diags	3691	3666	0	25 (list, grid)
comprehensive_v2_mpas_analysis	856	0	856 (list)	0
comprehensive_v3_e3sm_diags	5210	5185	0	25 (list, grid)
comprehensive_v3_mpas_analysis	856	0	856 (list)	0
comprehensive_v3_pcmdi_diags	617	400	0	217 (list, grid)

Results Analysis

The 25 mismatched images of the e3sm_diags tasks and the 856 missing images of the mpas_analysis tasks are consistent with the 2026-05-04 Compy E3SM Unified 1.13.0rc8 results.

The pcmdi_diags numbers do differ from those results, but they are consistent with the 2026-05-05 Compy pcmdi_diags focused test, try 2 results.

Since all errors noted here are consistent with what we would expect, we are now able to update the expected results on Compy.

Updating expected results

Setting up new expected results

We last set the expected results to the official results of E3SM Unified 1.12.0 here, on 2026-03-31.

ls -lt /compyfs/www/zppy_test_resources/
# The files are from Dec. 1
# This is consistent with how we updated the expected results on 2026-03-31.
ls -lt /compyfs/www/fors729/zppy_test_resources_previous
# Mar 31 16:45 expected_results_until_20260331
# Dec  1 14:45 expected_results_for_unified_1.12.0
# Nov 18 09:16 expected_results_until_20250429_from_unified_1.11.1

# First, let's copy over the old expected results #############################
cd /compyfs/www/

# Unlike on Perlmutter, we don't have write-access to this level of the directory tree on Compy.
# That means we can't simply run:
# mv zppy_test_resources/ zppy_test_resources_previous/expected_results_until_20260508
# Because then zppy_test_resources would be gone

# Backup the current contents somewhere writable under the existing tree
mkdir -p fors729/zppy_test_resources_previous/expected_results_until_20260508

# Move everything out of zppy_test_resources/, including hidden files
# From ChatGPT:
# * = "all visible files"
# dotglob = "actually, include hidden ones too"
# nullglob = "and don’t break if there are none"
shopt -s dotglob nullglob
mv zppy_test_resources/* fors729/zppy_test_resources_previous/expected_results_until_20260508/
shopt -u dotglob nullglob
ls -lt fors729/zppy_test_resources_previous/
# Now present: expected_results_until_20260508

# Second, update the expected results #########################################

# Let's update the simpler tests' results first:
cd ~/ez/zppy
git status
# Good, correct branch: test-zppy-main-plus-unified-1-13rc10
# Good, correct env: Unified rc10

ls tests/integration/generated/update_*_compy.sh
# tests/integration/generated/update_archive_expected_files_compy.sh
# tests/integration/generated/update_bash_generation_expected_files_compy.sh
# tests/integration/generated/update_campaign_expected_files_compy.sh
# tests/integration/generated/update_defaults_expected_files_compy.sh
# tests/integration/generated/update_weekly_expected_files_compy.sh

chmod 755 tests/integration/generated/update_bash_generation_expected_files_compy.sh
./tests/integration/generated/update_bash_generation_expected_files_compy.sh
# 1 passed in 2.19s

chmod 755 tests/integration/generated/update_campaign_expected_files_compy.sh
./tests/integration/generated/update_campaign_expected_files_compy.sh
# 6 passed in 7.50s

chmod 755 tests/integration/generated/update_defaults_expected_files_compy.sh
./tests/integration/generated/update_defaults_expected_files_compy.sh
1 passed in 2.02s

# test_last_year doesn't need an update script because its cfg is stand-alone:
pytest -vv tests/integration/test_last_year.py
# 1 passed in 1.32s

# Now update the expected files for the major tests:

# NOTE: `cp -a fors729/zppy_test_resources_previous/expected_results_for_unified_1.12.0/. zppy_test_resources/`
# took FOUR HOURS to complete, according to our note in the 2026-04-08 Compy test:
# https://github.com/E3SM-Project/zppy/discussions/802?sort=new#discussioncomment-16508424

# To make sure exiting the SSH connection doesn't halt the progress,
# we can use a compute node or screen.
# For reference:
scontrol show partition short | grep MaxTime
# MaxTime=02:00:00
scontrol show partition slurm | grep MaxTime
# MaxTime=4-00:00:00

# Let's go with `screen`:
screen -ls # No Sockets found
screen
screen -ls # 392427.pts-2.compy-e    (Attached)
# Re-establish env:
bash # Run bash so we're in an isolated subshell
source ~/.bash_profile
source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc10_compy.sh
chmod 755 tests/integration/generated/update_weekly_expected_files_compy.sh
./tests/integration/generated/update_weekly_expected_files_compy.sh
# CTRL A D
screen -ls # 392427.pts-2.compy-e	(Detached)

screen -R
# cp: cannot stat ‘/compyfs/www/fors729/zppy_weekly_legacy_3.1.0_comprehensive_v2_www/zppy_main_branch_plus_unified_1_13rc10/v2.LR.historical_0201’: No such file or directory
# ./tests/integration/generated/update_weekly_expected_files_compy.sh: line 56: cd: /compyfs/www/zppy_test_resources/expected_legacy_3.1.0_comprehensive_v2: No such file or directory
# cp: cannot stat ‘/compyfs/www/fors729/zppy_weekly_legacy_3.1.0_comprehensive_v3_www/zppy_main_branch_plus_unified_1_13rc10/v3.LR.historical_0051’: No such file or directory
# ./tests/integration/generated/update_weekly_expected_files_compy.sh: line 56: cd: /compyfs/www/zppy_test_resources/expected_legacy_3.1.0_comprehensive_v3: No such file or directory
# cp: cannot stat ‘/compyfs/www/fors729/zppy_weekly_legacy_3.1.0_bundles_www/zppy_main_branch_plus_unified_1_13rc10/v3.LR.historical_0051’: No such file or directory
# ./tests/integration/generated/update_weekly_expected_files_compy.sh: line 56: cd: /compyfs/www/zppy_test_resources/expected_legacy_3.1.0_bundles: No such file or directory
# cp: cannot stat ‘/compyfs/www/fors729/zppy_weekly_legacy_3.0.0_comprehensive_v2_www/zppy_main_branch_plus_unified_1_13rc10/v2.LR.historical_0201’: No such file or directory
# ./tests/integration/generated/update_weekly_expected_files_compy.sh: line 56: cd: /compyfs/www/zppy_test_resources/expected_legacy_3.0.0_comprehensive_v2: No such file or directory
# cp: cannot stat ‘/compyfs/www/fors729/zppy_weekly_legacy_3.0.0_comprehensive_v3_www/zppy_main_branch_plus_unified_1_13rc10/v3.LR.historical_0051’: No such file or directory
# ./tests/integration/generated/update_weekly_expected_files_compy.sh: line 56: cd: /compyfs/www/zppy_test_resources/expected_legacy_3.0.0_comprehensive_v3: No such file or directory
# cp: cannot stat ‘/compyfs/www/fors729/zppy_weekly_legacy_3.0.0_bundles_www/zppy_main_branch_plus_unified_1_13rc10/v3.LR.historical_0051’: No such file or directory
# ./tests/integration/generated/update_weekly_expected_files_compy.sh: line 56: cd: /compyfs/www/zppy_test_resources/expected_legacy_3.0.0_bundles: No such file or directory


# Yes, this does makes sense, because we didn't run any of the legacy tests.

exit # Exit bash subshell
exit # Exit screen
cd /compyfs/www/zppy_test_resources
ls -lt 
# Everything says updated 5/8 except for expected_bash_files, which says 4/9
# There's also no expected files for the legacy tests
# Since we plan to only run the non-legacy tests on Compy, that seems fine.

cd ~/ez/zppy
pytest tests/integration/test_bundles.py
# 2 passed in 0.64s

salloc --nodes=1 --partition=short --time=01:00:00 --account=e3sm
# Re-establish Unified env
bash
source ~/.bash_profile
source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc10_compy.sh
pytest tests/integration/test_images.py # 1 passed in 2527.18s (0:42:07)
cat test_images_summary.md

Complete summary table:

Test name	Total images	Correct images
bundles_e3sm_diags	1762	1762
bundles_global_time_series	3	3
bundles_ilamb	388	388
comprehensive_v2_e3sm_diags	3806	3806
comprehensive_v2_mpas_analysis	856	856
comprehensive_v2_global_time_series	12	12
comprehensive_v2_ilamb	776	776
comprehensive_v3_e3sm_diags	5369	5369
comprehensive_v3_mpas_analysis	1280	1280
comprehensive_v3_global_time_series	1404	1404
comprehensive_v3_ilamb	837	837
comprehensive_v3_pcmdi_diags	647	647

1 reply

forsyth2 May 9, 2026
Maintainer Author

✅ E3SM Unified 1.13.0rc10 passes on Compy.

forsyth2 · 2026-05-09T01:14:53Z

forsyth2
May 9, 2026
Maintainer Author

2026-05-08 Chrysalis E3SM Unified 1.13.0rc10 (zppy v3.2.0rc2)

Chrysalis: run tests on all 9 cfgs
Compy: run all 3 non-legacy test cfgs
Perlmutter: run just the non-legacy v3 test cfg

What expected results are currently on Compy?

The latest expected results for Compy & Perlmutter are still the E3SM Unified 1.12.0 official results. Chrysalis, however, has had its expected results updated much more recently, on 2026-03-30. We can confirm with:

cd /lcrc/group/e3sm/public_html/zppy_test_resources
ls -lt # Latest updates are from 3/30
cd /lcrc/group/e3sm/public_html/zppy_test_resources_previous
ls -lt # Latest update is from 3/30:
# expected_results_until_20260330
# These were the expected results until we updated them on 2026-03-30.

What changes are we testing?

Package	Changes since 3/30 (excluding version bump PRs)
NCO	Unclear
e3sm_to_cmip	E3SM-Project/e3sm_to_cmip#323, E3SM-Project/e3sm_to_cmip#344
e3sm_diags	E3SM-Project/e3sm_diags#1042, E3SM-Project/e3sm_diags#1031, E3SM-Project/e3sm_diags#1046, E3SM-Project/e3sm_diags#1045
mpas_analysis	MPAS-Dev/MPAS-Analysis#1139, MPAS-Dev/MPAS-Analysis#1141, MPAS-Dev/MPAS-Analysis#1142, MPAS-Dev/MPAS-Analysis#1143, MPAS-Dev/MPAS-Analysis#1144, MPAS-Dev/MPAS-Analysis#1146, MPAS-Dev/MPAS-Analysis#1147
zppy-interfaces	None
zppy	#788, #801, #804, #806, #810, #805, #812, #814

Testing process

Starting actual testing on 2026-05-08
Branch: zppy main branch
Environments: zppy & packages for all tasks -- E3SM Unified 1.13.0rc10

Set up environments

We can mostly skip this step, since all the packages will be tested using the E3SM Unified RC.

cd ~/ez/zppy-interfaces
git status
# On branch main
# nothing to commit, working directory clean

git fetch upstream main
git checkout main
git reset --hard upstream/main
git log --oneline | head -n 1
# 6b0758f Bump to 0.2.1rc1 (#48)
# Good, matches https://github.com/E3SM-Project/zppy-interfaces/commits/main

bash # Run bash so we're in an isolated subshell
source ~/.bashrc
source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh
# Check zppy-interfaces tests
pytest tests/unit/global_time_series/test_*.py # 10 passed, 2 warnings in 34.78s
# SyntaxWarning: invalid escape sequence '\w'
# SyntaxWarning: invalid escape sequence '\s'
pytest tests/unit/pcmdi_diags/test_*.py # 7 passed in 30.59s

# Stay in the bash subshell

zppy run

# zppy itself #################################################################
cd ~/ez/zppy
git status
# On branch main
# nothing to commit, working tree clean

git fetch upstream main
git checkout -b test-zppy-main-plus-unified-1-13rc10 upstream/main
git log --oneline | head -n 2
# f368ab7 Bump to 3.2.0rc2 (#816)
# f335f0e Merge pull request #814 from E3SM-Project/fixes-of-rc8
# Good, matches https://github.com/E3SM-Project/zppy/commits/main

/usr/lpp/mmfs/bin/mmlsquota -u ac.forsyth2 --block-size T fs2 # Have enough space on /lcrc/group/e3sm/
lcrc-quota # Have enough space on /home/ac.forsyth2/

# WE ARE ALREADY IN UNIFIED RC10
# DO NOT MAKE A DEV ENV

pixi list python # Check Python version
# python                    3.13.13

# Run zppy unit tests
pytest tests/test_*.py # 44 passed in 0.91s

# Edit tests/integration/utils.py:

TEST_SPECIFICS: Dict[str, Any] = {
    # This is the NCO path.
    # Keep as "" to use the production-version NCO commands.
    # Set to a specific path to use development-version NCO commands.
    "nco_path": "",
    # These are custom environment_commands for specific tasks.
    # Never set these to "", because they will print the line
    # `environment_commands = ""` for the corresponding task,
    # thus overriding the value set higher up in the cfg.
    # That is, there will be no environment set.
    # (`environment_commands = ""` only redirects to Unified
    # if specified under the [default] task)
    "e3sm_to_cmip_environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh",
    "diags_environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh",
    "mpas_analysis_environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh",
    "global_time_series_environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh",
    "livvkit_environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh",
    "pcmdi_diags_environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh",
    # This is the environment setup for other tasks.
    # Leave as "" to use the latest Unified environment.
    "environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh",
    # For a complete test, run the set of latest cfgs and at least one set of legacy cfgs
    "cfgs_to_run": [
        "weekly_bundles",
        "weekly_comprehensive_v2",
        "weekly_comprehensive_v3",
        "weekly_legacy_3.1.0_bundles",
        "weekly_legacy_3.1.0_comprehensive_v2",
        "weekly_legacy_3.1.0_comprehensive_v3",
        "weekly_legacy_3.0.0_bundles",
        "weekly_legacy_3.0.0_comprehensive_v2",
        "weekly_legacy_3.0.0_comprehensive_v3",
    ],
    "tasks_to_run": [
        "e3sm_diags",
        "mpas_analysis",
        "global_time_series",
        "ilamb",
        "livvkit",
        "pcmdi_diags",
    ],
    "unique_id": "zppy_main_branch_plus_unified_1_13rc10_try1",
}

git diff # Diff looks good
python tests/integration/utils.py
# CFG FILES HAVE BEEN GENERATED FROM TEMPLATES WITH THESE SETTINGS:
# UNIQUE_ID=zppy_main_branch_plus_unified_1_13rc10_try1
# nco_path=
# Reminder: `nco_path=''` => the production-version NCO commands will be used
# e3sm_to_cmip_environment_commands=source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh
# diags_environment_commands=source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh
# mpas_analysis_environment_commands=source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh
# global_time_series_environment_commands=source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh
# livvkit_environment_commands=source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh
# pcmdi_diags_environment_commands=source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh
# environment_commands=source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh
# Reminder: `environment_commands=''` => the latest E3SM Unified environment will be used

# For reference:
alias sq
# Perlmutter: alias sq='sqa -u forsyth'
# Compy:      alias sq='sqa -u fors729'
# Chrysalis:  alias sq='sqa -u ac.forsyth2'
alias sqa
# Perlmutter: alias sqa='squeue -o "%8u %.7a %.4D %.9P %8i %.2t %.10r %.10M %.10l %j" --sort=P,-t,-p'
# Compy:      alias sqa='squeue -o "%8u %.7a %.4D %.9P %7i %.2t %.10r %.10M %.10l %.8Q %j" --sort=P,-t,-p'
# Chrysalis:  alias sqa='squeue -o "%8u %.7a %.4D %.9P %7i %.2t %.10r %.10M %.10l %j" --sort=P,-t,-p'
sq
# No jobs currently queued
ls tests/integration/generated/test_weekly_*_chrysalis.cfg
# tests/integration/generated/test_weekly_bundles_chrysalis.cfg
# tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
# tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg

# tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg
# tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v2_chrysalis.cfg
# tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v3_chrysalis.cfg

# tests/integration/generated/test_weekly_legacy_3.1.0_bundles_chrysalis.cfg
# tests/integration/generated/test_weekly_legacy_3.1.0_comprehensive_v2_chrysalis.cfg
# tests/integration/generated/test_weekly_legacy_3.1.0_comprehensive_v3_chrysalis.cfg

# Launch jobs; start with the most important one, comprehensive_v3
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg

zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_comprehensive_v3_chrysalis.cfg

zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v3_chrysalis.cfg


sq | wc -l # Fri 05/08 13:19 => 233 - header row = 232 jobs
sq | wc -l # Fri 05/08 15:10 => 1 - header row = 0 jobs

Bundles part 2

# Check on bundles status
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/zppy_main_branch_plus_unified_1_13rc10_try1/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_bundles_output/zppy_main_branch_plus_unified_1_13rc10_try1/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/zppy_main_branch_plus_unified_1_13rc10_try1/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors

# Now, run bundles part 2
cd ~/ez/zppy
git status
# Good, correct branch: test-zppy-main-plus-unified-1-13rc10
# Good, correct env: Unified rc9

zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg
sq | wc -l # Fri 05/08 15:13 => 7 - header row = 6 jobs

Review finished runs

grep -n "output =" tests/integration/generated/test_weekly_*_chrysalis.cfg
# tests/integration/generated/test_weekly_bundles_chrysalis.cfg:36:output = "/lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/zppy_main_branch_plus_unified_1_13rc10_try1/v3.LR.historical_0051"
# tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg:11:output = "/lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_try1/v2.LR.historical_0201"
# tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg:13:output = "/lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc10_try1/v3.LR.historical_0051"

# tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg:38:output = "/lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/zppy_main_branch_plus_unified_1_13rc10_try1/v3.LR.historical_0051"
# tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v2_chrysalis.cfg:13:output = "/lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_try1/v2.LR.historical_0201"
# tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v3_chrysalis.cfg:20:output = "/lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc10_try1/v3.LR.historical_0051"

# tests/integration/generated/test_weekly_legacy_3.1.0_bundles_chrysalis.cfg:38:output = "/lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_bundles_output/zppy_main_branch_plus_unified_1_13rc10_try1/v3.LR.historical_0051"
# tests/integration/generated/test_weekly_legacy_3.1.0_comprehensive_v2_chrysalis.cfg:13:output = "/lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_try1/v2.LR.historical_0201"
# tests/integration/generated/test_weekly_legacy_3.1.0_comprehensive_v3_chrysalis.cfg:15:output = "/lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc10_try1/v3.LR.historical_0051"

### v2  ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_try1/v2.LR.historical_0201/post/scripts
grep -v "OK" *status # Good, no errors

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_try1/v2.LR.historical_0201/post/scripts
grep -v "OK" *status # Good, no errors

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_try1/v2.LR.historical_0201/post/scripts
grep -v "OK" *status # Good, no errors

### v3 ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc10_try1/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc10_try1/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc10_try1/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors

### bundles ###
sq | wc -l # Fri 05/08 15:21 => 1 - header row = 0 jobs

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/zppy_main_branch_plus_unified_1_13rc10_try1/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, still no errors

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_bundles_output/zppy_main_branch_plus_unified_1_13rc10_try1/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, still no errors

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/zppy_main_branch_plus_unified_1_13rc10_try1/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, still no errors

post/scripts directory status: NO ERRORS

Python tests

cd ~/ez/zppy
git status
# Good, correct branch: test-zppy-main-plus-unified-1-13rc10
# Good, correct env: Unified rc8
ls tests/integration/test_*.py

# Errors look to be expected based on code changes since 3/30:
pytest tests/integration/test_bash_generation.py # 1 failed in 2.15s
pytest tests/integration/test_campaign.py # 6 failed in 3.14s
pytest tests/integration/test_defaults.py # 1 failed in 1.02s
pytest tests/integration/test_bundles.py # 1 failed, 1 passed in 0.60s 

# Passing
pytest tests/integration/test_last_year.py # 1 passed in 0.68s

# Compute node image tests were introduced in https://github.com/E3SM-Project/zppy/pull/752.
salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm
# Re-establish Unified env
bash
source ~/.bashrc
source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh
pytest tests/integration/test_images.py # 1 failed in 391.53s (0:06:31)
# FAILED tests/integration/test_images.py::test_images - PIL.UnidentifiedImageError: cannot identify image file '/lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy...
# 1 failed in 2461.11s (0:41:01)
cat early_test_images_summary.md

Output:

>               raise self._exception
E               PIL.UnidentifiedImageError: cannot identify image file '/lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_legacy_3.1.0_bundles_www/zppy_main_branch_plus_unified_1_13rc10_try1/v3.LR.historical_0051/e3sm_diags/atm_monthly_180x360_aave/model_vs_obs_1985-1986/diurnal_cycle/TRMM-3B43v-7_3hr/TRMM-3B43v-7_3hr-PRECT-DJF-global.png'

/lcrc/soft/climate/e3sm-unified/e3smu_1_13_0rc10/chrysalis/pixi/.pixi/envs/default/lib/python3.13/concurrent/futures/_base.py:401: UnidentifiedImageError
--------------------------------------------------- Captured stdout call ---------------------------------------------------
Preparing weekly cfg tests
Preparing legacy 3.1.0 cfg tests
Preparing legacy 3.0.0 cfg tests
Running 9 tests in parallel
Individual test logs will be written to test_<name>.log files
✓ Completed: bundles (3 tasks) (log: test_bundles.log)
✗ Test legacy_3.1.0_bundles generated an exception: cannot identify image file '/lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_legacy_3.1.0_bundles_www/zppy_main_branch_plus_unified_1_13rc10_try1/v3.LR.historical_0051/e3sm_diags/atm_monthly_180x360_aave/model_vs_obs_1985-1986/diurnal_cycle/TRMM-3B43v-7_3hr/TRMM-3B43v-7_3hr-PRECT-DJF-global.png'
Copy the output of early_test_images_summary.md to a Pull Request comment
Copy the output of early_test_images_summary.md to a Pull Request comment
================================================= short test summary info ==================================================
FAILED tests/integration/test_images.py::test_images - PIL.UnidentifiedImageError: cannot identify image file '/lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy...
============================================== 1 failed in 391.53s (0:06:31) ===============================================

Test name	Total images	Correct images	Missing images
bundles_e3sm_diags	1762	657	1105 (list)
bundles_global_time_series	3	3	0
bundles_ilamb	388	388	0

Let's try re-launching the test.

exit # exit bash subshell
exit # exit interactive node
git status
git add -A
git commit -m "Testing" --no-verify

# Apply a diff to tests/integration/image_checker.py
git diff
# diff --git a/tests/integration/image_checker.py b/tests/integration/image_checker.py
# index 9327a459..0344722d 100644
# --- a/tests/integration/image_checker.py
# +++ b/tests/integration/image_checker.py
# @@ -256,6 +256,10 @@ def _compare_actual_and_expected(
#      except FileNotFoundError:
#          missing_images.append(image_name)
#          return
# +    except Exception as e:
# +        print(f"Warning: could not open actual image {path_to_actual_png}: {e}")
# +        missing_images.append(image_name)
# +        return
#      expected_png = Image.open(path_to_expected_png).convert("RGB")
#      diff = ImageChops.difference(actual_png, expected_png)
git add -A
git commit -m "Fix image checker" --no-verify # Unified env doesn't have our pre-commit checks

salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm
# Re-establish Unified env
bash
source ~/.bashrc
source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh
pytest tests/integration/test_images.py # 1 failed in 400.20s (0:06:40)
cat test_images_summary.md

Output:

--------------------------------------------------- Captured stdout call ---------------------------------------------------
Preparing weekly cfg tests
Preparing legacy 3.1.0 cfg tests
Preparing legacy 3.0.0 cfg tests
Running 9 tests in parallel
Individual test logs will be written to test_<name>.log files
✓ Completed: bundles (3 tasks) (log: test_bundles.log)
✓ Completed: legacy_3.1.0_bundles (3 tasks) (log: test_legacy_3.1.0_bundles.log)
✓ Completed: comprehensive_v2 (4 tasks) (log: test_comprehensive_v2.log)
✓ Completed: legacy_3.1.0_comprehensive_v2 (4 tasks) (log: test_legacy_3.1.0_comprehensive_v2.log)
✓ Completed: legacy_3.0.0_comprehensive_v2 (4 tasks) (log: test_legacy_3.0.0_comprehensive_v2.log)
✓ Completed: legacy_3.0.0_bundles (3 tasks) (log: test_legacy_3.0.0_bundles.log)
✓ Completed: legacy_3.1.0_comprehensive_v3 (4 tasks) (log: test_legacy_3.1.0_comprehensive_v3.log)
✓ Completed: legacy_3.0.0_comprehensive_v3 (4 tasks) (log: test_legacy_3.0.0_comprehensive_v3.log)
✓ Completed: comprehensive_v3 (5 tasks) (log: test_comprehensive_v3.log)
Copy the output of test_images_summary.md to a Pull Request comment

Test Summary:
Test                                                    Total    Correct     Status
----------------------------------------------------------------------------------
bundles_e3sm_diags                                       1762        657     ✗ FAIL
bundles_global_time_series                                  3          3     ✓ PASS
bundles_ilamb                                             388        388     ✓ PASS
comprehensive_v2_e3sm_diags                              3791        937     ✗ FAIL
comprehensive_v2_mpas_analysis                            856          0     ✗ FAIL
comprehensive_v2_global_time_series                        12         12     ✓ PASS
comprehensive_v2_ilamb                                    776        776     ✓ PASS
comprehensive_v3_e3sm_diags                              5354       1797     ✗ FAIL
comprehensive_v3_mpas_analysis                            856          0     ✗ FAIL
comprehensive_v3_global_time_series                      1404       1404     ✓ PASS
comprehensive_v3_ilamb                                    837        837     ✓ PASS
comprehensive_v3_pcmdi_diags                              647        400     ✗ FAIL
legacy_3.1.0_bundles_e3sm_diags                          1762        656     ✗ FAIL
legacy_3.1.0_bundles_global_time_series                     3          3     ✓ PASS
legacy_3.1.0_bundles_ilamb                                388        388     ✓ PASS
legacy_3.1.0_comprehensive_v2_e3sm_diags                 3791        936     ✗ FAIL
legacy_3.1.0_comprehensive_v2_mpas_analysis               856          0     ✗ FAIL
legacy_3.1.0_comprehensive_v2_global_time_series           12         12     ✓ PASS
legacy_3.1.0_comprehensive_v2_ilamb                       776        776     ✓ PASS
legacy_3.1.0_comprehensive_v3_e3sm_diags                 5350       1797     ✗ FAIL
legacy_3.1.0_comprehensive_v3_mpas_analysis               856          0     ✗ FAIL
legacy_3.1.0_comprehensive_v3_global_time_series         1404       1404     ✓ PASS
legacy_3.1.0_comprehensive_v3_ilamb                       837        837     ✓ PASS
legacy_3.0.0_bundles_e3sm_diags                          1762        655     ✗ FAIL
legacy_3.0.0_bundles_global_time_series                     3          3     ✓ PASS
legacy_3.0.0_bundles_ilamb                                388        388     ✓ PASS
legacy_3.0.0_comprehensive_v2_e3sm_diags                 3791        937     ✗ FAIL
legacy_3.0.0_comprehensive_v2_mpas_analysis               856          0     ✗ FAIL
legacy_3.0.0_comprehensive_v2_global_time_series           12         12     ✓ PASS
legacy_3.0.0_comprehensive_v2_ilamb                       776        776     ✓ PASS
legacy_3.0.0_comprehensive_v3_e3sm_diags                 5350       1797     ✗ FAIL
legacy_3.0.0_comprehensive_v3_mpas_analysis               856          0     ✗ FAIL
legacy_3.0.0_comprehensive_v3_global_time_series           90         90     ✓ PASS
legacy_3.0.0_comprehensive_v3_ilamb                       837        837     ✓ PASS
----------------------------------------------------------------------------------

⚠ Some tests had mismatched or missing images. Check individual log files for details.
================================================= short test summary info ==================================================
FAILED tests/integration/test_images.py::test_images - assert 1762 == 657
============================================== 1 failed in 400.20s (0:06:40) ===============================================

Complete summary table:

Test name	Total images	Correct images	Missing images	Mismatched images
bundles_e3sm_diags	1762	657	1105 (list)	0
bundles_global_time_series	3	3	0	0
bundles_ilamb	388	388	0	0
comprehensive_v2_e3sm_diags	3791	937	2854 (list)	0
comprehensive_v2_mpas_analysis	856	0	856 (list)	0
comprehensive_v2_global_time_series	12	12	0	0
comprehensive_v2_ilamb	776	776	0	0
comprehensive_v3_e3sm_diags	5354	1797	3557 (list)	0
comprehensive_v3_mpas_analysis	856	0	856 (list)	0
comprehensive_v3_global_time_series	1404	1404	0	0
comprehensive_v3_ilamb	837	837	0	0
comprehensive_v3_pcmdi_diags	647	400	0	247 (list, grid)
legacy_3.1.0_bundles_e3sm_diags	1762	656	1106 (list)	0
legacy_3.1.0_bundles_global_time_series	3	3	0	0
legacy_3.1.0_bundles_ilamb	388	388	0	0
legacy_3.1.0_comprehensive_v2_e3sm_diags	3791	936	2855 (list)	0
legacy_3.1.0_comprehensive_v2_mpas_analysis	856	0	856 (list)	0
legacy_3.1.0_comprehensive_v2_global_time_series	12	12	0	0
legacy_3.1.0_comprehensive_v2_ilamb	776	776	0	0
legacy_3.1.0_comprehensive_v3_e3sm_diags	5350	1797	3553 (list)	0
legacy_3.1.0_comprehensive_v3_mpas_analysis	856	0	856 (list)	0
legacy_3.1.0_comprehensive_v3_global_time_series	1404	1404	0	0
legacy_3.1.0_comprehensive_v3_ilamb	837	837	0	0
legacy_3.0.0_bundles_e3sm_diags	1762	655	1107 (list)	0
legacy_3.0.0_bundles_global_time_series	3	3	0	0
legacy_3.0.0_bundles_ilamb	388	388	0	0
legacy_3.0.0_comprehensive_v2_e3sm_diags	3791	937	2854 (list)	0
legacy_3.0.0_comprehensive_v2_mpas_analysis	856	0	856 (list)	0
legacy_3.0.0_comprehensive_v2_global_time_series	12	12	0	0
legacy_3.0.0_comprehensive_v2_ilamb	776	776	0	0
legacy_3.0.0_comprehensive_v3_e3sm_diags	5350	1797	3553 (list)	0
legacy_3.0.0_comprehensive_v3_mpas_analysis	856	0	856 (list)	0
legacy_3.0.0_comprehensive_v3_global_time_series	90	90	0	0
legacy_3.0.0_comprehensive_v3_ilamb	837	837	0	0

Summary table -- only failing image-check tests

The image checking test initially encountered a broken file, so I had to make a change to the testing code. The diff is only a few lines; I'll include it in a PR.

Test name	Total images	Correct images	Missing images	Mismatched images
bundles_e3sm_diags	1762	657	1105 (list)	0
comprehensive_v2_e3sm_diags	3791	937	2854 (list)	0
comprehensive_v2_mpas_analysis	856	0	856 (list)	0
comprehensive_v3_e3sm_diags	5354	1797	3557 (list)	0
comprehensive_v3_mpas_analysis	856	0	856 (list)	0
comprehensive_v3_pcmdi_diags	647	400	0	247 (list, grid)
legacy_3.1.0_bundles_e3sm_diags	1762	656	1106 (list)	0
legacy_3.1.0_comprehensive_v2_e3sm_diags	3791	936	2855 (list)	0
legacy_3.1.0_comprehensive_v2_mpas_analysis	856	0	856 (list)	0
legacy_3.1.0_comprehensive_v3_e3sm_diags	5350	1797	3553 (list)	0
legacy_3.1.0_comprehensive_v3_mpas_analysis	856	0	856 (list)	0
legacy_3.0.0_bundles_e3sm_diags	1762	655	1107 (list)	0
legacy_3.0.0_comprehensive_v2_e3sm_diags	3791	937	2854 (list)	0
legacy_3.0.0_comprehensive_v2_mpas_analysis	856	0	856 (list)	0
legacy_3.0.0_comprehensive_v3_e3sm_diags	5350	1797	3553 (list)	0
legacy_3.0.0_comprehensive_v3_mpas_analysis	856	0	856 (list)	0

Let's organize this by task:

pcmdi_diags

Test name	Total images	Correct images	Missing images	Mismatched images
comprehensive_v3_pcmdi_diags	647	400	0	247 (list, grid)

Let's compare this with what we saw for pcmdi_diags on 2026-05-08 Compy E3SM Unified 1.13.0rc10 (zppy v3.2.0rc2), 2026-05-04 Compy E3SM Unified 1.13.0rc8, 2026-05-04 Perlmutter E3SM Unified 1.13.0rc8:

Machine/Unified env	Total	Correct	Missing	Mismatched
Chrysalis, Unified rc10	647	400	0	247
Compy, Unified rc10	617	400	0	217
Compy, Unified rc8	617	335	182	100
Perlmutter, Unified rc8	617	387	97	133

Debugging

So, Chrysalis has added 30 more images to check, all of which end up mismatched. (As always, that's under the assumption it didn't fix other mismatched images. In that case, it's possible this made more than 30 new images fail to match).

Let's see if we can figure out what the 30 files are.

Copying those both to Perlmutter as chrysalis_rc10_pcmdi_mismatched.txt and compy_rc10_pcmdi_mismatched.txt respectively, we get:

wc -l chrysalis_rc10_pcmdi_mismatched.txt 
# 247 chrysalis_rc10_pcmdi_mismatched.txt
wc -l compy_rc10_pcmdi_mismatched.txt 
# 217 compy_rc10_pcmdi_mismatched.txt
diff chrysalis_rc10_pcmdi_mismatched.txt compy_rc10_pcmdi_mismatched.txt

which gives:

3,4d2
< pcmdi_diags/model_vs_obs/CLIM_patttern/global/AC/ta-200_global_AC.png
< pcmdi_diags/model_vs_obs/CLIM_patttern/global/AC/ta-850_global_AC.png
13,14d10
< pcmdi_diags/model_vs_obs/CLIM_patttern/global/DJF/ta-200_global_DJF.png
< pcmdi_diags/model_vs_obs/CLIM_patttern/global/DJF/ta-850_global_DJF.png
23,24d18
< pcmdi_diags/model_vs_obs/CLIM_patttern/global/JJA/ta-200_global_JJA.png
< pcmdi_diags/model_vs_obs/CLIM_patttern/global/JJA/ta-850_global_JJA.png
33,34d26
< pcmdi_diags/model_vs_obs/CLIM_patttern/global/MAM/ta-200_global_MAM.png
< pcmdi_diags/model_vs_obs/CLIM_patttern/global/MAM/ta-850_global_MAM.png
43,44d34
< pcmdi_diags/model_vs_obs/CLIM_patttern/global/SON/ta-200_global_SON.png
< pcmdi_diags/model_vs_obs/CLIM_patttern/global/SON/ta-850_global_SON.png
53,54d42
< pcmdi_diags/model_vs_obs/CLIM_patttern/land/AC/ta-200_land_AC.png
< pcmdi_diags/model_vs_obs/CLIM_patttern/land/AC/ta-850_land_AC.png
63,64d50
< pcmdi_diags/model_vs_obs/CLIM_patttern/land/DJF/ta-200_land_DJF.png
< pcmdi_diags/model_vs_obs/CLIM_patttern/land/DJF/ta-850_land_DJF.png
73,74d58
< pcmdi_diags/model_vs_obs/CLIM_patttern/land/JJA/ta-200_land_JJA.png
< pcmdi_diags/model_vs_obs/CLIM_patttern/land/JJA/ta-850_land_JJA.png
83,84d66
< pcmdi_diags/model_vs_obs/CLIM_patttern/land/MAM/ta-200_land_MAM.png
< pcmdi_diags/model_vs_obs/CLIM_patttern/land/MAM/ta-850_land_MAM.png
93,94d74
< pcmdi_diags/model_vs_obs/CLIM_patttern/land/SON/ta-200_land_SON.png
< pcmdi_diags/model_vs_obs/CLIM_patttern/land/SON/ta-850_land_SON.png
103,104d82
< pcmdi_diags/model_vs_obs/CLIM_patttern/ocean/AC/ta-200_ocean_AC.png
< pcmdi_diags/model_vs_obs/CLIM_patttern/ocean/AC/ta-850_ocean_AC.png
113,114d90
< pcmdi_diags/model_vs_obs/CLIM_patttern/ocean/DJF/ta-200_ocean_DJF.png
< pcmdi_diags/model_vs_obs/CLIM_patttern/ocean/DJF/ta-850_ocean_DJF.png
123,124d98
< pcmdi_diags/model_vs_obs/CLIM_patttern/ocean/JJA/ta-200_ocean_JJA.png
< pcmdi_diags/model_vs_obs/CLIM_patttern/ocean/JJA/ta-850_ocean_JJA.png
133,134d106
< pcmdi_diags/model_vs_obs/CLIM_patttern/ocean/MAM/ta-200_ocean_MAM.png
< pcmdi_diags/model_vs_obs/CLIM_patttern/ocean/MAM/ta-850_ocean_MAM.png
143,144d114
< pcmdi_diags/model_vs_obs/CLIM_patttern/ocean/SON/ta-200_ocean_SON.png
< pcmdi_diags/model_vs_obs/CLIM_patttern/ocean/SON/ta-850_ocean_SON.png
169d138
< pcmdi_diags/model_vs_obs/MOV_metric/MOV_compose/SON/MOV_compose_PSA2_SON_eof3.png
180a150
> pcmdi_diags/model_vs_obs/MOV_metric/MOV_pattern/DJF/MOV_pattern_PSA1_DJF_eof3.png
183d152
< pcmdi_diags/model_vs_obs/MOV_metric/MOV_pattern/DJF/MOV_pattern_SAM_DJF_eof3.png
184a154
> pcmdi_diags/model_vs_obs/MOV_metric/MOV_pattern/JJA/MOV_pattern_PSA1_JJA_eof3.png
188a159
> pcmdi_diags/model_vs_obs/MOV_metric/MOV_pattern/JJA/MOV_pattern_SAM_JJA_eof3.png
190a162
> pcmdi_diags/model_vs_obs/MOV_metric/MOV_pattern/MAM/MOV_pattern_PSA2_MAM_eof2.png
192c164
< pcmdi_diags/model_vs_obs/MOV_metric/MOV_pattern/MAM/MOV_pattern_SAM_MAM_eof2.png
---
> pcmdi_diags/model_vs_obs/MOV_metric/MOV_pattern/SON/MOV_pattern_PSA1_SON_eof3.png
195d166
< pcmdi_diags/model_vs_obs/MOV_metric/MOV_pattern/SON/MOV_pattern_SAM_SON_eof3.png
213d183

diff chrysalis_rc10_pcmdi_mismatched.txt compy_rc10_pcmdi_mismatched.txt > diff_output.txt
grep "<" diff_output.txt | wc -l
# 35
grep ">" diff_output.txt | wc -l
# 5

that's under the assumption it didn't fix other mismatched images

It turns out that assumption doesn't hold. Rather, working with 30 new files, it created 35 more mismatched files and fixed 5 mismatched files, as compared to Compy.

I think we'd just need @zhangshixuan1987 to review the diff grid here. Unlike on Compy, these diffs represent changes only since 2026-03-30, not since E3SM Unified 1.12.0 was released in Fall 2025.

It's unclear why Chrysalis has 30 more images to review than Compy does; they must have been added to Chrysalis's expected results after E3SM Unified 1.12.0 was released.

mpas_analysis

Test name	Total images	Missing images
comprehensive_v2_mpas_analysis	856	856 (list)
comprehensive_v3_mpas_analysis	856	856 (list)
legacy_3.1.0_comprehensive_v2_mpas_analysis	856	856 (list)
legacy_3.1.0_comprehensive_v3_mpas_analysis	856	856 (list)
legacy_3.0.0_comprehensive_v2_mpas_analysis	856	856 (list)
legacy_3.0.0_comprehensive_v3_mpas_analysis	856	856 (list)

These appear in line with what we've seen on Compy with Unified rc10:

856 missing images of the mpas_analysis tasks are consistent with the 2026-05-04 Compy E3SM Unified 1.13.0rc8 results

So, I don't think these are a problem. We just need to update the expected results to find them in the correct location.

e3sm_diags

Test name	Total images	Correct images	Missing images
bundles_e3sm_diags	1762	657	1105 (list)
comprehensive_v2_e3sm_diags	3791	937	2854 (list)
comprehensive_v3_e3sm_diags	5354	1797	3557 (list)
legacy_3.1.0_bundles_e3sm_diags	1762	656	1106 (list)
legacy_3.1.0_comprehensive_v2_e3sm_diags	3791	936	2855 (list)
legacy_3.1.0_comprehensive_v3_e3sm_diags	5350	1797	3553 (list)
legacy_3.0.0_bundles_e3sm_diags	1762	655	1107 (list)
legacy_3.0.0_comprehensive_v2_e3sm_diags	3791	937	2854 (list)
legacy_3.0.0_comprehensive_v3_e3sm_diags	5350	1797	3553 (list)

Unlike the mpas_anlaysis or even pcmdi_diags results, this is a major departure from what we've seen on the other machines.

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc10_try1/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # No errors
ls e3sm_diags*.o*
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1987-1988.o1214586
# e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1987-1988_vs_1985-1986.o1214587
# e3sm_diags_lnd_monthly_mvm_lnd_model_vs_model_1987-1988_vs_1985-1986.o1214588
tail -n 20 e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1987-1988.o1214586
# concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
tail -n 20 e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1987-1988_vs_1985-1986.o1214587
# concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
tail -n 20 e3sm_diags_lnd_monthly_mvm_lnd_model_vs_model_1987-1988_vs_1985-1986.o1214588
# Doesn't seem to have an error

grep -l "concurrent.futures.process.BrokenProcessPool" e3sm_diags*.o*
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1987-1988.o1214586
# e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1987-1988_vs_1985-1986.o1214587

# We can review the list of missing images for non-legacy v3:
# https://web.lcrc.anl.gov/public/e3sm/diagnostic_output//ac.forsyth2/zppy_weekly_comprehensive_v3_www/zppy_main_branch_plus_unified_1_13rc10_try1/v3.LR.historical_0051/image_check_failures_comprehensive_v3/e3sm_diags/missing_images.txt
# Let's look for e3sm_diags_lnd_monthly_mvm_lnd_model_vs_model_1987-1988_vs_1985-1986
# "lnd" doesn't appear in this file at all.
# So, yes, this must contribute to the 1,797 correct images.

@chengzhuzhang While e3sm_diags can produce some correct images, it's missing many more. The issue appears related to concurrent.futures.process.BrokenProcessPool.

Action items

Me: make PR for the image checker fix
@zhangshixuan1987: Confirm the pcmdi_diags diffs look reasonable. NOTE: Unlike on Compy, these diffs represent changes only since 2026-03-30, not since E3SM Unified 1.12.0 was released in Fall 2025.
@chengzhuzhang: Do you know what's causing the concurrent.futures.process.BrokenProcessPool?

12 replies

xylar May 11, 2026
Collaborator

Yeah, #497 doesn't suggest that it should be removed.

I don't understand OMPI_MCA_pmix=isolated. It seems worth finding out if that's something we should be setting for Chrysalis generally or if it's specific to this problem.

xylar May 11, 2026
Collaborator

Also, did you actually test that that helps or is it just a theory for now?

chengzhuzhang May 11, 2026
Maintainer

I don't know about option 1, but curious about option 2, about 3 years ago we started to use srun to run e3sm_diags in zppy, to avoid an MPI initiation issue. #485 (comment). I'm trying to look at the logs now.

forsyth2 May 11, 2026
Maintainer Author

did you actually test that that helps or is it just a theory for now

I just tested now trying out a small reproducer. (I was initially asking if it was even worth trying to test). Unfortunately, the reproducer has revealed that's definitely not the fix. So that means I need to keep debugging to have a worthwhile reproducer script...

I'm still unclear if the issue is on the zppy side or the e3sm_diags side, which also makes debugging harder.

forsyth2 May 11, 2026
Maintainer Author

I think this issue has grown outside the scope of this thread; I opened a new issue: #819

forsyth2 · 2026-05-11T20:09:06Z

forsyth2
May 11, 2026
Maintainer Author

2026-05-08 Perlmutter E3SM Unified 1.13.0rc10 (zppy v3.2.0rc2)

Chrysalis: run tests on all 9 cfgs
Compy: run all 3 non-legacy test cfgs
Perlmutter: run just the non-legacy v3 test cfg

What expected results are currently on Compy?

The latest expected results are still the E3SM Unified 1.12.0 official results.

What changes are we testing?

Package	Changes since last test
NCO	Unified rc10 instead of rc9
e3sm_to_cmip	Unified rc10 instead of rc9
e3sm_diags	Unified rc10 instead of rc9
mpas_analysis	Unified rc10 instead of rc9
zppy-interfaces	Unified rc10 instead of rc9
zppy	Unified rc10 instead of rc9 (in both RCs: zppy v3.2.0rc2)

Testing process

Starting actual testing on 2026-05-08
Branch: zppy main branch
Environments: zppy & packages for all tasks -- E3SM Unified 1.13.0rc10 (zppy v3.2.0rc2)

Set up environments

We can mostly skip this step, since all the packages will be tested using the E3SM Unified RC.

cd ~/ez/zppy-interfaces
git status
# # On branch main
# nothing to commit, working directory clean
git fetch upstream main
git checkout main
git reset --hard upstream/main
git log --oneline | head -n 1
# 6b0758f Bump to 0.2.1rc1 (#48)
# Good, matches https://github.com/E3SM-Project/zppy-interfaces/commits/main
bash # Run bash so we're in an isolated subshell
source ~/.bash_profile.ext
source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc10_pm-cpu.sh
# Check zppy-interfaces tests
pytest tests/unit/global_time_series/test_*.py # 10 passed, 2 warnings in 11.51s
# SyntaxWarning: invalid escape sequence '\w'
# SyntaxWarning: invalid escape sequence '\s'
pytest tests/unit/pcmdi_diags/test_*.py # 7 passed in 4.65s

# Stay in the bash subshell

zppy run

# zppy itself #################################################################
cd ~/ez/zppy
git status
# On branch test-zppy-main-plus-unified-1-13rc9
# nothing to commit, working tree clean

git fetch upstream main
git checkout -b test-zppy-main-plus-unified-1-13rc10 upstream/main
git log --oneline | head -n 2
# f368ab71 Bump to 3.2.0rc2 (#816)
# f335f0e3 Merge pull request #814 from E3SM-Project/fixes-of-rc8
# Good, matches https://github.com/E3SM-Project/zppy/commits/main

showquota # Have enough space on home and pscratch
# (Currently in /global/homes/f/forsyth/ez/zppy)

# WE ARE ALREADY IN UNIFIED RC10
# DO NOT MAKE A DEV ENV

pixi list python # Check Python version
# python                    3.13.13

# Run zppy unit tests
pytest tests/test_*.py # 44 passed in 0.23s

# Edit tests/integration/utils.py:

TEST_SPECIFICS: Dict[str, Any] = {
    # This is the NCO path.
    # Keep as "" to use the production-version NCO commands.
    # Set to a specific path to use development-version NCO commands.
    "nco_path": "",
    # These are custom environment_commands for specific tasks.
    # Never set these to "", because they will print the line
    # `environment_commands = ""` for the corresponding task,
    # thus overriding the value set higher up in the cfg.
    # That is, there will be no environment set.
    # (`environment_commands = ""` only redirects to Unified
    # if specified under the [default] task)
    "e3sm_to_cmip_environment_commands": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc10_pm-cpu.sh",
    "diags_environment_commands": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc10_pm-cpu.sh",
    "mpas_analysis_environment_commands": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc10_pm-cpu.sh",
    "global_time_series_environment_commands": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc10_pm-cpu.sh",
    "livvkit_environment_commands": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc10_pm-cpu.sh",
    "pcmdi_diags_environment_commands": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc10_pm-cpu.sh",
    # This is the environment setup for other tasks.
    # Leave as "" to use the latest Unified environment.
    "environment_commands": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc10_pm-cpu.sh",
    # For a complete test, run the set of latest cfgs and at least one set of legacy cfgs
    "cfgs_to_run": [
        # "weekly_bundles",
        # "weekly_comprehensive_v2",
        "weekly_comprehensive_v3",
        # "weekly_legacy_3.1.0_bundles",
        # "weekly_legacy_3.1.0_comprehensive_v2",
        # "weekly_legacy_3.1.0_comprehensive_v3",
        # "weekly_legacy_3.0.0_bundles",
        # "weekly_legacy_3.0.0_comprehensive_v2",
        # "weekly_legacy_3.0.0_comprehensive_v3",
    ],
    "tasks_to_run": [
        "e3sm_diags",
        "mpas_analysis",
        "global_time_series",
        "ilamb",
        "livvkit",
        "pcmdi_diags",
    ],
    "unique_id": "zppy_main_branch_plus_unified_1_13rc10",
}

git diff # Diff looks good
python tests/integration/utils.py
# CFG FILES HAVE BEEN GENERATED FROM TEMPLATES WITH THESE SETTINGS:
# UNIQUE_ID=zppy_main_branch_plus_unified_1_13rc10
# nco_path=
# Reminder: `nco_path=''` => the production-version NCO commands will be used
# e3sm_to_cmip_environment_commands=source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc10_pm-cpu.sh
# diags_environment_commands=source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc10_pm-cpu.sh
# mpas_analysis_environment_commands=source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc10_pm-cpu.sh
# global_time_series_environment_commands=source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc10_pm-cpu.sh
# livvkit_environment_commands=source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc10_pm-cpu.sh
# pcmdi_diags_environment_commands=source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc10_pm-cpu.sh
# environment_commands=source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc10_pm-cpu.sh
# Reminder: `environment_commands=''` => the latest E3SM Unified environment will be used

# For reference:
alias sq
# Perlmutter: alias sq='sqa -u forsyth'
# Compy:      alias sq='sqa -u fors729'
alias sqa
# Perlmutter: alias sqa='squeue -o "%8u %.7a %.4D %.9P %8i %.2t %.10r %.10M %.10l %j" --sort=P,-t,-p'
# Compy:      alias sqa='squeue -o "%8u %.7a %.4D %.9P %7i %.2t %.10r %.10M %.10l %.8Q %j" --sort=P,-t,-p'
sq
# No jobs currently queued

# This is the only cfg we're running on Perlmutter:
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_pm-cpu.cfg

sq | wc -l # Fri 05/08 11:15 => 56 - header row = 55 jobs
sq | wc -l # Fri 05/08 12:31 => 17 - header row = 16 jobs
# HERE

# Picking up Mon 05/11 ########################################################
sq | wc -l # Mon 05/11 10:21 => 1 - header row = 0 jobs

Review finished runs

# Re-establish Unified env
bash
source ~/.bash_profile.ext
source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc10_pm-cpu.sh

### v3 ###
cd /global/cfs/cdirs/e3sm/forsyth/zppy_weekly_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc10/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors

post/scripts directory status: NO ERRORS

Python tests

cd ~/ez/zppy
git status
# Good, correct branch: test-zppy-main-plus-unified-1-13rc10
# Good, correct env: Unified rc10
ls tests/integration/test_*.py

# Errors look to be expected based on code changes during this development period:
pytest tests/integration/test_bash_generation.py # 1 failed in 1.25s
pytest tests/integration/test_campaign.py # 6 failed in 3.39s
pytest tests/integration/test_defaults.py # 1 failed in 0.75s

# Non-applicable (we didn't do the necessary `zppy` runs for this test):
# pytest tests/integration/test_bundles.py 

# Passing
pytest tests/integration/test_last_year.py
# 1 passed in 0.55s

# We'll now proceed with the image checker test.
# On Chrysalis, we were running `salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm`
# Compute node image tests were introduced in https://github.com/E3SM-Project/zppy/pull/752.
# Referring to notes from the week that merged, it looks like we want to use this line to get a Perlmutter compute note:
salloc --nodes=1 --qos=interactive --time=01:00:00 --constraint=cpu --account=e3sm
# Re-establish Unified env
bash
source ~/.bash_profile.ext
source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc10_pm-cpu.sh
pytest tests/integration/test_images.py # 1 failed in 1883.75s (0:31:23)
cat test_images_summary.md

Output:

------------------------------------------ Captured stdout call -------------------------------------------
Preparing weekly cfg tests
Preparing legacy 3.1.0 cfg tests
Preparing legacy 3.0.0 cfg tests
Running 1 tests in parallel
Individual test logs will be written to test_<name>.log files
✓ Completed: comprehensive_v3 (5 tasks) (log: test_comprehensive_v3.log)
Copy the output of test_images_summary.md to a Pull Request comment

Test Summary:
Test                                                    Total    Correct     Status
----------------------------------------------------------------------------------
comprehensive_v3_e3sm_diags                              5210       5185     ✗ FAIL
comprehensive_v3_mpas_analysis                            856          0     ✗ FAIL
comprehensive_v3_global_time_series                      1404       1404     ✓ PASS
comprehensive_v3_ilamb                                    837        837     ✓ PASS
comprehensive_v3_pcmdi_diags                              617        400     ✗ FAIL
----------------------------------------------------------------------------------

⚠ Some tests had mismatched or missing images. Check individual log files for details.
========================================= short test summary info =========================================
FAILED tests/integration/test_images.py::test_images - assert 5210 == 5185
===================================== 1 failed in 1883.75s (0:31:23) ======================================

Complete summary table:

Test name	Total images	Correct images	Missing images	Mismatched images
comprehensive_v3_e3sm_diags	5210	5185	0	25 (list, grid)
comprehensive_v3_mpas_analysis	856	0	856 (list)	0
comprehensive_v3_global_time_series	1404	1404	0	0
comprehensive_v3_ilamb	837	837	0	0
comprehensive_v3_pcmdi_diags	617	400	0	217 (list, grid)

Summary table -- only failing image-check tests

Test name	Total images	Correct images	Missing images	Mismatched images
comprehensive_v3_e3sm_diags	5210	5185	0	25 (list, grid)
comprehensive_v3_mpas_analysis	856	0	856 (list)	0
comprehensive_v3_pcmdi_diags	617	400	0	217 (list, grid)

Results Analysis

The 25 mismatched images for e3sm_diags, 856 missing images for mpas_analysis, and 217 mismatched images for pcmdi_diags all match up with what we saw in the 2026-05-08 Compy E3SM Unified 1.13.0rc10 (zppy v3.2.0rc2) results. We had already determined those are to be the new expectation.

Since all errors noted here are consistent with what we would expect, we are now able to update the expected results on Perlmutter.

Updating expected results

Setting up new expected results

We last set the expected results to the official results of E3SM Unified 1.12.0 here, on 2026-03-31.

ls -lt /global/cfs/cdirs/e3sm/www/zppy_test_resources/
# The files are from Mar. 31
# This is consistent with how we updated the expected results on 2026-03-31.
ls -lt /global/cfs/cdirs/e3sm/www/zppy_test_resources_previous
# Mar 31 16:27 expected_results_until_20260331
# Dec  1 18:43 expected_results_for_unified_1.12.0
# Nov 17 13:21 expected_results_until_20250422_from_unified_1.11.1
# Apr 22  2025 expected_results_for_unified_1.11.1
# Apr 22  2025 expected_results_until_2025_04_22
# Mar 24  2025 expected_results_until_20250324_with_unified_1.11.0rc13
ls -lt /global/cfs/cdirs/e3sm/www/zppy_test_resources_previous/expected_results_until_20260331
# Notice: similar contents to /global/cfs/cdirs/e3sm/www/zppy_test_resources/

# First, let's copy over the old expected results #############################
cd /global/cfs/cdirs/e3sm/www/
mv zppy_test_resources/ zppy_test_resources_previous/expected_results_until_20260511 # mv is faster than cp
mkdir zppy_test_resources # Remake the official expected_results dir
ls -lt zppy_test_resources_previous/expected_results_until_20260511
# Good, contents are from 3/31.

# Second, update the expected results #########################################
# Let's update the simpler tests' results first:
cd ~/ez/zppy
git status
# Good, correct branch: test-zppy-main-plus-unified-1-13rc10
# Good, correct env: Unified rc10

ls tests/integration/generated/update_*_pm-cpu.sh
# tests/integration/generated/update_archive_expected_files_pm-cpu.sh
# tests/integration/generated/update_bash_generation_expected_files_pm-cpu.sh
# tests/integration/generated/update_campaign_expected_files_pm-cpu.sh
# tests/integration/generated/update_defaults_expected_files_pm-cpu.sh
# tests/integration/generated/update_weekly_expected_files_pm-cpu.sh

chmod 755 tests/integration/generated/update_bash_generation_expected_files_pm-cpu.sh
./tests/integration/generated/update_bash_generation_expected_files_pm-cpu.sh
# 1 passed in 1.08s

chmod 755 tests/integration/generated/update_campaign_expected_files_pm-cpu.sh
./tests/integration/generated/update_campaign_expected_files_pm-cpu.sh
# 6 passed in 3.25s

chmod 755 tests/integration/generated/update_defaults_expected_files_pm-cpu.sh
./tests/integration/generated/update_defaults_expected_files_pm-cpu.sh
# 1 passed in 0.77s

# test_last_year doesn't need an update script because its cfg is stand-alone:
pytest -vv tests/integration/test_last_year.py
# 1 passed in 0.57s

# Now update the expected files for the major tests:

# To make sure exiting the SSH connection doesn't halt the progress,
# we can use screen.
screen -ls # No Sockets found
screen
screen -ls # 1082087.pts-69.login11  (Attached)
# Re-establish env:
bash # Run bash so we're in an isolated subshell
source ~/.bash_profile.ext
source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc10_pm-cpu.sh
chmod 755 tests/integration/generated/update_weekly_expected_files_pm-cpu.sh
./tests/integration/generated/update_weekly_expected_files_pm-cpu.sh
# CTRL A D
screen -R
# Took less than 25 min to run on Perlmutter
screen -R
# cp: cannot stat '/global/cfs/cdirs/e3sm/www/forsyth/zppy_weekly_comprehensive_v2_www/zppy_main_branch_plus_unified_1_13rc10/v2.LR.historical_0201': No such file or directory
# ./tests/integration/generated/update_weekly_expected_files_pm-cpu.sh: line 56: cd: /global/cfs/cdirs/e3sm/www/zppy_test_resources/expected_comprehensive_v2: No such file or directory
# cp: cannot stat '/global/cfs/cdirs/e3sm/www/forsyth/zppy_weekly_bundles_www/zppy_main_branch_plus_unified_1_13rc10/v3.LR.historical_0051': No such file or directory
# cp: cannot stat '/global/cfs/cdirs/e3sm/forsyth/zppy_weekly_bundles_output/zppy_main_branch_plus_unified_1_13rc10/v3.LR.historical_0051/post/scripts/bundle*.bash': No such file or directory
# cp: cannot stat '/global/cfs/cdirs/e3sm/www/forsyth/zppy_weekly_legacy_3.1.0_comprehensive_v2_www/zppy_main_branch_plus_unified_1_13rc10/v2.LR.historical_0201': No such file or directory
# ./tests/integration/generated/update_weekly_expected_files_pm-cpu.sh: line 56: cd: /global/cfs/cdirs/e3sm/www/zppy_test_resources/expected_legacy_3.1.0_comprehensive_v2: No such file or directory
# cp: cannot stat '/global/cfs/cdirs/e3sm/www/forsyth/zppy_weekly_legacy_3.1.0_comprehensive_v3_www/zppy_main_branch_plus_unified_1_13rc10/v3.LR.historical_0051': No such file or directory
# ./tests/integration/generated/update_weekly_expected_files_pm-cpu.sh: line 56: cd: /global/cfs/cdirs/e3sm/www/zppy_test_resources/expected_legacy_3.1.0_comprehensive_v3: No such file or directory
# cp: cannot stat '/global/cfs/cdirs/e3sm/www/forsyth/zppy_weekly_legacy_3.1.0_bundles_www/zppy_main_branch_plus_unified_1_13rc10/v3.LR.historical_0051': No such file or directory
# ./tests/integration/generated/update_weekly_expected_files_pm-cpu.sh: line 56: cd: /global/cfs/cdirs/e3sm/www/zppy_test_resources/expected_legacy_3.1.0_bundles: No such file or directory
# cp: cannot stat '/global/cfs/cdirs/e3sm/www/forsyth/zppy_weekly_legacy_3.0.0_comprehensive_v2_www/zppy_main_branch_plus_unified_1_13rc10/v2.LR.historical_0201': No such file or directory
# ./tests/integration/generated/update_weekly_expected_files_pm-cpu.sh: line 56: cd: /global/cfs/cdirs/e3sm/www/zppy_test_resources/expected_legacy_3.0.0_comprehensive_v2: No such file or directory
# cp: cannot stat '/global/cfs/cdirs/e3sm/www/forsyth/zppy_weekly_legacy_3.0.0_comprehensive_v3_www/zppy_main_branch_plus_unified_1_13rc10/v3.LR.historical_0051': No such file or directory
# ./tests/integration/generated/update_weekly_expected_files_pm-cpu.sh: line 56: cd: /global/cfs/cdirs/e3sm/www/zppy_test_resources/expected_legacy_3.0.0_comprehensive_v3: No such file or directory
# cp: cannot stat '/global/cfs/cdirs/e3sm/www/forsyth/zppy_weekly_legacy_3.0.0_bundles_www/zppy_main_branch_plus_unified_1_13rc10/v3.LR.historical_0051': No such file or directory
# ./tests/integration/generated/update_weekly_expected_files_pm-cpu.sh: line 56: cd: /global/cfs/cdirs/e3sm/www/zppy_test_resources/expected_legacy_3.0.0_bundles: No such file or directory

# Yes, this does makes sense, because we exclusively ran the non-legacy v3 test.\
# That's 8 tests we didn't run and the script obviously can't stat the www paths or the corresponding expected results dir for those 8 tests.
# That accounts for the 16 `cannot stat` lines.
ls -lt /global/cfs/cdirs/e3sm/www/zppy_test_resources
# Good, no longer empty.
# Everything says updated 5/11 except for expected_bash_files, which says 4/27

exit # Exit bash subshell
exit # Exit screen
screen -ls # No Sockets found

cd ~/ez/zppy
# Still can't run this, because never ran the corresponding zppy jobs:
# pytest tests/integration/test_bundles.py

salloc --nodes=1 --qos=interactive --time=01:00:00 --constraint=cpu --account=e3sm
# Re-establish Unified env
bash
source ~/.bash_profile.ext
source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc10_pm-cpu.sh
pytest tests/integration/test_images.py # 1 passed in 2404.40s (0:40:04)
cat test_images_summary.md

Test name	Total images	Correct images
comprehensive_v3_e3sm_diags	5369	5369
comprehensive_v3_mpas_analysis	1280	1280
comprehensive_v3_global_time_series	1404	1404
comprehensive_v3_ilamb	837	837
comprehensive_v3_pcmdi_diags	647	647

✅ E3SM Unified 1.13.0rc10 passes on Perlmutter.

0 replies

forsyth2 · 2026-05-13T00:04:39Z

forsyth2
May 13, 2026
Maintainer Author

2026-05-12 Compy E3SM Unified 1.13.0rc10, post-fix (zppy v3.2.0rc2)

Chrysalis: run tests on all 9 cfgs
Compy: run all 3 non-legacy test cfgs
Perlmutter: run just the non-legacy v3 test cfg

What expected results are currently on Compy?

Compy's expected results were updated after the last test run -- 2026-05-08 Compy E3SM Unified 1.13.0rc10 (zppy v3.2.0rc2).

What changes are we testing?

Since 2026-05-08 Compy E3SM Unified 1.13.0rc10 (zppy v3.2.0rc2), the only change has been the updating of the Unified load script, per the discussion in #819.

We also just merged #818, which only affects testing code.

Testing process

Starting actual testing on 2026-05-12
Branch: zppy main branch
Environments: zppy & packages for all tasks -- E3SM Unified 1.13.0rc10

Set up environments

We can mostly skip this step, since all the packages will be tested using the E3SM Unified RC.

cd ~/ez/zppy-interfaces
git status
# # On branch main
# nothing to commit, working directory clean

# REMINDER: On Compy, you must NOT specify the branch, 
# otherwise it won't actually fetch the latest commits!!
git fetch upstream 
git checkout main
git reset --hard upstream/main
git log --oneline | head -n 1
# 6b0758f Bump to 0.2.1rc1 (#48)
# Good, matches https://github.com/E3SM-Project/zppy-interfaces/commits/main

bash # Run bash so we're in an isolated subshell
source ~/.bash_profile
source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc10_compy.sh
# Check zppy-interfaces tests
pytest tests/unit/global_time_series/test_*.py # 10 passed, 2 warnings in 20.57s
# SyntaxWarning: invalid escape sequence '\w'
# SyntaxWarning: invalid escape sequence '\s'
pytest tests/unit/pcmdi_diags/test_*.py # 7 passed in 9.57s

# Stay in the bash subshell

zppy run

# zppy itself #################################################################
cd ~/ez/zppy
git status
# On branch test-zppy-main-plus-unified-1-13rc10
git add -A
git commit -m "Testing" --no-verify # Clean up uncommitted changes

git fetch upstream # Remember, do NOT specify the branch on Compy!
git checkout -b test-zppy-main-plus-unified-1-13rc10-fix upstream/main
git log --oneline | head -n 3
# 885d53c Fix image checker (#818)
# f368ab7 Bump to 3.2.0rc2 (#816)
# f335f0e Merge pull request #814 from E3SM-Project/fixes-of-rc8
# Good, matches https://github.com/E3SM-Project/zppy/commits/main

lfs quota -hu fors729 /compyfs # Have enough space on /compyfs
# Not sure how to check space on /qfs
# (Currently in /qfs/people/fors729/ez/zppy)

# WE ARE ALREADY IN UNIFIED RC10
# DO NOT MAKE A DEV ENV

pixi list python # Check Python version
# python                    3.13.13

# Run zppy unit tests
pytest tests/test_*.py # 44 passed in 0.67s

# Edit tests/integration/utils.py:

TEST_SPECIFICS: Dict[str, Any] = {
    # This is the NCO path.
    # Keep as "" to use the production-version NCO commands.
    # Set to a specific path to use development-version NCO commands.
    "nco_path": "",
    # These are custom environment_commands for specific tasks.
    # Never set these to "", because they will print the line
    # `environment_commands = ""` for the corresponding task,
    # thus overriding the value set higher up in the cfg.
    # That is, there will be no environment set.
    # (`environment_commands = ""` only redirects to Unified
    # if specified under the [default] task)
    "e3sm_to_cmip_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc10_compy.sh",
    "diags_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc10_compy.sh",
    "mpas_analysis_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc10_compy.sh",
    "global_time_series_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc10_compy.sh",
    "livvkit_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc10_compy.sh",
    "pcmdi_diags_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc10_compy.sh",
    # This is the environment setup for other tasks.
    # Leave as "" to use the latest Unified environment.
    "environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc10_compy.sh",
    # For a complete test, run the set of latest cfgs and at least one set of legacy cfgs
    "cfgs_to_run": [
        "weekly_bundles",
        "weekly_comprehensive_v2",
        "weekly_comprehensive_v3",
        # "weekly_legacy_3.1.0_bundles",
        # "weekly_legacy_3.1.0_comprehensive_v2",
        # "weekly_legacy_3.1.0_comprehensive_v3",
        # "weekly_legacy_3.0.0_bundles",
        # "weekly_legacy_3.0.0_comprehensive_v2",
        # "weekly_legacy_3.0.0_comprehensive_v3",
    ],
    "tasks_to_run": [
        "e3sm_diags",
        "mpas_analysis",
        "global_time_series",
        "ilamb",
        "livvkit",
        "pcmdi_diags",
    ],
    "unique_id": "zppy_main_branch_plus_unified_1_13rc10_fix",
}

git diff # Diff looks good
python tests/integration/utils.py
# CFG FILES HAVE BEEN GENERATED FROM TEMPLATES WITH THESE SETTINGS:
# UNIQUE_ID=zppy_main_branch_plus_unified_1_13rc10_fix
# nco_path=
# Reminder: `nco_path=''` => the production-version NCO commands will be used
# e3sm_to_cmip_environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc10_compy.sh
# diags_environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc10_compy.sh
# mpas_analysis_environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc10_compy.sh
# global_time_series_environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc10_compy.sh
# livvkit_environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc10_compy.sh
# pcmdi_diags_environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc10_compy.sh
# environment_commands=source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc10_compy.sh
# Reminder: `environment_commands=''` => the latest E3SM Unified environment will be used

# For reference:
alias sq
# Perlmutter: alias sq='sqa -u forsyth'
# Compy:      alias sq='sqa -u fors729'
alias sqa
# Perlmutter: alias sqa='squeue -o "%8u %.7a %.4D %.9P %8i %.2t %.10r %.10M %.10l %j" --sort=P,-t,-p'
# Compy:      alias sqa='squeue -o "%8u %.7a %.4D %.9P %7i %.2t %.10r %.10M %.10l %.8Q %j" --sort=P,-t,-p'
sq
# No jobs currently queued

# Launch jobs; start with the most important one, comprehensive_v3
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_compy.cfg
zppy -c tests/integration/generated/test_weekly_bundles_compy.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_compy.cfg

sq | wc -l # Tue 05/12 10:01 => 85 - header row = 84 jobs
sq | wc -l # Tue 05/12 11:48 => 12 - header row = 11 jobs
sq | wc -l # Tue 05/12 12:59 => 1 - header row = 0 jobs

Bundles part 2

# Check on bundles status
cd /compyfs/fors729/zppy_weekly_bundles_output/zppy_main_branch_plus_unified_1_13rc10_fix/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors

# Now, run bundles part 2
cd ~/ez/zppy
git status
# Good, correct branch: test-zppy-main-plus-unified-1-13rc10-fix
# Good, correct env: Unified rc10

zppy -c tests/integration/generated/test_weekly_bundles_compy.cfg
sq | wc -l # Tue 05/12 12:59 => 3 - header row = 2 jobs

Review finished runs

### v2  ###
cd /compyfs/fors729/zppy_weekly_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_fix/v2.LR.historical_0201/post/scripts
grep -v "OK" *status # Good, no errors

### v3 ###
cd /compyfs/fors729/zppy_weekly_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc10_fix/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors

### bundles ###
sq | wc -l # Tue 05/12 13:14 => 1 - header row = 0 jobs
cd /compyfs/fors729/zppy_weekly_bundles_output/zppy_main_branch_plus_unified_1_13rc10_fix/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, still no errors after running bundles part 2

post/scripts directory status: NO ERRORS

Python tests

cd ~/ez/zppy
git status
# Good, correct branch: test-zppy-main-plus-unified-1-13rc10-fix
# Good, correct env: Unified rc10
ls tests/integration/test_*.py

# Passing
pytest tests/integration/test_bash_generation.py # 1 passed in 10.35s
pytest tests/integration/test_campaign.py # 6 passed in 17.69s
pytest tests/integration/test_defaults.py # 1 passed in 2.16s
pytest tests/integration/test_last_year.py # 1 passed in 1.63s
pytest tests/integration/test_bundles.py # 2 passed in 1.23s

# We'll now proceed with the image checker test.
# Compute node image tests were introduced in https://github.com/E3SM-Project/zppy/pull/752.
# Referring to notes from the week that merged, 
# it looks like we want to use this line to get a Compy compute note:
# salloc --nodes=1 --partition=short --time=00:30:00 --account=e3sm
salloc --nodes=1 --partition=short --time=01:00:00 --account=e3sm
# Re-establish Unified env
bash
source ~/.bash_profile
source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc10_compy.sh
pytest tests/integration/test_images.py # 1 passed in 3297.44s (0:54:57)
cat test_images_summary.md

Complete summary table:

Test name	Total images	Correct images
bundles_e3sm_diags	1762	1762
bundles_global_time_series	3	3
bundles_ilamb	388	388
comprehensive_v2_e3sm_diags	3806	3806
comprehensive_v2_mpas_analysis	856	856
comprehensive_v2_global_time_series	12	12
comprehensive_v2_ilamb	776	776
comprehensive_v3_e3sm_diags	5369	5369
comprehensive_v3_mpas_analysis	1280	1280
comprehensive_v3_global_time_series	1404	1404
comprehensive_v3_ilamb	837	837
comprehensive_v3_pcmdi_diags	647	647

Results Analysis

✅ E3SM Unified 1.13.0rc10 (fixed script) passes on Compy. Since tests pass, there is no need to update expected results.

0 replies

forsyth2 · 2026-05-13T00:06:22Z

forsyth2
May 13, 2026
Maintainer Author

2026-05-12 Chrysalis E3SM Unified 1.13.0rc10, post-fix (zppy v3.2.0rc2)

Chrysalis: run tests on all 9 cfgs
Compy: run all 3 non-legacy test cfgs
Perlmutter: run just the non-legacy v3 test cfg

What expected results are currently on Chrysalis?

The latest expected results for Compy & Perlmutter are still the E3SM Unified 1.12.0 official results. Chrysalis, however, has had its expected results updated much more recently, on 2026-03-30. We can confirm with:

cd /lcrc/group/e3sm/public_html/zppy_test_resources
ls -lt # Latest updates are from 3/30
cd /lcrc/group/e3sm/public_html/zppy_test_resources_previous
ls -lt # Latest update is from 3/30:
# expected_results_until_20260330
# These were the expected results until we updated them on 2026-03-30.

What changes are we testing?

Since 2026-05-08 Chrysalis E3SM Unified 1.13.0rc10 (zppy v3.2.0rc2), the only change has been the updating of the Unified load script, per the discussion in #819.

We also just merged #818, which only affects testing code.

Testing process

Starting actual testing on 2026-05-12
Branch: zppy main branch
Environments: zppy & packages for all tasks -- E3SM Unified 1.13.0rc10

Set up environments

We can mostly skip this step, since all the packages will be tested using the E3SM Unified RC.

cd ~/ez/zppy-interfaces
git status
# On branch main
# nothing to commit, working directory clean

git fetch upstream main
git checkout main
git reset --hard upstream/main
git log --oneline | head -n 1
# 6b0758f Bump to 0.2.1rc1 (#48)
# Good, matches https://github.com/E3SM-Project/zppy-interfaces/commits/main

bash # Run bash so we're in an isolated subshell
source ~/.bashrc
source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh
# Check zppy-interfaces tests
pytest tests/unit/global_time_series/test_*.py # 10 passed, 2 warnings in 34.46s
# SyntaxWarning: invalid escape sequence '\w'
# SyntaxWarning: invalid escape sequence '\s'
pytest tests/unit/pcmdi_diags/test_*.py # 7 passed in 47.95s

# Stay in the bash subshell

zppy run

# zppy itself #################################################################
cd ~/ez/zppy
git status
# On branch fixes-of-rc10
git add -A
git commit -m "Testing" --no-verify # Clean up workspace

git fetch upstream main
git checkout -b test-zppy-main-plus-unified-1-13rc10-fix upstream/main
git log --oneline | head -n 3
# 885d53cb Fix image checker (#818)
# f368ab71 Bump to 3.2.0rc2 (#816)
# f335f0e3 Merge pull request #814 from E3SM-Project/fixes-of-rc8
# Good, matches https://github.com/E3SM-Project/zppy/commits/main

/usr/lpp/mmfs/bin/mmlsquota -u ac.forsyth2 --block-size T fs2 # Have enough space on /lcrc/group/e3sm/
lcrc-quota # Have enough space on /home/ac.forsyth2/

# WE ARE ALREADY IN UNIFIED RC10
# DO NOT MAKE A DEV ENV

pixi list python # Check Python version
# python                    3.13.13

# Run zppy unit tests
pytest tests/test_*.py # 44 passed in 3.02s

# Edit tests/integration/utils.py:

TEST_SPECIFICS: Dict[str, Any] = {
    # This is the NCO path.
    # Keep as "" to use the production-version NCO commands.
    # Set to a specific path to use development-version NCO commands.
    "nco_path": "",
    # These are custom environment_commands for specific tasks.
    # Never set these to "", because they will print the line
    # `environment_commands = ""` for the corresponding task,
    # thus overriding the value set higher up in the cfg.
    # That is, there will be no environment set.
    # (`environment_commands = ""` only redirects to Unified
    # if specified under the [default] task)
    "e3sm_to_cmip_environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh",
    "diags_environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh",
    "mpas_analysis_environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh",
    "global_time_series_environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh",
    "livvkit_environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh",
    "pcmdi_diags_environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh",
    # This is the environment setup for other tasks.
    # Leave as "" to use the latest Unified environment.
    "environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh",
    # For a complete test, run the set of latest cfgs and at least one set of legacy cfgs
    "cfgs_to_run": [
        "weekly_bundles",
        "weekly_comprehensive_v2",
        "weekly_comprehensive_v3",
        "weekly_legacy_3.1.0_bundles",
        "weekly_legacy_3.1.0_comprehensive_v2",
        "weekly_legacy_3.1.0_comprehensive_v3",
        "weekly_legacy_3.0.0_bundles",
        "weekly_legacy_3.0.0_comprehensive_v2",
        "weekly_legacy_3.0.0_comprehensive_v3",
    ],
    "tasks_to_run": [
        "e3sm_diags",
        "mpas_analysis",
        "global_time_series",
        "ilamb",
        "livvkit",
        "pcmdi_diags",
    ],
    "unique_id": "zppy_main_branch_plus_unified_1_13rc10_fix",
}

git diff # Diff looks good
python tests/integration/utils.py
# CFG FILES HAVE BEEN GENERATED FROM TEMPLATES WITH THESE SETTINGS:
# UNIQUE_ID=zppy_main_branch_plus_unified_1_13rc10_fix
# nco_path=
# Reminder: `nco_path=''` => the production-version NCO commands will be used
# e3sm_to_cmip_environment_commands=source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh
# diags_environment_commands=source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh
# mpas_analysis_environment_commands=source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh
# global_time_series_environment_commands=source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh
# livvkit_environment_commands=source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh
# pcmdi_diags_environment_commands=source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh
# environment_commands=source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh
# Reminder: `environment_commands=''` => the latest E3SM Unified environment will be used

# For reference:
alias sq
# Perlmutter: alias sq='sqa -u forsyth'
# Compy:      alias sq='sqa -u fors729'
# Chrysalis:  alias sq='sqa -u ac.forsyth2'
alias sqa
# Perlmutter: alias sqa='squeue -o "%8u %.7a %.4D %.9P %8i %.2t %.10r %.10M %.10l %j" --sort=P,-t,-p'
# Compy:      alias sqa='squeue -o "%8u %.7a %.4D %.9P %7i %.2t %.10r %.10M %.10l %.8Q %j" --sort=P,-t,-p'
# Chrysalis:  alias sqa='squeue -o "%8u %.7a %.4D %.9P %7i %.2t %.10r %.10M %.10l %j" --sort=P,-t,-p'
sq
# No jobs currently queued
ls tests/integration/generated/test_weekly_*_chrysalis.cfg # See all cfgs

# Launch jobs from ALL cfgs; start with the most important one, comprehensive_v3
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg

zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_comprehensive_v3_chrysalis.cfg

zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v3_chrysalis.cfg


sq | wc -l # Tue 05/12 09:48 => 233 - header row = 232 jobs
sq | wc -l # Tue 05/12 11:48 => 3 - header row = 2 jobs
sq | wc -l # Tue 05/12 15:46 => 1 - header row = 0 jobs

Bundles part 2

# Check on bundles status
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/zppy_main_branch_plus_unified_1_13rc10_fix/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_bundles_output/zppy_main_branch_plus_unified_1_13rc10_fix/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/zppy_main_branch_plus_unified_1_13rc10_fix/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors

# Now, run bundles part 2
cd ~/ez/zppy
git status
# Good, correct branch: test-zppy-main-plus-unified-1-13rc10-fix
# Good, correct env: Unified rc10

zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg
sq | wc -l # Tue 05/12 15:47 => 7 - header row = 6 jobs

Review finished runs

### v2  ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_fix/v2.LR.historical_0201/post/scripts
grep -v "OK" *status # Good, no errors

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_fix/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.status:RUNNING 1216551
# => See Debugging section later

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_fix/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.status:RUNNING 1216471
# => See Debugging section later

### v3 ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc10_fix/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc10_fix/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc10_fix/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors

### bundles ###
sq | wc -l # Tue 05/12 16:01 => 1 - header row = 0 jobs

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/zppy_main_branch_plus_unified_1_13rc10_fix/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, still no errors

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_bundles_output/zppy_main_branch_plus_unified_1_13rc10_fix/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, still no errors

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/zppy_main_branch_plus_unified_1_13rc10_fix/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, still no errors

Debugging

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_fix/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.status:RUNNING 1216551
tail e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.o*
# 2026-05-12 13:31:38,402 [WARNING]: warnings.py(_showwarnmsg:110) >> /lcrc/soft/climate/e3sm-unified/e3smu_1_13_0rc10/chrysalis/pixi/.pixi/envs/default/lib/python3.13/site-packages/cartopy/mpl/geoaxes.py:856: UserWarning: Attempting to set identical low and high xlims makes transformation singular; automatically expanding.
#   self.set_xlim([x1, x2])
# 2026-05-12 13:31:42,553 [INFO]: utils.py(_save_main_plot:122) >> Plot saved in: model_vs_obs_1982-1983/lat_lon/ERA5/ERA5-TREFHT-SON-land.png
# srun: Job step aborted: Waiting up to 92 seconds for job step to finish.
# slurmstepd: error: *** STEP 1216551.0 ON chr-0093 CANCELLED AT 2026-05-12T17:44:37 DUE TO TIME LIMIT ***
# slurmstepd: error: *** JOB 1216551 ON chr-0093 CANCELLED AT 2026-05-12T17:44:37 DUE TO TIME LIMIT ***
# srun: got SIGCONT
# srun: forcing job termination

grep -n 2026-05-12 e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.o* | head -n 1
# 55:2026-05-12 12:45:06,226 [INFO]: e3sm_diags_driver.py(_log_diagnostic_run_info:525) >> 

# So, this actually only runs from 12:45 to 13:31, or about 45 minutes. So the remaining 4h15m of its time, it's just hanging.

###############################################################################
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_fix/v2.LR.historical_0201/post/scripts
grep -v "OK" *status # Good, no errors

# Compare working version to hanging version -- output:
diff e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.o1216437 /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_fix/v2.LR.historical_0201/post/scripts/e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.o1216551 | wc -l
# 52,375 lines in the diff

# Compare working version to hanging version -- source scripts:
diff e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.bash /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_fix/v2.LR.historical_0201/post/scripts/e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.bash | wc -l
# Only 56 lines here
diff e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.bash /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_fix/v2.LR.historical_0201/post/scripts/e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.bash
# That gives:

< #SBATCH  --output=/lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_fix/v2.LR.historical_0201/post/scripts/e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.o%j
---
> #SBATCH  --output=/lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_fix/v2.LR.historical_0201/post/scripts/e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.o%j
30c30
< cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_fix/v2.LR.historical_0201/post/scripts
---
> cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_fix/v2.LR.historical_0201/post/scripts
99c99
< www="/lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v2_www/zppy_main_branch_plus_unified_1_13rc10_fix"
---
> www="/lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_www/zppy_main_branch_plus_unified_1_13rc10_fix"
128c128
<     cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_fix/v2.LR.historical_0201/post/scripts
---
>     cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_fix/v2.LR.historical_0201/post/scripts
147c147
<     cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_fix/v2.LR.historical_0201/post/scripts
---
>     cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_fix/v2.LR.historical_0201/post/scripts
157c157
< climo_dir_source=/lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_fix/v2.LR.historical_0201/post/atm/180x360_aave/clim/2yr
---
> climo_dir_source=/lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_fix/v2.LR.historical_0201/post/atm/180x360_aave/clim/2yr
164c164
< climo_diurnal_dir_source=/lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_fix/v2.LR.historical_0201/post/atm/180x360_aave/clim_diurnal_8xdaily/2yr
---
> climo_diurnal_dir_source=/lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_fix/v2.LR.historical_0201/post/atm/180x360_aave/clim_diurnal_8xdaily/2yr
168c168
< ts_dir_primary=/lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_fix/v2.LR.historical_0201/post/atm/180x360_aave/ts/monthly/2yr
---
> ts_dir_primary=/lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_fix/v2.LR.historical_0201/post/atm/180x360_aave/ts/monthly/2yr
170c170
< ts_daily_dir=/lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_fix/v2.LR.historical_0201/post/atm/180x360_aave/ts/daily/2yr
---
> ts_daily_dir=/lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_fix/v2.LR.historical_0201/post/atm/180x360_aave/ts/daily/2yr
172c172
< ts_rof_dir_primary="/lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_fix/v2.LR.historical_0201/post/rof/native/ts/monthly/2yr"
---
> ts_rof_dir_primary="/lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_fix/v2.LR.historical_0201/post/rof/native/ts/monthly/2yr"
295c295
< tc_param.test_data_path = "/lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_fix/v2.LR.historical_0201/post/atm/tc-analysis_${Y1}_${Y2}"
---
> tc_param.test_data_path = "/lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_fix/v2.LR.historical_0201/post/atm/tc-analysis_${Y1}_${Y2}"
326c326
<   cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_fix/v2.LR.historical_0201/post/scripts
---
>   cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_fix/v2.LR.historical_0201/post/scripts
340c340
<   cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_fix/v2.LR.historical_0201/post/scripts
---
>   cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_fix/v2.LR.historical_0201/post/scripts
350c350
<   cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_fix/v2.LR.historical_0201/post/scripts
---
>   cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_fix/v2.LR.historical_0201/post/scripts

# Compare working version to hanging version -- provenance cfgs:
diff provenance.20260512_164716_827216.cfg /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_fix/v2.LR.historical_0201/post/scripts/provenance.20260512_164807_787558.cfg | wc -l
# Only 19 lines here
diff provenance.20260512_164716_827216.cfg /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_fix/v2.LR.historical_0201/post/scripts/provenance.20260512_164807_787558.cfg
# That gives:

> # NOTE: This parameter was added AFTER v3.0.0.
> # It's only in this cfg so we can continue to test latest code versions on a legacy cfg.
11c13
< output = "/lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_fix/v2.LR.historical_0201"
---
> output = "/lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_fix/v2.LR.historical_0201"
14c16
< www = "/lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v2_www/zppy_main_branch_plus_unified_1_13rc10_fix"
---
> www = "/lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_www/zppy_main_branch_plus_unified_1_13rc10_fix"
145c147
<   reference_data_path = "/lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_fix/v2.LR.historical_0201/post/atm/180x360_aave/clim"
---
>   reference_data_path = "/lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_fix/v2.LR.historical_0201/post/atm/180x360_aave/clim"
162c164
<   reference_data_path = "/lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_fix/v2.LR.historical_0201/post/lnd/180x360_aave/clim"
---
>   reference_data_path = "/lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_fix/v2.LR.historical_0201/post/lnd/180x360_aave/clim"

post/scripts directory status: Mostly successful, but unfortunately e3sm_diags hangs on the legacy 3.0.0 & 3.1.0 E3SMv2 test cfgs. I did some debugging (see section immediately above), but I'm not much closer to an answer. I thought for sure we've encountered hangs like this before, but I'm not getting any good matches searching GitHub or Slack.

Nevertheless, let's proceed with the Python tests, so we can at least test the working cfgs and working tasks on these two particular cfgs:

Python tests

cd ~/ez/zppy
git status
# Good, correct branch: test-zppy-main-plus-unified-1-13rc10-fix
# Good, correct env: Unified rc10
ls tests/integration/test_*.py

# Errors look to be expected based on code changes since 3/30:
pytest tests/integration/test_bash_generation.py # 1 failed in 2.07s
pytest tests/integration/test_campaign.py # 6 failed in 2.48s
pytest tests/integration/test_defaults.py # 1 failed in 0.52s
pytest tests/integration/test_bundles.py # 1 failed, 1 passed in 0.25s

# Passing
pytest tests/integration/test_last_year.py # 1 passed in 0.36s

# Compute node image tests were introduced in https://github.com/E3SM-Project/zppy/pull/752.
salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm
# Re-establish Unified env
bash
source ~/.bashrc
source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh
pytest tests/integration/test_images.py # 1 failed in 715.82s (0:11:55)
cat test_images_summary.md

Output:

------------------------------------------------------- Captured stdout call -------------------------------------------------------
Preparing weekly cfg tests
Preparing legacy 3.1.0 cfg tests
Preparing legacy 3.0.0 cfg tests
Running 9 tests in parallel
Individual test logs will be written to test_<name>.log files
✓ Completed: bundles (3 tasks) (log: test_bundles.log)
✓ Completed: legacy_3.1.0_bundles (3 tasks) (log: test_legacy_3.1.0_bundles.log)
✓ Completed: legacy_3.1.0_comprehensive_v2 (4 tasks) (log: test_legacy_3.1.0_comprehensive_v2.log)
✓ Completed: comprehensive_v2 (4 tasks) (log: test_comprehensive_v2.log)
✓ Completed: legacy_3.0.0_bundles (3 tasks) (log: test_legacy_3.0.0_bundles.log)
✓ Completed: legacy_3.0.0_comprehensive_v2 (4 tasks) (log: test_legacy_3.0.0_comprehensive_v2.log)
✓ Completed: legacy_3.1.0_comprehensive_v3 (4 tasks) (log: test_legacy_3.1.0_comprehensive_v3.log)
✓ Completed: comprehensive_v3 (5 tasks) (log: test_comprehensive_v3.log)
✓ Completed: legacy_3.0.0_comprehensive_v3 (4 tasks) (log: test_legacy_3.0.0_comprehensive_v3.log)
Copy the output of test_images_summary.md to a Pull Request comment

Test Summary:
Test                                                    Total    Correct     Status
----------------------------------------------------------------------------------
bundles_e3sm_diags                                       1762       1762     ✓ PASS
bundles_global_time_series                                  3          3     ✓ PASS
bundles_ilamb                                             388        388     ✓ PASS
comprehensive_v2_e3sm_diags                              3791       3766     ✗ FAIL
comprehensive_v2_mpas_analysis                            856          0     ✗ FAIL
comprehensive_v2_global_time_series                        12         12     ✓ PASS
comprehensive_v2_ilamb                                    776        776     ✓ PASS
comprehensive_v3_e3sm_diags                              5354       5329     ✗ FAIL
comprehensive_v3_mpas_analysis                            856          0     ✗ FAIL
comprehensive_v3_global_time_series                      1404       1404     ✓ PASS
comprehensive_v3_ilamb                                    837        837     ✓ PASS
comprehensive_v3_pcmdi_diags                              647        400     ✗ FAIL
legacy_3.1.0_bundles_e3sm_diags                          1762       1762     ✓ PASS
legacy_3.1.0_bundles_global_time_series                     3          3     ✓ PASS
legacy_3.1.0_bundles_ilamb                                388        388     ✓ PASS
legacy_3.1.0_comprehensive_v2_e3sm_diags                 3791       2117     ✗ FAIL
legacy_3.1.0_comprehensive_v2_mpas_analysis               856          0     ✗ FAIL
legacy_3.1.0_comprehensive_v2_global_time_series           12         12     ✓ PASS
legacy_3.1.0_comprehensive_v2_ilamb                       776        776     ✓ PASS
legacy_3.1.0_comprehensive_v3_e3sm_diags                 5350       5330     ✗ FAIL
legacy_3.1.0_comprehensive_v3_mpas_analysis               856          0     ✗ FAIL
legacy_3.1.0_comprehensive_v3_global_time_series         1404       1404     ✓ PASS
legacy_3.1.0_comprehensive_v3_ilamb                       837        837     ✓ PASS
legacy_3.0.0_bundles_e3sm_diags                          1762       1762     ✓ PASS
legacy_3.0.0_bundles_global_time_series                     3          3     ✓ PASS
legacy_3.0.0_bundles_ilamb                                388        388     ✓ PASS
legacy_3.0.0_comprehensive_v2_e3sm_diags                 3791       2117     ✗ FAIL
legacy_3.0.0_comprehensive_v2_mpas_analysis               856          0     ✗ FAIL
legacy_3.0.0_comprehensive_v2_global_time_series           12         12     ✓ PASS
legacy_3.0.0_comprehensive_v2_ilamb                       776        776     ✓ PASS
legacy_3.0.0_comprehensive_v3_e3sm_diags                 5350       5330     ✗ FAIL
legacy_3.0.0_comprehensive_v3_mpas_analysis               856          0     ✗ FAIL
legacy_3.0.0_comprehensive_v3_global_time_series           90         90     ✓ PASS
legacy_3.0.0_comprehensive_v3_ilamb                       837        837     ✓ PASS
----------------------------------------------------------------------------------

⚠ Some tests had mismatched or missing images. Check individual log files for details.
===================================================== short test summary info ======================================================
FAILED tests/integration/test_images.py::test_images - assert 3791 == 3766
================================================== 1 failed in 715.82s (0:11:55) ===================================================

Complete summary table:

Test name	Total images	Correct images	Missing images	Mismatched images
bundles_e3sm_diags	1762	1762	0	0
bundles_global_time_series	3	3	0	0
bundles_ilamb	388	388	0	0
comprehensive_v2_e3sm_diags	3791	3766	0	25 (list, grid)
comprehensive_v2_mpas_analysis	856	0	856 (list)	0
comprehensive_v2_global_time_series	12	12	0	0
comprehensive_v2_ilamb	776	776	0	0
comprehensive_v3_e3sm_diags	5354	5329	0	25 (list, grid)
comprehensive_v3_mpas_analysis	856	0	856 (list)	0
comprehensive_v3_global_time_series	1404	1404	0	0
comprehensive_v3_ilamb	837	837	0	0
comprehensive_v3_pcmdi_diags	647	400	0	247 (list, grid)
legacy_3.1.0_bundles_e3sm_diags	1762	1762	0	0
legacy_3.1.0_bundles_global_time_series	3	3	0	0
legacy_3.1.0_bundles_ilamb	388	388	0	0
legacy_3.1.0_comprehensive_v2_e3sm_diags	3791	2117	1669 (list)	5 (list, grid)
legacy_3.1.0_comprehensive_v2_mpas_analysis	856	0	856 (list)	0
legacy_3.1.0_comprehensive_v2_global_time_series	12	12	0	0
legacy_3.1.0_comprehensive_v2_ilamb	776	776	0	0
legacy_3.1.0_comprehensive_v3_e3sm_diags	5350	5330	0	20 (list, grid)
legacy_3.1.0_comprehensive_v3_mpas_analysis	856	0	856 (list)	0
legacy_3.1.0_comprehensive_v3_global_time_series	1404	1404	0	0
legacy_3.1.0_comprehensive_v3_ilamb	837	837	0	0
legacy_3.0.0_bundles_e3sm_diags	1762	1762	0	0
legacy_3.0.0_bundles_global_time_series	3	3	0	0
legacy_3.0.0_bundles_ilamb	388	388	0	0
legacy_3.0.0_comprehensive_v2_e3sm_diags	3791	2117	1669 (list)	5 (list, grid)
legacy_3.0.0_comprehensive_v2_mpas_analysis	856	0	856 (list)	0
legacy_3.0.0_comprehensive_v2_global_time_series	12	12	0	0
legacy_3.0.0_comprehensive_v2_ilamb	776	776	0	0
legacy_3.0.0_comprehensive_v3_e3sm_diags	5350	5330	0	20 (list, grid)
legacy_3.0.0_comprehensive_v3_mpas_analysis	856	0	856 (list)	0
legacy_3.0.0_comprehensive_v3_global_time_series	90	90	0	0
legacy_3.0.0_comprehensive_v3_ilamb	837	837	0	0

Summary table, if we consider only failing image-check tests:

Test name	Total images	Correct images	Missing images	Mismatched images
comprehensive_v2_e3sm_diags	3791	3766	0	25 (list, grid)
comprehensive_v2_mpas_analysis	856	0	856 (list)	0
comprehensive_v3_e3sm_diags	5354	5329	0	25 (list, grid)
comprehensive_v3_mpas_analysis	856	0	856 (list)	0
comprehensive_v3_pcmdi_diags	647	400	0	247 (list, grid)
legacy_3.1.0_comprehensive_v2_e3sm_diags	3791	2117	1669 (list)	5 (list, grid)
legacy_3.1.0_comprehensive_v2_mpas_analysis	856	0	856 (list)	0
legacy_3.1.0_comprehensive_v3_e3sm_diags	5350	5330	0	20 (list, grid)
legacy_3.1.0_comprehensive_v3_mpas_analysis	856	0	856 (list)	0
legacy_3.0.0_comprehensive_v2_e3sm_diags	3791	2117	1669 (list)	5 (list, grid)
legacy_3.0.0_comprehensive_v2_mpas_analysis	856	0	856 (list)	0
legacy_3.0.0_comprehensive_v3_e3sm_diags	5350	5330	0	20 (list, grid)
legacy_3.0.0_comprehensive_v3_mpas_analysis	856	0	856 (list)	0

Summary table -- only failing image-check tests, sorted by task

mpas_analysis

Test name	Total images	Missing images
comprehensive_v2_mpas_analysis	856	856 (list)
comprehensive_v3_mpas_analysis	856	856 (list)
legacy_3.1.0_comprehensive_v2_mpas_analysis	856	856 (list)
legacy_3.1.0_comprehensive_v3_mpas_analysis	856	856 (list)
legacy_3.0.0_comprehensive_v2_mpas_analysis	856	856 (list)
legacy_3.0.0_comprehensive_v3_mpas_analysis	856	856 (list)

The 856 missing images are to be expected, based on the 2026-05-08 Chrysalis E3SM Unified 1.13.0rc10 (zppy v3.2.0rc2) results

pcmdi_diags

Test name	Total images	Correct images	Missing images	Mismatched images
comprehensive_v3_pcmdi_diags	647	400	0	247 (list, grid)

The 247 mismatched images are to be expected, based on the 2026-05-08 Chrysalis E3SM Unified 1.13.0rc10 (zppy v3.2.0rc2) results

e3sm_diags

Test name	Total images	Correct images	Missing images	Mismatched images
comprehensive_v2_e3sm_diags	3791	3766	0	25 (list, grid)
comprehensive_v3_e3sm_diags	5354	5329	0	25 (list, grid)
legacy_3.1.0_comprehensive_v2_e3sm_diags	3791	2117	1669 (list)	5 (list, grid)
legacy_3.1.0_comprehensive_v3_e3sm_diags	5350	5330	0	20 (list, grid)
legacy_3.0.0_comprehensive_v2_e3sm_diags	3791	2117	1669 (list)	5 (list, grid)
legacy_3.0.0_comprehensive_v3_e3sm_diags	5350	5330	0	20 (list, grid)

A few things to note here:

The non-legacy v2 & v3 tests each have 25 mismatched images, which is to be expected, based on what we saw on the 2026-05-08 Compy E3SM Unified 1.13.0rc10 (zppy v3.2.0rc2) results
We didn't run the legacy tests on either Compy or Perlmutter
- legacy_3.1.0_comprehensive_v3_e3sm_diags & legacy_3.0.0_comprehensive_v3_e3sm_diags: I think the 20 mismatched images here are analagous and to be expected.
- legacy_3.1.0_comprehensive_v2_e3sm_diags & legacy_3.0.0_comprehensive_v2_e3sm_diags have errors because they hung and didn't finish. I'm actually surprised any images are present.

Results Analysis

Everything seems to be working except for legacy_3.1.0_comprehensive_v2_e3sm_diags and legacy_3.0.0_comprehensive_v2_e3sm_diags, which hung after ~45 minutes and never finished running.

I did some initial debugging (see above), but I can't find an obvious cause. It might be worthwhile to rerun the same bash file and see if the job hangs again. But if it doesn't hang, then we'd have a non-reproducible bug.

❌ E3SM Unified 1.13.0rc10 (fixed script) is not fully passing on Chrysalis.

15 replies

zhangshixuan1987 May 13, 2026
Collaborator

@forsyth2 @chengzhuzhang: Hi Ryan and Jill, in my view, having multiprocessing = False work correctly is good to have for robustness and debugging. However, I would not recommend using multiprocessing = False as the default or suggested setting for PCMDI diagnostics, since it can make the workflow much slower. For example, the mean-climate diagnostics include more than 20 variables, and looping through them serially is not efficient.

forsyth2 May 13, 2026
Maintainer Author

Hi @zhangshixuan1987 -- just for clarity, I haven't changed any multiprocessing settings for pcmdi_diags here, only in a one-off test case for a single e3sm_diags job.

chengzhuzhang May 13, 2026
Maintainer

@forsyth2 for proceed with testing, I think we should do 3:
3. Re-launch the entire Chrysalis test suite to have a clean start. Update expected results based on that.

This is the cleanest and most straightforward approach, and it should also help confirm whether the previous hang was just a one-time glitch.

Otherwise option 2 can also help to confirm.
2. Just re-sbatch legacy 3.1.0's e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.bash with the correct setting of multiprocessing and re-run the image checker. Update expected results based on that.
If the hang re-occurs, we will have to investigate further.

forsyth2 May 13, 2026
Maintainer Author

@chengzhuzhang Thanks, I'll get the full test suite going. That is definitely best for complete coverage.

zhangshixuan1987 May 13, 2026
Collaborator

Hi @zhangshixuan1987 -- just for clarity, I haven't changed any multiprocessing settings for pcmdi_diags here, only in a one-off test case for a single e3sm_diags job.

I apologize for the misunderstanding. Please ignore my comments then.

forsyth2 · 2026-05-13T17:09:42Z

forsyth2
May 13, 2026
Maintainer Author

2026-05-12 Perlmutter E3SM Unified 1.13.0rc10, post-fix (zppy v3.2.0rc2)

Chrysalis: run tests on all 9 cfgs
Compy: run all 3 non-legacy test cfgs
Perlmutter: run just the non-legacy v3 test cfg

What expected results are currently on Compy?

Perlmutter's expected results were updated after the last test run -- 2026-05-08 Perlmutter E3SM Unified 1.13.0rc10 (zppy v3.2.0rc2).

What changes are we testing?

Since 2026-05-08 Perlmutter E3SM Unified 1.13.0rc10 (zppy v3.2.0rc2), the only change has been the updating of the Unified load script, per the discussion in #819.

We also just merged #818, which only affects testing code.

Testing process

Starting actual testing on 2026-05-12
Branch: zppy main branch
Environments: zppy & packages for all tasks -- E3SM Unified 1.13.0rc10 (zppy v3.2.0rc2)

Set up environments

We can mostly skip this step, since all the packages will be tested using the E3SM Unified RC.

cd ~/ez/zppy-interfaces
git status
# # On branch main
# nothing to commit, working directory clean
git fetch upstream main
git checkout main
git reset --hard upstream/main
git log --oneline | head -n 1
# 6b0758f Bump to 0.2.1rc1 (#48)
# Good, matches https://github.com/E3SM-Project/zppy-interfaces/commits/main
bash # Run bash so we're in an isolated subshell
source ~/.bash_profile.ext
source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc10_pm-cpu.sh
# Check zppy-interfaces tests
pytest tests/unit/global_time_series/test_*.py # 10 passed, 2 warnings in 9.57s
# SyntaxWarning: invalid escape sequence '\w'
# SyntaxWarning: invalid escape sequence '\s'
pytest tests/unit/pcmdi_diags/test_*.py #  7 passed in 7.91s

# Stay in the bash subshell

zppy run

# zppy itself #################################################################
cd ~/ez/zppy
git status
# On branch test-zppy-main-plus-unified-1-13rc10
git add -A
git commit -m "Testing" --no-verify # Clean up uncommitted changes

git fetch upstream main
git checkout -b test-zppy-main-plus-unified-1-13rc10-fix upstream/main
git log --oneline | head -n 3
# 885d53cb Fix image checker (#818)
# f368ab71 Bump to 3.2.0rc2 (#816)
# f335f0e3 Merge pull request #814 from E3SM-Project/fixes-of-rc8
# Good, matches https://github.com/E3SM-Project/zppy/commits/main

showquota # Have enough space on home and pscratch
# (Currently in /global/homes/f/forsyth/ez/zppy)

# WE ARE ALREADY IN UNIFIED RC10
# DO NOT MAKE A DEV ENV

pixi list python # Check Python version
# python                    3.13.13

# Run zppy unit tests
pytest tests/test_*.py # 44 passed in 0.46s

# Edit tests/integration/utils.py:

TEST_SPECIFICS: Dict[str, Any] = {
    # This is the NCO path.
    # Keep as "" to use the production-version NCO commands.
    # Set to a specific path to use development-version NCO commands.
    "nco_path": "",
    # These are custom environment_commands for specific tasks.
    # Never set these to "", because they will print the line
    # `environment_commands = ""` for the corresponding task,
    # thus overriding the value set higher up in the cfg.
    # That is, there will be no environment set.
    # (`environment_commands = ""` only redirects to Unified
    # if specified under the [default] task)
    "e3sm_to_cmip_environment_commands": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc10_pm-cpu.sh",
    "diags_environment_commands": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc10_pm-cpu.sh",
    "mpas_analysis_environment_commands": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc10_pm-cpu.sh",
    "global_time_series_environment_commands": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc10_pm-cpu.sh",
    "livvkit_environment_commands": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc10_pm-cpu.sh",
    "pcmdi_diags_environment_commands": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc10_pm-cpu.sh",
    # This is the environment setup for other tasks.
    # Leave as "" to use the latest Unified environment.
    "environment_commands": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc10_pm-cpu.sh",
    # For a complete test, run the set of latest cfgs and at least one set of legacy cfgs
    "cfgs_to_run": [
        # "weekly_bundles",
        # "weekly_comprehensive_v2",
        "weekly_comprehensive_v3",
        # "weekly_legacy_3.1.0_bundles",
        # "weekly_legacy_3.1.0_comprehensive_v2",
        # "weekly_legacy_3.1.0_comprehensive_v3",
        # "weekly_legacy_3.0.0_bundles",
        # "weekly_legacy_3.0.0_comprehensive_v2",
        # "weekly_legacy_3.0.0_comprehensive_v3",
    ],
    "tasks_to_run": [
        "e3sm_diags",
        "mpas_analysis",
        "global_time_series",
        "ilamb",
        "livvkit",
        "pcmdi_diags",
    ],
    "unique_id": "zppy_main_branch_plus_unified_1_13rc10_fix",
}

git diff # Diff looks good
python tests/integration/utils.py
# UNIQUE_ID=zppy_main_branch_plus_unified_1_13rc10_fix
# nco_path=
# Reminder: `nco_path=''` => the production-version NCO commands will be used
# e3sm_to_cmip_environment_commands=source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc10_pm-cpu.sh
# diags_environment_commands=source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc10_pm-cpu.sh
# mpas_analysis_environment_commands=source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc10_pm-cpu.sh
# global_time_series_environment_commands=source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc10_pm-cpu.sh
# livvkit_environment_commands=source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc10_pm-cpu.sh
# pcmdi_diags_environment_commands=source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc10_pm-cpu.sh
# environment_commands=source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc10_pm-cpu.sh
# Reminder: `environment_commands=''` => the latest E3SM Unified environment will be used

# For reference:
alias sq
# Perlmutter: alias sq='sqa -u forsyth'
# Compy:      alias sq='sqa -u fors729'
alias sqa
# Perlmutter: alias sqa='squeue -o "%8u %.7a %.4D %.9P %8i %.2t %.10r %.10M %.10l %j" --sort=P,-t,-p'
# Compy:      alias sqa='squeue -o "%8u %.7a %.4D %.9P %7i %.2t %.10r %.10M %.10l %.8Q %j" --sort=P,-t,-p'
sq
# No jobs currently queued

# This is the only cfg we're running on Perlmutter:
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_pm-cpu.cfg

sq | wc -l # Tue 05/12 10:08 => 56 - header row = 55 jobs
sq | wc -l # Tue 05/12 11:47 => 50 - header row = 49 jobs

# Picking up Wed 5/13 ########################################################
sq | wc -l # Wed 05/13 09:28 => 1 - header row = 0 jobs

Review finished runs

# Re-establish Unified env
bash
source ~/.bash_profile.ext
source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc10_pm-cpu.sh

### v3 ###
cd /global/cfs/cdirs/e3sm/forsyth/zppy_weekly_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc10_fix/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors

post/scripts directory status: NO ERRORS

Python tests

cd ~/ez/zppy
git status
# Good, correct branch: test-zppy-main-plus-unified-1-13rc10-fix
# Good, correct env: Unified rc10
ls tests/integration/test_*.py

# Non-applicable (we didn't do the necessary `zppy` runs for this test):
# pytest tests/integration/test_bundles.py 

# Passing
pytest tests/integration/test_bash_generation.py # 1 passed in 1.15s
pytest tests/integration/test_campaign.py # 6 passed in 3.24s
pytest tests/integration/test_defaults.py # 1 passed in 0.56s
pytest tests/integration/test_last_year.py # 1 passed in 0.55s

# We'll now proceed with the image checker test.
# On Chrysalis, we were running `salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm`
# Compute node image tests were introduced in https://github.com/E3SM-Project/zppy/pull/752.
# Referring to notes from the week that merged, it looks like we want to use this line to get a Perlmutter compute note:
salloc --nodes=1 --qos=interactive --time=01:00:00 --constraint=cpu --account=e3sm
# Re-establish Unified env
bash
source ~/.bash_profile.ext
source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.13.0rc10_pm-cpu.sh
pytest tests/integration/test_images.py # 1 passed in 1963.77s (0:32:43)
cat test_images_summary.md

Complete summary table:

Test name	Total images	Correct images
comprehensive_v3_e3sm_diags	5369	5369
comprehensive_v3_mpas_analysis	1280	1280
comprehensive_v3_global_time_series	1404	1404
comprehensive_v3_ilamb	837	837
comprehensive_v3_pcmdi_diags	647	647

Results Analysis

✅ E3SM Unified 1.13.0rc10 (fixed script) passes on Perlmutter. Since tests pass, there is no need to update expected results.

0 replies

forsyth2 · 2026-05-13T23:41:46Z

forsyth2
May 13, 2026
Maintainer Author

2026-05-12 Chrysalis E3SM Unified 1.13.0rc10, post-fix (zppy v3.2.0rc2), try 2

Chrysalis: run tests on all 9 cfgs
Compy: run all 3 non-legacy test cfgs
Perlmutter: run just the non-legacy v3 test cfg

What expected results are currently on Chrysalis?

The latest expected results for Compy & Perlmutter are still the E3SM Unified 1.12.0 official results. Chrysalis, however, has had its expected results updated much more recently, on 2026-03-30.

What changes are we testing?

There have been no changes since try 1 -- 2026-05-12 Chrysalis E3SM Unified 1.13.0rc10, post-fix (zppy v3.2.0rc2)

Therefore, this test will let us know if #822 is truly an intermittent, non-reproducible bug

Possible outcomes:

Hang happens again: we should hold try to figure out a fix, and where that fix would go (e.g., in e3sm_diags, zppy, or Unified load script)
Hang doesn't happen: presumably then all tests will pass aside from expected failures. Then, we can update the expected results on Chrysalis and move forward with production releases.

Testing process

Starting actual testing on 2026-05-13
Branch: zppy main branch
Environments: zppy & packages for all tasks -- E3SM Unified 1.13.0rc10

Set up environments

We can mostly skip this step, since all the packages will be tested using the E3SM Unified RC.

cd ~/ez/zppy-interfaces
git status
# On branch main
# nothing to commit, working directory clean

git fetch upstream main
git checkout main
git reset --hard upstream/main
git log --oneline | head -n 1
# 6b0758f Bump to 0.2.1rc1 (#48)
# Good, matches https://github.com/E3SM-Project/zppy-interfaces/commits/main

bash # Run bash so we're in an isolated subshell
source ~/.bashrc
source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh
# Check zppy-interfaces tests
pytest tests/unit/global_time_series/test_*.py # 10 passed, 2 warnings in 25.63s
# SyntaxWarning: invalid escape sequence '\w'
# SyntaxWarning: invalid escape sequence '\s'
pytest tests/unit/pcmdi_diags/test_*.py # 7 passed in 11.41s

# Stay in the bash subshell

zppy run

# zppy itself #################################################################
cd ~/ez/zppy
git status
# On branch test-zppy-main-plus-unified-1-13rc10-fix
git add -A
git commit -m "Testing" --no-verify # Clean up workspace

git fetch upstream main
git checkout -b test-zppy-main-plus-unified-1-13rc10-fix-try2 upstream/main
git log --oneline | head -n 3
# 885d53cb Fix image checker (#818)
# f368ab71 Bump to 3.2.0rc2 (#816)
# f335f0e3 Merge pull request #814 from E3SM-Project/fixes-of-rc8
# Good, matches https://github.com/E3SM-Project/zppy/commits/main

/usr/lpp/mmfs/bin/mmlsquota -u ac.forsyth2 --block-size T fs2 # Have enough space on /lcrc/group/e3sm/
lcrc-quota # Have enough space on /home/ac.forsyth2/

# WE ARE ALREADY IN UNIFIED RC10
# DO NOT MAKE A DEV ENV

pixi list python # Check Python version
# python                    3.13.13

# Run zppy unit tests
pytest tests/test_*.py # 44 passed in 0.70s

# Edit tests/integration/utils.py:

TEST_SPECIFICS: Dict[str, Any] = {
    # This is the NCO path.
    # Keep as "" to use the production-version NCO commands.
    # Set to a specific path to use development-version NCO commands.
    "nco_path": "",
    # These are custom environment_commands for specific tasks.
    # Never set these to "", because they will print the line
    # `environment_commands = ""` for the corresponding task,
    # thus overriding the value set higher up in the cfg.
    # That is, there will be no environment set.
    # (`environment_commands = ""` only redirects to Unified
    # if specified under the [default] task)
    "e3sm_to_cmip_environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh",
    "diags_environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh",
    "mpas_analysis_environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh",
    "global_time_series_environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh",
    "livvkit_environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh",
    "pcmdi_diags_environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh",
    # This is the environment setup for other tasks.
    # Leave as "" to use the latest Unified environment.
    "environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh",
    # For a complete test, run the set of latest cfgs and at least one set of legacy cfgs
    "cfgs_to_run": [
        "weekly_bundles",
        "weekly_comprehensive_v2",
        "weekly_comprehensive_v3",
        "weekly_legacy_3.1.0_bundles",
        "weekly_legacy_3.1.0_comprehensive_v2",
        "weekly_legacy_3.1.0_comprehensive_v3",
        "weekly_legacy_3.0.0_bundles",
        "weekly_legacy_3.0.0_comprehensive_v2",
        "weekly_legacy_3.0.0_comprehensive_v3",
    ],
    "tasks_to_run": [
        "e3sm_diags",
        "mpas_analysis",
        "global_time_series",
        "ilamb",
        "livvkit",
        "pcmdi_diags",
    ],
    "unique_id": "zppy_main_branch_plus_unified_1_13rc10_fix_try2",
}

git diff # Diff looks good
python tests/integration/utils.py
# CFG FILES HAVE BEEN GENERATED FROM TEMPLATES WITH THESE SETTINGS:
# UNIQUE_ID=zppy_main_branch_plus_unified_1_13rc10_fix_try2
# nco_path=
# Reminder: `nco_path=''` => the production-version NCO commands will be used
# e3sm_to_cmip_environment_commands=source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh
# diags_environment_commands=source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh
# mpas_analysis_environment_commands=source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh
# global_time_series_environment_commands=source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh
# livvkit_environment_commands=source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh
# pcmdi_diags_environment_commands=source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh
# environment_commands=source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh
# Reminder: `environment_commands=''` => the latest E3SM Unified environment will be used

# For reference:
alias sq
# Perlmutter: alias sq='sqa -u forsyth'
# Compy:      alias sq='sqa -u fors729'
# Chrysalis:  alias sq='sqa -u ac.forsyth2'
alias sqa
# Perlmutter: alias sqa='squeue -o "%8u %.7a %.4D %.9P %8i %.2t %.10r %.10M %.10l %j" --sort=P,-t,-p'
# Compy:      alias sqa='squeue -o "%8u %.7a %.4D %.9P %7i %.2t %.10r %.10M %.10l %.8Q %j" --sort=P,-t,-p'
# Chrysalis:  alias sqa='squeue -o "%8u %.7a %.4D %.9P %7i %.2t %.10r %.10M %.10l %j" --sort=P,-t,-p'
sq
# No jobs currently queued
ls tests/integration/generated/test_weekly_*_chrysalis.cfg # See all cfgs

# Launch jobs from ALL cfgs; start with the most important one, comprehensive_v3
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg

zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_comprehensive_v3_chrysalis.cfg

zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v3_chrysalis.cfg


sq | wc -l # Wed 05/13 12:07 => 233 - header row = 232 jobs
sq | wc -l # Wed 05/13 14:34 => 1 - header row = 0 jobs

Bundles part 2

# Check on bundles status
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/zppy_main_branch_plus_unified_1_13rc10_fix_try2/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_bundles_output/zppy_main_branch_plus_unified_1_13rc10_fix_try2/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/zppy_main_branch_plus_unified_1_13rc10_fix_try2/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors

# Now, run bundles part 2
cd ~/ez/zppy
git status
# Good, correct branch: test-zppy-main-plus-unified-1-13rc10-fix-try2
# Good, correct env: Unified rc10

zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg
sq | wc -l # Wed 05/13 14:37 => 7 - header row = 6 jobs

Review finished runs

### v2  ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_fix_try2/v2.LR.historical_0201/post/scripts
grep -v "OK" *status # Good, no errors

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_fix_try2/v2.LR.historical_0201/post/scripts
grep -v "OK" *status # Good, no errors -- this is an improvement over the last run!

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_fix_try2/v2.LR.historical_0201/post/scripts
grep -v "OK" *status # Good, no errors -- this is an improvement over the last run!

### v3 ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc10_fix_try2/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc10_fix_try2/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc10_fix_try2/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors

### bundles ###
sq | wc -l # Wed 05/13 14:46 => 1 - header row = 0 jobs

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/zppy_main_branch_plus_unified_1_13rc10_fix_try2/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, still no errors

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_bundles_output/zppy_main_branch_plus_unified_1_13rc10_fix_try2/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, still no errors

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/zppy_main_branch_plus_unified_1_13rc10_fix_try2/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, still no errors

post/scripts directory status: NO ERRORS. This is evidence that #822 is indeed an intermittent, non-reproducible bug.

Python tests

cd ~/ez/zppy
git status
# Good, correct branch: test-zppy-main-plus-unified-1-13rc10-fix-try2
# Good, correct env: Unified rc10
ls tests/integration/test_*.py

# Errors look to be expected based on code changes since 3/30:
pytest tests/integration/test_bash_generation.py # 1 failed in 2.31s
pytest tests/integration/test_campaign.py # 6 failed in 2.96s
pytest tests/integration/test_defaults.py # 1 failed in 1.37s
pytest tests/integration/test_bundles.py # 1 failed, 1 passed in 0.68s

# Passing
pytest tests/integration/test_last_year.py # 1 passed in 1.23s

# Compute node image tests were introduced in https://github.com/E3SM-Project/zppy/pull/752.
salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm
# Re-establish Unified env
bash
source ~/.bashrc
source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh
pytest tests/integration/test_images.py # 1 failed in 761.38s (0:12:41)
# HERE 14:52

cat test_images_summary.md

Output:

------------------------------------------------------------------------------------ Captured stdout call -------------------------------------------------------------------------------------
Preparing weekly cfg tests
Preparing legacy 3.1.0 cfg tests
Preparing legacy 3.0.0 cfg tests
Running 9 tests in parallel
Individual test logs will be written to test_<name>.log files
✓ Completed: legacy_3.1.0_bundles (3 tasks) (log: test_legacy_3.1.0_bundles.log)
✓ Completed: bundles (3 tasks) (log: test_bundles.log)
✓ Completed: comprehensive_v2 (4 tasks) (log: test_comprehensive_v2.log)
✓ Completed: legacy_3.1.0_comprehensive_v2 (4 tasks) (log: test_legacy_3.1.0_comprehensive_v2.log)
✓ Completed: legacy_3.0.0_bundles (3 tasks) (log: test_legacy_3.0.0_bundles.log)
✓ Completed: legacy_3.0.0_comprehensive_v2 (4 tasks) (log: test_legacy_3.0.0_comprehensive_v2.log)
✓ Completed: legacy_3.1.0_comprehensive_v3 (4 tasks) (log: test_legacy_3.1.0_comprehensive_v3.log)
✓ Completed: legacy_3.0.0_comprehensive_v3 (4 tasks) (log: test_legacy_3.0.0_comprehensive_v3.log)
✓ Completed: comprehensive_v3 (5 tasks) (log: test_comprehensive_v3.log)
Copy the output of test_images_summary.md to a Pull Request comment

Test Summary:
Test                                                    Total    Correct     Status
----------------------------------------------------------------------------------
bundles_e3sm_diags                                       1762       1762     ✓ PASS
bundles_global_time_series                                  3          3     ✓ PASS
bundles_ilamb                                             388        388     ✓ PASS
comprehensive_v2_e3sm_diags                              3791       3766     ✗ FAIL
comprehensive_v2_mpas_analysis                            856          0     ✗ FAIL
comprehensive_v2_global_time_series                        12         12     ✓ PASS
comprehensive_v2_ilamb                                    776        776     ✓ PASS
comprehensive_v3_e3sm_diags                              5354       5329     ✗ FAIL
comprehensive_v3_mpas_analysis                            856          0     ✗ FAIL
comprehensive_v3_global_time_series                      1404       1404     ✓ PASS
comprehensive_v3_ilamb                                    837        837     ✓ PASS
comprehensive_v3_pcmdi_diags                              647        400     ✗ FAIL
legacy_3.1.0_bundles_e3sm_diags                          1762       1762     ✓ PASS
legacy_3.1.0_bundles_global_time_series                     3          3     ✓ PASS
legacy_3.1.0_bundles_ilamb                                388        388     ✓ PASS
legacy_3.1.0_comprehensive_v2_e3sm_diags                 3791       3766     ✗ FAIL
legacy_3.1.0_comprehensive_v2_mpas_analysis               856          0     ✗ FAIL
legacy_3.1.0_comprehensive_v2_global_time_series           12         12     ✓ PASS
legacy_3.1.0_comprehensive_v2_ilamb                       776        776     ✓ PASS
legacy_3.1.0_comprehensive_v3_e3sm_diags                 5350       5330     ✗ FAIL
legacy_3.1.0_comprehensive_v3_mpas_analysis               856          0     ✗ FAIL
legacy_3.1.0_comprehensive_v3_global_time_series         1404       1404     ✓ PASS
legacy_3.1.0_comprehensive_v3_ilamb                       837        837     ✓ PASS
legacy_3.0.0_bundles_e3sm_diags                          1762       1762     ✓ PASS
legacy_3.0.0_bundles_global_time_series                     3          3     ✓ PASS
legacy_3.0.0_bundles_ilamb                                388        388     ✓ PASS
legacy_3.0.0_comprehensive_v2_e3sm_diags                 3791       3766     ✗ FAIL
legacy_3.0.0_comprehensive_v2_mpas_analysis               856          0     ✗ FAIL
legacy_3.0.0_comprehensive_v2_global_time_series           12         12     ✓ PASS
legacy_3.0.0_comprehensive_v2_ilamb                       776        776     ✓ PASS
legacy_3.0.0_comprehensive_v3_e3sm_diags                 5350       5330     ✗ FAIL
legacy_3.0.0_comprehensive_v3_mpas_analysis               856          0     ✗ FAIL
legacy_3.0.0_comprehensive_v3_global_time_series           90         90     ✓ PASS
legacy_3.0.0_comprehensive_v3_ilamb                       837        837     ✓ PASS
----------------------------------------------------------------------------------

⚠ Some tests had mismatched or missing images. Check individual log files for details.
=================================================================================== short test summary info ===================================================================================
FAILED tests/integration/test_images.py::test_images - assert 3791 == 3766
================================================================================ 1 failed in 761.38s (0:12:41) ================================================================================

Complete summary table:

Test name	Total images	Correct images	Missing images	Mismatched images
bundles_e3sm_diags	1762	1762	0	0
bundles_global_time_series	3	3	0	0
bundles_ilamb	388	388	0	0
comprehensive_v2_e3sm_diags	3791	3766	0	25 (list, grid)
comprehensive_v2_mpas_analysis	856	0	856 (list)	0
comprehensive_v2_global_time_series	12	12	0	0
comprehensive_v2_ilamb	776	776	0	0
comprehensive_v3_e3sm_diags	5354	5329	0	25 (list, grid)
comprehensive_v3_mpas_analysis	856	0	856 (list)	0
comprehensive_v3_global_time_series	1404	1404	0	0
comprehensive_v3_ilamb	837	837	0	0
comprehensive_v3_pcmdi_diags	647	400	0	247 (list, grid)
legacy_3.1.0_bundles_e3sm_diags	1762	1762	0	0
legacy_3.1.0_bundles_global_time_series	3	3	0	0
legacy_3.1.0_bundles_ilamb	388	388	0	0
legacy_3.1.0_comprehensive_v2_e3sm_diags	3791	3766	0	25 (list, grid)
legacy_3.1.0_comprehensive_v2_mpas_analysis	856	0	856 (list)	0
legacy_3.1.0_comprehensive_v2_global_time_series	12	12	0	0
legacy_3.1.0_comprehensive_v2_ilamb	776	776	0	0
legacy_3.1.0_comprehensive_v3_e3sm_diags	5350	5330	0	20 (list, grid)
legacy_3.1.0_comprehensive_v3_mpas_analysis	856	0	856 (list)	0
legacy_3.1.0_comprehensive_v3_global_time_series	1404	1404	0	0
legacy_3.1.0_comprehensive_v3_ilamb	837	837	0	0
legacy_3.0.0_bundles_e3sm_diags	1762	1762	0	0
legacy_3.0.0_bundles_global_time_series	3	3	0	0
legacy_3.0.0_bundles_ilamb	388	388	0	0
legacy_3.0.0_comprehensive_v2_e3sm_diags	3791	3766	0	25 (list, grid)
legacy_3.0.0_comprehensive_v2_mpas_analysis	856	0	856 (list)	0
legacy_3.0.0_comprehensive_v2_global_time_series	12	12	0	0
legacy_3.0.0_comprehensive_v2_ilamb	776	776	0	0
legacy_3.0.0_comprehensive_v3_e3sm_diags	5350	5330	0	20 (list, grid)
legacy_3.0.0_comprehensive_v3_mpas_analysis	856	0	856 (list)	0
legacy_3.0.0_comprehensive_v3_global_time_series	90	90	0	0
legacy_3.0.0_comprehensive_v3_ilamb	837	837	0	0

Summary table, if we consider only failing image-check tests:

Test name	Total images	Correct images	Missing images	Mismatched images
comprehensive_v2_e3sm_diags	3791	3766	0	25 (list, grid)
comprehensive_v2_mpas_analysis	856	0	856 (list)	0
comprehensive_v3_e3sm_diags	5354	5329	0	25 (list, grid)
comprehensive_v3_mpas_analysis	856	0	856 (list)	0
comprehensive_v3_pcmdi_diags	647	400	0	247 (list, grid)
legacy_3.1.0_comprehensive_v2_e3sm_diags	3791	3766	0	25 (list, grid)
legacy_3.1.0_comprehensive_v2_mpas_analysis	856	0	856 (list)	0
legacy_3.1.0_comprehensive_v3_e3sm_diags	5350	5330	0	20 (list, grid)
legacy_3.1.0_comprehensive_v3_mpas_analysis	856	0	856 (list)	0
legacy_3.0.0_comprehensive_v2_e3sm_diags	3791	3766	0	25 (list, grid)
legacy_3.0.0_comprehensive_v2_mpas_analysis	856	0	856 (list)	0
legacy_3.0.0_comprehensive_v3_e3sm_diags	5350	5330	0	20 (list, grid)
legacy_3.0.0_comprehensive_v3_mpas_analysis	856	0	856 (list)	0

Summary table -- only failing image-check tests, sorted by task

mpas_analysis

Test name	Total images	Missing images
comprehensive_v2_mpas_analysis	856	856 (list)
comprehensive_v3_mpas_analysis	856	856 (list)
legacy_3.1.0_comprehensive_v2_mpas_analysis	856	856 (list)
legacy_3.1.0_comprehensive_v3_mpas_analysis	856	856 (list)
legacy_3.0.0_comprehensive_v2_mpas_analysis	856	856 (list)
legacy_3.0.0_comprehensive_v3_mpas_analysis	856	856 (list)

These are consistent with 2026-05-12 Chrysalis E3SM Unified 1.13.0rc10, post-fix (zppy v3.2.0rc2) -- we just need to update the expected results now.

pcmdi_diags

Test name	Total images	Correct images	Missing images	Mismatched images
comprehensive_v3_pcmdi_diags	647	400	0	247 (list, grid)

These are consistent with 2026-05-12 Chrysalis E3SM Unified 1.13.0rc10, post-fix (zppy v3.2.0rc2) -- we just need to update the expected results now.

e3sm_diags

Test name	Total images	Correct images	Mismatched images
comprehensive_v2_e3sm_diags	3791	3766	25 (list, grid)
comprehensive_v3_e3sm_diags	5354	5329	25 (list, grid)
legacy_3.1.0_comprehensive_v2_e3sm_diags	3791	3766	25 (list, grid)
legacy_3.1.0_comprehensive_v3_e3sm_diags	5350	5330	20 (list, grid)
legacy_3.0.0_comprehensive_v2_e3sm_diags	3791	3766	25 (list, grid)
legacy_3.0.0_comprehensive_v3_e3sm_diags	5350	5330	20 (list, grid)

These are consistent with 2026-05-12 Chrysalis E3SM Unified 1.13.0rc10, post-fix (zppy v3.2.0rc2), with the added benefit that the two hanging jobs now run to completion. We just need to update the expected results now.

Updating expected results

Setting up new expected results

We last set the expected results to the official results of E3SM Unified 1.12.0 here, on 2026-03-31.

ls -lt /lcrc/group/e3sm/public_html/zppy_test_resources/
# The files are from Mar. 30
# This is consistent with how we updated the expected results on 2026-03-30.
ls -lt /lcrc/group/e3sm/public_html/zppy_test_resources_previous/
# Mar 30 17:12 expected_results_until_20260330
# ...
ls -lt /lcrc/group/e3sm/public_html/zppy_test_resources_previous/expected_results_until_20260330
# Notice: similar contents to /lcrc/group/e3sm/public_html/zppy_test_resources/

# First, let's copy over the old expected results #############################
cd /lcrc/group/e3sm/public_html/
mv zppy_test_resources/ zppy_test_resources_previous/expected_results_until_20260513 # mv is faster than cp
mkdir zppy_test_resources
ls -lt zppy_test_resources_previous/expected_results_until_20260513
# Good, contents are from 3/30

# Second, update the expected results #########################################
# Let's update the simpler tests' results first:
cd ~/ez/zppy
git status
# Good, correct branch: test-zppy-main-plus-unified-1-13rc10-fix-try2
# Good, correct env: Unified rc10

ls tests/integration/generated/update_*_chrysalis.sh
# tests/integration/generated/update_archive_expected_files_chrysalis.sh # ignore this one
# tests/integration/generated/update_bash_generation_expected_files_chrysalis.sh
# tests/integration/generated/update_campaign_expected_files_chrysalis.sh
# tests/integration/generated/update_defaults_expected_files_chrysalis.sh
# tests/integration/generated/update_weekly_expected_files_chrysalis.sh

# Unlike on Perlmutter, these are already runnable, so no need for `chmod 755`

./tests/integration/generated/update_bash_generation_expected_files_chrysalis.sh # 1 passed in 1.81s
./tests/integration/generated/update_campaign_expected_files_chrysalis.sh # 6 passed in 2.91s 
./tests/integration/generated/update_defaults_expected_files_chrysalis.sh # 1 passed in 0.82s

./tests/integration/generated/update_weekly_expected_files_chrysalis.sh # ~25 minutes to run
ls /lcrc/group/e3sm/public_html/zppy_test_resources
# expected_bash_files                     expected_legacy_3.1.0_comprehensive_v3                 image_list_expected_legacy_3.1.0_comprehensive_v3.txt
# expected_bundles                        image_list_expected_bundles.txt                        test_campaign_cryosphere_expected_files
# expected_comprehensive_v2               image_list_expected_comprehensive_v2.txt               test_campaign_cryosphere_override_expected_files
# expected_comprehensive_v3               image_list_expected_comprehensive_v3.txt               test_campaign_high_res_v1_expected_files
# expected_legacy_3.0.0_bundles           image_list_expected_legacy_3.0.0_bundles.txt           test_campaign_none_expected_files
# expected_legacy_3.0.0_comprehensive_v2  image_list_expected_legacy_3.0.0_comprehensive_v2.txt  test_campaign_water_cycle_expected_files
# expected_legacy_3.0.0_comprehensive_v3  image_list_expected_legacy_3.0.0_comprehensive_v3.txt  test_campaign_water_cycle_override_expected_files
# expected_legacy_3.1.0_bundles           image_list_expected_legacy_3.1.0_bundles.txt           test_defaults_expected_files
# expected_legacy_3.1.0_comprehensive_v2  image_list_expected_legacy_3.1.0_comprehensive_v2.txt

# Good, we have `expected_` dirs for all 9 cfgs.

cd ~/ez/zppy
# test_last_year doesn't need an update script because its cfg is stand-alone:
pytest -vv tests/integration/test_last_year.py # 1 passed in 2.00s
pytest tests/integration/test_bundles.py # 2 passed in 1.27s
salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm

# On a separate Chrysalis tab:
scontrol show partition debug | grep TotalNodes # TotalNodes=20
# `squeue` shows someone is using all 20 debug-partition nodes for another ~1.5 hours.
scontrol show partition compute | grep TotalNodes # TotalNodes=251
# `squeue` shows only 136 compute-partition nodes are in use

# Cancel the previous salloc, and try again using compute-partition, with a shorter time window too
salloc --nodes=1 --partition=compute --time=00:45:00 --account=e3sm
# Re-establishe Unified env:
bash # Run bash so we're in an isolated subshell
source ~/.bashrc
source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh
pytest tests/integration/test_images.py # 1 passed in 740.19s (0:12:20)
cat test_images_summary.md

Test name	Total images	Correct images
bundles_e3sm_diags	1762	1762
bundles_global_time_series	3	3
bundles_ilamb	388	388
comprehensive_v2_e3sm_diags	3806	3806
comprehensive_v2_mpas_analysis	856	856
comprehensive_v2_global_time_series	12	12
comprehensive_v2_ilamb	776	776
comprehensive_v3_e3sm_diags	5369	5369
comprehensive_v3_mpas_analysis	1280	1280
comprehensive_v3_global_time_series	1404	1404
comprehensive_v3_ilamb	837	837
comprehensive_v3_pcmdi_diags	647	647
legacy_3.1.0_bundles_e3sm_diags	1762	1762
legacy_3.1.0_bundles_global_time_series	3	3
legacy_3.1.0_bundles_ilamb	388	388
legacy_3.1.0_comprehensive_v2_e3sm_diags	3806	3806
legacy_3.1.0_comprehensive_v2_mpas_analysis	856	856
legacy_3.1.0_comprehensive_v2_global_time_series	12	12
legacy_3.1.0_comprehensive_v2_ilamb	776	776
legacy_3.1.0_comprehensive_v3_e3sm_diags	5365	5365
legacy_3.1.0_comprehensive_v3_mpas_analysis	856	856
legacy_3.1.0_comprehensive_v3_global_time_series	1404	1404
legacy_3.1.0_comprehensive_v3_ilamb	837	837
legacy_3.0.0_bundles_e3sm_diags	1762	1762
legacy_3.0.0_bundles_global_time_series	3	3
legacy_3.0.0_bundles_ilamb	388	388
legacy_3.0.0_comprehensive_v2_e3sm_diags	3806	3806
legacy_3.0.0_comprehensive_v2_mpas_analysis	856	856
legacy_3.0.0_comprehensive_v2_global_time_series	12	12
legacy_3.0.0_comprehensive_v2_ilamb	776	776
legacy_3.0.0_comprehensive_v3_e3sm_diags	5365	5365
legacy_3.0.0_comprehensive_v3_mpas_analysis	856	856
legacy_3.0.0_comprehensive_v3_global_time_series	90	90
legacy_3.0.0_comprehensive_v3_ilamb	837	837

✅ E3SM Unified 1.13.0rc10 (fixed script), try2 is fully passing on Chrysalis! I've updated expected results for Chrysalis accordingly.

3 replies

forsyth2 May 13, 2026
Maintainer Author

Just for reference: It appears the list/grid links above no longer work because I used mv instead of cp to update the expected results. That's ok though, since we already knew those diffs were to be expected.

chengzhuzhang May 13, 2026
Maintainer

This is great news, thanks for testing again! That’s a relief for me.

forsyth2 May 13, 2026
Maintainer Author

I'm wondering if it has anything to do with Chrysalis newly back online -- i.e., maybe a node/system problem? In any case, at least it seems to be working now!

forsyth2 · 2026-05-21T01:13:32Z

forsyth2
May 21, 2026
Maintainer Author

2026-05-19 Chrysalis E3SM Unified 1.13.0 official results

E3SM Unified 1.13.0 has been released. Let's produce official results to refer back to.

Chrysalis: run tests on all 9 cfgs
Compy: run all 3 non-legacy test cfgs
Perlmutter: run just the non-legacy v3 test cfg

What expected results are currently on Chrysalis?

Chrysalis's expected results were updated on 2026-05-12 Chrysalis E3SM Unified 1.13.0rc10, post-fix (zppy v3.2.0rc2), try 2.

What changes are new?

Package	Changes since 5/8 run up until production releases
NCO	Unclear
e3sm_to_cmip	None
e3sm_diags	E3SM-Project/e3sm_diags#1055, E3SM-Project/e3sm_diags#1056
mpas_analysis	MPAS-Dev/MPAS-Analysis#1148
zppy-interfaces	E3SM-Project/zppy-interfaces#50
zppy	#818, #825, #826

Testing process

Starting on 2026-05-19
Branch: zppy main branch
Environments: zppy & packages for all tasks -- production releases (E3SM Unified 1.13.0, zppy v3.2.0)

Set up environments

We can mostly skip this step, since all the packages will be tested using E3SM Unified rather than through dev environments.

cd ~/ez/zppy-interfaces
git status
# On branch v0.2.1
# nothing to commit, working tree clean
git fetch upstream main
git checkout main
git reset --hard upstream/main
git log --oneline | head -n 1
# a458338 Bump to 0.2.1 (#50)
# Good, that's the production release
bash # Run bash so we're in an isolated subshell
source ~/.bashrc
source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh

# Check zppy-interfaces tests
pytest tests/unit/global_time_series/test_*.py # 10 passed, 2 warnings in 31.36s
# SyntaxWarning: invalid escape sequence '\w'
# SyntaxWarning: invalid escape sequence '\s'
pytest tests/unit/pcmdi_diags/test_*.py # 7 passed in 8.42s
# Stay in the bash subshell

zppy run

# zppy itself #################################################################
cd ~/ez/zppy
git status
# On branch issue-794-update-agents-md
# nothing to commit, working tree clean

git fetch upstream main
# Go to https://github.com/E3SM-Project/zppy/commits/main
# Get the SHA from the production release commit
git checkout -b official-results-unified-1-13 17ce9335fe2c828a13992df4d2f3a5631edcba2d
git log --oneline | head -n 1
# 17ce9335 Bump to 3.2.0 (#826)
# Good, that's the production release

/usr/lpp/mmfs/bin/mmlsquota -u ac.forsyth2 --block-size T fs2 # Have enough space on /lcrc/group/e3sm/
lcrc-quota # Have enough space on /home/ac.forsyth2/

# WE ARE ALREADY IN THE UNIFIED ENV.
# DO NOT MAKE A DEV ENV.

pixi list python # Check Python version
# python                    3.13.13

# Run zppy unit tests
pytest tests/test_*.py # 44 passed in 0.87s

# Edit tests/integration/utils.py:

TEST_SPECIFICS: Dict[str, Any] = {
    # This is the NCO path.
    # Keep as "" to use the production-version NCO commands.
    # Set to a specific path to use development-version NCO commands.
    "nco_path": "",
    # These are custom environment_commands for specific tasks.
    # Never set these to "", because they will print the line
    # `environment_commands = ""` for the corresponding task,
    # thus overriding the value set higher up in the cfg.
    # That is, there will be no environment set.
    # (`environment_commands = ""` only redirects to Unified
    # if specified under the [default] task)
    "e3sm_to_cmip_environment_commands": "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh",
    "diags_environment_commands": "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh",
    "mpas_analysis_environment_commands": "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh",
    "global_time_series_environment_commands": "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh",
    "livvkit_environment_commands": "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh",
    "pcmdi_diags_environment_commands": "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh",
    # This is the environment setup for other tasks.
    # Leave as "" to use the latest Unified environment.
    "environment_commands": "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh",
    # For a complete test, run the set of latest cfgs and at least one set of legacy cfgs
    "cfgs_to_run": [
        "weekly_bundles",
        "weekly_comprehensive_v2",
        "weekly_comprehensive_v3",
        "weekly_legacy_3.1.0_bundles",
        "weekly_legacy_3.1.0_comprehensive_v2",
        "weekly_legacy_3.1.0_comprehensive_v3",
        "weekly_legacy_3.0.0_bundles",
        "weekly_legacy_3.0.0_comprehensive_v2",
        "weekly_legacy_3.0.0_comprehensive_v3",
    ],
    "tasks_to_run": [
        "e3sm_diags",
        "mpas_analysis",
        "global_time_series",
        "ilamb",
        "livvkit",
        "pcmdi_diags",
    ],
    "unique_id": "official_results_unified_1_13",
}

git diff # Diff looks good
python tests/integration/utils.py
# CFG FILES HAVE BEEN GENERATED FROM TEMPLATES WITH THESE SETTINGS:
# UNIQUE_ID=official_results_unified_1_13
# nco_path=
# Reminder: `nco_path=''` => the production-version NCO commands will be used
# e3sm_to_cmip_environment_commands=source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh
# diags_environment_commands=source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh
# mpas_analysis_environment_commands=source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh
# global_time_series_environment_commands=source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh
# livvkit_environment_commands=source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh
# pcmdi_diags_environment_commands=source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh
# environment_commands=source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh
# Reminder: `environment_commands=''` => the latest E3SM Unified environment will be used


# For reference:
alias sq
# Perlmutter: alias sq='sqa -u forsyth'
# Compy:      alias sq='sqa -u fors729'
# Chrysalis:  alias sq='sqa -u ac.forsyth2'
alias sqa
# Perlmutter: alias sqa='squeue -o "%8u %.7a %.4D %.9P %8i %.2t %.10r %.10M %.10l %j" --sort=P,-t,-p'
# Compy:      alias sqa='squeue -o "%8u %.7a %.4D %.9P %7i %.2t %.10r %.10M %.10l %.8Q %j" --sort=P,-t,-p'
# Chrysalis:  alias sqa='squeue -o "%8u %.7a %.4D %.9P %7i %.2t %.10r %.10M %.10l %j" --sort=P,-t,-p'
sq
# No jobs currently queued
ls tests/integration/generated/test_weekly_*_chrysalis.cfg # See all cfgs

# Launch jobs from ALL cfgs; start with the most important one, comprehensive_v3
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg

zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_comprehensive_v3_chrysalis.cfg

zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v3_chrysalis.cfg


sq | wc -l # Tue 05/19 18:57 => 233 - header row = 232 jobs


# Picking up 2026-05-20 #######################################################
sq | wc -l # Wed 05/20 08:49 => 1 - header row = 0 jobs

Bundles part 2

# Check on bundles status
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/official_results_unified_1_13/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_bundles_output/official_results_unified_1_13/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/official_results_unified_1_13/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors

# Now, run bundles part 2
cd ~/ez/zppy
git status
# Good, correct branch: official-results-unified-1-13
# Need to re-establish Unified env:
bash # Run bash so we're in an isolated subshell
source ~/.bashrc
source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh


zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg
sq | wc -l # Wed 05/20 08:52 => 7 - header row = 6 jobs

Review finished runs

### v2  ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/official_results_unified_1_13/v2.LR.historical_0201/post/scripts
grep -v "OK" *status # Good, no errors

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/official_results_unified_1_13/v2.LR.historical_0201/post/scripts
grep -v "OK" *status # Good, no errors

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_comprehensive_v2_output/official_results_unified_1_13/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.status:RUNNING 1219760

# See debugging section below.

### v3 ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/official_results_unified_1_13/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/official_results_unified_1_13/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_comprehensive_v3_output/official_results_unified_1_13/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors

### bundles ###
sq | wc -l # Wed 05/20 09:03 => 1 - header row = 0 jobs

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/official_results_unified_1_13/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, still no errors

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/official_results_unified_1_13/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, still no errors

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_bundles_output/official_results_unified_1_13/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, still no errors

Debugging

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_comprehensive_v2_output/official_results_unified_1_13/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.status:RUNNING 1219760
tail e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.o1219760
# 2026-05-19 22:05:49,490 [WARNING]: warnings.py(_showwarnmsg:110) >> /lcrc/soft/climate/e3sm-unified/e3smu_1_13_0/chrysalis/pixi/.pixi/envs/default/lib/python3.13/site-packages/cartopy/mpl/geoaxes.py:856: UserWarning: Attempting to set identical low and high xlims makes transformation singular; automatically expanding.
#   self.set_xlim([x1, x2])

# 2026-05-19 22:05:53,577 [INFO]: utils.py(_save_main_plot:122) >> Plot saved in: model_vs_obs_1982-1983/lat_lon/ERA5/ERA5-TREFHT-SON-land.png
# slurmstepd: error: *** STEP 1219760.0 ON chr-0185 CANCELLED AT 2026-05-20T02:18:58 DUE TO TIME LIMIT ***
# srun: Job step aborted: Waiting up to 92 seconds for job step to finish.
# slurmstepd: error: *** JOB 1219760 ON chr-0185 CANCELLED AT 2026-05-20T02:18:58 DUE TO TIME LIMIT ***
# srun: got SIGCONT
# srun: forcing job termination

grep "\-\-time" e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.bash
#SBATCH  --time=5:00:00

# So, it hit a 5 hour time limit that ended at 5/19 22:05. When did we start?
grep -n 2026-05-19 e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.o1219760 | head -n 1
# 55:2026-05-19 21:19:35,603 [INFO]: e3sm_diags_driver.py(_log_diagnostic_run_info:525) >> 
# We started at 21:19, so we actually ran for only about 45 minutes before hanging.

# Let's compare to the hanging issue we saw before,
# from [2026-05-12 Chrysalis E3SM Unified 1.13.0rc10, post-fix (zppy v3.2.0rc2)](https://github.com/E3SM-Project/zppy/discussions/802#discussioncomment-16898214)
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/zppy_main_branch_plus_unified_1_13rc10_fix/v2.LR.historical_0201/post/scripts
tail e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.o1216551

# Yes, this looks like the same error. We should be able to simply rerun it.
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_comprehensive_v2_output/official_results_unified_1_13/v2.LR.historical_0201/post/scripts
sbatch e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.bash
sq | wc -l # Wed 05/20 09:05 => 2 - header row = 1 jobs
sq | wc -l # Wed 05/20 09:55 => 1 - header row = 0 jobs
grep -v "OK" *status # No errors now! 
# It really was an intermittent bug, we changed nothing else.

post/scripts directory status: One error -- e3sm_diags once again hits the intermittent bug of #822. Simply rerunning the affected job via sbatch resolves the issue, so there is a workaround. But this does mean we shouldn't forget about this issue.

Python tests

cd ~/ez/zppy
git status
# Good, correct branch: official-results-unified-1-13
# Good, correct env: Unified
ls tests/integration/test_*.py

# Passing:
pytest tests/integration/test_bash_generation.py # 1 passed in 2.41s
pytest tests/integration/test_campaign.py # 6 passed in 2.96s
pytest tests/integration/test_defaults.py # 1 passed in 0.51s
pytest tests/integration/test_last_year.py # 1 passed in 0.61s

# Need to update epxected results:
pytest tests/integration/test_bundles.py # 1 failed, 1 passed in 0.41s
# -  if echo "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh" | grep -Eq 'test_e3sm_unified|load_latest_e3sm_unified'; then
# +  if echo "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh" | grep -Eq 'test_e3sm_unified|load_latest_e3sm_unified'; then

# Ok, f"diff -bu -I 'zppy_weekly_bundles_output/' {actual_directory}/bundle1.bash {expected_directory}/bundle1.bash"
# is just catching the environment name being used. 
# If we can somehow make the test ignore that line, that would be useful.
# But in any case, this isn't a real failure. 
# (And we saw the same thing on Compy).
# We'll update the expected results.

salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm
# Re-establish Unified env
bash
source ~/.bashrc
source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh
pytest tests/integration/test_images.py # 1 failed in 812.54s (0:13:32)
cat test_images_summary.md

Output:

---------------------------------------- Captured stdout call -----------------------------------------
--- /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/official_results_unified_1_13/v3.LR.historical_0051/post/scripts/bundle1.bash	2026-05-19 20:55:51.330642000 -0500
+++ /lcrc/group/e3sm/public_html/zppy_test_resources/expected_bundles/bundle_files/bundle1.bash	2026-05-13 17:43:09.553706000 -0500
@@ -42,7 +42,7 @@
 
   # Important: we need the quotes since environment_commands might
   # include multiple commands, separated by semi-colon.
-  if echo "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh" | grep -Eq 'test_e3sm_unified|load_latest_e3sm_unified'; then
+  if echo "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.13.0rc10_chrysalis.sh" | grep -Eq 'test_e3sm_unified|load_latest_e3sm_unified'; then
     pkg_manager="pixi"
   else
     pkg_manager="conda"
======================================= short test summary info =======================================
FAILED tests/integration/test_bundles.py::test_bundles_bash_file_content - assert 256 == 0
===================================== 1 failed, 1 passed in 0.41s =====================================

Complete summary table:

Test name	Total images	Correct images	Missing images
bundles_e3sm_diags	1762	1762	0
bundles_global_time_series	3	3	0
bundles_ilamb	388	388	0
comprehensive_v2_e3sm_diags	3806	3696	110 (list)
comprehensive_v2_mpas_analysis	856	856	0
comprehensive_v2_global_time_series	12	12	0
comprehensive_v2_ilamb	776	776	0
comprehensive_v3_e3sm_diags	5369	5259	110 (list)
comprehensive_v3_mpas_analysis	1280	1280	0
comprehensive_v3_global_time_series	1404	1404	0
comprehensive_v3_ilamb	837	837	0
comprehensive_v3_pcmdi_diags	647	647	0
legacy_3.1.0_bundles_e3sm_diags	1762	1762	0
legacy_3.1.0_bundles_global_time_series	3	3	0
legacy_3.1.0_bundles_ilamb	388	388	0
legacy_3.1.0_comprehensive_v2_e3sm_diags	3806	3696	110 (list)
legacy_3.1.0_comprehensive_v2_mpas_analysis	856	856	0
legacy_3.1.0_comprehensive_v2_global_time_series	12	12	0
legacy_3.1.0_comprehensive_v2_ilamb	776	776	0
legacy_3.1.0_comprehensive_v3_e3sm_diags	5365	5255	110 (list)
legacy_3.1.0_comprehensive_v3_mpas_analysis	856	856	0
legacy_3.1.0_comprehensive_v3_global_time_series	1404	1404	0
legacy_3.1.0_comprehensive_v3_ilamb	837	837	0
legacy_3.0.0_bundles_e3sm_diags	1762	1762	0
legacy_3.0.0_bundles_global_time_series	3	3	0
legacy_3.0.0_bundles_ilamb	388	388	0
legacy_3.0.0_comprehensive_v2_e3sm_diags	3806	3696	110 (list)
legacy_3.0.0_comprehensive_v2_mpas_analysis	856	856	0
legacy_3.0.0_comprehensive_v2_global_time_series	12	12	0
legacy_3.0.0_comprehensive_v2_ilamb	776	776	0
legacy_3.0.0_comprehensive_v3_e3sm_diags	5365	5255	110 (list)
legacy_3.0.0_comprehensive_v3_mpas_analysis	856	856	0
legacy_3.0.0_comprehensive_v3_global_time_series	90	90	0
legacy_3.0.0_comprehensive_v3_ilamb	837	837	0

Summary table -- only failing image-check tests, sorted by task

e3sm_diags

Test name	Total images	Correct images	Missing images
comprehensive_v2_e3sm_diags	3806	3696	110 (list)
comprehensive_v3_e3sm_diags	5369	5259	110 (list)
legacy_3.1.0_comprehensive_v2_e3sm_diags	3806	3696	110 (list)
legacy_3.1.0_comprehensive_v3_e3sm_diags	5365	5255	110 (list)
legacy_3.0.0_comprehensive_v2_e3sm_diags	3806	3696	110 (list)
legacy_3.0.0_comprehensive_v3_e3sm_diags	5365	5255	110 (list)

It looks like these 110 missing images on every e3sm_diags task can be explained by this PR which was merged since the last run: E3SM-Project/e3sm_diags#1055

Updating expected results

Setting up new expected results

We last set updated Chrysalis expected results during 2026-05-12 Chrysalis E3SM Unified 1.13.0rc10, post-fix (zppy v3.2.0rc2), try 2, on May 13.

ls -lt /lcrc/group/e3sm/public_html/zppy_test_resources/
# The files are from May 13
# This is consistent with how we updated the expected results on 2026-03-30.
ls -lt /lcrc/group/e3sm/public_html/zppy_test_resources_previous/
# May 13 17:29 expected_results_until_20260513
# ...
ls -lt /lcrc/group/e3sm/public_html/zppy_test_resources_previous/expected_results_until_20260513
# Notice: similar contents to /lcrc/group/e3sm/public_html/zppy_test_resources/

# First, let's copy over the old expected results #############################
cd /lcrc/group/e3sm/public_html/
cp -r zppy_test_resources/ zppy_test_resources_previous/expected_results_until_20260520 # Took between 20 and 60 minutes
ls -lt zppy_test_resources_previous/expected_results_until_20260520
# Contents are from 5/20 rather than 5/13, but that's because we copied over files, rather than just moving them

# Second, update the expected results #########################################
# Let's update the simpler tests' results first:
cd ~/ez/zppy
git status
# Good, correct branch: test-zppy-main-plus-unified-1-13rc10-fix-try2
# Good, correct env: Unified

ls tests/integration/generated/update_*_chrysalis.sh
# tests/integration/generated/update_archive_expected_files_chrysalis.sh
# tests/integration/generated/update_bash_generation_expected_files_chrysalis.sh
# tests/integration/generated/update_campaign_expected_files_chrysalis.sh
# tests/integration/generated/update_defaults_expected_files_chrysalis.sh
# tests/integration/generated/update_weekly_expected_files_chrysalis.sh

# Unlike on Perlmutter, these are already runnable, so no need for `chmod 755`

# May need to run these scripts multiple times to make sure they get the test passing.
./tests/integration/generated/update_bash_generation_expected_files_chrysalis.sh # 1 passed in 0.49s
./tests/integration/generated/update_campaign_expected_files_chrysalis.sh # 6 passed in 2.52s
./tests/integration/generated/update_defaults_expected_files_chrysalis.sh # 1 passed in 0.45s

./tests/integration/generated/update_weekly_expected_files_chrysalis.sh # ~40 minutes to run
ls /lcrc/group/e3sm/public_html/zppy_test_resources
# expected_bash_files                       image_list_expected_legacy_3.0.0_bundles.txt
# expected_bundles                          image_list_expected_legacy_3.0.0_comprehensive_v2.txt
# expected_comprehensive_v2                 image_list_expected_legacy_3.0.0_comprehensive_v3.txt
# expected_comprehensive_v3                 image_list_expected_legacy_3.1.0_bundles.txt
# expected_legacy_3.0.0_bundles             image_list_expected_legacy_3.1.0_comprehensive_v2.txt
# expected_legacy_3.0.0_comprehensive_v2    image_list_expected_legacy_3.1.0_comprehensive_v3.txt
# expected_legacy_3.0.0_comprehensive_v3    test_campaign_cryosphere_expected_files
# expected_legacy_3.1.0_bundles             test_campaign_cryosphere_override_expected_files
# expected_legacy_3.1.0_comprehensive_v2    test_campaign_high_res_v1_expected_files
# expected_legacy_3.1.0_comprehensive_v3    test_campaign_none_expected_files
# image_list_expected_bundles.txt           test_campaign_water_cycle_expected_files
# image_list_expected_comprehensive_v2.txt  test_campaign_water_cycle_override_expected_files
# image_list_expected_comprehensive_v3.txt  test_defaults_expected_files

# Good, we have `expected_` dirs for all 9 cfgs.

cd ~/ez/zppy
# test_last_year doesn't need an update script because its cfg is stand-alone:
pytest -vv tests/integration/test_last_year.py # 1 passed in 1.60s
pytest tests/integration/test_bundles.py # 2 passed in 0.35s
salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm
# Re-establishe Unified env:
bash # Run bash so we're in an isolated subshell
source ~/.bashrc
source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh
pytest tests/integration/test_images.py # 1 passed in 805.37s (0:13:25)
cat test_images_summary.md

Test name	Total images	Correct images
bundles_e3sm_diags	1762	1762
bundles_global_time_series	3	3
bundles_ilamb	388	388
comprehensive_v2_e3sm_diags	3806	3806
comprehensive_v2_mpas_analysis	856	856
comprehensive_v2_global_time_series	12	12
comprehensive_v2_ilamb	776	776
comprehensive_v3_e3sm_diags	5369	5369
comprehensive_v3_mpas_analysis	1280	1280
comprehensive_v3_global_time_series	1404	1404
comprehensive_v3_ilamb	837	837
comprehensive_v3_pcmdi_diags	647	647
legacy_3.1.0_bundles_e3sm_diags	1762	1762
legacy_3.1.0_bundles_global_time_series	3	3
legacy_3.1.0_bundles_ilamb	388	388
legacy_3.1.0_comprehensive_v2_e3sm_diags	3806	3806
legacy_3.1.0_comprehensive_v2_mpas_analysis	856	856
legacy_3.1.0_comprehensive_v2_global_time_series	12	12
legacy_3.1.0_comprehensive_v2_ilamb	776	776
legacy_3.1.0_comprehensive_v3_e3sm_diags	5365	5365
legacy_3.1.0_comprehensive_v3_mpas_analysis	856	856
legacy_3.1.0_comprehensive_v3_global_time_series	1404	1404
legacy_3.1.0_comprehensive_v3_ilamb	837	837
legacy_3.0.0_bundles_e3sm_diags	1762	1762
legacy_3.0.0_bundles_global_time_series	3	3
legacy_3.0.0_bundles_ilamb	388	388
legacy_3.0.0_comprehensive_v2_e3sm_diags	3806	3806
legacy_3.0.0_comprehensive_v2_mpas_analysis	856	856
legacy_3.0.0_comprehensive_v2_global_time_series	12	12
legacy_3.0.0_comprehensive_v2_ilamb	776	776
legacy_3.0.0_comprehensive_v3_e3sm_diags	5365	5365
legacy_3.0.0_comprehensive_v3_mpas_analysis	856	856
legacy_3.0.0_comprehensive_v3_global_time_series	90	90
legacy_3.0.0_comprehensive_v3_ilamb	837	837

exit # Exit bash shell
exit # Exit compute note

# Third, let's copy the expected results into a special official results dir ##
cd /lcrc/group/e3sm/public_html/
ls -lt zppy_test_resources_previous
# Includes: expected_results_for_unified_1.12.0
cp -r zppy_test_resources/ zppy_test_resources_previous/expected_results_for_unified_1.13.0
# No space left on device, after ~5 min

# Need to clear up inodes
rm -rf zppy_test_resources_previous/expected_results_for_unified_1.13.0
# Try again:
cp -r zppy_test_resources/ zppy_test_resources_previous/expected_results_for_unified_1.13.0

ls zppy_test_resources_previous/expected_results_for_unified_1.13.0
# Good, has results for all 9 cfgs

✅ Official results for E3SM Unified 1.13.0 on Chrysalis can be found at:

/lcrc/group/e3sm/public_html/zppy_test_resources_previous/expected_results_for_unified_1.13.0

5 replies

forsyth2 May 21, 2026
Maintainer Author

@chengzhuzhang This is for the E3SM Unified 1.13.0 official results, extending E3SM Unified 1.12.0 official results.

I've also completed Compy's official results. Perlmutter is down today, so I still need to make its official results.

I did run into two issues while producing these official results, both of which were low-impact:

The intermittent, non-reproducible hanging e3sm_diags bug is still around. The workaround if it occurs is to just sbatch the timed-out bash file again.
There were 110 missing images, which I determined to be because of Add and use new cosp files e3sm_diags#1055, and were therefore to be expected. Unfortunately, since I've updated the expected results, the list links above no longer work, but all 110 missing images were COSP-related and I found them in that PR.

chengzhuzhang May 21, 2026
Maintainer

Hey @forsyth2, Thanks for reporting this. I'm wondering how many hanging e3sm_diags runs happened, and on which machines? And yes, the images diff is caused by the new cosp files being used.

forsyth2 May 21, 2026
Maintainer Author

@chengzhuzhang

how many hanging e3sm_diags runs happened, and on which machines?

Chrysalis -- only one job this time, which is less than the two jobs it originally happened on. So, that's yet more evidence it's some intermittent bug. The job this time was e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983 for legacy_3.1.0_comprehensive_v2. It does seem to only ever be the legacy cfgs (3.0.0 and/or 3.1.0) though. And recall we aren't running the legacy tests on the other two machines, per the tiered testing strategy:

Chrysalis: run tests on all 9 cfgs
Compy: run all 3 non-legacy test cfgs
Perlmutter: run just the non-legacy v3 test cfg

Compy -- no hangs.

Perlmutter -- haven't run yet (and I don't think I'll be able to get to that until tomorrow afternoon or next week).

yes, the images diff is caused by the new cosp files being used.

Great, thanks for confirming.

chengzhuzhang May 21, 2026
Maintainer

thanks for documenting this!

forsyth2 May 21, 2026
Maintainer Author

Perlmutter -- haven't run yet

I had a moment to restart it (the maintenance day yesterday cleared the job queue). I don't imagine I'll have results until late tomorrow or next week though.

forsyth2 · 2026-05-21T01:14:33Z

forsyth2
May 21, 2026
Maintainer Author

2026-05-19 Compy E3SM Unified 1.13.0 official results

E3SM Unified 1.13.0 has been released. Let's produce official results to refer back to.

Chrysalis: run tests on all 9 cfgs
Compy: run all 3 non-legacy test cfgs
Perlmutter: run just the non-legacy v3 test cfg

What expected results are currently on Compy?

Compy's expected results were updated after the last test run -- 2026-05-08 Compy E3SM Unified 1.13.0rc10 (zppy v3.2.0rc2).

What changes are new?

Package	Changes since 5/8 run up until production releases
NCO	Unclear
e3sm_to_cmip	None
e3sm_diags	E3SM-Project/e3sm_diags#1055, E3SM-Project/e3sm_diags#1056
mpas_analysis	MPAS-Dev/MPAS-Analysis#1148
zppy-interfaces	E3SM-Project/zppy-interfaces#50
zppy	#818, #825, #826

Testing process

Starting on 2026-05-19
Branch: zppy main branch
Environments: zppy & packages for all tasks -- production releases (E3SM Unified 1.13.0, zppy v3.2.0)

Set up environments

We can mostly skip this step, since all the packages will be tested using E3SM Unified rather than through dev environments.

cd ~/ez/zppy-interfaces
git status
# On branch main
# nothing to commit, working directory clean

# REMINDER: On Compy, you must NOT specify the branch, 
# otherwise it won't actually fetch the latest commits!!
git fetch upstream
git checkout main
git reset --hard upstream/main
git log --oneline | head -n 1
# a458338 Bump to 0.2.1 (#50)
# Good, that's the production release
bash # Run bash so we're in an isolated subshell
source ~/.bash_profile
source /share/apps/E3SM/conda_envs/load_latest_e3sm_unified_compy.sh

# Check zppy-interfaces tests
pytest tests/unit/global_time_series/test_*.py # 10 passed, 2 warnings in 16.01s
# SyntaxWarning: invalid escape sequence '\w'
# SyntaxWarning: invalid escape sequence '\s'
pytest tests/unit/pcmdi_diags/test_*.py # 7 passed in 9.78s
# Stay in the bash subshell

zppy run

# zppy itself #################################################################
cd ~/ez/zppy
git status
# On branch test-zppy-main-plus-unified-1-13rc10-fix
git add -A
git commit -m "Testing" --no-verify # Clean up uncommitted changes


git fetch upstream # Remember, do NOT specify the branch on Compy!
# Go to https://github.com/E3SM-Project/zppy/commits/main
# Get the SHA from the production release commit
git checkout -b official-results-unified-1-13 17ce9335fe2c828a13992df4d2f3a5631edcba2d
git log --oneline | head -n 1
# 17ce933 Bump to 3.2.0 (#826)
# Good, that's the production release

lfs quota -hu fors729 /compyfs # Have enough space on /compyfs
# Not sure how to check space on /qfs
# (Currently in /qfs/people/fors729/ez/zppy)

# WE ARE ALREADY IN THE UNIFIED ENV.
# DO NOT MAKE A DEV ENV.

pixi list python # Check Python version
# python                    3.13.13

# Run zppy unit tests
pytest tests/test_*.py # 44 passed in 0.97s

# Edit tests/integration/utils.py:

TEST_SPECIFICS: Dict[str, Any] = {
    # This is the NCO path.
    # Keep as "" to use the production-version NCO commands.
    # Set to a specific path to use development-version NCO commands.
    "nco_path": "",
    # These are custom environment_commands for specific tasks.
    # Never set these to "", because they will print the line
    # `environment_commands = ""` for the corresponding task,
    # thus overriding the value set higher up in the cfg.
    # That is, there will be no environment set.
    # (`environment_commands = ""` only redirects to Unified
    # if specified under the [default] task)
    "e3sm_to_cmip_environment_commands": "source /share/apps/E3SM/conda_envs/load_latest_e3sm_unified_compy.sh",
    "diags_environment_commands": "source /share/apps/E3SM/conda_envs/load_latest_e3sm_unified_compy.sh",
    "mpas_analysis_environment_commands": "source /share/apps/E3SM/conda_envs/load_latest_e3sm_unified_compy.sh",
    "global_time_series_environment_commands": "source /share/apps/E3SM/conda_envs/load_latest_e3sm_unified_compy.sh",
    "livvkit_environment_commands": "source /share/apps/E3SM/conda_envs/load_latest_e3sm_unified_compy.sh",
    "pcmdi_diags_environment_commands": "source /share/apps/E3SM/conda_envs/load_latest_e3sm_unified_compy.sh",
    # This is the environment setup for other tasks.
    # Leave as "" to use the latest Unified environment.
    "environment_commands": "source /share/apps/E3SM/conda_envs/load_latest_e3sm_unified_compy.sh",
    # For a complete test, run the set of latest cfgs and at least one set of legacy cfgs
    "cfgs_to_run": [
        "weekly_bundles",
        "weekly_comprehensive_v2",
        "weekly_comprehensive_v3",
        # "weekly_legacy_3.1.0_bundles",
        # "weekly_legacy_3.1.0_comprehensive_v2",
        # "weekly_legacy_3.1.0_comprehensive_v3",
        # "weekly_legacy_3.0.0_bundles",
        # "weekly_legacy_3.0.0_comprehensive_v2",
        # "weekly_legacy_3.0.0_comprehensive_v3",
    ],
    "tasks_to_run": [
        "e3sm_diags",
        "mpas_analysis",
        "global_time_series",
        "ilamb",
        "livvkit",
        "pcmdi_diags",
    ],
    "unique_id": "official_results_unified_1_13",
}

git diff # Diff looks good
python tests/integration/utils.py
# CFG FILES HAVE BEEN GENERATED FROM TEMPLATES WITH THESE SETTINGS:
# UNIQUE_ID=official_results_unified_1_13
# nco_path=
# Reminder: `nco_path=''` => the production-version NCO commands will be used
# e3sm_to_cmip_environment_commands=source /share/apps/E3SM/conda_envs/load_latest_e3sm_unified_compy.sh
# diags_environment_commands=source /share/apps/E3SM/conda_envs/load_latest_e3sm_unified_compy.sh
# mpas_analysis_environment_commands=source /share/apps/E3SM/conda_envs/load_latest_e3sm_unified_compy.sh
# global_time_series_environment_commands=source /share/apps/E3SM/conda_envs/load_latest_e3sm_unified_compy.sh
# livvkit_environment_commands=source /share/apps/E3SM/conda_envs/load_latest_e3sm_unified_compy.sh
# pcmdi_diags_environment_commands=source /share/apps/E3SM/conda_envs/load_latest_e3sm_unified_compy.sh
# environment_commands=source /share/apps/E3SM/conda_envs/load_latest_e3sm_unified_compy.sh
# Reminder: `environment_commands=''` => the latest E3SM Unified environment will be used

# For reference:
alias sq
# Perlmutter: alias sq='sqa -u forsyth'
# Compy:      alias sq='sqa -u fors729'
alias sqa
# Perlmutter: alias sqa='squeue -o "%8u %.7a %.4D %.9P %8i %.2t %.10r %.10M %.10l %j" --sort=P,-t,-p'
# Compy:      alias sqa='squeue -o "%8u %.7a %.4D %.9P %7i %.2t %.10r %.10M %.10l %.8Q %j" --sort=P,-t,-p'
sq
# No jobs currently queued

# Launch jobs; start with the most important one, comprehensive_v3
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_compy.cfg
zppy -c tests/integration/generated/test_weekly_bundles_compy.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_compy.cfg

sq | wc -l # Tue 05/19 18:41 => 84 - header row = 83 jobs

# Picking up 2026-05-20 #######################################################
sq | wc -l # Tue 05/20 09:13 => 1 - header row = 0 jobs

Bundles part 2

# Check on bundles status
cd /compyfs/fors729/zppy_weekly_bundles_output/official_results_unified_1_13/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors

# Now, run bundles part 2
cd ~/ez/zppy
git status
# Good, correct branch: official-results-unified-1-13
# Need to re-establish Unified env:
bash # Run bash so we're in an isolated subshell
source ~/.bash_profile
source /share/apps/E3SM/conda_envs/load_latest_e3sm_unified_compy.sh


zppy -c tests/integration/generated/test_weekly_bundles_compy.cfg
sq | wc -l # Wed 05/20 09:15 => 3 - header row = 2 jobs

Review finished runs

### v2  ###
cd /compyfs/fors729/zppy_weekly_comprehensive_v2_output/official_results_unified_1_13/v2.LR.historical_0201/post/scripts
grep -v "OK" *status # Good, no errors

### v3 ###
cd /compyfs/fors729/zppy_weekly_comprehensive_v3_output/official_results_unified_1_13/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors

### bundles ###
sq | wc -l # Wed 05/20 09:24 => 1 - header row = 0 jobs
cd /compyfs/fors729/zppy_weekly_bundles_output/official_results_unified_1_13/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, still no errors after running bundles part 2

post/scripts directory status: NO ERRORS

Python tests

cd ~/ez/zppy
git status
# Good, correct branch: official-results-unified-1-13
# Good, correct env: Unified
ls tests/integration/test_*.py

# Passing:
pytest tests/integration/test_bash_generation.py # 1 passed in 3.17s
pytest tests/integration/test_campaign.py # 6 passed in 42.18s
pytest tests/integration/test_defaults.py # 1 passed in 1.90s
pytest tests/integration/test_last_year.py # 1 passed in 1.02s

# Need to update epxected results:
pytest tests/integration/test_bundles.py # 1 failed, 1 passed in 1.77s
# -  if echo "source /share/apps/E3SM/conda_envs/load_latest_e3sm_unified_compy.sh" | grep -Eq 'test_e3sm_unified|load_latest_e3sm_unified'; then
# +  if echo "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc10_compy.sh" | grep -Eq 'test_e3sm_unified|load_latest_e3sm_unified'; then

# Ok, `f"diff -bu -I 'zppy_weekly_bundles_output/' {actual_directory}/bundle1.bash {expected_directory}/bundle1.bash"` 
# is just catching the environment name being used. 
# If we can somehow make the test ignore that line, that would be useful.
# But in any case, this isn't a real failure. 
# We'll update the expected results.

# We'll now proceed with the image checker test.
salloc --nodes=1 --partition=short --time=01:00:00 --account=e3sm
# Re-establish Unified env
bash
source ~/.bash_profile
source /share/apps/E3SM/conda_envs/load_latest_e3sm_unified_compy.sh
pytest tests/integration/test_images.py 
# tests/integration/test_images.py salloc: Job 759260 has exceeded its time limit and its allocation has been revoked.
# Hangup

# Try again, with a longer time limit
salloc --nodes=1 --partition=short --time=02:00:00 --account=e3sm
bash
source ~/.bash_profile
source /share/apps/E3SM/conda_envs/load_latest_e3sm_unified_compy.sh
pytest tests/integration/test_images.py # 1 failed in 2654.32s (0:44:14)
cat test_images_summary.md

Output:

---------------------------------------- Captured stdout call -----------------------------------------
Preparing weekly cfg tests
Preparing legacy 3.1.0 cfg tests
Preparing legacy 3.0.0 cfg tests
Running 3 tests in parallel
Individual test logs will be written to test_<name>.log files
✓ Completed: bundles (3 tasks) (log: test_bundles.log)
✓ Completed: comprehensive_v2 (4 tasks) (log: test_comprehensive_v2.log)
✓ Completed: comprehensive_v3 (5 tasks) (log: test_comprehensive_v3.log)
Copy the output of test_images_summary.md to a Pull Request comment

Test Summary:
Test                                                    Total    Correct     Status
----------------------------------------------------------------------------------
bundles_e3sm_diags                                       1762       1762     ✓ PASS
bundles_global_time_series                                  3          3     ✓ PASS
bundles_ilamb                                             388        388     ✓ PASS
comprehensive_v2_e3sm_diags                              3806       3696     ✗ FAIL
comprehensive_v2_mpas_analysis                            856        856     ✓ PASS
comprehensive_v2_global_time_series                        12         12     ✓ PASS
comprehensive_v2_ilamb                                    776        776     ✓ PASS
comprehensive_v3_e3sm_diags                              5369       5259     ✗ FAIL
comprehensive_v3_mpas_analysis                           1280       1280     ✓ PASS
comprehensive_v3_global_time_series                      1404       1404     ✓ PASS
comprehensive_v3_ilamb                                    837        837     ✓ PASS
comprehensive_v3_pcmdi_diags                              647        647     ✓ PASS
----------------------------------------------------------------------------------

⚠ Some tests had mismatched or missing images. Check individual log files for details.
======================================= short test summary info =======================================
FAILED tests/integration/test_images.py::test_images - assert 3806 == 3696
=================================== 1 failed in 2654.32s (0:44:14) ====================================

Complete summary table:

Test name	Total images	Correct images	Missing images
bundles_e3sm_diags	1762	1762	0
bundles_global_time_series	3	3	0
bundles_ilamb	388	388	0
comprehensive_v2_e3sm_diags	3806	3696	110 (list)
comprehensive_v2_mpas_analysis	856	856	0
comprehensive_v2_global_time_series	12	12	0
comprehensive_v2_ilamb	776	776	0
comprehensive_v3_e3sm_diags	5369	5259	110 (list)
comprehensive_v3_mpas_analysis	1280	1280	0
comprehensive_v3_global_time_series	1404	1404	0
comprehensive_v3_ilamb	837	837	0
comprehensive_v3_pcmdi_diags	647	647	0

Summary table -- only failing image-check tests, sorted by task

e3sm_diags

Test name	Total images	Correct images	Missing images	Mismatched images
comprehensive_v2_e3sm_diags	3806	3696	110 (list)	0
comprehensive_v3_e3sm_diags	5369	5259	110 (list)	0

Same note I made for Chrysalis: It looks like these 110 missing images on every e3sm_diags task can be explained by this PR which was merged since the last run: E3SM-Project/e3sm_diags#1055

Updating expected results

Setting up new expected results

We last set updated Compy expected results during 2026-05-08 Compy E3SM Unified 1.13.0rc10 (zppy v3.2.0rc2), on May 8.

ls -lt /compyfs/www/zppy_test_resources/
# The files are from May 8
# This is consistent with how we updated the expected results on 2026-05-08
ls -lt /compyfs/www/fors729/zppy_test_resources_previous
# May  8 14:47 expected_results_until_20260508

# First, let's copy over the old expected results #############################
cd /compyfs/www/

# Unlike on Perlmutter, we don't have write-access to this level of the directory tree on Compy.
# That means we can't simply run:
# mv zppy_test_resources/ zppy_test_resources_previous/expected_results_until_20260520
# Because then zppy_test_resources would be gone

# Backup the current contents somewhere writable under the existing tree
mkdir -p fors729/zppy_test_resources_previous/expected_results_until_20260520

# Move everything out of zppy_test_resources/, including hidden files
# From ChatGPT:
# * = "all visible files"
# dotglob = "actually, include hidden ones too"
# nullglob = "and don’t break if there are none"
shopt -s dotglob nullglob
mv zppy_test_resources/* fors729/zppy_test_resources_previous/expected_results_until_20260520/
shopt -u dotglob nullglob
ls -lt fors729/zppy_test_resources_previous/
# Now present: May 20 11:46 expected_results_until_20260520

# Second, update the expected results #########################################

# Let's update the simpler tests' results first:
cd ~/ez/zppy
git status
# Good, correct branch: official-results-unified-1-13
# Good, correct env: Unified rc10

ls tests/integration/generated/update_*_compy.sh
# tests/integration/generated/update_archive_expected_files_compy.sh
# tests/integration/generated/update_bash_generation_expected_files_compy.sh
# tests/integration/generated/update_campaign_expected_files_compy.sh
# tests/integration/generated/update_defaults_expected_files_compy.sh
# tests/integration/generated/update_weekly_expected_files_compy.sh

# May need to run these scripts multiple times to make sure they get the test passing.
chmod 755 tests/integration/generated/update_bash_generation_expected_files_compy.sh
./tests/integration/generated/update_bash_generation_expected_files_compy.sh
# 1 passed in 1.98s

chmod 755 tests/integration/generated/update_campaign_expected_files_compy.sh
./tests/integration/generated/update_campaign_expected_files_compy.sh
# 6 passed in 7.28s

chmod 755 tests/integration/generated/update_defaults_expected_files_compy.sh
./tests/integration/generated/update_defaults_expected_files_compy.sh
# 1 passed in 1.29s

# test_last_year doesn't need an update script because its cfg is stand-alone:
pytest -vv tests/integration/test_last_year.py
# 1 passed in 1.26s

# Now update the expected files for the major tests:

# NOTE: `cp -a fors729/zppy_test_resources_previous/expected_results_for_unified_1.12.0/. zppy_test_resources/`
# took FOUR HOURS to complete, according to our note in the 2026-04-08 Compy test:
# https://github.com/E3SM-Project/zppy/discussions/802?sort=new#discussioncomment-16508424

screen -ls # No Sockets found
screen
screen -ls # 392427.pts-2.compy-e    (Attached)
# Re-establish env:
bash # Run bash so we're in an isolated subshell
source ~/.bash_profile
source /share/apps/E3SM/conda_envs/load_latest_e3sm_unified_compy.sh
chmod 755 tests/integration/generated/update_weekly_expected_files_compy.sh
./tests/integration/generated/update_weekly_expected_files_compy.sh
# CTRL A D
screen -ls # 64194.pts-1.compy-e	(Detached)

# HERE 11:52

screen -R
# cp: cannot stat ‘/compyfs/www/fors729/zppy_weekly_legacy_3.1.0_comprehensive_v2_www/official_results_unified_1_13/v2.LR.historical_0201’: No such file or directory
# ./tests/integration/generated/update_weekly_expected_files_compy.sh: line 56: cd: /compyfs/www/zppy_test_resources/expected_legacy_3.1.0_comprehensive_v2: No such file or directory
# cp: cannot stat ‘/compyfs/www/fors729/zppy_weekly_legacy_3.1.0_comprehensive_v3_www/official_results_unified_1_13/v3.LR.historical_0051’: No such file or directory
# ./tests/integration/generated/update_weekly_expected_files_compy.sh: line 56: cd: /compyfs/www/zppy_test_resources/expected_legacy_3.1.0_comprehensive_v3: No such file or directory
# cp: cannot stat ‘/compyfs/www/fors729/zppy_weekly_legacy_3.1.0_bundles_www/official_results_unified_1_13/v3.LR.historical_0051’: No such file or directory
# ./tests/integration/generated/update_weekly_expected_files_compy.sh: line 56: cd: /compyfs/www/zppy_test_resources/expected_legacy_3.1.0_bundles: No such file or directory
# cp: cannot stat ‘/compyfs/www/fors729/zppy_weekly_legacy_3.0.0_comprehensive_v2_www/official_results_unified_1_13/v2.LR.historical_0201’: No such file or directory
# ./tests/integration/generated/update_weekly_expected_files_compy.sh: line 56: cd: /compyfs/www/zppy_test_resources/expected_legacy_3.0.0_comprehensive_v2: No such file or directory
# cp: cannot stat ‘/compyfs/www/fors729/zppy_weekly_legacy_3.0.0_comprehensive_v3_www/official_results_unified_1_13/v3.LR.historical_0051’: No such file or directory
# ./tests/integration/generated/update_weekly_expected_files_compy.sh: line 56: cd: /compyfs/www/zppy_test_resources/expected_legacy_3.0.0_comprehensive_v3: No such file or directory
# cp: cannot stat ‘/compyfs/www/fors729/zppy_weekly_legacy_3.0.0_bundles_www/official_results_unified_1_13/v3.LR.historical_0051’: No such file or directory
# ./tests/integration/generated/update_weekly_expected_files_compy.sh: line 56: cd: /compyfs/www/zppy_test_resources/expected_legacy_3.0.0_bundles: No such file or directory


# Yes, this does makes sense, because we didn't run any of the legacy tests.

exit # Exit bash subshell
exit # Exit screen
cd /compyfs/www/zppy_test_resources
ls -lt 
# Everything says updated 5/20
# There's also no expected files for the legacy tests, since we didn't run those.

cd ~/ez/zppy
pytest tests/integration/test_bundles.py
# 2 passed in 1.05s

salloc --nodes=1 --partition=short --time=01:00:00 --account=e3sm
# Re-establish Unified env
bash
source ~/.bash_profile
source /share/apps/E3SM/conda_envs/load_latest_e3sm_unified_compy.sh
pytest tests/integration/test_images.py # 1 passed in 2478.29s (0:41:18)
cat test_images_summary.md

Complete summary table:

Test name	Total images	Correct images
bundles_e3sm_diags	1762	1762
bundles_global_time_series	3	3
bundles_ilamb	388	388
comprehensive_v2_e3sm_diags	3806	3806
comprehensive_v2_mpas_analysis	856	856
comprehensive_v2_global_time_series	12	12
comprehensive_v2_ilamb	776	776
comprehensive_v3_e3sm_diags	5369	5369
comprehensive_v3_mpas_analysis	1280	1280
comprehensive_v3_global_time_series	1404	1404
comprehensive_v3_ilamb	837	837
comprehensive_v3_pcmdi_diags	647	647

exit # Exit bash shell
exit # Exit compute note

# Third, let's copy the expected results into a special official results dir ##
cd /compyfs/www/
ls -lt fors729/zppy_test_resources_previous
# Includes: expected_results_until_20260520

screen
cp -r zppy_test_resources/ fors729/zppy_test_resources_previous/expected_results_for_unified_1.13.0 # ~1h20m to run
# CTRL A D
screen -ls
# 185057.pts-1.compy-e	(Detached)

screen -R
exit

ls fors729/zppy_test_resources_previous/expected_results_for_unified_1.13.0
# Has results for all 9 cfgs

✅ Official results for E3SM Unified 1.13.0 on Compy can be found at:

/compyfs/www/fors729/zppy_test_resources_previous/expected_results_for_unified_1.13.0

0 replies

forsyth2 · 2026-05-26T19:34:45Z

forsyth2
May 26, 2026
Maintainer Author

2026-05-21 Perlmutter E3SM Unified 1.13.0 official results

E3SM Unified 1.13.0 has been released. Let's produce official results to refer back to.

Chrysalis: run tests on all 9 cfgs
Compy: run all 3 non-legacy test cfgs
Perlmutter: run just the non-legacy v3 test cfg

What expected results are currently on Perlmutter?

Perlmutter's expected results were updated on 2026-05-08 Perlmutter E3SM Unified 1.13.0rc10 (zppy v3.2.0rc2).

What changes are new?

Package	Changes since 5/8 run up until production releases
NCO	Unclear
e3sm_to_cmip	None
e3sm_diags	E3SM-Project/e3sm_diags#1055, E3SM-Project/e3sm_diags#1056
mpas_analysis	MPAS-Dev/MPAS-Analysis#1148
zppy-interfaces	E3SM-Project/zppy-interfaces#50
zppy	#818, #825, #826

Testing process

Starting on 2026-05-21 (had to restart because Perlmutter was down on 5/20)
Branch: zppy main branch
Environments: zppy & packages for all tasks -- production releases (E3SM Unified 1.13.0, zppy v3.2.0)

Set up environments

We can mostly skip this step, since all the packages will be tested using E3SM Unified rather than through dev environments.

cd ~/ez/zppy-interfaces
git status
# On branch main
# nothing to commit, working directory clean
git fetch upstream main
git reset --hard upstream/main
git log --oneline | head -n 1
# a458338 Bump to 0.2.1 (#50)
# Good, that's the production release
bash # Run bash so we're in an isolated subshell
source ~/.bash_profile.ext
source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh

# Check zppy-interfaces tests
pytest tests/unit/global_time_series/test_*.py # 10 passed, 2 warnings in 11.89s
# SyntaxWarning: invalid escape sequence '\w'
# SyntaxWarning: invalid escape sequence '\s'
pytest tests/unit/pcmdi_diags/test_*.py # 7 passed in 4.82s
# Stay in the bash subshell

zppy run

# zppy itself #################################################################
cd ~/ez/zppy
git status # On branch official-results-unified-1-13
git add -A
git commit -m "Testing" --no-verify # Commit uncommitted changes

git fetch upstream main
# Go to https://github.com/E3SM-Project/zppy/commits/main
# Get the SHA from the production release commit
git checkout -b official-results-unified-1-13-try2 17ce9335fe2c828a13992df4d2f3a5631edcba2d
git log --oneline | head -n 1
# 17ce9335 Bump to 3.2.0 (#826)
# Good, that's the production release

showquota # Have enough space on home and pscratch
# (Currently in /global/homes/f/forsyth/ez/zppy)

# WE ARE ALREADY IN THE UNIFIED ENV.
# DO NOT MAKE A DEV ENV.

pixi list python # Check Python version
# python                    3.13.13

# Run zppy unit tests
pytest tests/test_*.py # 44 passed in 0.24s

# Edit tests/integration/utils.py:

TEST_SPECIFICS: Dict[str, Any] = {
    # This is the NCO path.
    # Keep as "" to use the production-version NCO commands.
    # Set to a specific path to use development-version NCO commands.
    "nco_path": "",
    # These are custom environment_commands for specific tasks.
    # Never set these to "", because they will print the line
    # `environment_commands = ""` for the corresponding task,
    # thus overriding the value set higher up in the cfg.
    # That is, there will be no environment set.
    # (`environment_commands = ""` only redirects to Unified
    # if specified under the [default] task)
    "e3sm_to_cmip_environment_commands": "source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh",
    "diags_environment_commands": "source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh",
    "mpas_analysis_environment_commands": "source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh",
    "global_time_series_environment_commands": "source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh",
    "livvkit_environment_commands": "source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh",
    "pcmdi_diags_environment_commands": "source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh",
    # This is the environment setup for other tasks.
    # Leave as "" to use the latest Unified environment.
    "environment_commands": "source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh",
    # For a complete test, run the set of latest cfgs and at least one set of legacy cfgs
    "cfgs_to_run": [
        # "weekly_bundles",
        # "weekly_comprehensive_v2",
        "weekly_comprehensive_v3",
        # "weekly_legacy_3.1.0_bundles",
        # "weekly_legacy_3.1.0_comprehensive_v2",
        # "weekly_legacy_3.1.0_comprehensive_v3",
        # "weekly_legacy_3.0.0_bundles",
        # "weekly_legacy_3.0.0_comprehensive_v2",
        # "weekly_legacy_3.0.0_comprehensive_v3",
    ],
    "tasks_to_run": [
        "e3sm_diags",
        "mpas_analysis",
        "global_time_series",
        "ilamb",
        "livvkit",
        "pcmdi_diags",
    ],
    "unique_id": "official_results_unified_1_13_try2",
}

git diff # Diff looks good
python tests/integration/utils.py
# CFG FILES HAVE BEEN GENERATED FROM TEMPLATES WITH THESE SETTINGS:
# UNIQUE_ID=official_results_unified_1_13_try2
# nco_path=
# Reminder: `nco_path=''` => the production-version NCO commands will be used
# e3sm_to_cmip_environment_commands=source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh
# diags_environment_commands=source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh
# mpas_analysis_environment_commands=source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh
# global_time_series_environment_commands=source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh
# livvkit_environment_commands=source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh
# pcmdi_diags_environment_commands=source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh
# environment_commands=source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh
# Reminder: `environment_commands=''` => the latest E3SM Unified environment will be used

# For reference:
alias sq
# Perlmutter: alias sq='sqa -u forsyth'
# Compy:      alias sq='sqa -u fors729'
alias sqa
# Perlmutter: alias sqa='squeue -o "%8u %.7a %.4D %.9P %8i %.2t %.10r %.10M %.10l %j" --sort=P,-t,-p'
# Compy:      alias sqa='squeue -o "%8u %.7a %.4D %.9P %7i %.2t %.10r %.10M %.10l %.8Q %j" --sort=P,-t,-p'
sq
# No jobs currently queued

# This is the only cfg we're running on Perlmutter:
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_pm-cpu.cfg

sq | wc -l # Thu 05/21 14:06 => 56 - header row = 55 jobs

# Picking up Tue 5/26 #########################################################
sq | wc -l # Tue 05/26 10:31 => 1 - header row = 0 jobs

Review finished runs

# Re-establish Unified env
bash
source ~/.bash_profile.ext
source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh

### v3 ###
cd /global/cfs/cdirs/e3sm/forsyth/zppy_weekly_comprehensive_v3_output/official_results_unified_1_13_try2/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors

post/scripts directory status: NO ERRORS

Python tests

cd ~/ez/zppy
git status
# Good, correct branch: official-results-unified-1-13-try2
# Good, correct env: Unified env
ls tests/integration/test_*.py

# Non-applicable (we didn't do the necessary `zppy` runs for this test):
# pytest tests/integration/test_bundles.py 

# Passing
pytest tests/integration/test_bash_generation.py # 1 passed in 1.23s
pytest tests/integration/test_campaign.py # 6 passed in 3.17s
pytest tests/integration/test_defaults.py # 1 passed in 0.59s
pytest tests/integration/test_last_year.py # 1 passed in 0.53s

# We'll now proceed with the image checker test.
salloc --nodes=1 --qos=interactive --time=01:00:00 --constraint=cpu --account=e3sm
# Re-establish Unified env
bash
source ~/.bash_profile.ext
source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh
pytest tests/integration/test_images.py # 1 failed in 1099.39s (0:18:19)
cat test_images_summary.md

Output:

----------------------------------------------- Captured stdout call ------------------------------------------------
Preparing weekly cfg tests
Preparing legacy 3.1.0 cfg tests
Preparing legacy 3.0.0 cfg tests
Running 1 tests in parallel
Individual test logs will be written to test_<name>.log files
✓ Completed: comprehensive_v3 (5 tasks) (log: test_comprehensive_v3.log)
Copy the output of test_images_summary.md to a Pull Request comment

Test Summary:
Test                                                    Total    Correct     Status
----------------------------------------------------------------------------------
comprehensive_v3_e3sm_diags                              5369       5259     ✗ FAIL
comprehensive_v3_mpas_analysis                           1280       1280     ✓ PASS
comprehensive_v3_global_time_series                      1404       1404     ✓ PASS
comprehensive_v3_ilamb                                    837        837     ✓ PASS
comprehensive_v3_pcmdi_diags                              647        647     ✓ PASS
----------------------------------------------------------------------------------

⚠ Some tests had mismatched or missing images. Check individual log files for details.
============================================== short test summary info ==============================================
FAILED tests/integration/test_images.py::test_images - assert 5369 == 5259
========================================== 1 failed in 1099.39s (0:18:19) ===========================================

Complete summary table:

Test name	Total images	Correct images	Missing images
comprehensive_v3_e3sm_diags	5369	5259	110 (list)
comprehensive_v3_mpas_analysis	1280	1280	0
comprehensive_v3_global_time_series	1404	1404	0
comprehensive_v3_ilamb	837	837	0
comprehensive_v3_pcmdi_diags	647	647	0

Summary table -- only failing image-check tests, sorted by task

e3sm_diags

Test name	Total images	Correct images	Missing images	Mismatched images
comprehensive_v3_e3sm_diags	5369	5259	110 (list)	0

These missing images are consistent with what saw on Chrysalis and Compy. The 110 missing images on the e3sm_diags task can be explained by this PR which was merged since the last run: E3SM-Project/e3sm_diags#1055

Updating expected results

Setting up new expected results

We last updated Perlmutter expected results during 2026-05-08 Perlmutter E3SM Unified 1.13.0rc10 (zppy v3.2.0rc2), on May 11.

ls -lt /global/cfs/cdirs/e3sm/www/zppy_test_resources/
# The files are from May 11
# This is consistent with how we updated the expected results on 2026-05-11
ls -lt /global/cfs/cdirs/e3sm/www/zppy_test_resources_previous
# May 11 11:35 expected_results_until_20260511
ls -lt /global/cfs/cdirs/e3sm/www/zppy_test_resources_previous/expected_results_until_20260511
# Notice: similar contents to /global/cfs/cdirs/e3sm/www/zppy_test_resources/
# (Less content though, because we ran fewer tests)

# First, let's copy over the old expected results #############################
cd /global/cfs/cdirs/e3sm/www/
mv zppy_test_resources/ zppy_test_resources_previous/expected_results_until_20260526 # mv is faster than cp
mkdir zppy_test_resources # Remake the official expected_results dir
ls -lt zppy_test_resources_previous/expected_results_until_20260526
# Good, contents are from 5/11

# Second, update the expected results #########################################

# Let's update the simpler tests' results first:
cd ~/ez/zppy
git status
# Good, correct branch: official-results-unified-1-13-try2
# Good, correct env: Unified env

ls tests/integration/generated/update_*_pm-cpu.sh
# tests/integration/generated/update_archive_expected_files_pm-cpu.sh
# tests/integration/generated/update_bash_generation_expected_files_pm-cpu.sh
# tests/integration/generated/update_campaign_expected_files_pm-cpu.sh
# tests/integration/generated/update_defaults_expected_files_pm-cpu.sh
# tests/integration/generated/update_weekly_expected_files_pm-cpu.sh


chmod 755 tests/integration/generated/update_bash_generation_expected_files_pm-cpu.sh
./tests/integration/generated/update_bash_generation_expected_files_pm-cpu.sh 
# Have to run twice, so we can re-build the expected results
# 1 passed in 0.60s

# We have to rerun the tests because the expected results dir is now empty,
# because we ran `mv zppy_test_resources/` rather than `cp zppy_test_resources/`.

chmod 755 tests/integration/generated/update_campaign_expected_files_pm-cpu.sh
pytest tests/integration/test_campaign.py # Fails simply because we have no expected results
./tests/integration/generated/update_campaign_expected_files_pm-cpu.sh
# 6 passed in 3.05s

chmod 755 tests/integration/generated/update_defaults_expected_files_pm-cpu.sh
pytest tests/integration/test_defaults.py # Fails simply because we have no expected results
./tests/integration/generated/update_defaults_expected_files_pm-cpu.sh
# 1 passed in 0.55s

# test_last_year doesn't need an update script because its cfg is stand-alone:
pytest -vv tests/integration/test_last_year.py
# 1 passed in 0.53s


# Now update the expected files for the major tests:

# To make sure exiting the SSH connection doesn't halt the progress,
# we can use screen.
screen -ls # No Sockets found
screen
screen -ls # 1082087.pts-69.login11  (Attached)
# Re-establish env:
bash # Run bash so we're in an isolated subshell
source ~/.bash_profile.ext
source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh
chmod 755 tests/integration/generated/update_weekly_expected_files_pm-cpu.sh
./tests/integration/generated/update_weekly_expected_files_pm-cpu.sh # About 25 minutes to run
# CTRL A D
screen -ls # 646678.pts-42.login07	(Detached)

screen -R
# cp: cannot stat '/global/cfs/cdirs/e3sm/www/forsyth/zppy_weekly_comprehensive_v2_www/official_results_unified_1_13_try2/v2.LR.historical_0201': No such file or directory
# ./tests/integration/generated/update_weekly_expected_files_pm-cpu.sh: line 56: cd: /global/cfs/cdirs/e3sm/www/zppy_test_resources/expected_comprehensive_v2: No such file or directory
# cp: cannot stat '/global/cfs/cdirs/e3sm/www/forsyth/zppy_weekly_bundles_www/official_results_unified_1_13_try2/v3.LR.historical_0051': No such file or directory
# cp: cannot stat '/global/cfs/cdirs/e3sm/forsyth/zppy_weekly_bundles_output/official_results_unified_1_13_try2/v3.LR.historical_0051/post/scripts/bundle*.bash': No such file or directory
# cp: cannot stat '/global/cfs/cdirs/e3sm/www/forsyth/zppy_weekly_legacy_3.1.0_comprehensive_v2_www/official_results_unified_1_13_try2/v2.LR.historical_0201': No such file or directory
# ./tests/integration/generated/update_weekly_expected_files_pm-cpu.sh: line 56: cd: /global/cfs/cdirs/e3sm/www/zppy_test_resources/expected_legacy_3.1.0_comprehensive_v2: No such file or directory
# cp: cannot stat '/global/cfs/cdirs/e3sm/www/forsyth/zppy_weekly_legacy_3.1.0_comprehensive_v3_www/official_results_unified_1_13_try2/v3.LR.historical_0051': No such file or directory
# ./tests/integration/generated/update_weekly_expected_files_pm-cpu.sh: line 56: cd: /global/cfs/cdirs/e3sm/www/zppy_test_resources/expected_legacy_3.1.0_comprehensive_v3: No such file or directory
# cp: cannot stat '/global/cfs/cdirs/e3sm/www/forsyth/zppy_weekly_legacy_3.1.0_bundles_www/official_results_unified_1_13_try2/v3.LR.historical_0051': No such file or directory
# ./tests/integration/generated/update_weekly_expected_files_pm-cpu.sh: line 56: cd: /global/cfs/cdirs/e3sm/www/zppy_test_resources/expected_legacy_3.1.0_bundles: No such file or directory
# cp: cannot stat '/global/cfs/cdirs/e3sm/www/forsyth/zppy_weekly_legacy_3.0.0_comprehensive_v2_www/official_results_unified_1_13_try2/v2.LR.historical_0201': No such file or directory
# ./tests/integration/generated/update_weekly_expected_files_pm-cpu.sh: line 56: cd: /global/cfs/cdirs/e3sm/www/zppy_test_resources/expected_legacy_3.0.0_comprehensive_v2: No such file or directory
# cp: cannot stat '/global/cfs/cdirs/e3sm/www/forsyth/zppy_weekly_legacy_3.0.0_comprehensive_v3_www/official_results_unified_1_13_try2/v3.LR.historical_0051': No such file or directory
# ./tests/integration/generated/update_weekly_expected_files_pm-cpu.sh: line 56: cd: /global/cfs/cdirs/e3sm/www/zppy_test_resources/expected_legacy_3.0.0_comprehensive_v3: No such file or directory
# cp: cannot stat '/global/cfs/cdirs/e3sm/www/forsyth/zppy_weekly_legacy_3.0.0_bundles_www/official_results_unified_1_13_try2/v3.LR.historical_0051': No such file or directory
# ./tests/integration/generated/update_weekly_expected_files_pm-cpu.sh: line 56: cd: /global/cfs/cdirs/e3sm/www/zppy_test_resources/expected_legacy_3.0.0_bundles: No such file or directory

# Yes, this does makes sense, because we exclusively ran the non-legacy v3 test.
# That's 8 tests we didn't run and the script obviously can't stat the www paths or the corresponding expected results dir for those 8 tests.
# That accounts for the 16 `cannot stat` lines.
ls -lt /global/cfs/cdirs/e3sm/www/zppy_test_resources
# Good, no longer empty.
# Everything says updated 5/26


exit # Exit bash subshell
exit # Exit screen
screen -ls # No Sockets found


cd ~/ez/zppy
# Still can't run this, because never ran the corresponding zppy jobs:
# pytest tests/integration/test_bundles.py

salloc --nodes=1 --qos=interactive --time=01:00:00 --constraint=cpu --account=e3sm
# Re-establish Unified env
bash
source ~/.bash_profile.ext
source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh
pytest tests/integration/test_images.py # 1 passed in 883.08s (0:14:43)
cat test_images_summary.md

Complete summary table:

Test name	Total images	Correct images
comprehensive_v3_e3sm_diags	5367	5367
comprehensive_v3_mpas_analysis	1280	1280
comprehensive_v3_global_time_series	1404	1404
comprehensive_v3_ilamb	837	837
comprehensive_v3_pcmdi_diags	647	647

exit # Exit bash shell
exit # Exit compute note

# Third, let's copy the expected results into a special official results dir ##
cd /global/cfs/cdirs/e3sm/www/
ls -lt zppy_test_resources_previous
# May 26 11:14 expected_results_until_20260526
cp -r zppy_test_resources/ zppy_test_resources_previous/expected_results_for_unified_1.13.0 
# <5 min to run
ls zppy_test_resources_previous/
# Good, expected_results_for_unified_1.13.0 exists
ls -lt zppy_test_resources_previous/expected_results_for_unified_1.13.0
# Has results for the 1 cfg we ran

✅ Official results for E3SM Unified 1.13.0 on Perlmutter can be found at:

/global/cfs/cdirs/e3sm/www/zppy_test_resources_previous/expected_results_for_unified_1.13.0

1 reply

forsyth2 May 26, 2026
Maintainer Author

@chengzhuzhang Just for reference -- I successfully completed the Perlmutter official results for Unified 1.13.0. (Equivalent posts: Chrysalis, Compy)

Uh oh!

E3SM Unified 1.13.0 testing #802

Uh oh!

Uh oh!

forsyth2 Apr 1, 2026 Maintainer

Replies: 33 comments · 135 replies

Uh oh!

Uh oh!

forsyth2 Apr 1, 2026 Maintainer Author

2026-03-31 Chrysalis

What changes are we testing?

Testing process

Results analysis

What changed (the critical behavior shift)

Why this caused the slowdown

1. You killed xarray/dask’s lazy reuse

2. _subset_vars_and_load() now forces full eager loads repeatedly

3. Derived variables amplify the problem

4. You likely broke backend caching semantics

Why this looks like “stalling”

The core mistake

What likely went wrong in design

❌ Closing in every layer

❌ Closing both ds_subset and ds

What to fix (high-impact)

1. Do NOT close in _subset_vars_and_load

2. Only close at the outermost lifecycle

3. Keep open_mfdataset() alive longer

4. Optional: remove scheduler="sync"

TL;DR

Uh oh!

xylar Apr 2, 2026 Collaborator

Uh oh!

forsyth2 Apr 2, 2026 Maintainer Author

Uh oh!

Uh oh!

chengzhuzhang Apr 2, 2026 Maintainer

Uh oh!

forsyth2 Apr 2, 2026 Maintainer Author

Uh oh!

xylar Apr 2, 2026 Collaborator

Uh oh!

Uh oh!

forsyth2 Apr 9, 2026 Maintainer Author

2026-04-08 Compy

What expected results are currently on Compy?

What changes are we testing?

Testing process

Results analysis

Comprehensive v3 test

Comprehensive v2 test

Comprehensive bundles test

Legacy 3.1.0 tests

Legacy 3.0.0 tests

Fixes to make

Uh oh!

Uh oh!

chengzhuzhang Apr 10, 2026 Maintainer

Uh oh!

chengzhuzhang Apr 10, 2026 Maintainer

Uh oh!

forsyth2 Apr 10, 2026 Maintainer Author

Uh oh!

xylar Apr 10, 2026 Collaborator

Uh oh!

forsyth2 Apr 10, 2026 Maintainer Author

Uh oh!

Uh oh!

forsyth2 Apr 14, 2026 Maintainer Author

2026-04-13 Compy E3SM Unified 1.13.0rc5

What expected results are currently on Compy?

What changes are we testing?

Testing process

Results analysis

Comprehensive v3 test

Comprehensive v2 test

Comprehensive bundles test

Legacy 3.1.0 tests

Legacy 3.0.0 tests

Fixes to make

forsyth2
Apr 1, 2026
Maintainer

Replies: 33 comments 135 replies

forsyth2
Apr 1, 2026
Maintainer Author

2. `_subset_vars_and_load()` now forces full eager loads repeatedly

❌ Closing both `ds_subset` and `ds`

1. Do NOT close in `_subset_vars_and_load`

3. Keep `open_mfdataset()` alive longer

4. Optional: remove `scheduler="sync"`

xylar Apr 2, 2026
Collaborator

forsyth2 Apr 2, 2026
Maintainer Author

chengzhuzhang Apr 2, 2026
Maintainer

forsyth2 Apr 2, 2026
Maintainer Author

xylar Apr 2, 2026
Collaborator

forsyth2
Apr 9, 2026
Maintainer Author

chengzhuzhang Apr 10, 2026
Maintainer

chengzhuzhang Apr 10, 2026
Maintainer

forsyth2 Apr 10, 2026
Maintainer Author

xylar Apr 10, 2026
Collaborator

forsyth2 Apr 10, 2026
Maintainer Author

forsyth2
Apr 14, 2026
Maintainer Author

forsyth2 Apr 14, 2026
Maintainer Author

TonyB9000
Apr 14, 2026
Collaborator

chengzhuzhang Apr 14, 2026
Maintainer

forsyth2 Apr 14, 2026
Maintainer Author