Replies: 30 comments 163 replies
-
Friday 2024-10-18Note that before we get to testing a release candidate for Unified, we'll need to get v3 test data on to Perlmutter and Compy too. On Chrysalis: # Set up branch
cd ez/zppy
git status
# Check no file changes will persist when we switch branches.
git fetch upstream main
git checkout -b test_zppy_weekly_20241018 upstream/main
git log
# Last commit, from 10/15: Improve carryover dependency handling (#623)
# Set up diags to use
cd ../e3sm_diags
git checkout main
git fetch upstream
git reset --hard upstream/main
git log
# Last commit, from 10/15: make --closedcontourcmd resolution specific (#824)
mamba clean --all
# Press `y` twice
mamba env create -f conda-env/dev.yml -n e3sm_diags_20241018
conda activate e3sm_diags_20241018
pip install .
# Set up testing files
cd ../zppy
# Edit tests/integration/utils.py:
# UNIQUE_ID = "test_zppy_weekly_20241018"
# For get_chyrsalis_expansions: "diags_environment_commands": "source /home/ac.forsyth2/miniconda3/etc/profile.d/conda.sh; conda activate e3sm_diags_20241018",
# Keep this as is: generate_cfgs(unified_testing=False, dry_run=False)
# Set up zppy environment
mamba clean --all
mamba env create -f conda/dev.yml -n zppy_dev_weekly_20241018
conda activate zppy_dev_weekly_20241018
pip install .
# Run unit tests
python -u -m unittest tests/test_*.py
# All 22 tests pass
# Run zppy to produce actual results to compare in the integration tests
python tests/integration/utils.py
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 1st part of bundles cfg
# Once those all finish:
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 2nd part of bundles cfg
# Once that finishes:
# Check output
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/test_zppy_weekly_20241018/v3.LR.historical_0051/post/scripts/
grep -v "OK" *status
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/test_zppy_weekly_20241018/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/test_zppy_weekly_20241018/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Run integration tests
python -u -m unittest tests/integration/test_*.pyThis gets us these results: We can see the diff is from the formatting change introduced in #627, so this is expected. Let's update the expected results. That gives: ✅ This week's testing of |
Beta Was this translation helpful? Give feedback.
0 replies
-
Friday 2024-10-25Commits since 2024-10-18:
# Set up branch
cd ~/ez/zppy
git status
# Check no file changes will persist when we switch branches.
git fetch upstream main
git checkout -b test_zppy_weekly_20241025 upstream/main
git log
# Last commit, from 10/15: Improve carryover dependency handling (#623)
# Set up diags to use
cd ../e3sm_diags
git checkout main
git status
# Check no file changes will persist when we switch branches.
git fetch upstream
git reset --hard upstream/main
git log
# Last commit, from 10/23: Fixing African easterly wave density plots in TC analysis (#851)
mamba clean --all --y
mamba env create -f conda-env/dev.yml -n e3sm_diags_20241025
conda activate e3sm_diags_20241025
pip install .
# Set up testing files
cd ../zppy
# NOTE: CURRENTLY MANUAL STEP --could potentially pass in arguments to utils.py?
# Edit tests/integration/utils.py:
# UNIQUE_ID = "test_zppy_weekly_20241025"
# For get_chyrsalis_expansions: "diags_environment_commands": "source /home/ac.forsyth2/miniconda3/etc/profile.d/conda.sh; conda activate e3sm_diags_20241025",
# Keep this as is: generate_cfgs(unified_testing=False, dry_run=False)
# Set up zppy environment
mamba clean --all --y
mamba env create -f conda/dev.yml -n zppy_dev_weekly_20241025 # This command took a while
conda activate zppy_dev_weekly_20241025
pip install .
# Run unit tests
python -u -m unittest tests/test_*.py
# All 22 tests pass
# Run zppy to produce actual results to compare in the integration tests
python tests/integration/utils.py
# That prints:
# CFG FILES HAVE BEEN GENERATED FROM TEMPLATES WITH THESE SETTINGS:
# UNIQUE_ID=test_zppy_weekly_20241025
# unified_testing=False
# diags_environment_commands=source /home/ac.forsyth2/miniconda3/etc/profile.d/conda.sh; conda activate e3sm_diags_20241025
# e3sm_to_cmip_environment_commands=
# environment_commands=
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 1st part of bundles cfg
# NOTE: CURRENTLY MANUAL STEP -- check that those 3 zppy runs have finished. Try bash `wait command? Just keep running `squeue` every hour until there are no jobs left?
# Once those all finish: (took about 1h15m)
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 2nd part of bundles cfg
# NOTE: CURRENTLY MANUAL STEP -- check that that zppy run has finished. Try bash `wait command? Just keep running `squeue` every hour until there are no jobs left?
# Once that finishes: (took about 10m)
# Check output
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/test_zppy_weekly_20241025/v3.LR.historical_0051/post/scripts/
grep -v "OK" *status
# e3sm_diags_lnd_monthly_mvm_lnd_model_vs_model_1987-1988_vs_1985-1986.status:ERROR (1)
cat e3sm_diags_lnd_monthly_mvm_lnd_model_vs_model_1987-1988_vs_1985-1986.o614397
# cp: cannot stat '/lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/test_zppy_weekly_20241025/v3.LR.historical_0051/post/lnd/180x360_aave/clim/2yr/v3.LR.historical_0051_*_1987??_1988??_climo.nc': No such file or directory
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/test_zppy_weekly_20241025/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# e3sm_diags_lnd_monthly_mvm_lnd_model_vs_model_1982-1983_vs_1980-1981.status:ERROR (1)
cat e3sm_diags_lnd_monthly_mvm_lnd_model_vs_model_1982-1983_vs_1980-1981.o614424
# cp: cannot stat '/lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/test_zppy_weekly_20241025/v2.LR.historical_0201/post/lnd/180x360_aave/clim/2yr/v2.LR.historical_0201_*_1982??_1983??_climo.nc': No such file or directory
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/test_zppy_weekly_20241025/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Run integration tests
cd ~/ez/zppy
python -u -m unittest tests/integration/test_*.pyThat gives: The The Let's look at the image check failures: Let's try the directory paths: Clearly, image check failures aren't being routed to the right place. |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
Bug report is at #635. |
Beta Was this translation helpful? Give feedback.
-
Friday 2024-11-01Commits since 2024-10-25:
# Set up branch
cd ~/ez/zppy
git status
# Check no file changes will persist when we switch branches.
git fetch upstream main
git checkout -b test_zppy_weekly_20241101 upstream/main
git log
# Last commit, from 10/30: Updates to diags handling (#633)
# Set up diags to use
cd ../e3sm_diags
git checkout main
git status
# Check no file changes will persist when we switch branches.
git fetch upstream
git reset --hard upstream/main
git log
# Last commit, from 10/29: add_TCO_60S60N to default lat-lon (#879)
mamba clean --all --y
mamba env create -f conda-env/dev.yml -n e3sm_diags_20241101
conda activate e3sm_diags_20241101
pip install .
# Set up testing files
cd ../zppy
# NOTE: CURRENTLY MANUAL STEP --could potentially pass in arguments to utils.py?
# Edit tests/integration/utils.py:
# UNIQUE_ID = "test_zppy_weekly_20241101"
# For get_chyrsalis_expansions: "diags_environment_commands": "source /home/ac.forsyth2/miniconda3/etc/profile.d/conda.sh; conda activate e3sm_diags_20241101",
# Keep this as is: generate_cfgs(unified_testing=False, dry_run=False)
# Set up zppy environment
mamba clean --all --y
mamba env create -f conda/dev.yml -n zppy_dev_weekly_20241101 # This command took a while
conda activate zppy_dev_weekly_20241101
pip install .
# Run unit tests
python -u -m unittest tests/test_*.py
# All 23 tests pass (increased from 22)
# Run zppy to produce actual results to compare in the integration tests
python tests/integration/utils.py
# That prints:
# CFG FILES HAVE BEEN GENERATED FROM TEMPLATES WITH THESE SETTINGS:
# UNIQUE_ID=test_zppy_weekly_20241101
# unified_testing=False
# diags_environment_commands=source /home/ac.forsyth2/miniconda3/etc/profile.d/conda.sh; conda activate e3sm_diags_20241101
# e3sm_to_cmip_environment_commands=
# environment_commands=
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 1st part of bundles cfg
# NOTE: CURRENTLY MANUAL STEP -- check that those 3 zppy runs have finished. Try bash `wait command? Just keep running `squeue` every hour until there are no jobs left?
# Once those all finish: (took about 1h15m)
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 2nd part of bundles cfg
# NOTE: CURRENTLY MANUAL STEP -- check that that zppy run has finished. Try bash `wait command? Just keep running `squeue` every hour until there are no jobs left?
# Once that finishes: (took about 10m)
# Check output
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/test_zppy_weekly_20241101/v3.LR.historical_0051/post/scripts/
grep -v "OK" *status
# No errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/test_zppy_weekly_20241101/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# No errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/test_zppy_weekly_20241101/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# No errors
# Run integration tests
cd ~/ez/zppy
python -u -m unittest tests/integration/test_*.pyThat gives: The diff is from the template file change in https://github.com/E3SM-Project/zppy/pull/633/files#diff-fbd1002ca5f9034c321e896884792a69fc7e4a9fc14e826a3954dc81a3d6d76d, so this is expected. Let's update the expected results. That test passes now. As for the image check failures (attempted to format this hierarchically, but only semi-successfully):
So, all image check failures are accounted for. Let's update the expected results. After about 20 minutes, that gives: All expected results have been updated. ✅ This week's testing of |
Beta Was this translation helpful? Give feedback.
0 replies
-
Friday 2024-11-22This is the first Weekly test since the merging of major refactor splitting zppy into zppy itself (workflow coordination) and zppy-interfaces ("last-mile" code improving E3SM-specific usability of external packages). Check commit historyCommits since 2024-11-01 (last Weekly test of zppy:
However, I'm going to conciously test Set up branch# Set up branch
cd ~/ez/zppy
git status
# Check no file changes will persist when we switch branches.
git fetch upstream main
git checkout -b test_zppy_weekly_20241122 upstream/main
git log
# Last commit, from 11/22: zppy-interfaces refactor (#642)Set up zppy-interfaces envcd ~/ez/zppy-interfaces
git status
# Check no file changes will persist when we switch branches.
git fetch upstream main
git checkout -b test_zi_weekly_20241122 upstream/main
git log
# Last commit, from 11/22: Initial implementation (#1)
conda clean --all --y
conda env create -f conda/dev.yml -n zi_dev_weekly_20241122
conda activate zi_dev_weekly_20241122
pip install .Set up e3sm_diags envcd ~/ez/e3sm_diags
git status
# Check no file changes will persist when we switch branches.
git fetch upstream
git checkout main
git reset --hard upstream/main
git log
# Last commit, from 11/20: Release 2.12.1 (#893)
# NOTE: As mentioned earlier, let's use the Diags code that was used for the last testing
git reset --hard 3f5b03627aef703b4f7d8463cd5faed3c0806566
git log
# Last commit, from 10/29: add_TCO_60S60N to default lat-lon (#879)
conda clean --all --y
conda env create -f conda-env/dev.yml -n e3sm_diags_from_20241029
conda activate e3sm_diags_from_20241029
pip install .Set up zppy envcd ~/ez/zppy
conda clean --all --y
conda env create -f conda/dev.yml -n zppy_dev_weekly_20241122
conda activate zppy_dev_weekly_20241122
pip install .Unit tests for zppy-interfacescd ~/ez/zppy-interfaces
conda activate zi_dev_weekly_20241122
pytest tests/global_time_series/test_*.py
# 7 passed in 25.86sUnit tests for zppycd ~/ez/zppy
conda activate zppy_dev_weekly_20241122
pytest tests/test_*.py
# 23 passed in 0.63sIntegration testing for zppy# NOTE: CURRENTLY MANUAL STEP --could potentially pass in arguments to utils.py?
# Edit tests/integration/utils.py:
# UNIQUE_ID = "test_zppy_weekly_20241122"
# For get_chyrsalis_expansions:
# "diags_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate e3sm_diags_from_20241029",
# "global_time_series_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate zi_dev_weekly_20241122",
# Keep this as is: generate_cfgs(unified_testing=False, dry_run=False)
# Run zppy to produce actual results to compare in the integration tests
python tests/integration/utils.py
# That prints:
# CFG FILES HAVE BEEN GENERATED FROM TEMPLATES WITH THESE SETTINGS:
# UNIQUE_ID=test_zppy_weekly_20241122
# unified_testing=False
# diags_environment_commands=source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate e3sm_diags_from_20241029
# e3sm_to_cmip_environment_commands=
# environment_commands=
# Reminder: `e3sm_to_cmip_environment_commands=''` => the environment of the `ts` task will be used
# Reminder: `environment_commands=''` => the latest E3SM Unified environment will be used
# NOTE: Created https://github.com/E3SM-Project/zppy/issues/643 to also print out `global_time_series_environment_commands`
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 1st part of bundles cfg
# NOTE: CURRENTLY MANUAL STEP -- check that those 3 zppy runs have finished. Try bash `wait command? Just keep running `squeue` every hour until there are no jobs left?
# Once those all finish:
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 2nd part of bundles cfg
# NOTE: CURRENTLY MANUAL STEP -- check that that zppy run has finished. Try bash `wait command? Just keep running `squeue` every hour until there are no jobs left?
# Once that finishes:
# Check output
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/test_zppy_weekly_20241122/v3.LR.historical_0051/post/scripts/
grep -v "OK" *status
# No errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/test_zppy_weekly_20241122/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# No errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/test_zppy_weekly_20241122/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# No errors
# Run integration tests
cd ~/ez/zppy
python -u -m unittest tests/integration/test_*.py
# This needs to be converted to pytest still -- see https://github.com/E3SM-Project/zppy/issues/644That gives: Errorsls tests/integration/test_*.py
# tests/integration/test_bash_generation.py tests/integration/test_defaults.py tests/integration/test_weekly.py
# tests/integration/test_campaign.py tests/integration/test_last_year.py
python -u -m unittest tests/integration/test_bash_generation.py
# FAILED (failures=1)
# The filled-out global-time-series templates have of course changed
# `inclusions/` has been added to relevant paths
# We just need to update the expected results
python -u -m unittest tests/integration/test_campaign.py
# FAILED (failures=5)
# `inclusions/` has been added to relevant paths
# We just need to update the expected results
python -u -m unittest tests/integration/test_defaults.py
# FAILED (failures=1)
# The filled-out global-time-series templates have of course changed
# `inclusions/` has been added to relevant paths
# We just need to update the expected results
python -u -m unittest tests/integration/test_last_year.py
# OKFor Weekly tests, let's look at the image check failures
These issues should be simple to fix. Next week, I'll address them and then re-run the Weekly test suite. Specifically:
|
Beta Was this translation helpful? Give feedback.
1 reply
-
Rerun on branch
|
Beta Was this translation helpful? Give feedback.
-
|
Note that #520 is to automate the |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
NOTE: The weekly tests' expected results were updated as part of the merging of #650 today (2024-12-18). All tests pass. |
Beta Was this translation helpful? Give feedback.
0 replies
-
2025-01-14 testing for v3.0.0rc1# Set up branch
cd ~/ez/zppy
git status
# Check no file changes will persist when we switch branches.
git fetch upstream main
git checkout -b test_zppy_weekly_20250114 upstream/main
git log
# Last commit, from 1/14: Update PR template (#658)
# Set up and test zppy-interfaces env
cd ~/ez/zppy-interfaces
git status
# Check no file changes will persist when we switch branches.
git fetch upstream main
git checkout -b test_zi_weekly_20250114 upstream/main
git log
# Last commit, from 1/14: Update PR template (#12)
conda clean --all --y
conda env create -f conda/dev.yml -n zi_dev_weekly_20250114
conda activate zi_dev_weekly_20250114
pip install .
pytest tests/unit/global_time_series/test_*.py
# 10 passed in 20.28s
# Set up e3sm_diags env
cd ~/ez/e3sm_diags
git status
# Check no file changes will persist when we switch branches.
git fetch upstream
# https://github.com/E3SM-Project/zppy/pull/651 (zppy using CDAT-migrated Diags) hasn't merged yet
# Because https://github.com/E3SM-Project/e3sm_diags/pull/907 (fixes for CDAT-migrated Diags) hasn't merged yet.
# So, we can't use the latest E3SM Diags.
# https://github.com/E3SM-Project/e3sm_diags/pull/902 (CDAT-migrated Diags) has 42 commits
# The first is:
# CDAT Migration Phase 2: Refactor core utilities and lat_lon set (#677) c090d34ae96dcf43159174ead5c7688385cc70fb
# From 2024-12-04
# The last commit before that was:
# Release 2.12.1 (#893) ca41b0e5d913610c88410928951f1ed11c75663f
# From 2024-11-20
git checkout -b diags_pre_cdat_migration ca41b0e5d913610c88410928951f1ed11c75663f
git log
# Confirmed last commit, from 11/20: Release 2.12.1 (#893)
conda clean --all --y
conda env create -f conda-env/dev.yml -n e3sm_diags_pre_cdat_migration
conda activate e3sm_diags_pre_cdat_migration
pip install .
# Set up zppy env
cd ~/ez/zppy
conda clean --all --y
conda env create -f conda/dev.yml -n zppy_dev_weekly_20250114
conda activate zppy_dev_weekly_20250114
pip install .
# Unit tests for zppy
pytest tests/test_*.py
# 23 passed, 1 warning in 0.40s
# (DeprecationWarning: invalid escape sequence \.)
# Integration testing for zppy
# Edit tests/integration/utils.py:
# UNIQUE_ID = "test_zppy_weekly_20250114"
# "diags_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate e3sm_diags_pre_cdat_migration",
# "global_time_series_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate zi_dev_weekly_20250114",
# Keep this as-is: generate_cfgs(unified_testing=False, dry_run=False)
# Run zppy to produce actual results to compare in the integration tests
python tests/integration/utils.py
# That prints:
# CFG FILES HAVE BEEN GENERATED FROM TEMPLATES WITH THESE SETTINGS:
# UNIQUE_ID=test_zppy_weekly_20250114
# unified_testing=False
# diags_environment_commands=source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate e3sm_diags_pre_cdat_migration
# global_time_series_environment_commands=source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate zi_dev_weekly_20250114
# environment_commands=
# Reminder: `environment_commands=''` => the latest E3SM Unified environment will be used
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 1st part of bundles cfg
# Wait to finish
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 2nd part of bundles cfg
# Wait to finish
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/test_zppy_weekly_20250114/v3.LR.historical_0051/post/scripts/
grep -v "OK" *status
# No errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/test_zppy_weekly_20250114/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# No errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/test_zppy_weekly_20250114/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# No errors
cd ~/ez/zppy
ls tests/integration/test_*.py
pytest tests/integration/test_bash_generation.py
# 1 failed in 1.30s
# Diffs are expected
pytest tests/integration/test_campaign.py
# 3 failed, 3 passed in 2.33s
# Diffs are expected
pytest tests/integration/test_defaults.py
# 1 failed in 1.25s
# Diffs are expected
pytest tests/integration/test_last_year.py
# 1 passed in 0.27s
pytest tests/integration/test_weekly.py
# FAILED tests/integration/test_weekly.py::test_comprehensive_v2_images - AssertionError
# FAILED tests/integration/test_weekly.py::test_comprehensive_v3_images - AssertionError
# 2 failed, 3 passed in 1290.08s (0:21:30)
# test_comprehensive_v2_images
# Total number of images checked: 4376
# Missing images: 2
# e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/qbo/QBO-model-vs-model/qbo_diags.png
# e3sm_diags/atm_monthly_180x360_aave/model_vs_obs_1982-1983/qbo/QBO-ERA-Interim/qbo_diags.png
# test_comprehensive_v3_images
# Total number of images checked: 5216
# Missing images: 2
# e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/qbo/QBO-model-vs-model/qbo_diags.png
# e3sm_diags/atm_monthly_180x360_aave/model_vs_obs_1987-1988/qbo/QBO-ERA-Interim/qbo_diags.png
# Mismatched images: 14
# global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_s_lnd_1.png
# global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_glb_lnd_3.png
# global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_s_lnd_3.png
# global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_glb_lnd_0.png
# global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_glb_lnd_2.png
# global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_n_lnd_1.png
# global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_s_original.png
# global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_s_lnd_0.png
# global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_n_lnd_0.png
# global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_s_lnd_2.png
# global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_glb_lnd_1.png
# global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_n_original.png
# global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_n_lnd_2.png
# global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_n_lnd_3.png
# Let's look at the image diffs
# https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v2_www/test_zppy_weekly_20250114/v2.LR.historical_0201/image_check_failures_comprehensive_v2/ is empty
# https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v3_www/test_zppy_weekly_20250114/v3.LR.historical_0051/image_check_failures_comprehensive_v3/global_time_series/ has diffs
# The diffs appear to be due to the unit changes introduced by zppy_interfaces/global_time_series/zppy_land_fields.csv
# What about the missing E3SM images though?
# QBO isn't showing up on:
# https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v2_www/test_zppy_weekly_20250114/v2.LR.historical_0201/e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/viewer/
# https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v2_www/test_zppy_weekly_20250114/v2.LR.historical_0201/e3sm_diags/atm_monthly_180x360_aave/model_vs_obs_1982-1983/viewer/
# https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v3_www/test_zppy_weekly_20250114/v3.LR.historical_0051/e3sm_diags/atm_monthly_180x360_aave/model_vs_obs_1987-1988/viewer/
# https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v3_www/test_zppy_weekly_20250114/v3.LR.historical_0051/e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/viewer/
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/test_zppy_weekly_20250114/v3.LR.historical_0051/post/scripts
grep -in qbo e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1987-1988.o665758
# 1642:2025-01-14 15:38:56,458 [ERROR]: core_parameter.py(_run_diag:269) >> Error in e3sm_diags.driver.qbo_driver
# 1646: File "/gpfs/fs1/home/ac.forsyth2/miniforge3/envs/e3sm_diags_pre_cdat_migration/lib/python3.10/site-packages/e3sm_diags/driver/qbo_driver.py", line 261, in run_diag
# 1648: File "/gpfs/fs1/home/ac.forsyth2/miniforge3/envs/e3sm_diags_pre_cdat_migration/lib/python3.10/site-packages/e3sm_diags/driver/qbo_driver.py", line 170, in get_psd_from_wavelet
# Traceback (most recent call last):
# File "/gpfs/fs1/home/ac.forsyth2/miniforge3/envs/e3sm_diags_pre_cdat_migration/lib/python3.10/site-packages/e3sm_diags/parameter/core_parameter.py", line 266, in _run_diag
# single_result = module.run_diag(self)
# File "/gpfs/fs1/home/ac.forsyth2/miniforge3/envs/e3sm_diags_pre_cdat_migration/lib/python3.10/site-packages/e3sm_diags/driver/qbo_driver.py", line 261, in run_diag
# test["wave_period"], test_wavelet = get_psd_from_wavelet(test_detrended_data)
# File "/gpfs/fs1/home/ac.forsyth2/miniforge3/envs/e3sm_diags_pre_cdat_migration/lib/python3.10/site-packages/e3sm_diags/driver/qbo_driver.py", line 170, in get_psd_from_wavelet
# cwtmatr = scipy.signal.cwt(data, scipy.signal.morlet2, widths=widths, w=deg)
# AttributeError: module 'scipy.signal' has no attribute 'cwt'
grep -in cwt e3sm_diags*.o*
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1987-1988.o665758:1649: cwtmatr = scipy.signal.cwt(data, scipy.signal.morlet2, widths=widths, w=deg)
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1987-1988.o665758:1650:AttributeError: module 'scipy.signal' has no attribute 'cwt'
# e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1987-1988_vs_1985-1986.o665759:2297: cwtmatr = scipy.signal.cwt(data, scipy.signal.morlet2, widths=widths, w=deg)
# e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1987-1988_vs_1985-1986.o665759:2298:AttributeError: module 'scipy.signal' has no attribute 'cwt'
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/test_zppy_weekly_20250114/v2.LR.historical_0201/post/scripts
grep -in cwt e3sm_diags*.o*
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.o665789:1790: cwtmatr = scipy.signal.cwt(data, scipy.signal.morlet2, widths=widths, w=deg)
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.o665789:1791:AttributeError: module 'scipy.signal' has no attribute 'cwt'
# e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1980-1981_vs_1980-1981.o665790:3483: cwtmatr = scipy.signal.cwt(data, scipy.signal.morlet2, widths=widths, w=deg)
# e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1980-1981_vs_1980-1981.o665790:3484:AttributeError: module 'scipy.signal' has no attribute 'cwt' |
Beta Was this translation helpful? Give feedback.
18 replies
-
|
Ok latest testing appears to be working, so I'm going to updated expected results and get the Testing details# Set up branch
cd ~/ez/zppy
git status
# Check no file changes will persist when we switch branches.
git fetch upstream main
git fetch xylar
git checkout -b test_zppy_weekly_20250116 upstream/main
git log
# Last commit, from 1/14: Update PR template (#658)
# Add Xylar's commit from https://github.com/E3SM-Project/zppy/pull/659
git cherry-pick 629289e8f532fc2f66d6c9ccb3637fb43179c8c9
git log
# Xylar's commit is there
# Set up and test zppy-interfaces env
cd ~/ez/zppy-interfaces
git status
# Check no file changes will persist when we switch branches.
git fetch upstream main
git checkout -b test_zi_weekly_20250116 upstream/main
git log
# Last commit, from 1/16: Bump to 0.1.1 (#13)
conda clean --all --y
conda env create -f conda/dev.yml -n zi_dev_weekly_20250116
conda activate zi_dev_weekly_20250116
pip install .
pytest tests/unit/global_time_series/test_*.py
# 10 passed in 22.99s
# Set up e3sm_diags env
cd ~/ez/e3sm_diags
git status
# Check no file changes will persist when we switch branches.
git checkout diags_pre_cdat_migration_plus_constraint
# Branch created earlier
git log
# Confirm last 2 commits:
# 1/15: Add scipy constraint
# 11/20: Release 2.12.1 (#893)
# Use existing conda env
conda activate e3sm_diags_pre_cdat_migration_plus_constraint
pip install .
# Set up zppy env
cd ~/ez/zppy
conda clean --all --y
conda env create -f conda/dev.yml -n zppy_dev_weekly_20250116
conda activate zppy_dev_weekly_20250116
pip install .
# Unit tests for zppy
pytest tests/test_*.py
# 23 passed in 0.23s
# Integration testing for zppy
# Edit tests/integration/utils.py:
# UNIQUE_ID = "test_zppy_weekly_20250116"
# "diags_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate e3sm_diags_pre_cdat_migration_plus_constraint",
# "global_time_series_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate zi_dev_weekly_20250116",
# Keep this as-is: generate_cfgs(unified_testing=False, dry_run=False)
# Run zppy to produce actual results to compare in the integration tests
python tests/integration/utils.py
# That prints:
# CFG FILES HAVE BEEN GENERATED FROM TEMPLATES WITH THESE SETTINGS:
# UNIQUE_ID=test_zppy_weekly_20250116
# unified_testing=False
# diags_environment_commands=source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate e3sm_diags_pre_cdat_migration_plus_constraint
# global_time_series_environment_commands=source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate zi_dev_weekly_20250116
# environment_commands=
# Reminder: `environment_commands=''` => the latest E3SM Unified environment will be used
# Clear earlier dependent tasks
scancel -u ac.forsyth2
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 1st part of bundles cfg
# Wait to finish
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 2nd part of bundles cfg
# Wait to finish
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/test_zppy_weekly_20250116/v3.LR.historical_0051/post/scripts/
grep -v "OK" *status
# No errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/test_zppy_weekly_20250116/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# No errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/test_zppy_weekly_20250116/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# No errors
cd ~/ez/zppy
ls tests/integration/test_*.py
pytest tests/integration/test_bash_generation.py
# 1 failed in 2.44s
# Diffs are expected
pytest tests/integration/test_campaign.py
# 5 failed, 1 passed in 2.50s
# Diffs are expected
pytest tests/integration/test_defaults.py
# 1 failed in 1.01s
# Diffs are expected
pytest tests/integration/test_last_year.py
# 1 passed in 1.06s
pytest tests/integration/test_weekly.py
# FAILED tests/integration/test_weekly.py::test_comprehensive_v3_images - AssertionError
# 1 failed, 4 passed in 1293.95s (0:21:33)The errors are: Let's look at the image diffs. https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v3_www/test_zppy_weekly_20250116/v3.LR.historical_0051/image_check_failures_comprehensive_v3/global_time_series/ has diffs, which continue to appear to be due to the unit changes introduced by zppy_interfaces/global_time_series/zppy_land_fields.csv (or else minor changes from pixel shifting) |
Beta Was this translation helpful? Give feedback.
-
Updated expected results# Remembered to copy over the old expected results!
cd /lcrc/group/e3sm/public_html
cp -r zppy_test_resources zppy_test_resources_previous/expected_results_until_20250116
cd ~/ez/zppy
chmod u+x tests/integration/generated/update_*_chrysalis.sh
./tests/integration/generated/update_bash_generation_expected_files_chrysalis.sh
# 1 passed in 1.49s
./tests/integration/generated/update_campaign_expected_files_chrysalis.sh
# 1 failed, 5 passed in 1.74s
rm -rf /lcrc/group/e3sm/public_html/zppy_test_resources/test_campaign_high_res_v1_expected_files
mkdir -p /lcrc/group/e3sm/public_html/zppy_test_resources/test_campaign_high_res_v1_expected_files
mv test_campaign_high_res_v1_output/post/scripts/*.settings /lcrc/group/e3sm/public_html/zppy_test_resources/test_campaign_high_res_v1_expected_files
pytest tests/integration/test_campaign.py
# 6 passed in 1.47s
./tests/integration/generated/update_defaults_expected_files_chrysalis.sh
# 1 passed in 0.29s
./tests/integration/generated/update_weekly_expected_files_chrysalis.sh
# 5 passed in 1298.62s (0:21:38) |
Beta Was this translation helpful? Give feedback.
-
|
@forsyth2, I've seen some more of these |
Beta Was this translation helpful? Give feedback.
-
|
I added MPAS-Dev/MPAS-Analysis#1052 to try to keep tack of it. |
Beta Was this translation helpful? Give feedback.
-
|
Sounds good, thanks @xylar |
Beta Was this translation helpful? Give feedback.
-
Thread for E3SM Unified 1.11.0 testingWhere is testing data currently?
2025-01-21source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.11.0rc2_chrysalis.sh
conda list
conda list | grep -E "zppy-interfaces|zppy|zstash"
# zppy 3.0.0rc1 pyh51c0ceb_0 conda-forge/label/zppy_dev
# zppy-interfaces 0.1.1 pyhd8ed1ab_0 conda-forge
# zstash 1.4.4rc1 pyh9caca29_0 conda-forge/label/zstash_devTo do:
|
Beta Was this translation helpful? Give feedback.
42 replies
-
|
@chengzhuzhang If you don't already have something like this, I made a script to compare actual/expected/diff of files in the Image diff grid script# Stand-alone python script
from math import ceil
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
import matplotlib.backends.backend_pdf
import os
def make_image_diff_grid(directory):
files = os.listdir(directory)
prefixes = [f.split("_diff.png")[0] for f in files if f.endswith("_diff.png")]
rows = len(prefixes)
#assert len(files) == 3*rows + 1, f"Expected 3 files per prefix plus grid, got {len(files)} files"
cols = 3 # actual, expected, diff
print(f"Constructing a {rows}x{cols} grid of image diffs")
pdf = matplotlib.backends.backend_pdf.PdfPages(f"{directory}/image_diff_grid.pdf")
rows_per_page = 4
num_pages = ceil(rows/rows_per_page)
for page in range(num_pages):
fig, axes = plt.subplots(rows_per_page, cols)
print(f"Page {page}")
for i, ax_row in enumerate(axes):
count = page*3 + i
if count > len(prefixes)-1:
break
print(f"prefixes[{i}]: {prefixes[count]}")
if len(prefixes[count]) > 40:
ax_row[0].set_title(f"{prefixes[count][:40]}...", fontsize=6) # First 40 chars
else:
ax_row[0].set_title(prefixes[count], fontsize=6) # Full title
img = mpimg.imread(f"{directory}/{prefixes[count]}_actual.png")
ax_row[0].imshow(img)
ax_row[0].set_xticks([])
ax_row[0].set_yticks([])
img = mpimg.imread(f"{directory}/{prefixes[count]}_expected.png")
ax_row[1].imshow(img)
ax_row[1].set_xticks([])
ax_row[1].set_yticks([])
img = mpimg.imread(f"{directory}/{prefixes[count]}_diff.png")
ax_row[2].imshow(img)
ax_row[2].set_xticks([])
ax_row[2].set_yticks([])
fig.tight_layout()
pdf.savefig(1)
plt.clf()
pdf.close()
if __name__ == "__main__":
path = "/lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v3_www/test_zppy_rc3_unified_rc8/v3.LR.historical_0051/image_check_failures_comprehensive_v3/mpas_analysis/ts_1985-1995_climo_1990-1995/sea_ice/"
#path = "/lcrc/group/e3sm/public_html//diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v3_www/test_zppy_rc3_unified_rc8/v3.LR.historical_0051/image_check_failures_comprehensive_v3/ilamb/_1985-1988/EcosystemandCarbonCycle/EcosystemRespiration/FLUXCOM/"
make_image_diff_grid(path)(Also at #685) |
Beta Was this translation helpful? Give feedback.
-
|
the diffs from @forsyth2 great idea to have over-viewer capability for quickly viewing all image diffs. Yes, it would be useful to walk through all directoreis, with each diff sets marked with path/filename. |
Beta Was this translation helpful? Give feedback.
-
Testing E3SM Unified 1.11.0rc10Summary on https://acme-climate.atlassian.net/wiki/spaces/IPD/pages/4984012840/zppy+test+results ChrysalisImage counts, table form. Click the mismatched image link to go to the image diff grid, generated using the script from #685.
@chengzhuzhang From what I can tell, the @xylar The Chrysalis detailed testing steps# 2025-02-20
source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.11.0rc10_chrysalis.sh
conda list | grep -E "e3sm_diags|zppy-interfaces|zppy"
# e3sm_diags 3.0.0rc3 pyh2ed6286_0 conda-forge/label/e3sm_diags_dev
# zppy 3.0.0rc3 pyh51c0ceb_0 conda-forge/label/zppy_dev
# zppy-interfaces 0.1.1 pyhd8ed1ab_0 conda-forge
# Unified rc8->rc10 => e3sm_diags: rc2->rc3, zppy & zppy-interfaces: no change
# The last important commit for...
# https://github.com/E3SM-Project/e3sm_diags/commits/main -- 1 important merges since 3.0.0rc3: #943
# https://github.com/E3SM-Project/zppy-interfaces/commits/main -- 0 important merges since 0.1.1
# https://github.com/E3SM-Project/zppy/commits/main -- 0 important merges since 3.0.0rc3
cd ~/ez/zppy
git status
# Make sure no changes will persist
git fetch upstream main
git checkout -b test_unified_rc10_20250220 upstream/main
git log
# Latest commit, from 2/13: Factor integration tests (#684)
# 2nd-to-last commit, from 2/12: Bump to 3.0.0rc3 (#683)
# Unit tests for zppy
pytest tests/test_*.py
# 23 passed, 1 warning in 0.67s
# /gpfs/fs1/home/ac.forsyth2/ez/zppy/zppy/utils.py:191: DeprecationWarning: invalid escape sequence '\.'
# Edit tests/integration/utils.py
# UNIQUE_ID = "test_unified_rc10_20250220"
# "diags_environment_commands": "",
# "environment_commands_test": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.11.0rc10_chrysalis.sh",
# "global_time_series_environment_commands": "",
# Set: generate_cfgs(unified_testing=True, dry_run=False)
# Keep:
# "user_input_v2": "/lcrc/group/e3sm/ac.forsyth2/",
# "user_input_v3": "/lcrc/group/e3sm2/ac.wlin/",
python tests/integration/utils.py
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 1st part of bundles cfg
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 2nd part of bundles cfg
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/test_unified_rc10_20250220/v3.LR.historical_0051/post/scripts/
grep -v "OK" *status
# No errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/test_unified_rc10_20250220/v2.LR.historical_0201/post/scripts/
grep -v "OK" *status
# No errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/test_unified_rc10_20250220/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# No errors
cd ~/ez/zppy
ls tests/integration/test_*.py
pytest tests/integration/test_bash_generation.py
# 0/1 passed (diffs are expected)
pytest tests/integration/test_campaign.py
# 1/6 passed (diffs are expected)
pytest tests/integration/test_defaults.py
# 0/1 passed (diffs are expected)
pytest tests/integration/test_last_year.py
# 1/1 passed
pytest tests/integration/test_weekly.py
# 0/5 passedThat gives: Image counts, table form:
mkdir /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/test_zppy_rc3_unified_rc10_results
cd /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/test_zppy_rc3_unified_rc10_results
# Copy image checker failing file lists to this directoryNow, we can use the new image_diff_grid script to look at the mismatched images:
|
Beta Was this translation helpful? Give feedback.
-
|
Yes, all MPAS-Analysis diffs look expected and unproblematic to me. |
Beta Was this translation helpful? Give feedback.
-
|
@forsyth2 Thank you for generating the PDF to summarize the diffs. This is extremely helpful for a quick glance review. |
Beta Was this translation helpful? Give feedback.
-
Testing zppy using Unified 1.11.0rc13 on Chrysalis2025-03-13 I ran the full intergration test suite on Chrysalis. (At the moment, I still have 12 jobs waiting on Perlmutter and while Compy has finished running jobs I haven't had a chance to run All unit tests pass. Excluding To see all the steps I did to test on Chrysalis, expand this section: Testing steps# 2025-03-13
source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.11.0rc13_chrysalis.sh
conda list | grep -E "e3sm_diags|zppy-interfaces|zppy"
# e3sm_diags 3.0.0rc4 pyh2ed6286_0 conda-forge/label/e3sm_diags_dev
# zppy 3.0.0rc5 pyh51c0ceb_0 conda-forge/label/zppy_dev
# zppy-interfaces 0.1.1 pyhd8ed1ab_0 conda-forge
# Unified rc10->rc13 ===> e3sm_diags: rc3->rc4, zppy: rc3->rc5, zppy-interfaces: no change
# The last important commit for...
# https://github.com/E3SM-Project/e3sm_diags/commits/main -- 0 important merges since 3.0.0rc4
# https://github.com/E3SM-Project/zppy-interfaces/commits/main -- 0 important merges since 0.1.1
# https://github.com/E3SM-Project/zppy/commits/main -- 0 important merges since 3.0.0rc5
cd ~/ez/zppy
git status
# Make sure no changes will persist
git fetch upstream main
git checkout -b test_unified_rc13_20250313 upstream/main
git log
# Latest commit, from 3/11: Bump to 3.0.0rc5 (#691)
# Unit tests for zppy
pytest tests/test_*.py
# 25 passed, 1 warning in 0.81s
# /gpfs/fs1/home/ac.forsyth2/ez/zppy/zppy/utils.py:191: DeprecationWarning: invalid escape sequence '\.'
# Edit tests/integration/utils.py
# UNIQUE_ID = "test_unified_rc13_20250313"
# "diags_environment_commands": "",
# "environment_commands_test": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.11.0rc13_chrysalis.sh",
# "global_time_series_environment_commands": "",
# Set: generate_cfgs(unified_testing=True, dry_run=False)
# Keep:
# "user_input_v2": "/lcrc/group/e3sm/ac.forsyth2/",
# "user_input_v3": "/lcrc/group/e3sm2/ac.wlin/",
python tests/integration/utils.py
git add -A # To make it clear if there are later changes
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 1st part of bundles cfg
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 2nd part of bundles cfg
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/test_unified_rc13_20250313/v3.LR.historical_0051/post/scripts/
grep -v "OK" *status
# No errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/test_unified_rc13_20250313/v2.LR.historical_0201/post/scripts/
grep -v "OK" *status
# No errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/test_unified_rc13_20250313/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# No errors
cd ~/ez/zppy
ls tests/integration/test_*.py
pytest tests/integration/test_bash_generation.py
# 0/1 passed (diffs are expected)
pytest tests/integration/test_campaign.py
# 1/6 passed (diffs are expected)
pytest tests/integration/test_defaults.py
# 0/1 passed (diffs are expected)
pytest tests/integration/test_last_year.py
# 1/1 passed
pytest tests/integration/test_weekly.py
# 0/5 passed
cd /home/ac.forsyth2/ez/zppy/tests/integration
# Copy over image_diff_grid.py from https://github.com/E3SM-Project/zppy/pull/685
# Make some edits for use here
time python image_diff_grid.py
# real 13m42.580s
# Needed to make some more edits to get all PDFs to actually work
# Each PDF needs to be created as a distinct python call
# Otherwise, only the first generates
# run `python image_diff_grid.py` for each set of diffs individuallyThe rest of this message will concern Image counts, table form:
Challenges to easily displaying test outputPlease read this section if you are curious about the difficulties in compiling test results and displaying them in any sort of human-friendly manner. Challenges compiling and displaying resultsFirst and foremost, we usually do not have so many diffs to work through. I think because the CDAT migration touched every part of In any case, the sheer number of diffs has made combing through test results quite time consuming. I have tried to automate some of this, such as #685 for creating high-level overviews of diffs. More automation is possible, but there is always the tradeoff between just getting something done for this particular round of testing vs producing something more robust that can be used in the future many times.
e3sm_diags@chengzhuzhang @tomvothecoder Please review this section. If the diffs are expected, I can go ahead and update the expected results for e3sm_diagsMissing imagestest_comprehensive_v2_e3sm_diags_images: 81 missing
test_comprehensive_v3_e3sm_diags_images: 102 missing
Was Mismatched imagesI did a very tedious visual inspection of the diff PDFs to try to make it as clear as possible what the diffs are. See tables below. These use the image diff grid with 5 rows per page, but thanks to the Explanation of column titles (columns 5, 7, 8, 9 are probably most relevant for you two, columns 2/3/4/6 were really for checking my arithmetic)
test_comprehensive_v2_e3sm_diags_images: 979 mismatched images -- image diff grid
Notice the cumulative diff above, 980, is 1 more than the expected number of mismatched images, 979. test_comprehensive_v3_e3sm_diags_images: 1069 mismatched images -- image diff grid
Notice the cumulative diff above, 1070, is 1 more than the expected number of mismatched images, 1069. test_bundles_e3sm_diags_images: 492 mismatched images
Notice the cumulative diff above, 495, is 3 more than the expected number of mismatched images, 492. mpas_analysis@xylar Please review this section. If the diffs are expected, I can go ahead and update the expected results for I know you already reviewed the mpas_analysisMissing imagestest_comprehensive_v2_mpas_analysis_images: 722 missing (compare to 708 in the
test_comprehensive_v3_mpas_analysis_images: 740 missing (compare to 726 in the
In both cases, it looks like 14 images went missing since your Mismatched imagestest_comprehensive_v2_mpas_analysis_images: 41 mismatched images -- image diff grid
test_comprehensive_v3_mpas_analysis_images: 41 mismatched images -- image diff grid
ilamb@chengzhuzhang Please review this section. If the diffs are expected, I can go ahead and update the expected results for ilambMissing imagesNo missing images Mismatched imagesAs in test_comprehensive_v2_ilamb_images: 20 mismatched images -- image_diff_grid
test_comprehensive_v3_ilamb_images: 24 mismatched images -- image_diff_grid
test_bundles_ilamb_images: 16 mismatched images -- image_diff_grid
|
Beta Was this translation helpful? Give feedback.
16 replies
-
|
Re: Perlmutter and Compy testing -- so far no errors in jobs, except that |
Beta Was this translation helpful? Give feedback.
-
|
I went through the diffs and I think we have fixed all issues that we identified from zppy v3.0.0 test results, remaining issues are under investigation (I tend to believe most are caused by the cdms regridding/expected data issue) and may not be needed or included in this e3sm_diags v3.0.0 release. ERA5-TREFHT-ANN-land.png Resolved: ERA5-TREFHT-ANN-land.png_diff.png More metrics difference in zonal_mean_2d: RMSE CORR diffs fixed, still have diffs in test mean and ref mean (updated resolved). |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @chengzhuzhang!! If there are still some unresolved diffs, should I hold off on updating the "official" expected results? Or are those good to update now? |
Beta Was this translation helpful? Give feedback.
-
|
Perlmutter and Compy testing appear fine. Unit tests pass, no errors appear in output files, only expected diffs show up in the generated bash comparison tests.
It's hard to give a status on the weekly tests because of this; the expected results are pretty out of date. We can discuss doing weekly testing on all 3 machines, but just running on Chrysalis weekly is time-consuming. Perlmutter in particular takes up to a day to get nodes. Overall, I'm happy with Perlmutter testing steps# 2025-03-13
source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.11.0rc13_pm-cpu.sh
conda list | grep -E "e3sm_diags|zppy-interfaces|zppy"
# e3sm_diags 3.0.0rc4 pyh2ed6286_0 conda-forge/label/e3sm_diags_dev
# zppy 3.0.0rc5 pyh51c0ceb_0 conda-forge/label/zppy_dev
# zppy-interfaces 0.1.1 pyhd8ed1ab_0 conda-forge
# Unified rc2->rc13 ===> e3sm_diags: rc1->rc4, zppy: rc2->rc5, zppy-interfaces: no change
# The last important commit for...
# https://github.com/E3SM-Project/e3sm_diags/commits/main -- 0 important merges since 3.0.0rc4
# https://github.com/E3SM-Project/zppy-interfaces/commits/main -- 0 important merges since 0.1.1
# https://github.com/E3SM-Project/zppy/commits/main -- 0 important merges since 3.0.0rc5
cd ~/ez/zppy
git status
# Make sure no changes will persist
git fetch upstream main
git checkout -b test_unified_rc13_20250313 upstream/main
git log
# Latest commit, from 3/11: Bump to 3.0.0rc5 (#691)
# Unit tests for zppy
pytest tests/test_*.py
# 25 passed, 1 warning in 0.46s
# /global/u1/f/forsyth/ez/zppy/zppy/utils.py:191: DeprecationWarning: invalid escape sequence '\.'
# Edit tests/integration/utils.py
# UNIQUE_ID = "test_unified_rc13_20250313"
# "diags_environment_commands": "",
# "environment_commands_test": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.11.0rc13_pm-cpu.sh",
# "global_time_series_environment_commands": "",
# Set: generate_cfgs(unified_testing=True, dry_run=False)
# Keep:
# "user_input_v2": "/global/cfs/cdirs/e3sm/forsyth/",
# "user_input_v3": "/global/cfs/cdirs/e3sm/forsyth/",
python tests/integration/utils.py
git add -A # To make it clear if there are later changes
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_pm-cpu.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_pm-cpu.cfg
zppy -c tests/integration/generated/test_weekly_bundles_pm-cpu.cfg # Runs 1st part of bundles cfg
# Picking up 2025-03-14
source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.11.0rc13_pm-cpu.sh
zppy -c tests/integration/generated/test_weekly_bundles_pm-cpu.cfg # Runs 2nd part of bundles cfg
cd /global/cfs/cdirs/e3sm/forsyth/zppy_weekly_comprehensive_v3_output/test_unified_rc13_20250313/v3.LR.historical_0051/post/scripts/
grep -v "OK" *status
# No errors
cd /global/cfs/cdirs/e3sm/forsyth/zppy_weekly_comprehensive_v2_output/test_unified_rc13_20250313/v2.LR.historical_0201/post/scripts/
grep -v "OK" *status
# global_time_series_1980-1990.status:WAITING 36829915
# mpas_analysis_ts_1980-1990_climo_1985-1990.status:RUNNING 36829914
tail -n 1 mpas_analysis_ts_1980-1990_climo_1985-1990.o36829914
cd /global/cfs/cdirs/e3sm/forsyth/zppy_weekly_bundles_output/test_unified_rc13_20250313/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# No errors
# Compy had a similar runtime issue with mpas_analysis. Let's rerun zppy.
cd ~/ez/zppy
emacs tests/integration/generated/test_weekly_comprehensive_v2_pm-cpu.cfg
# output = "/global/cfs/cdirs/e3sm/forsyth/zppy_weekly_comprehensive_v2_output/test_unified_rc13_20250313/v2.LR.historical_0201"
# www = "/global/cfs/cdirs/e3sm/www/forsyth/zppy_weekly_comprehensive_v2_www/test_unified_rc13_20250313"
# We'll keep those paths, but let's move the old files
cd /global/cfs/cdirs/e3sm/forsyth/zppy_weekly_comprehensive_v2_output/
mv test_unified_rc13_20250313 test_unified_rc13_20250313_try1
cd /global/cfs/cdirs/e3sm/www/forsyth/zppy_weekly_comprehensive_v2_www/
mv test_unified_rc13_20250313 test_unified_rc13_20250313_try1
cd ~/ez/zppy
emacs tests/integration/generated/test_weekly_comprehensive_v2_pm-cpu.cfg
# Originally, `walltime = "01:00:00"` for `mpas_analysis` task, which is double what Compy had
# So, looks like we need to extend this even more than on Compy
# Manually set `walltime = "02:00:00"` for `mpas_analysis` task
git diff
# The only diff is the walltime change for mpas_analysis
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_pm-cpu.cfg
# Picking up 2025-03-17
source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.11.0rc13_pm-cpu.sh
cd /global/cfs/cdirs/e3sm/forsyth/zppy_weekly_comprehensive_v3_output/test_unified_rc13_20250313/v3.LR.historical_0051/post/scripts/
grep -v "OK" *status
# global_time_series_1985-1995.status:WAITING 36829860
# mpas_analysis_ts_1985-1995_climo_1990-1995.status:RUNNING 36829858
cd /global/cfs/cdirs/e3sm/forsyth/zppy_weekly_comprehensive_v2_output/test_unified_rc13_20250313/v2.LR.historical_0201/post/scripts/
grep -v "OK" *status
# No errors
cd /global/cfs/cdirs/e3sm/forsyth/zppy_weekly_bundles_output/test_unified_rc13_20250313/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# No errors
# Strange, v2 test seems fixed, but now v3 has problems?
cd /global/cfs/cdirs/e3sm/forsyth/zppy_weekly_comprehensive_v3_output/test_unified_rc13_20250313/v3.LR.historical_0051/post/scripts/
tail -n 1 mpas_analysis_ts_1985-1995_climo_1990-1995.o36829858
# slurmstepd: error: *** JOB 36829858 ON nid006677 CANCELLED AT 2025-03-14T00:43:08 DUE TO TIME LIMIT ***
emacs mpas_analysis_ts_1985-1995_climo_1990-1995.bash
#SBATCH --time=01:00:00
cd ~/ez/zppy
emacs tests/integration/generated/test_weekly_comprehensive_v3_pm-cpu.cfg
# output = "/global/cfs/cdirs/e3sm/forsyth/zppy_weekly_comprehensive_v3_output/test_unified_rc13_20250313/v3.LR.historical_0051"
# www = "/global/cfs/cdirs/e3sm/www/forsyth/zppy_weekly_comprehensive_v3_www/test_unified_rc13_20250313"
# We'll keep those paths, but let's move the old files
cd /global/cfs/cdirs/e3sm/forsyth/zppy_weekly_comprehensive_v3_output/
mv test_unified_rc13_20250313 test_unified_rc13_20250313_try1
cd /global/cfs/cdirs/e3sm/www/forsyth/zppy_weekly_comprehensive_v3_www/
mv test_unified_rc13_20250313 test_unified_rc13_20250313_try1
cd ~/ez/zppy
emacs tests/integration/generated/test_weekly_comprehensive_v3_pm-cpu.cfg
# Manually set `walltime = "02:00:00"` for `mpas_analysis` task
git diff
# The only diff is the walltime change for mpas_analysis in both v2 and v3
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_pm-cpu.cfg
# Picking up 2025-03-19
cd /global/cfs/cdirs/e3sm/forsyth/zppy_weekly_comprehensive_v3_output/test_unified_rc13_20250313/v3.LR.historical_0051/post/scripts/
grep -v "OK" *status
# No errors
cd ~/ez/zppy
source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.11.0rc13_pm-cpu.sh
ls tests/integration/test_*.py
mkdir ~/ez/zppy_test_unified_1.11.0rc13
pytest tests/integration/test_bash_generation.py &> ~/ez/zppy_test_unified_1.11.0rc13/output_bash_generation.txt
# 0/1 passed (diffs are expected)
pytest tests/integration/test_campaign.py &> ~/ez/zppy_test_unified_1.11.0rc13/output_campaign.txt
grep "^-" ~/ez/zppy_test_unified_1.11.0rc13/output_campaign.txt # Actual
grep "^+" ~/ez/zppy_test_unified_1.11.0rc13/output_campaign.txt # Expected
# 0/6 passed (diffs are expected)
# 1 more failure than on Chrysalis
pytest tests/integration/test_defaults.py &> ~/ez/zppy_test_unified_1.11.0rc13/output_defaults.txt
# 0/1 passed (diffs are expected)
pytest tests/integration/test_last_year.py &> ~/ez/zppy_test_unified_1.11.0rc13/output_last_year.txt
# 1/1 passed
pytest tests/integration/test_weekly.py &> ~/ez/zppy_test_unified_1.11.0rc13/output_weekly.txt
# 2/14 passed
# Expected results are so far out of date as to have different expected years, so there's nothing to check againstCompy testing steps# 2025-03-13
source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.11.0rc13_compy.sh
conda list | grep -E "e3sm_diags|zppy-interfaces|zppy"
# e3sm_diags 3.0.0rc4 pyh2ed6286_0 conda-forge/label/e3sm_diags_dev
# zppy 3.0.0rc5 pyh51c0ceb_0 conda-forge/label/zppy_dev
# zppy-interfaces 0.1.1 pyhd8ed1ab_0 conda-forge
# Unified rc2->rc13 ===> e3sm_diags: rc1->rc4, zppy: rc2->rc5, zppy-interfaces: no change
# The last important commit for...
# https://github.com/E3SM-Project/e3sm_diags/commits/main -- 0 important merges since 3.0.0rc4
# https://github.com/E3SM-Project/zppy-interfaces/commits/main -- 0 important merges since 0.1.1
# https://github.com/E3SM-Project/zppy/commits/main -- 0 important merges since 3.0.0rc5
cd ~/ez/zppy
git status
# Make sure no changes will persist
git fetch upstream
# NOTE: On Compy specifically, do NOT do `git fetch upstream main`,
# as it won't actually get the latest commits!
git checkout -b test_unified_rc13_20250313 upstream/main
git log
# Latest commit, from 3/11: Bump to 3.0.0rc5 (#691)
# Unit tests for zppy
pytest tests/test_*.py
# 25 passed, 1 warning in 0.77s
# /qfs/people/fors729/ez/zppy/zppy/utils.py:191: DeprecationWarning: invalid escape sequence '\.'
# Edit tests/integration/utils.py
# UNIQUE_ID = "test_unified_rc13_20250313"
# "diags_environment_commands": "",
# "environment_commands_test": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.11.0rc13_compy.sh",
# "global_time_series_environment_commands": "",
# "user_input_v3": "/compyfs/fors729/zppy_test_data/",
# Set: generate_cfgs(unified_testing=True, dry_run=False)
# Keep:
# "user_input_v2": "/compyfs/fors729/",
python tests/integration/utils.py
git add -A # To make it clear if there are later changes
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_compy.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_compy.cfg
zppy -c tests/integration/generated/test_weekly_bundles_compy.cfg # Runs 1st part of bundles cfg
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 2nd part of bundles cfg
# Picking up 2025-03-14
cd /compyfs/fors729/zppy_weekly_comprehensive_v3_output/test_unified_rc13_20250313/v3.LR.historical_0051/post/scripts/
grep -v "OK" *status
# global_time_series_1985-1995.status:WAITING 664560
# mpas_analysis_ts_1985-1989_climo_1985-1989.status:RUNNING 664558
# mpas_analysis_ts_1985-1995_climo_1990-1995.status:WAITING 664559
tail -n 1 mpas_analysis_ts_1985-1989_climo_1985-1989.o664558
# slurmstepd: error: *** JOB 664558 ON n0066 CANCELLED AT 2025-03-13T10:12:51 DUE TO TIME LIMIT ***
emacs mpas_analysis_ts_1985-1989_climo_1985-1989.bash
#SBATCH --time=00:30:00
# Matches /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/test_unified_rc13_20250313/v3.LR.historical_0051/post/scripts/zppy_weekly_comprehensive_v3_output/test_unified_rc13_20250313/v3.LR.historical_0051/post/scripts> emacs mpas_analysis_ts_1985-1989_climo_1985-1989.bash
# So, Compy just takes longer
# Note the other two waiting tasks are waiting for this task.
cd /compyfs/fors729/zppy_weekly_comprehensive_v2_output/test_unified_rc13_20250313/v2.LR.historical_0201/post/scripts/
grep -v "OK" *status
# No errors
cd /compyfs/fors729/zppy_weekly_bundles_output/test_unified_rc13_20250313/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# No errors
source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.11.0rc13_compy.sh
cd ~/ez/zppy
emacs tests/integration/generated/test_weekly_comprehensive_v3_compy.cfg
# Manually set `walltime = "01:00:00"` for `mpas_analysis` task
cd /compyfs/fors729/zppy_weekly_comprehensive_v3_output/test_unified_rc13_20250313/v3.LR.historical_0051/post/scripts/
grep -v "OK" *status
# global_time_series_1985-1995.status:WAITING 664560
# mpas_analysis_ts_1985-1989_climo_1985-1989.status:RUNNING 664558
# mpas_analysis_ts_1985-1995_climo_1990-1995.status:WAITING 664559
rm mpas_analysis_ts_1985-1995_climo_1990-1995.status mpas_analysis_ts_1985-1989_climo_1985-1989.status global_time_series_1985-1995.status
cd ~/ez/zppy
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_compy.cfg
sq
# USER ACCOUNT NODE PARTITION JOBID ST REASON TIME TIME_LIMIT PRIORITY NAME
# fors729 e3sm 1 slurm 664706 PD Priority 0:00 1:00:00 364 mpas_analysis_ts_1985-1989_climo_1985-1989
# fors729 e3sm 1 slurm 664707 PD Dependency 0:00 1:00:00 364 mpas_analysis_ts_1985-1995_climo_1990-1995
# fors729 e3sm 1 short 664708 PD Dependency 0:00 30:00 364 global_time_series_1985-1995
sq
# USER ACCOUNT NODE PARTITION JOBID ST REASON TIME TIME_LIMIT PRIORITY NAME
# fors729 e3sm 1 slurm 664707 PD Dependency 0:00 1:00:00 367 mpas_analysis_ts_1985-1995_climo_1990-1995
# fors729 e3sm 1 short 664708 PD Dependency 0:00 30:00 367 global_time_series_1985-1995
# Two tasks left, both waiting for a dependency.
cd /compyfs/fors729/zppy_weekly_comprehensive_v3_output/test_unified_rc13_20250313/v3.LR.historical_0051/post/scripts/
grep -v "OK" *status
# global_time_series_1985-1995.status:WAITING 664708
# mpas_analysis_ts_1985-1989_climo_1985-1989.status:ERROR (1)
# mpas_analysis_ts_1985-1995_climo_1990-1995.status:WAITING 664707
tail -n 1 mpas_analysis_ts_1985-1989_climo_1985-1989.o664706
# There were errors in task climatologyMapSeaIceThickSH: remapMpasClimatology
# It's possible that the previous run of mpas_analysis altered files in the directory
# and that may now be causing conflicts.
# Let's try rerunning fresh.
cd ~/ez/zppy
emacs tests/integration/generated/test_weekly_comprehensive_v3_compy.cfg
# output = "/compyfs/fors729/zppy_weekly_comprehensive_v3_output/test_unified_rc13_20250313/v3.LR.historical_0051"
# www = "/compyfs/www/fors729/zppy_weekly_comprehensive_v3_www/test_unified_rc13_20250313"
# We'll keep those paths, but let's move the old files
cd /compyfs/fors729/zppy_weekly_comprehensive_v3_output
mv test_unified_rc13_20250313 test_unified_rc13_20250313_try1
cd /compyfs/www/fors729/zppy_weekly_comprehensive_v3_www/
mv test_unified_rc13_20250313 test_unified_rc13_20250313_try1
cd ~/ez/zppy
git diff
# The only diff is still the walltime change for mpas_analysis
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_compy.cfg
sq
# USER ACCOUNT NODE PARTITION JOBID ST REASON TIME TIME_LIMIT PRIORITY NAME
# fors729 e3sm 1 slurm 664707 PD Dependency 0:00 1:00:00 385 mpas_analysis_ts_1985-1995_climo_1990-1995
# fors729 e3sm 1 short 664708 PD Dependency 0:00 30:00 385 global_time_series_1985-1995
scancel -u fors729
# Looks like we need an even longer walltime
cd /compyfs/fors729/zppy_weekly_comprehensive_v3_output
mv test_unified_rc13_20250313 test_unified_rc13_20250313_try2
cd /compyfs/www/fors729/zppy_weekly_comprehensive_v3_www/
mv test_unified_rc13_20250313 test_unified_rc13_20250313_try2
cd ~/ez/zppy
emacs tests/integration/generated/test_weekly_comprehensive_v3_compy.cfg
# walltime = "02:00:00"
git diff
# The only diff is the new walltime for mpas_analysis
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_compy.cfg
cd /compyfs/fors729/zppy_weekly_comprehensive_v3_output/test_unified_rc13_20250313/v3.LR.historical_0051/post/scripts/
grep -v "OK" *status
# No errors
# Picking up 2025-03-17
cd ~/ez/zppy
source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.11.0rc13_compy.sh
ls tests/integration/test_*.py
mkdir ~/ez/zppy_test_unified_1.11.0rc13
pytest tests/integration/test_bash_generation.py &> ~/ez/zppy_test_unified_1.11.0rc13/output_bash_generation.txt
# 0/1 passed (diffs are expected)
pytest tests/integration/test_campaign.py &> ~/ez/zppy_test_unified_1.11.0rc13/output_campaign.txt
grep "^-" ~/ez/zppy_test_unified_1.11.0rc13/output_campaign.txt # Actual
grep "^+" ~/ez/zppy_test_unified_1.11.0rc13/output_campaign.txt # Expected
# 0/6 passed (diffs are expected)
# 1 more failure than on Chrysalis
pytest tests/integration/test_defaults.py &> ~/ez/zppy_test_unified_1.11.0rc13/output_defaults.txt
# 0/1 passed (diffs are expected)
pytest tests/integration/test_last_year.py &> ~/ez/zppy_test_unified_1.11.0rc13/output_last_year.txt
# 1/1 passed
pytest tests/integration/test_weekly.py &> ~/ez/zppy_test_unified_1.11.0rc13/output_weekly.txt
# 2/14 passed
# Expected results are so far out of date as to have different expected years, so there's nothing to check against |
Beta Was this translation helpful? Give feedback.
-
|
@chengzhuzhang Re: questions on runtime: Found with
Walltime conversions:
Walltime increases:
Directory pathsChrysalis: Perlmutter: Compy: |
Beta Was this translation helpful? Give feedback.
-
Chrysalis test 2025-04-01: latest
|
| test name | missing | mismatched | correct |
|---|---|---|---|
| test_comprehensive_v2_e3sm_diags_images | 0 | 46 | 2875 |
| test_comprehensive_v2_global_time_series_images | 0 | 5 | 1 |
| test_comprehensive_v3_e3sm_diags_images | 0 | 54 | 3696 |
| test_comprehensive_v3_global_time_series_images | 0 | 14 | 1 |
| test_bundles_global_time_series_images | 0 | 2 | 1 |
Most of the image mismatches seem ok to be. There are a couple concerns, which might not be big deals:
test_comprehensive_v3_e3sm_diags_images: The OMEGA diffs are decently noticeable. Perhaps expected?test_comprehensive_v3_global_time_series_images: lnd plots are plotted in the PDF in the order they appear in the lnd csv, rather than the order they are listed in the cfg.
Details of Chrysalis mismatched images
test_comprehensive_v2_e3sm_diags_images mismatched -- none of the diffs appear to be important. Must be a result of e3sm_diags/driver/utils/regrid.py updates in https://github.com/E3SM-Project/e3sm_diags/pull/958/files or https://github.com/E3SM-Project/e3sm_diags/pull/959/files. Note the stats do change by small amounts for some of the diffs.
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/lat_lon/model_vs_model/v2.LR.historical_0201-T-850-DJF-global.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/lat_lon/model_vs_model/v2.LR.historical_0201-SST-JJA-global.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/lat_lon/model_vs_model/v2.LR.historical_0201-U-850-DJF-global.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/lat_lon/model_vs_model/v2.LR.historical_0201-T-850-JJA-global.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/lat_lon/model_vs_model/v2.LR.historical_0201-TREFHT-SON-land.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/lat_lon/model_vs_model/v2.LR.historical_0201-TREFHT-ANN-land.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/lat_lon/model_vs_model/v2.LR.historical_0201-SST-MAM-global.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/lat_lon/model_vs_model/v2.LR.historical_0201-TREFMXAV-SON-land.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/lat_lon/model_vs_model/v2.LR.historical_0201-U-850-JJA-global.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/lat_lon/model_vs_model/v2.LR.historical_0201-U-850-ANN-global.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/lat_lon/model_vs_model/v2.LR.historical_0201-TREFHT-JJA-land.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/lat_lon/model_vs_model/v2.LR.historical_0201-U-850-MAM-global.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/lat_lon/model_vs_model/v2.LR.historical_0201-U-850-SON-global.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/lat_lon/model_vs_model/v2.LR.historical_0201-SST-ANN-global.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/lat_lon/model_vs_model/v2.LR.historical_0201-TREF_range-DJF-land.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/lat_lon/model_vs_model/v2.LR.historical_0201-T-850-SON-global.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/lat_lon/model_vs_model/v2.LR.historical_0201-TREFMXAV-MAM-land.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/lat_lon/model_vs_model/v2.LR.historical_0201-TREFMXAV-ANN-land.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/lat_lon/model_vs_model/v2.LR.historical_0201-TREF_range-SON-land.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/lat_lon/model_vs_model/v2.LR.historical_0201-TREF_range-MAM-land.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/lat_lon/model_vs_model/v2.LR.historical_0201-TREFHT-DJF-land.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/lat_lon/model_vs_model/v2.LR.historical_0201-OMEGA-850-JJA-global.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/lat_lon/model_vs_model/v2.LR.historical_0201-TAUXY-JJA-ocean.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/lat_lon/model_vs_model/v2.LR.historical_0201-OMEGA-850-MAM-global.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/lat_lon/model_vs_model/v2.LR.historical_0201-TREFHT-MAM-land.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/lat_lon/model_vs_model/v2.LR.historical_0201-T-850-MAM-global.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/lat_lon/model_vs_model/v2.LR.historical_0201-TREF_range-ANN-land.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/lat_lon/model_vs_model/v2.LR.historical_0201-SST-DJF-global.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/lat_lon/model_vs_model/v2.LR.historical_0201-TREFMXAV-DJF-land.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/lat_lon/model_vs_model/v2.LR.historical_0201-TREFMNAV-DJF-land.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/lat_lon/model_vs_model/v2.LR.historical_0201-TREFMNAV-ANN-land.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/lat_lon/model_vs_model/v2.LR.historical_0201-OMEGA-850-ANN-global.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/lat_lon/model_vs_model/v2.LR.historical_0201-SST-SON-global.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/lat_lon/model_vs_model/v2.LR.historical_0201-OMEGA-850-DJF-global.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/lat_lon/model_vs_model/v2.LR.historical_0201-TAUXY-ANN-ocean.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/lat_lon/model_vs_model/v2.LR.historical_0201-T-850-ANN-global.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/lat_lon/model_vs_model/v2.LR.historical_0201-TREFMXAV-JJA-land.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/lat_lon/model_vs_model/v2.LR.historical_0201-TREFMNAV-MAM-land.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/lat_lon/model_vs_model/v2.LR.historical_0201-TREF_range-JJA-land.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/lat_lon/model_vs_model/v2.LR.historical_0201-OMEGA-850-SON-global.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/lat_lon/model_vs_model/v2.LR.historical_0201-TREFMNAV-SON-land.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/lat_lon/model_vs_model/v2.LR.historical_0201-TREFMNAV-JJA-land.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/lat_lon/model_vs_model/v2.LR.historical_0201-TAUXY-SON-ocean.png
e3sm_diags/atm_monthly_180x360_aave/model_vs_obs_1982-1983/lat_lon/CRU_IPCC/CRU-TREFHT-DJF-land_60S90N.png
e3sm_diags/atm_monthly_180x360_aave/model_vs_obs_1982-1983/lat_lon/CRU_IPCC/CRU-TREFHT-JJA-land_60S90N.png
e3sm_diags/atm_monthly_180x360_aave/model_vs_obs_1982-1983/lat_lon/CRU_IPCC/CRU-TREFHT-ANN-land_60S90N.png
test_comprehensive_v2_global_time_series_images mismatched -- looks like the plots just expanded/shrunk horizontally a barely noticeable amount.
global_time_series/global_time_series_1980-1990_results/v2.LR.historical_0201_n_original.png
global_time_series/global_time_series_1980-1990_results/v2.LR.historical_0201_s_lnd.png
global_time_series/global_time_series_1980-1990_results/v2.LR.historical_0201_n_lnd.png
global_time_series/global_time_series_1980-1990_results/v2.LR.historical_0201_glb_lnd.png
global_time_series/global_time_series_1980-1990_results/v2.LR.historical_0201_s_original.png
test_comprehensive_v3_e3sm_diags_images mismatched -- Most of the diffs appear to be unimportant. Must be a result of e3sm_diags/driver/utils/regrid.py updates in https://github.com/E3SM-Project/e3sm_diags/pull/958/files or https://github.com/E3SM-Project/e3sm_diags/pull/959/files. Note the stats do change by small amounts for some of the diffs. The OMEGA diffs are noticeable though.
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/viewer/taylor-diagram-data/JJA_metrics_taylor_diag_amip.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/viewer/taylor-diagram-data/JJA_metrics_taylor_diag_historical.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/viewer/taylor-diagram-data/SON_metrics_taylor_diag_historical.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/viewer/taylor-diagram-data/ANN_metrics_taylor_diag_amip.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/viewer/taylor-diagram-data/ANN_metrics_taylor_diag_historical.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/viewer/taylor-diagram-data/SON_metrics_taylor_diag_amip.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/lat_lon/model_vs_model/v3.LR.historical_0051-TREF_range-JJA-land.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/lat_lon/model_vs_model/v3.LR.historical_0051-U-850-SON-global.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/lat_lon/model_vs_model/v3.LR.historical_0051-TREFMNAV-ANN-land.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/lat_lon/model_vs_model/v3.LR.historical_0051-SST-DJF-global.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/lat_lon/model_vs_model/v3.LR.historical_0051-TREFMXAV-MAM-land.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/lat_lon/model_vs_model/v3.LR.historical_0051-U-850-JJA-global.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/lat_lon/model_vs_model/v3.LR.historical_0051-TREFMXAV-DJF-land.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/lat_lon/model_vs_model/v3.LR.historical_0051-TREF_range-MAM-land.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/lat_lon/model_vs_model/v3.LR.historical_0051-TREFMNAV-SON-land.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/lat_lon/model_vs_model/v3.LR.historical_0051-TREFMNAV-MAM-land.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/lat_lon/model_vs_model/v3.LR.historical_0051-SST-SON-global.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/lat_lon/model_vs_model/v3.LR.historical_0051-U-850-DJF-global.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/lat_lon/model_vs_model/v3.LR.historical_0051-TREF_range-ANN-land.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/lat_lon/model_vs_model/v3.LR.historical_0051-TREFMNAV-JJA-land.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/lat_lon/model_vs_model/v3.LR.historical_0051-U-850-MAM-global.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/lat_lon/model_vs_model/v3.LR.historical_0051-TAUXY-SON-ocean.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/lat_lon/model_vs_model/v3.LR.historical_0051-OMEGA-850-SON-global.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/lat_lon/model_vs_model/v3.LR.historical_0051-OMEGA-850-DJF-global.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/lat_lon/model_vs_model/v3.LR.historical_0051-SST-JJA-global.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/lat_lon/model_vs_model/v3.LR.historical_0051-OMEGA-850-ANN-global.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/lat_lon/model_vs_model/v3.LR.historical_0051-TAUXY-JJA-ocean.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/lat_lon/model_vs_model/v3.LR.historical_0051-TREFHT-DJF-land.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/lat_lon/model_vs_model/v3.LR.historical_0051-TAUXY-ANN-ocean.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/lat_lon/model_vs_model/v3.LR.historical_0051-TREFMNAV-DJF-land.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/lat_lon/model_vs_model/v3.LR.historical_0051-T-850-ANN-global.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/lat_lon/model_vs_model/v3.LR.historical_0051-T-850-SON-global.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/lat_lon/model_vs_model/v3.LR.historical_0051-TREFHT-JJA-land.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/lat_lon/model_vs_model/v3.LR.historical_0051-TAUXY-MAM-ocean.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/lat_lon/model_vs_model/v3.LR.historical_0051-TREFHT-MAM-land.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/lat_lon/model_vs_model/v3.LR.historical_0051-U-850-ANN-global.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/lat_lon/model_vs_model/v3.LR.historical_0051-SST-MAM-global.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/lat_lon/model_vs_model/v3.LR.historical_0051-OMEGA-850-MAM-global.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/lat_lon/model_vs_model/v3.LR.historical_0051-TAUXY-DJF-ocean.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/lat_lon/model_vs_model/v3.LR.historical_0051-TREFHT-SON-land.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/lat_lon/model_vs_model/v3.LR.historical_0051-TREF_range-DJF-land.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/lat_lon/model_vs_model/v3.LR.historical_0051-SST-ANN-global.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/lat_lon/model_vs_model/v3.LR.historical_0051-TREFMXAV-SON-land.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/lat_lon/model_vs_model/v3.LR.historical_0051-TREFMXAV-ANN-land.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/lat_lon/model_vs_model/v3.LR.historical_0051-OMEGA-850-JJA-global.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/lat_lon/model_vs_model/v3.LR.historical_0051-T-850-JJA-global.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/lat_lon/model_vs_model/v3.LR.historical_0051-T-850-MAM-global.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/lat_lon/model_vs_model/v3.LR.historical_0051-TREFMXAV-JJA-land.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/lat_lon/model_vs_model/v3.LR.historical_0051-TREF_range-SON-land.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/lat_lon/model_vs_model/v3.LR.historical_0051-T-850-DJF-global.png
e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/lat_lon/model_vs_model/v3.LR.historical_0051-TREFHT-ANN-land.png
e3sm_diags/atm_monthly_180x360_aave/model_vs_obs_1987-1988/lat_lon/CRU_IPCC/CRU-TREFHT-DJF-land_60S90N.png
e3sm_diags/atm_monthly_180x360_aave/model_vs_obs_1987-1988/lat_lon/CRU_IPCC/CRU-TREFHT-JJA-land_60S90N.png
e3sm_diags/atm_monthly_180x360_aave/model_vs_obs_1987-1988/lat_lon/CRU_IPCC/CRU-TREFHT-ANN-land_60S90N.png
test_comprehensive_v3_global_time_series_images mismatched -- the plots are printing in a different order, so it is hard to compare any diff. It appears the order now matches the order they appear in the lnd csv. Compare the PDF with the CSV. I suspect this is because of the ("lnd", list(map(lambda v: v.variable_name, requested_variables.vars_land))), change in https://github.com/E3SM-Project/zppy-interfaces/pull/16/files.
global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_s_lnd_1.png
global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_glb_lnd_3.png
global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_s_lnd_3.png
global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_glb_lnd_0.png
global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_glb_lnd_2.png
global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_n_lnd_1.png
global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_s_original.png
global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_s_lnd_0.png
global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_n_lnd_0.png
global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_s_lnd_2.png
global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_glb_lnd_1.png
global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_n_original.png
global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_n_lnd_2.png
global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_n_lnd_3.png
(Also, for reference, note that n_original is an example of displaying blank plots in the multi-plot PDF, because regionally averaged data isn't available.)
test_bundles_global_time_series_images mismatched -- looks like the plots just expanded/shrunk horizontally a barely noticeable amount.
global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_s_original.png
global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_n_original.png
Beta Was this translation helpful? Give feedback.
10 replies
-
|
Okay, Thanks! The baseline is rc13, then it explains why there are so much less diffs. And also it looks like the diffs are expected. |
Beta Was this translation helpful? Give feedback.
-
|
Yep, that is why I wanted to get the expected results updated ASAP -- much easier to compare against. And ok, that's great! |
Beta Was this translation helpful? Give feedback.
-
|
The two tables above, for "test 2025-04-01: latest zppy, latest zppy-interfaces, e3sm_diags v3.0.0, Unified 1.11.0rc13 elsewhere", are based on the code as of #697 on the zppy commit log and E3SM-Project/zppy-interfaces#16 on the zi commit log. There have been no further commits on That leaves the ilamb commit (https://github.com/E3SM-Project/zppy/pull/702/files) as the one real functional change that hasn't been tested in the integration test suite. |
Beta Was this translation helpful? Give feedback.
-
|
Detailed change summary:
|
Beta Was this translation helpful? Give feedback.
-
|
@chengzhuzhang Auto-generated table for new Chrysalis run; matches earlier table, so no new issues. I'll merge the fix noted at #702 (comment) + #701, and get started on the final releases. Summary of test resultsDiff subdir is where to find the lists of missing/mismatched images, the image diff grid, and the individual diffs.
|
Beta Was this translation helpful? Give feedback.
-
Unified 1.11.1 official resultsThese machines have "official" test results documented for E3SM Unified 1.11.1:
@chengzhuzhang Did HadISST data on Compy get updated recently? As mentioned in today's meeting, I updated the Chrysalis expected results more recently than the Compy/Perlmutter results. When I ran the zppy tests using Unified 1.11.1, the tests pass on Chrysalis. On Perlmutter, the diffs match up exactly with what Chrysalis had before I updated the expected results for Chrysalis (meaning I'm comfortable updating Perlmutter's expected results as well). Compy however, has had 122 diffs added -- all of which are related to model-vs-obs HadISST. List of diffsThese v2 e3sm_diags diffs were found in Compy run4 but not in Compy run2: These v3 e3sm_diags diffs were found in Compy run4 but not in Compy run2: These bundles e3sm_diags diffs were found in Compy run4 but not in Compy run2: Where to find the actual diffs (subdirectories have actual,diff,expected images):
Code changes (again, these code changes affected neither Chrysalis's results or Perlmutter's results -- only Compy's results):
Comparison scriptfrom typing import Dict, List, Tuple
class Results(object):
def __init__(self, machine:str, run_name: str):
self.machine: str = machine
self.run_name: str = run_name
self.base_path: str = f"/home/ac.forsyth2/ez/unified_1.11.1_results_comparison/{self.run_name}/{self.machine}"
self.file_dict: Dict[str, Dict[str, List[str]]] = {}
def read_files(self, test: str, task: str, file_type: str) -> List[str]:
mismatched_files: List[str] = []
with open(f"{self.base_path}/{test}_{task}_{file_type}.txt", 'r') as file:
for line in file:
line = line.strip()
# For some reason, the list of mismatched files wasn't made for some runs.
# The files can still be listed via `find . -type f -name '*_diff.png'` in the proper directory.
if file_type == "found_diffs":
line = f"{task}/" + line[2:-9] # Remove leading "./" and trailing "_diff.png"
mismatched_files.append(line)
return sorted(mismatched_files)
def filter_list(list1: List[str], list2: List[str]) -> Tuple[List[str], List[str], List[str]]:
left_only: List[str] = []
right_only: List[str] = []
in_both: List[str] = []
for item in list1:
if item in list2:
in_both.append(item)
else:
left_only.append(item)
for item in list2:
if item not in list1:
right_only.append(item)
return left_only, right_only, in_both
def compare():
print("2025-03-13")
print("Run 1: E3SM Unified 1.10.0rc13")
print("zppy: E3SM Unified 1.10.0rc13 (zppy v3.0.0rc4)")
print("zppy-interfaces: E3SM Unified 1.10.0rc13 (zppy-interfaces v0.1.1)")
print("e3sm_diags: E3SM Unified 1.10.0rc13 (e3sm_diags v3.0.0rc4)")
print("Everything else: E3SM Unified 1.10.0rc13")
print("Results:")
print("Chrysalis: Test failures, but they were determined to be ok. Updated expected results. Comment: https://github.com/E3SM-Project/zppy/discussions/634#discussioncomment-12495202")
print("Compy: Test failures, but they were determined to be ok. Updated expected results. Comment: https://github.com/E3SM-Project/zppy/discussions/634#discussioncomment-12554970")
print("Perlmutter: Test failures, but they were determined to be ok. Updated expected results. Comment: https://github.com/E3SM-Project/zppy/discussions/634#discussioncomment-12554970")
print("####################################################################")
print("Then we added the following:")
print("zppy: https://github.com/E3SM-Project/zppy/pull/696, https://github.com/E3SM-Project/zppy/pull/697")
print("zppy-interfaces: https://github.com/E3SM-Project/zppy-interfaces/pull/16")
print("e3sm_diags: https://github.com/E3SM-Project/e3sm_diags/pull/953, https://github.com/E3SM-Project/e3sm_diags/pull/955, https://github.com/E3SM-Project/e3sm_diags/pull/958, https://github.com/E3SM-Project/e3sm_diags/pull/959, https://github.com/E3SM-Project/e3sm_diags/pull/925, https://github.com/E3SM-Project/e3sm_diags/pull/956 (freezing e3sm_diags at v3.0.0)")
print("2025-04-01")
print("Run 2: latest zppy, latest zppy-interfaces, e3sm_diags v3.0.0, Unified 1.11.0rc13 elsewhere")
print("zppy: zppy_dev_20250401")
print("zppy-interfaces: zi_dev_20250401")
print("e3sm_diags: e3sm_diags_dev_v3.0.0")
print("Everything else: E3SM Unified 1.10.0rc13")
print("Results:")
print("Chrysalis: 100 mismatched images introduced in e3sm_diags, 21 mismatched images introduced in global_time_series. Did NOT update expected results. Comment: https://github.com/E3SM-Project/zppy/discussions/634#discussioncomment-12694142")
print("Compy: 100 mismatched images introduced in e3sm_diags, 21 mismatched images introduced in global_time_series. Also: 1 missing v3 ILAMB file -- `ilamb/_1985-1988/HydrologyCycle/Permafrost/NSIDC/legend_bias.png` -- which is expected based on the removal of NSIDC in https://github.com/E3SM-Project/zppy/pull/696/files. Did NOT update expected results. Comment: https://github.com/E3SM-Project/zppy/discussions/634#discussioncomment-12705106")
print("Perlmutter: Was already on next round of testing by the time nodes were allocated. Didn't continue with this run. Did NOT update expected results. Comment: N/A")
chrysalis_run2 = Results("chrysalis", "run2")
chrysalis_run2.file_dict = {
"v2": {
"e3sm_diags": chrysalis_run2.read_files("v2", "e3sm_diags", "mismatched"),
"global_time_series": chrysalis_run2.read_files("v2", "global_time_series", "mismatched"),
},
"v3": {
"e3sm_diags": chrysalis_run2.read_files("v3", "e3sm_diags", "mismatched"),
"global_time_series": chrysalis_run2.read_files("v3", "global_time_series", "mismatched"),
},
"bundles": {
"global_time_series": chrysalis_run2.read_files("bundles", "global_time_series", "mismatched"),
}
}
assert len(chrysalis_run2.file_dict["v2"]["e3sm_diags"]) == 46
assert len(chrysalis_run2.file_dict["v2"]["global_time_series"]) == 5
assert len(chrysalis_run2.file_dict["v3"]["e3sm_diags"]) == 54
assert len(chrysalis_run2.file_dict["v3"]["global_time_series"]) == 14
assert len(chrysalis_run2.file_dict["bundles"]["global_time_series"]) == 2
compy_run2 = Results("compy", "run2")
compy_run2.file_dict = {
"v2": {
"e3sm_diags": compy_run2.read_files("v2", "e3sm_diags", "mismatched"),
"global_time_series": compy_run2.read_files("v2", "global_time_series", "mismatched"),
},
"v3": {
"e3sm_diags": compy_run2.read_files("v3", "e3sm_diags", "mismatched"),
"global_time_series": compy_run2.read_files("v3", "global_time_series", "mismatched"),
},
"bundles": {
"global_time_series": compy_run2.read_files("bundles", "global_time_series", "mismatched"),
}
}
assert len(compy_run2.file_dict["v2"]["e3sm_diags"]) == 46
assert len(compy_run2.file_dict["v2"]["global_time_series"]) == 5
assert len(compy_run2.file_dict["v3"]["e3sm_diags"]) == 54
assert len(compy_run2.file_dict["v3"]["global_time_series"]) == 14
assert len(compy_run2.file_dict["bundles"]["global_time_series"]) == 2
assert chrysalis_run2.file_dict["v2"]["e3sm_diags"] == compy_run2.file_dict["v2"]["e3sm_diags"]
assert chrysalis_run2.file_dict["v2"]["global_time_series"] == compy_run2.file_dict["v2"]["global_time_series"]
assert chrysalis_run2.file_dict["v3"]["e3sm_diags"] == compy_run2.file_dict["v3"]["e3sm_diags"]
assert chrysalis_run2.file_dict["v3"]["global_time_series"] == compy_run2.file_dict["v3"]["global_time_series"]
assert chrysalis_run2.file_dict["bundles"]["global_time_series"] == compy_run2.file_dict["bundles"]["global_time_series"]
print("Chrysalis run2 and Compy run2 had the exact same set of mismatched files.")
print("Compy has a descrepancy: Missing `ilamb/_1985-1988/HydrologyCycle/Permafrost/NSIDC/legend_bias.png`. However, this is expected based on the code changes.")
print("####################################################################")
print("Then we added the following:")
print("zppy: https://github.com/E3SM-Project/zppy/pull/699, https://github.com/E3SM-Project/zppy/pull/700, https://github.com/E3SM-Project/zppy/pull/702 + manaual fix to 702 in bash script")
print("zppy-interfaces: N/A")
print("e3sm_diags: N/A")
print("2025-04-03")
print("Run 3: NEW latest zppy, latest zppy-interfaces, e3sm_diags v3.0.0, Unified 1.11.0rc13 elsewhere")
print("zppy: zppy_dev_20250403")
print("zppy-interfaces: zi_dev_20250403")
print("e3sm_diags: e3sm_diags_dev_v3.0.0")
print("Everything else: E3SM Unified 1.10.0rc13")
print("Results:")
print("Chrysalis: No changes. Updated expected results. Comment: https://github.com/E3SM-Project/zppy/discussions/634#discussioncomment-12727813")
print("Compy: Didn't run")
print("Perlmutter: Didn't run")
chrysalis_run3 = Results("chrysalis", "run3")
chrysalis_run3.file_dict = {
"v2": {
"e3sm_diags": chrysalis_run3.read_files("v2", "e3sm_diags", "found_diffs"),
"global_time_series": chrysalis_run3.read_files("v2", "global_time_series", "found_diffs"),
},
"v3": {
"e3sm_diags": chrysalis_run3.read_files("v3", "e3sm_diags", "found_diffs"),
"global_time_series": chrysalis_run3.read_files("v3", "global_time_series", "found_diffs"),
},
"bundles": {
"global_time_series": chrysalis_run3.read_files("bundles", "global_time_series", "found_diffs"),
}
}
assert len(chrysalis_run3.file_dict["v2"]["e3sm_diags"]) == 46
assert len(chrysalis_run3.file_dict["v2"]["global_time_series"]) == 5
assert len(chrysalis_run3.file_dict["v3"]["e3sm_diags"]) == 54
assert len(chrysalis_run3.file_dict["v3"]["global_time_series"]) == 14
assert len(chrysalis_run3.file_dict["bundles"]["global_time_series"]) == 2
assert chrysalis_run3.file_dict["v2"]["e3sm_diags"] == chrysalis_run2.file_dict["v2"]["e3sm_diags"]
assert chrysalis_run3.file_dict["v2"]["global_time_series"] == chrysalis_run2.file_dict["v2"]["global_time_series"]
assert chrysalis_run3.file_dict["v3"]["e3sm_diags"] == chrysalis_run2.file_dict["v3"]["e3sm_diags"]
assert chrysalis_run3.file_dict["v3"]["global_time_series"] == chrysalis_run2.file_dict["v3"]["global_time_series"]
assert chrysalis_run3.file_dict["bundles"]["global_time_series"] == chrysalis_run2.file_dict["bundles"]["global_time_series"]
print("Chrysalis run2 and Chrysalis run3 had the exact same set of mismatched files.")
print("####################################################################")
print("Then we added the following:")
print("zppy: https://github.com/E3SM-Project/zppy/pull/694, https://github.com/E3SM-Project/zppy/pull/705, https://github.com/E3SM-Project/zppy/pull/701, https://github.com/E3SM-Project/zppy/pull/706, https://github.com/E3SM-Project/zppy/pull/707 (freezing zppy at v3.0.0)")
print("zppy-interfaces: https://github.com/E3SM-Project/zppy-interfaces/pull/24 (freezing zppy-interfaces at v0.1.2)")
print("e3sm_diags: N/A")
print("2025-04-15")
print("Run 4: E3SM Unified 1.11.1")
print("zppy: E3SM Unified 1.11.1 (zppy v3.0.0)")
print("zppy-interfaces: E3SM Unified 1.11.1 (zppy-interfaces v0.1.2)")
print("e3sm_diags: E3SM Unified 1.11.1 (e3sm_diags v3.0.0)")
print("Everything else: E3SM Unified 1.11.1")
print("Results:")
print("Chrysalis: No test failures. Made copy of expected results, but didn't update them (no update needed, because there were no diffs)")
print("Compy: 122 mismatched images introduced in e3sm_diags (100 => 222). All changes are due to HadISST. Did NOT YET update expected results.")
print("Perlmutter: 100 mismatched images introduced in e3sm_diags, 21 mismatched images introduced in global_time_series. Did NOT YET update expected results.")
compy_run4 = Results("compy", "run4")
compy_run4.file_dict = {
"v2": {
"e3sm_diags": compy_run4.read_files("v2", "e3sm_diags", "mismatched"),
"global_time_series": compy_run4.read_files("v2", "global_time_series", "mismatched"),
},
"v3": {
"e3sm_diags": compy_run4.read_files("v3", "e3sm_diags", "mismatched"),
"global_time_series": compy_run4.read_files("v3", "global_time_series", "mismatched"),
},
"bundles": {
"e3sm_diags": compy_run4.read_files("bundles", "e3sm_diags", "mismatched"),
"global_time_series": compy_run4.read_files("bundles", "global_time_series", "mismatched"),
}
}
assert len(compy_run4.file_dict["v2"]["e3sm_diags"]) == 77
assert len(compy_run4.file_dict["v2"]["global_time_series"]) == 5
assert len(compy_run4.file_dict["v3"]["e3sm_diags"]) == 85
assert len(compy_run4.file_dict["v3"]["global_time_series"]) == 14
assert len(compy_run4.file_dict["bundles"]["e3sm_diags"]) == 60
assert len(compy_run4.file_dict["bundles"]["global_time_series"]) == 2
print("Compy run4 has a discrepancy: 122 more diffs in e3sm_diags than found in compy run2.")
print("Both Compy run4 and Compy run2 have the missing ILAMB file: `ilamb/_1985-1988/HydrologyCycle/Permafrost/NSIDC/legend_bias.png`. Again, this is expected based on the code changes.")
assert compy_run4.file_dict["v2"]["global_time_series"] == compy_run2.file_dict["v2"]["global_time_series"]
assert compy_run4.file_dict["v3"]["global_time_series"] == compy_run2.file_dict["v3"]["global_time_series"]
assert compy_run4.file_dict["bundles"]["global_time_series"] == compy_run2.file_dict["bundles"]["global_time_series"]
left_only, right_only, in_both = filter_list(compy_run4.file_dict["v2"]["e3sm_diags"], compy_run2.file_dict["v2"]["e3sm_diags"])
assert len(right_only) == 0
assert len(in_both) == 46
assert len(left_only) == 31
print("These v2 e3sm_diags diffs were found in Compy run4 but not in Compy run2:")
for file in left_only:
print(file)
left_only, right_only, in_both = filter_list(compy_run4.file_dict["v3"]["e3sm_diags"], compy_run2.file_dict["v3"]["e3sm_diags"])
assert len(right_only) == 0
assert len(in_both) == 54
assert len(left_only) == 31
print("These v3 e3sm_diags diffs were found in Compy run4 but not in Compy run2:")
for file in left_only:
print(file)
assert len(compy_run4.file_dict["bundles"]["e3sm_diags"]) == 60
print("These bundles e3sm_diags diffs were found in Compy run4 but not in Compy run2:")
for file in compy_run4.file_dict["bundles"]["e3sm_diags"]:
print(file)
perlmutter_run4 = Results("perlmutter", "run4")
perlmutter_run4.file_dict = {
"v2": {
"e3sm_diags": perlmutter_run4.read_files("v2", "e3sm_diags", "mismatched"),
"global_time_series": perlmutter_run4.read_files("v2", "global_time_series", "mismatched"),
},
"v3": {
"e3sm_diags": perlmutter_run4.read_files("v3", "e3sm_diags", "mismatched"),
"global_time_series": perlmutter_run4.read_files("v3", "global_time_series", "mismatched"),
},
"bundles": {
"global_time_series": perlmutter_run4.read_files("bundles", "global_time_series", "mismatched"),
}
}
assert len(perlmutter_run4.file_dict["v2"]["e3sm_diags"]) == 46
assert len(perlmutter_run4.file_dict["v2"]["global_time_series"]) == 5
assert len(perlmutter_run4.file_dict["v3"]["e3sm_diags"]) == 54
assert len(perlmutter_run4.file_dict["v3"]["global_time_series"]) == 14
assert len(perlmutter_run4.file_dict["bundles"]["global_time_series"]) == 2
assert perlmutter_run4.file_dict["v2"]["e3sm_diags"] == chrysalis_run2.file_dict["v2"]["e3sm_diags"]
assert perlmutter_run4.file_dict["v2"]["global_time_series"] == chrysalis_run2.file_dict["v2"]["global_time_series"]
assert perlmutter_run4.file_dict["v3"]["e3sm_diags"] == chrysalis_run2.file_dict["v3"]["e3sm_diags"]
assert perlmutter_run4.file_dict["v3"]["global_time_series"] == chrysalis_run2.file_dict["v3"]["global_time_series"]
assert perlmutter_run4.file_dict["bundles"]["global_time_series"] == chrysalis_run2.file_dict["bundles"]["global_time_series"]
print("Chrysalis run2 and Perlmutter run4 had the exact same set of mismatched files.")
if __name__ == "__main__":
compare() |
Beta Was this translation helpful? Give feedback.
32 replies
-
|
I can do: cd /global/cfs/cdirs/e3sm/forsyth/. could you change group permission to e3sm for following? |
Beta Was this translation helpful? Give feedback.
-
|
Ok, looks updated now. |
Beta Was this translation helpful? Give feedback.
-
|
Could you try to recursively change all files and directories. I think it should be something like: |
Beta Was this translation helpful? Give feedback.
-
|
Running now; that's taking a little longer |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for sharing. I think I can confirm now, the issue came back, and there is another potential intermittent file-system specific issue with TC detection (which is probably not reproducible). I will file an zppy issue. |
Beta Was this translation helpful? Give feedback.
-
2025-06-13 threadUsing the commits of #720 to test the latest image checker functionality (diff bounding boxes on actual & expected, added legacy tests). In particular, we want to test the latest # 1. Set up environments
lcrc_conda # Function to set up conda locally
# 1a. e3sm_diags
cd ~/ez/e3sm_diags
git fetch upstream main
git status
# Check for uncommitted changes
git checkout main
git reset --hard upstream/main
git log
# Check that latest commit matches https://github.com/E3SM-Project/e3sm_diags/commits/main
conda clean --all --y
conda env create -f conda-env/dev.yml -n e3sm-diags-main-20250613
conda activate e3sm-diags-main-20250613
pip install .
# 1b. zppy-interfaces
# Latest commit on https://github.com/E3SM-Project/zppy-interfaces/commits/main
# is "Bump to 0.1.2", so we can just use Unified.
# 1c. zppy
# We're going to use the commits from #720 to try out new features of the image checker
# (diff bounding boxes on actual & expected, added legacy tests)
cd ~/ez/zppy
git status
# Check for uncommitted changes
git fetch upstream image-checker # branch for #720
git checkout -b test-weekly-20250613 upstream/image-checker
git log
# The 3 commits of the image-checker PR #720
# The provenance cfg PR #713
# (We saw in testing of the global_time_series split PR #722 that this will require updates to the expected results)
conda clean --all --y
conda env create -f conda/dev.yml -n zppy-test-weekly-20250613
conda activate zppy-test-weekly-20250613
pip install .
# 2. Run unit tests
pytest tests/test_*.py
# 25 passed, 1 warning in 0.30s
# /gpfs/fs1/home/ac.forsyth2/ez/zppy/zppy/utils.py:191: DeprecationWarning: invalid escape sequence \.
# 3. Set up zppy runs
# Edit tests/integration/utils.py:
# UNIQUE_ID = "test_weekly_20250613"
# "diags_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate e3sm-diags-main-20250613",
# "global_time_series_environment_commands": "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh",
python tests/integration/utils.py
# 4. Launch zppy jobs to produce actual images for image checker test
# 4a. Legacy tests
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg # Runs 1st part of bundles cfg
# 4b. Regular tests
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 1st part of bundles cfg
# TODO ########################################################################
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg # Runs 2nd part of bundles cfg
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 2nd part of bundles cfg
ls tests/integration/test_*.py # to see what tests are available to run
# 5. Run zppy integration tests
pytest tests/integration/test_bash_generation.py
pytest tests/integration/test_bundles.py
pytest tests/integration/test_campaign.py
pytest tests/integration/test_defaults.py
pytest tests/integration/test_last_year.py
pytest tests/integration/test_bundles.py
pytest tests/integration/test_images.py |
Beta Was this translation helpful? Give feedback.
7 replies
-
|
Thanks a lot! I will investigate the diffs before the release. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the testing results, @forsyth2! This missing enso_diags needs to be addressed. I filed an e3sm_diags issue E3SM-Project/e3sm_diags#990 which @tomvothecoder can help with a fix. We will hold the bug-fix release until this issue is addressed. |
Beta Was this translation helpful? Give feedback.
-
|
Sounds good. I notice on E3SM-Project/e3sm_diags#990 you wrote:
That's the legacy test, which I did not run the image checker on. I'm going to rerun the legacy tests today because I noticed on Friday I had gotten some paths/test names mixed. See above comment; the name I gave to the legacy tests has gone through several iterations and they're not all synced yet. For instance, a All of the missing/mismatched image lists I gave above are for the non-legacy tests. You should probably be paying more attention to |
Beta Was this translation helpful? Give feedback.
-
|
thanks. i saw the same enso_diags error, i will update the link.. |
Beta Was this translation helpful? Give feedback.
-
|
Ok that makes sense the error would appear in both, but yeah in terms of debugging, the non-legacy output will be better to go through. |
Beta Was this translation helpful? Give feedback.
-
2025-07-10 threadSetup# 1. Set up environments
lcrc_conda # Function to set up conda locally
# 1a. e3sm_diags
cd ~/ez/e3sm_diags
git fetch upstream main
git status
# Check for uncommitted changes
git checkout main
git reset --hard upstream/main
git log
# Check that latest commit matches https://github.com/E3SM-Project/e3sm_diags/commits/main
# From 7/9: Fix xcdat bounds dim issue with v0.9.1 (#992)
# Matches, good
conda clean --all --y
conda env create -f conda-env/dev.yml -n e3sm-diags-main-20250710
conda activate e3sm-diags-main-20250710
pip install .
# 1b. zppy-interfaces
cd ~/ez/zppy-interfaces
git fetch upstream main
git status
# Check for uncommitted changes
git checkout main
git reset --hard upstream/main
git log
# Check that latest commit matches https://github.com/E3SM-Project/zppy-interfaces/commits/main
# From 7/1: Update to latest conda-incubator/setup-miniconda
# Matches, good
conda clean --all --y
conda env create -f conda/dev.yml -n zi-main-20250710
conda activate zi-main-20250710
pip install .
pytest tests/unit/global_time_series/test_global_time_series.py
# ImportError: cannot import name 'construct_generic_variables' from 'zppy_interfaces.global_time_series.coupled_global' (/gpfs/fs1/home/ac.forsyth2/miniforge3/envs/zi-main-20250710/lib/python3.12/site-packages/zppy_interfaces/global_time_series/coupled_global/__init__.py)
# 1c. zppy
cd ~/ez/zppy
git status
# Check for uncommitted changes
git fetch upstream main
git checkout -b test-weekly-20250710 upstream/main
git log
# Check that latest commit matches https://github.com/E3SM-Project/zppy/commits/main
# From 6/18: Merge pull request #720 from E3SM-Project/image-checker
# Matches, good
conda clean --all --y
conda env create -f conda/dev.yml -n zppy-test-weekly-20250710
conda activate zppy-test-weekly-20250710
pip install .
# 2. Run unit tests
pytest tests/test_*.py
# 25 passed in 0.49s
# Because of the zppy-interfaces building issues, we'll just use Unified.
# There has only been one commit (only affecting GitHub workflows) on zppy-interfaces.
# Build issues: https://github.com/E3SM-Project/zppy-interfaces/issues/30#issuecomment-3058378312
# 3. Set up zppy runs
# Edit tests/integration/utils.py:
# UNIQUE_ID = "test_weekly_20250710"
# "diags_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate e3sm-diags-main-20250710",
# "global_time_series_environment_commands": "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh",
python tests/integration/utils.py
# 4. Launch zppy jobs to produce actual images for image checker test
# 4a. Legacy tests
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg # Runs 1st part of bundles cfg
# 4b. Regular tests
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 1st part of bundles cfg
# 4c. Run the second part of the bundles cfgs
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/test_weekly_20250710/v3.LR.historical_0051/post/scripts/ && grep -v "OK" *status
# No errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/test_weekly_20250710/v3.LR.historical_0051/post/scripts/ && grep -v "OK" *status
# No errors
cd ~/ez/zppy
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg # Runs 2nd part of bundles cfg
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 2nd part of bundles cfg
# 4d. Check output
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/test_weekly_20250710/v3.LR.historical_0051/post/scripts/ && grep -v "OK" *status
# No errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/test_weekly_20250710/v2.LR.historical_0201/post/scripts/ && grep -v "OK" *status
# No errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/test_weekly_20250710/v3.LR.historical_0051/post/scripts/ && grep -v "OK" *status
# No errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/test_weekly_20250710/v3.LR.historical_0051/post/scripts/ && grep -v "OK" *status
# No errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/test_weekly_20250710/v2.LR.historical_0201/post/scripts/ && grep -v "OK" *status
# No errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/test_weekly_20250710/v3.LR.historical_0051/post/scripts/ && grep -v "OK" *status
# No errors
# 5. Run zppy integration tests
cd ~/ez/zppy
ls tests/integration/test_*.py
# 5a. The simpler integration tests
# Just need to add provenance cfg to expected files.
# Actually, the provenance file is test-specific:
# provenance.20250617_005905_907190.cfg
# So, that will need to be accounted for in the expected file update...
pytest tests/integration/test_bash_generation.py
# 1 failed in 1.95s
pytest tests/integration/test_campaign.py
# 6 failed in 3.04s
pytest tests/integration/test_defaults.py
# 1 failed in 1.18s
pytest tests/integration/test_last_year.py
# 1 failed in 0.34s
# 5b. The integration tests that rely on the results from step 4
pytest tests/integration/test_bundles.py
# 2 passed in 0.17s
pytest tests/integration/test_images.py
# 1 failed in 3152.74s (0:52:32)
cat test_images_summary.mdSummary of test resultsDiff subdir is where to find the lists of missing/mismatched images, the image diff grid, and the individual diffs.
AnalysisI didn't update the expected results after the last run, probably because we determined ls -l /lcrc/group/e3sm/public_html/zppy_test_resources
# The legacy expected files were updated on June 5, because that was me adding them in the first place
# Everything else was last updated on April 4, around the time of the Unified release
ls -lt /lcrc/group/e3sm/public_html/zppy_test_resources_previous/
# Last update was "expected_results_for_unified_1.11.1" on 4/15
# Then, "expected_results_until_20250403" on 4/4So, diff comparison from the last run:
Diff of the diffscd /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/
diff zppy_weekly_comprehensive_v2_www/test_weekly_20250613/v2.LR.historical_0201/image_check_failures_comprehensive_v2/e3sm_diags/mismatched_images.txt zppy_weekly_comprehensive_v2_www/test_weekly_20250710/v2.LR.historical_0201/image_check_failures_comprehensive_v2/e3sm_diags/mismatched_images.txt diff zppy_weekly_comprehensive_v3_www/test_weekly_20250613/v3.LR.historical_0051/image_check_failures_comprehensive_v3/e3sm_diags/mismatched_images.txt zppy_weekly_comprehensive_v3_www/test_weekly_20250710/v3.LR.historical_0051/image_check_failures_comprehensive_v3/e3sm_diags/mismatched_images.txtdiff zppy_weekly_bundles_www/test_weekly_20250613/v3.LR.historical_0051/image_check_failures_bundles/e3sm_diags/mismatched_images.txt zppy_weekly_bundles_www/test_weekly_20250710/v3.LR.historical_0051/image_check_failures_bundles/e3sm_diags/mismatched_images.txt@chengzhuzhang It looks like the missing images issue has been resolved. The number of mismatches has increased though, all attributable to |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
Updated expected results since cd /lcrc/group/e3sm/public_html
cp -r zppy_test_resources zppy_test_resources_previous/expected_results_until_20250710
cd ~/ez/zppy
lcrc_conda
conda activate zppy-test-weekly-20250710
./tests/integration/generated/update_bash_generation_expected_files_chrysalis.sh
# Still fails, see https://github.com/E3SM-Project/zppy/issues/728
# `zppy -c tests/integration/test_bash_generation.cfg` is going to create a new provenance each time!
./tests/integration/generated/update_campaign_expected_files_chrysalis.sh
# Same as above
./tests/integration/generated/update_defaults_expected_files_chrysalis.sh
# Same as above
./tests/integration/generated/update_weekly_expected_files_chrysalis.sh
# Rerun these tests manually:
pytest tests/integration/test_bundles.py
# 2 passed in 0.73s
pytest tests/integration/test_images.py
# 17:13
# 1 passed in 2822.21s (0:47:02) |
Beta Was this translation helpful? Give feedback.
-
|
@forsyth2 thanks Ryan! I think the results are as expected. ENSO figures are no longer missing. And the difference is just because the position change of labels needed for proper layout of pdf to be saved. |
Beta Was this translation helpful? Give feedback.
0 replies
-
2025-08-22 threadSetup# 1. Set up environments
lcrc_conda # Function to set up conda locally
# 1a. e3sm_diags
cd ~/ez/e3sm_diags
git fetch upstream main
git status
# Check for uncommitted changes
git checkout main
git reset --hard upstream/main
git log
# Check that latest commit matches https://github.com/E3SM-Project/e3sm_diags/commits/main
# From 7/15: Bump to 3.0.1 (#993)
# Matches, good
conda clean --all --y
conda env create -f conda-env/dev.yml -n e3sm-diags-main-20250822
conda activate e3sm-diags-main-20250822
python -m pip install .
# 1b. zppy-interfaces
cd ~/ez/zppy-interfaces
git fetch upstream main
git status
# Check for uncommitted changes
git checkout main
git reset --hard upstream/main
git log
# Check that latest commit matches https://github.com/E3SM-Project/zppy-interfaces/commits/main
# From 8/11: Add missing packages to dependencies (#32)
# Matches, good
conda clean --all --y
conda env create -f conda/dev.yml -n zi-main-20250822
conda activate zi-main-20250822
python -m pip install .
pytest tests/unit/global_time_series/test_global_time_series.py
# 10 passed in 26.53s
# 1c. zppy
cd ~/ez/zppy
git status
# Check for uncommitted changes
git fetch upstream main
git checkout -b test-weekly-20250822 upstream/main
git log
# Check that latest commit matches https://github.com/E3SM-Project/zppy/commits/main
# From 8/18: fix e3sm_diags year looping for land model vs model (#731)
# Matches, good
conda clean --all --y
conda env create -f conda/dev.yml -n zppy-test-weekly-20250822
conda activate zppy-test-weekly-20250822
python -m pip install .
# 2. Run unit tests
pytest tests/test_*.py
# 25 passed, 1 warning in 0.42s
# /gpfs/fs1/home/ac.forsyth2/ez/zppy/zppy/utils.py:191: DeprecationWarning: invalid escape sequence \.
# 3. Set up zppy runs
# Edit tests/integration/utils.py:
# UNIQUE_ID = "test_weekly_20250822"
# "diags_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate e3sm-diags-main-20250822",
# "global_time_series_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate zi-main-20250822",
python tests/integration/utils.py
# 4. Launch zppy jobs to produce actual images for image checker test
# 4a. Legacy tests
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg # Runs 1st part of bundles cfg
# 4b. Regular tests
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 1st part of bundles cfg
# 4c. Run the second part of the bundles cfgs
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/test_weekly_20250822/v3.LR.historical_0051/post/scripts/ && grep -v "OK" *status
# No errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/test_weekly_20250822/v3.LR.historical_0051/post/scripts/ && grep -v "OK" *status
# No errors
cd ~/ez/zppy
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg # Runs 2nd part of bundles cfg
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 2nd part of bundles cfg
# 4d. Check output
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/test_weekly_20250822/v3.LR.historical_0051/post/scripts/ && grep -v "OK" *status
# No errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/test_weekly_20250822/v2.LR.historical_0201/post/scripts/ && grep -v "OK" *status
# No errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/test_weekly_20250822/v3.LR.historical_0051/post/scripts/ && grep -v "OK" *status
# No errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/test_weekly_20250822/v3.LR.historical_0051/post/scripts/ && grep -v "OK" *status
# No errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/test_weekly_20250822/v2.LR.historical_0201/post/scripts/ && grep -v "OK" *status
# No errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/test_weekly_20250822/v3.LR.historical_0051/post/scripts/ && grep -v "OK" *status
# No errors
# 5. Run zppy integration tests
cd ~/ez/zppy
ls tests/integration/test_*.py
# 5a. The simpler integration tests
# Just need to add provenance cfg to expected files.
# Actually, the provenance file is test-specific:
# provenance.20250617_005905_907190.cfg
# So, that will need to be accounted for in the expected file update...
pytest tests/integration/test_bash_generation.py
# 1 failed
# Other failures appear expected based on changes in #723
pytest tests/integration/test_campaign.py
# 6 failed
pytest tests/integration/test_defaults.py
# 1 failed
pytest tests/integration/test_last_year.py
# 1 failed
# 5b. The integration tests that rely on the results from step 4
pytest tests/integration/test_bundles.py
# 2 passed
pytest tests/integration/test_images.py
# 1 failed in 2697.61s (0:44:57)
cat test_images_summary.mdTable of test resultsDiff subdir is where to find the lists of missing/mismatched images, the image diff grid, and the individual diffs.
To get the web path, replace with For the non-legacy The diffs in
and the zppy PRs merged after 7/10:
@chengzhuzhang From a cursory look, the diurnal plots no longer have missing coloring at top/bottom, the Taylor Diagrams have lost data, the TC analysis plots have turned to a green background, the polar plots have some noticeable diffs in the diff plots. And as usual, some diffs are the annoyance of matplotlib plotting things just different enough for the test to pick it up. Given the above, I think we'll want to dive deeper into the diffs before updating the expected test results. |
Beta Was this translation helpful? Give feedback.
3 replies
-
The same, but running only with zppy changesRunning weekly tests on Chrysalis, but with last-tested diags, to see if changes are due to zppy or e3sm_diags. Setup# 1. Set up environments
lcrc_conda # Function to set up conda locally
# 1a. e3sm_diags
cd ~/ez/e3sm_diags
git fetch upstream main
git status
# Check for uncommitted changes
git checkout -b test-up-to-07-10 821c84e108f01a3d6c0951ba8cf4018ecb06231b
git log
# From 7/9: Fix xcdat bounds dim issue with v0.9.1 (#992)
conda clean --all --y
conda env create -f conda-env/dev.yml -n e3sm-diags-main-revert-20250710
conda activate e3sm-diags-main-revert-20250710
python -m pip install .
# 1b. zppy-interfaces
# We'll use the same environment as before.
# 1c. zppy
cd ~/ez/zppy
git status
# Check for uncommitted changes
git fetch upstream main
git checkout -b test-weekly-20250825 upstream/main
git log
# Check that latest commit matches https://github.com/E3SM-Project/zppy/commits/main
# From 8/18: fix e3sm_diags year looping for land model vs model (#731)
# Matches, good
# We'll use the same environment as before
conda activate zppy-test-weekly-20250822
python -m pip install .
# 2. Run unit tests
pytest tests/test_*.py
# 25 passed in 0.46s
# 3. Set up zppy runs
# Edit tests/integration/utils.py:
# UNIQUE_ID = "test_weekly_20250825"
# "diags_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate e3sm-diags-main-revert-20250710",
# "global_time_series_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate zi-main-20250822",
python tests/integration/utils.py
# 4. Launch zppy jobs to produce actual images for image checker test
# 4a. Legacy tests
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg # Runs 1st part of bundles cfg
# 4b. Regular tests
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 1st part of bundles cfg
# Pick back up 8/26
cd ~/ez/zppy
lcrc_conda
conda activate zppy-test-weekly-20250822
python -m pip install .
# 4c. Run the second part of the bundles cfgs
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/test_weekly_20250825/v3.LR.historical_0051/post/scripts/ && grep -v "OK" *status
# No errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/test_weekly_20250825/v3.LR.historical_0051/post/scripts/ && grep -v "OK" *status
# No errors
cd ~/ez/zppy
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg # Runs 2nd part of bundles cfg
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 2nd part of bundles cfg
# 4d. Check output
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/test_weekly_20250825/v3.LR.historical_0051/post/scripts/ && grep -v "OK" *status
# No errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/test_weekly_20250825/v2.LR.historical_0201/post/scripts/ && grep -v "OK" *status
# No errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/test_weekly_20250825/v3.LR.historical_0051/post/scripts/ && grep -v "OK" *status
# No errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/test_weekly_20250825/v3.LR.historical_0051/post/scripts/ && grep -v "OK" *status
# No errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/test_weekly_20250825/v2.LR.historical_0201/post/scripts/ && grep -v "OK" *status
# No errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/test_weekly_20250825/v3.LR.historical_0051/post/scripts/ && grep -v "OK" *status
# No errors
# 5. Run zppy integration tests
cd ~/ez/zppy
ls tests/integration/test_*.py
# 5a. The simpler integration tests
# Skip these
# 5b. The integration tests that rely on the results from step 4
pytest tests/integration/test_bundles.py
# 2 passed
pytest tests/integration/test_images.py
# Started at 2025-08-26 12:20
# 1 failed in 2763.00s (0:46:02)
cat test_images_summary.mdTable of test resultsDiff subdir is where to find the lists of missing/mismatched images, the image diff grid, and the individual diffs.
To get the web path, replace with For the non-legacy The diffs in
The markdown table is identical to the run above, so the changes must be due to the recent |
Beta Was this translation helpful? Give feedback.
-
|
@forsyth2 thanks for conducting the weekly testing. It looks like there might be a bug in e3sm_diags vertical regridding routine that resulted all 3D variable related figures went missing. I'm filing an issue in e3sm_diags for troubleshooting. |
Beta Was this translation helpful? Give feedback.
-
|
The color change of tc track plot is strange, which I need to take a deeper look. |
Beta Was this translation helpful? Give feedback.
-
2025-09-19 ThreadTesting processExpandlcrc_conda
cd ~/ez/e3sm_diags
git status
# Check for uncommitted changes
git fetch upstream main
git checkout main
git reset --hard upstream/main
git log
# Good, matches https://github.com/E3SM-Project/e3sm_diags/commits/main
rm -rf build
conda clean --all --y
conda env create -f conda-env/dev.yml -n e3sm-diags-main-20250919
conda activate e3sm-diags-main-20250919
python -m pip install .
cd ~/ez/zppy-interfaces
git status
# Check for uncommitted changes
git fetch upstream main
git checkout main
git reset --hard upstream/main
git log
# Good, matches https://github.com/E3SM-Project/zppy-interfaces/commits/main
rm -rf build
time conda clean --all --y
# Will remove 19 (1.5 MB) tarball(s).
# Will remove 1 index cache(s).
# Will remove 17 (4.7 MB) package(s).
#
# Hanging -- this command isn't strictly necessary, so let's just skip it.
# CTRL C
# real 28m13.056s
time conda env create -f conda/dev.yml -n zi-main-20250919 # real 31m10.990s
conda activate zi-main-20250919
time python -m pip install . # real 0m32.155s
pytest tests/unit/global_time_series/test_*.py
# 10 passed in 55.64s
cd ~/ez/zppy
git status
# Check for uncommitted changes
git fetch upstream main
git checkout main
git reset --hard upstream/main
git log
# Good, matches https://github.com/E3SM-Project/zppy/commits/main
rm -rf build
time conda clean --all --y
# CTRL C
# real 13m11.526s
time conda env create -f conda/dev.yml -n zppy-main-20250919 # real 18m26.351s
conda activate zppy-main-20250919
pytest tests/test_*.py
# 25 passed, 1 warning in 3.51s
# (warning is ok, and has appeared on `main` anyway)
# Fix that warning
time python -m pip install .
pytest tests/test_*.py
# 25 passed in 0.51s
# Add the tasks_to_run_copy fix to test_images
time python -m pip install .
pre-commit run --all-files
pytest tests/test_*.py
# 25 passed in 0.43s
# Edit tests/integration/utils.py:
# TEST_SPECIFICS: Dict[str, Any] = {
# "diags_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate e3sm-diags-main-20250919",
# "global_time_series_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate zi-main-20250919",
# "cfgs_to_run": [
# "weekly_bundles",
# "weekly_comprehensive_v2",
# "weekly_comprehensive_v3",
# "weekly_legacy_3.0.0_bundles",
# "weekly_legacy_3.0.0_comprehensive_v2",
# "weekly_legacy_3.0.0_comprehensive_v3",
# ],
# "tasks_to_run": ["e3sm_diags", "mpas_analysis", "global_time_series", "ilamb"],
# "unique_id": "weekly_test_20250919",
# }
python tests/integration/utils.py
pre-commit run --all-files
time python -m pip install .
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v3_chrysalis.cfg
# Check on bundles status
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/weekly_test_20250919/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/weekly_test_20250919/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
# Now, run bundles part 2
cd ~/ez/zppy
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg
# Review finished runs
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/weekly_test_20250919/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# global_time_series_1980-1990.status:WAITING 905818
# mpas_analysis_ts_1980-1984_climo_1980-1984.status:ERROR (1)
# mpas_analysis_ts_1980-1990_climo_1985-1990.status:WAITING 905817
emacs mpas_analysis_ts_1980-1984_climo_1980-1984.o905816
# There were errors in task climatologyMapMassFluxes: plot_seaIceFreshWaterFlux_ANN_arctic
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/weekly_test_20250919/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/weekly_test_20250919/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# global_time_series_1980-1990.status:WAITING 905881
# mpas_analysis_ts_1980-1984_climo_1980-1984.status:ERROR (1)
# mpas_analysis_ts_1980-1990_climo_1985-1990.status:WAITING 905880
emacs mpas_analysis_ts_1980-1984_climo_1980-1984.o905879
# ERROR in task climatologyMapSeaIceConcNH: plotJFM_arctic_extended_Bootstrap. See log file /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comp\
# rehensive_v2_output/weekly_test_20250919/v2.LR.historical_0201/post/scripts/../analysis/mpas_analysis/ts_1980-1984_climo_1980-1984/logs/climatologyMa\
# pSeaIceConcNH_plotJFM_arctic_extended_Bootstrap.log for details
#
# There were errors in 5 tasks: climatologyMapMassFluxes: plot_evaporationFlux_ANN_antarctic, climatologyMapArgoTemperature: plotANN_latlon_depth_-150,\
# climatologyMapArgoTemperature: plotJAS_latlon_depth_-150, climatologyMapArgoSalinity: plotANN_latlon_depth_-400, climatologyMapSeaIceConcNH: plotJFM\
# _arctic_extended_Bootstrap
# See log files in /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/weekly_test_20250919/v2.LR.historical_0201/post/script\
# s/../analysis/mpas_analysis/ts_1980-1984_climo_1980-1984/logs for details.
# The following commands may be helpful:
# cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/weekly_test_20250919/v2.LR.historical_0201/post/scripts/../analysi\
# s/mpas_analysis/ts_1980-1984_climo_1980-1984/logs
# grep Error *.log
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/weekly_test_20250919/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# global_time_series_1985-1995.status:WAITING 905911
# mpas_analysis_ts_1985-1989_climo_1985-1989.status:ERROR (1)
# mpas_analysis_ts_1985-1995_climo_1990-1995.status:WAITING 905910
emacs mpas_analysis_ts_1985-1989_climo_1985-1989.o905909
# ERROR in task climatologyMapSeaIceConcSH: plotJJA_antarctic_extended_Bootstrap. See log file /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_c\
# omprehensive_v3_output/weekly_test_20250919/v3.LR.historical_0051/post/scripts/../analysis/mpas_analysis/ts_1985-1989_climo_1985-1989/logs/climatolog\
# yMapSeaIceConcSH_plotJJA_antarctic_extended_Bootstrap.log for details
#
# There were errors in 14 tasks: climatologyMapMassFluxes: plot_evaporationFlux_JFM_latlon, climatologyMapVel: plotVelocityMagnitude_ANN_latlon_depth_-\
# 800, climatologyMapVel: plotVelocityMagnitude_ANN_latlon_depth_-2000, climatologyMapWoa: plotTemperature_JFM_latlon_depth_-1000, climatologyMapArgoTe\
# mperature: plotANN_latlon_depth_-50, climatologyMapArgoTemperature: plotJFM_latlon_depth_-100, climatologyMapArgoSalinity: plotANN_latlon_depth_-25, \
# climatologyMapArgoSalinity: plotJFM_latlon_depth_-200, climatologyMapArgoSalinity: plotJAS_latlon_depth_-150, climatologyMapArgoSalinity: plotJAS_lat\
# lon_depth_-200, climatologyMapArgoSalinity: plotJAS_latlon_depth_-600, climatologyMapMLD: plotJFM_latlon, osnapTransects: plotANN_OSNAP_West_salinity\
# , climatologyMapSeaIceConcSH: plotJJA_antarctic_extended_Bootstrap
# See log files in /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/weekly_test_20250919/v3.LR.historical_0051/post/script\
# s/../analysis/mpas_analysis/ts_1985-1989_climo_1985-1989/logs for details.
# The following commands may be helpful:
# cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/weekly_test_20250919/v3.LR.historical_0051/post/scripts/../analysi\
# s/mpas_analysis/ts_1985-1989_climo_1985-1989/logs
# grep Error *.log
# The 6 WAITING jobs account for the 6 jobs remaining on `squeue`; cancel those
# Review the completed bundles jobs
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/weekly_test_20250919/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/weekly_test_20250919/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
# Let's proceed with the test suite to see if any more errors exist.
cd ~/ez/zppy
ls tests/integration/test_*.py
pytest tests/integration/test_bash_generation.py
# 1 failed in 2.05s, diffs appear to be expected based on recent changes on `main`
pytest tests/integration/test_campaign.py
# 6 failed in 6.72s, diffs appear to be expected based on recent changes on `main`
pytest tests/integration/test_defaults.py
# 1 failed in 5.67s, diffs appear to be expected based on recent changes on `main`
pytest tests/integration/test_last_year.py
# 1 failed in 1.68s, diff is because the prov file is always unique
pytest tests/integration/test_bundles.py
# 2 passed in 2.72s
pytest tests/integration/test_images.py
# Copy the output of test_images_summary.md to a Pull Request comment
# ============================================================== short test summary info ===============================================================
# FAILED tests/integration/test_images.py::test_images - assert 3031 == 2949
# =========================================================== 1 failed in 2671.08s (0:44:31) ===========================================================
cat test_images_summary.mdComplete testing summaryExpandDiff subdir is where to find the lists of missing/mismatched images, the image diff grid, and the individual diffs.
Concise testing summary
Preliminary analysis
Initial debugging of `mpas_analysis` logsgrep -in error /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/weekly_test_20250919/v2.LR.historical_0201/post/scripts/mpas_analysis_ts_1980-1984_climo_1980-1984.o905816
# 102:ERROR in task climatologyMapMassFluxes: plot_seaIceFreshWaterFlux_ANN_arctic. See log file /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/weekly_test_20250919/v2.LR.historical_0201/post/scripts/../analysis/mpas_analysis/ts_1980-1984_climo_1980-1984/logs/climatologyMapMassFluxes_plot_seaIceFreshWaterFlux_ANN_arctic.log for details
# 481:There were errors in task climatologyMapMassFluxes: plot_seaIceFreshWaterFlux_ANN_arctic
grep -in error /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/weekly_test_20250919/v2.LR.historical_0201/post/scripts/../analysis/mpas_analysis/ts_1980-1984_climo_1980-1984/logs/climatologyMapMassFluxes_plot_seaIceFreshWaterFlux_ANN_arctic.log
# 30: raise ValueError(
# 31:ValueError: x and y arguments to pcolormesh cannot have non-finite values or be of type numpy.ma.MaskedArray with masked values
grep -in error /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/weekly_test_20250919/v2.LR.historical_0201/post/scripts/mpas_analysis_ts_1980-1984_climo_1980-1984.o905879
# 101:ERROR in task climatologyMapMassFluxes: plot_evaporationFlux_ANN_antarctic. See log file /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/weekly_test_20250919/v2.LR.historical_0201/post/scripts/../analysis/mpas_analysis/ts_1980-1984_climo_1980-1984/logs/climatologyMapMassFluxes_plot_evaporationFlux_ANN_antarctic.log for details
# 234:ERROR in task climatologyMapArgoTemperature: plotANN_latlon_depth_-150. See log file /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/weekly_test_20250919/v2.LR.historical_0201/post/scripts/../analysis/mpas_analysis/ts_1980-1984_climo_1980-1984/logs/climatologyMapArgoTemperature_plotANN_latlon_depth_-150.log for details
# 241:ERROR in task climatologyMapArgoTemperature: plotJAS_latlon_depth_-150. See log file /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/weekly_test_20250919/v2.LR.historical_0201/post/scripts/../analysis/mpas_analysis/ts_1980-1984_climo_1980-1984/logs/climatologyMapArgoTemperature_plotJAS_latlon_depth_-150.log for details
# 255:ERROR in task climatologyMapArgoSalinity: plotANN_latlon_depth_-400. See log file /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/weekly_test_20250919/v2.LR.historical_0201/post/scripts/../analysis/mpas_analysis/ts_1980-1984_climo_1980-1984/logs/climatologyMapArgoSalinity_plotANN_latlon_depth_-400.log for details
# 541:ERROR in task climatologyMapSeaIceConcNH: plotJFM_arctic_extended_Bootstrap. See log file /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/weekly_test_20250919/v2.LR.historical_0201/post/scripts/../analysis/mpas_analysis/ts_1980-1984_climo_1980-1984/logs/climatologyMapSeaIceConcNH_plotJFM_arctic_extended_Bootstrap.log for details
# 543:There were errors in 5 tasks: climatologyMapMassFluxes: plot_evaporationFlux_ANN_antarctic, climatologyMapArgoTemperature: plotANN_latlon_depth_-150, climatologyMapArgoTemperature: plotJAS_latlon_depth_-150, climatologyMapArgoSalinity: plotANN_latlon_depth_-400, climatologyMapSeaIceConcNH: plotJFM_arctic_extended_Bootstrap
# 547: grep Error *.log
grep -in error /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/weekly_test_20250919/v2.LR.historical_0201/post/scripts/../analysis/mpas_analysis/ts_1980-1984_climo_1980-1984/logs/climatologyMapMassFluxes_plot_evaporationFlux_ANN_antarctic.log
# 24: raise ValueError(
# 25:ValueError: x and y arguments to pcolormesh cannot have non-finite values or be of type numpy.ma.MaskedArray with masked values
grep -in error /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/weekly_test_20250919/v2.LR.historical_0201/post/scripts/../analysis/mpas_analysis/ts_1980-1984_climo_1980-1984/logs/climatologyMapArgoTemperature_plotANN_latlon_depth_-150.log
# 24: raise ValueError(
# 25:ValueError: x and y arguments to pcolormesh cannot have non-finite values or be of type numpy.ma.MaskedArray with masked values
grep -in error /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/weekly_test_20250919/v2.LR.historical_0201/post/scripts/../analysis/mpas_analysis/ts_1980-1984_climo_1980-1984/logs/climatologyMapArgoTemperature_plotJAS_latlon_depth_-150.log
# 24: raise ValueError(
# 25:ValueError: x and y arguments to pcolormesh cannot have non-finite values or be of type numpy.ma.MaskedArray with masked values
grep -in error /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/weekly_test_20250919/v2.LR.historical_0201/post/scripts/../analysis/mpas_analysis/ts_1980-1984_climo_1980-1984/logs/climatologyMapArgoSalinity_plotANN_latlon_depth_-400.log
# 24: raise ValueError(
# 25:ValueError: x and y arguments to pcolormesh cannot have non-finite values or be of type numpy.ma.MaskedArray with masked values
grep -in error /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/weekly_test_20250919/v2.LR.historical_0201/post/scripts/../analysis/mpas_analysis/ts_1980-1984_climo_1980-1984/logs/climatologyMapSeaIceConcNH_plotJFM_arctic_extended_Bootstrap.log
# 31:MemoryError: std::bad_alloc
grep -in error /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/weekly_test_20250919/v3.LR.historical_0051/post/scripts/mpas_analysis_ts_1985-1989_climo_1985-1989.o905909
# 103:ERROR in task climatologyMapMassFluxes: plot_evaporationFlux_JFM_latlon. See log file /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/weekly_test_20250919/v3.LR.historical_0051/post/scripts/../analysis/mpas_analysis/ts_1985-1989_climo_1985-1989/logs/climatologyMapMassFluxes_plot_evaporationFlux_JFM_latlon.log for details
# 230:ERROR in task climatologyMapVel: plotVelocityMagnitude_ANN_latlon_depth_-800. See log file /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/weekly_test_20250919/v3.LR.historical_0051/post/scripts/../analysis/mpas_analysis/ts_1985-1989_climo_1985-1989/logs/climatologyMapVel_plotVelocityMagnitude_ANN_latlon_depth_-800.log for details
# 234:ERROR in task climatologyMapVel: plotVelocityMagnitude_ANN_latlon_depth_-2000. See log file /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/weekly_test_20250919/v3.LR.historical_0051/post/scripts/../analysis/mpas_analysis/ts_1985-1989_climo_1985-1989/logs/climatologyMapVel_plotVelocityMagnitude_ANN_latlon_depth_-2000.log for details
# 266:ERROR in task climatologyMapWoa: plotTemperature_JFM_latlon_depth_-1000. See log file /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/weekly_test_20250919/v3.LR.historical_0051/post/scripts/../analysis/mpas_analysis/ts_1985-1989_climo_1985-1989/logs/climatologyMapWoa_plotTemperature_JFM_latlon_depth_-1000.log for details
# 280:ERROR in task climatologyMapArgoTemperature: plotANN_latlon_depth_-50. See log file /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/weekly_test_20250919/v3.LR.historical_0051/post/scripts/../analysis/mpas_analysis/ts_1985-1989_climo_1985-1989/logs/climatologyMapArgoTemperature_plotANN_latlon_depth_-50.log for details
# 301:ERROR in task climatologyMapArgoTemperature: plotJFM_latlon_depth_-100. See log file /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/weekly_test_20250919/v3.LR.historical_0051/post/scripts/../analysis/mpas_analysis/ts_1985-1989_climo_1985-1989/logs/climatologyMapArgoTemperature_plotJFM_latlon_depth_-100.log for details
# 308:ERROR in task climatologyMapArgoSalinity: plotANN_latlon_depth_-25. See log file /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/weekly_test_20250919/v3.LR.historical_0051/post/scripts/../analysis/mpas_analysis/ts_1985-1989_climo_1985-1989/logs/climatologyMapArgoSalinity_plotANN_latlon_depth_-25.log for details
# 323:ERROR in task climatologyMapArgoSalinity: plotJFM_latlon_depth_-200. See log file /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/weekly_test_20250919/v3.LR.historical_0051/post/scripts/../analysis/mpas_analysis/ts_1985-1989_climo_1985-1989/logs/climatologyMapArgoSalinity_plotJFM_latlon_depth_-200.log for details
# 333:ERROR in task climatologyMapArgoSalinity: plotJAS_latlon_depth_-150. See log file /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/weekly_test_20250919/v3.LR.historical_0051/post/scripts/../analysis/mpas_analysis/ts_1985-1989_climo_1985-1989/logs/climatologyMapArgoSalinity_plotJAS_latlon_depth_-150.log for details
# 335:ERROR in task climatologyMapArgoSalinity: plotJAS_latlon_depth_-200. See log file /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/weekly_test_20250919/v3.LR.historical_0051/post/scripts/../analysis/mpas_analysis/ts_1985-1989_climo_1985-1989/logs/climatologyMapArgoSalinity_plotJAS_latlon_depth_-200.log for details
# 338:ERROR in task climatologyMapArgoSalinity: plotJAS_latlon_depth_-600. See log file /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/weekly_test_20250919/v3.LR.historical_0051/post/scripts/../analysis/mpas_analysis/ts_1985-1989_climo_1985-1989/logs/climatologyMapArgoSalinity_plotJAS_latlon_depth_-600.log for details
# 339:ERROR in task climatologyMapMLD: plotJFM_latlon. See log file /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/weekly_test_20250919/v3.LR.historical_0051/post/scripts/../analysis/mpas_analysis/ts_1985-1989_climo_1985-1989/logs/climatologyMapMLD_plotJFM_latlon.log for details
# 399:ERROR in task osnapTransects: plotANN_OSNAP_West_salinity. See log file /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/weekly_test_20250919/v3.LR.historical_0051/post/scripts/../analysis/mpas_analysis/ts_1985-1989_climo_1985-1989/logs/osnapTransects_plotANN_OSNAP_West_salinity.log for details
# 582:ERROR in task climatologyMapSeaIceConcSH: plotJJA_antarctic_extended_Bootstrap. See log file /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/weekly_test_20250919/v3.LR.historical_0051/post/scripts/../analysis/mpas_analysis/ts_1985-1989_climo_1985-1989/logs/climatologyMapSeaIceConcSH_plotJJA_antarctic_extended_Bootstrap.log for details
# 584:There were errors in 14 tasks: climatologyMapMassFluxes: plot_evaporationFlux_JFM_latlon, climatologyMapVel: plotVelocityMagnitude_ANN_latlon_depth_-800, climatologyMapVel: plotVelocityMagnitude_ANN_latlon_depth_-2000, climatologyMapWoa: plotTemperature_JFM_latlon_depth_-1000, climatologyMapArgoTemperature: plotANN_latlon_depth_-50, climatologyMapArgoTemperature: plotJFM_latlon_depth_-100, climatologyMapArgoSalinity: plotANN_latlon_depth_-25, climatologyMapArgoSalinity: plotJFM_latlon_depth_-200, climatologyMapArgoSalinity: plotJAS_latlon_depth_-150, climatologyMapArgoSalinity: plotJAS_latlon_depth_-200, climatologyMapArgoSalinity: plotJAS_latlon_depth_-600, climatologyMapMLD: plotJFM_latlon, osnapTransects: plotANN_OSNAP_West_salinity, climatologyMapSeaIceConcSH: plotJJA_antarctic_extended_Bootstrap
# 588: grep Error *.log
# That's a lot to check; presumably we're running into the same issues as above.Minor fixesAs part of this round of testing, I do have a commit with some fixes that I'd like to merge: #736 Details# Edit tests/integration/utils.py
# Reset TEST_SPECIFICS
python tests/integration/utils.py
pre-commit run --all-files
# Looks like we're still on the `main` branch, so let's checkout a new one.
git checkout -b test-fixes
git add -A
git commit -m "Unit and integration test fixes"
git push upstream test-fixes
# Push commit for test fixes
# Make PR: https://github.com/E3SM-Project/zppy/pull/736 |
Beta Was this translation helpful? Give feedback.
1 reply
-
@chengzhuzhang I'm not sure who the current
@chengzhuzhang Let me know if I can merge this small PR. |
Beta Was this translation helpful? Give feedback.
-
|
@forsyth2 thanks for the testing results. @xylar could you help invest the mpas_analysis task? |
Beta Was this translation helpful? Give feedback.
2 replies
-
|
Hey @chengzhuzhang, Xylar's on vacation this week so I looked into the I was unable to reproduce them in the latest Since we were unable to reproduce the errors, I think we decided to wait until the next run of the full test suite to see if they happen again. If you want me to keep looking into this further before then, I'd be happy to. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for update here! @andrewdnolan yeah, we can just have an eye out to see if it re-occurs, I'm going to do the same with a couple of e3sm_diags diffs. |
Beta Was this translation helpful? Give feedback.
-
2025-09-26 ThreadTesting processExpandlcrc_conda
cd ~/ez/e3sm_diags
git status
# Check for uncommitted changes
# Branch: main
git fetch upstream main
git reset --hard upstream/main
git log
# Last commit: fixing clevs for tc; an error that misses one storm (#1009
# Good, matches https://github.com/E3SM-Project/e3sm_diags/commits/main
rm -rf build
conda clean --all --y # ~7 min to run
time conda env create -f conda-env/dev.yml -n e3sm-diags-main-20250926 # ~3 min to run
conda activate e3sm-diags-main-20250926
python -m pip install .
cd ~/ez/zppy-interfaces
git status
# Check for uncommitted changes
# Branch: add-pcmdi => Need to change branches
git fetch upstream main
git checkout main
git reset --hard upstream/main
git log
# Last commit: Add missing packages to dependencies (#32)
# Good, matches https://github.com/E3SM-Project/zppy-interfaces/commits/main
rm -rf build
conda clean --all --y # ~6 min to run
time conda env create -f conda/dev.yml -n zi-main-20250926 # ~3 min to run
conda activate zi-main-20250926
python -m pip install .
pytest tests/unit/global_time_series/test_*.py
# 10 passed in 24.22s
cd ~/ez/zppy
git status
# Check for uncommitted changes
# Branch: zppy_pcmdi_202505_rev => Need to change branches
git fetch upstream main
git checkout -b test-zppy-main-20250926 upstream/main
git log
# Last commit: Unit and integration test fixes (#736)
# Good, matches https://github.com/E3SM-Project/zppy/commits/main
rm -rf build
conda clean --all --y # ~7 min to run
time conda env create -f conda/dev.yml -n zppy-main-20250926 # real 18m26.351s
conda activate zppy-main-20250926
pytest tests/test_*.py
# 25 passed in 0.72s
# Edit tests/integration/utils.py:
# TEST_SPECIFICS: Dict[str, Any] = {
# "diags_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate e3sm-diags-main-20250926",
# "global_time_series_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate zi-main-20250926",
# "cfgs_to_run": [
# "weekly_bundles",
# "weekly_comprehensive_v2",
# "weekly_comprehensive_v3",
# "weekly_legacy_3.0.0_bundles",
# "weekly_legacy_3.0.0_comprehensive_v2",
# "weekly_legacy_3.0.0_comprehensive_v3",
# ],
# "tasks_to_run": ["e3sm_diags", "mpas_analysis", "global_time_series", "ilamb"],
# "unique_id": "weekly_test_20250926",
# }
python tests/integration/utils.py
pre-commit run --all-files
python -m pip install .
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v3_chrysalis.cfg
# Check on bundles status
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/weekly_test_20250926/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/weekly_test_20250926/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
# Now, run bundles part 2
cd ~/ez/zppy
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg
# Review finished runs
### v2 ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/weekly_test_20250926/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# global_time_series_1980-1990.status:ERROR (8)
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/weekly_test_20250926/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# global_time_series_1980-1990.status:ERROR (8)
### v3 ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/weekly_test_20250926/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/weekly_test_20250926/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
### bundles ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/weekly_test_20250926/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/weekly_test_20250926/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
# Let's review those global_time_series errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/weekly_test_20250926/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# KeyError: 'rgn'
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/weekly_test_20250926/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# KeyError: 'rgn'
# That's the same error that appeared in the pcmdi_diags test,
# so it looks like the problem is in fact on `main`.
# Let's proceed with the test suite to see if any more errors exist.
cd ~/ez/zppy
ls tests/integration/test_*.py
pytest tests/integration/test_bash_generation.py
# 1 failed in 1.95s, diffs appear to be expected based on recent changes on `main`
pytest tests/integration/test_campaign.py
# 6 failed in 2.16s, diffs appear to be expected based on recent changes on `main`
pytest tests/integration/test_defaults.py
# 1 failed in 0.34s, diffs appear to be expected based on recent changes on `main`
pytest tests/integration/test_last_year.py
# 1 failed in 0.34s, diff is because the prov file is always unique
pytest tests/integration/test_bundles.py
# 2 passed in 0.22s
pytest tests/integration/test_images.py
# Copy the output of test_images_summary.md to a Pull Request comment
# =================================================================== short test summary info ===================================================================
# FAILED tests/integration/test_images.py::test_images - assert 3031 == 2745
# =============================================================== 1 failed in 2803.42s (0:46:43) ================================================================
cat test_images_summary.mdComplete testing summaryExpandDiff subdir is where to find the lists of missing/mismatched images, the image diff grid, and the individual diffs.
Concise testing summary
Preliminary analysis
|
Beta Was this translation helpful? Give feedback.
2 replies
-
See E3SM-Project/zppy-interfaces#35 for debugging. |
Beta Was this translation helpful? Give feedback.
-
|
@chengzhuzhang I confirmed Charlie's fix worked (E3SM-Project/zppy-interfaces#35 (comment)), so I'll rerun the test cfgs now and update the expected results, so we have a usable baseline. |
Beta Was this translation helpful? Give feedback.
-
|
@forsyth2 thanks for the weekly testing. It appears that all the missing images from e3sm diags are model vs model plots for land variables. The plots are not missing instead because of the new PR to add land vars and grouping into categories based on land var csv, the images are saved in a different directory structure (e.g., viewer/lat_lon_land/veg-state/btran-global/ann.html). The mis-matched images are expected and can be explained by the code changes between two versions. |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
Great, thanks. I'm debugging the |
Beta Was this translation helpful? Give feedback.
-
2025-09-29 -- rerun of the 2025-09-26 testTesting ProcessExpandlcrc_conda
cd ~/ez/e3sm_diags
git status
# Check for uncommitted changes
# Branch: main
git fetch upstream main
git reset --hard upstream/main
git log
# Last commit (from today Mon 9/29): Add ARs and ETCs detection script (#997)
# To make sure we're producing exactly the same plots as before,
# let's use the code as of Friday 9/26.
git reset --hard 4c2908d336b59104a8b5a4af74ec02c6509b6a22
git log
# Last commit (from Wed 9/24): fixing clevs for tc; an error that misses one storm (#1009
# Good, matches https://github.com/E3SM-Project/e3sm_diags/commits/main
rm -rf build
conda clean --all --y
conda env create -f conda-env/dev.yml -n e3sm-diags-main-20250926-rerun
conda activate e3sm-diags-main-20250926-rerun
python -m pip install .
cd ~/ez/zppy-interfaces
git status
# Check for uncommitted changes
git fetch upstream main
git checkout main
git reset --hard upstream/main
git log
# Last commit (8/11): Add missing packages to dependencies (#32)
# Good, matches https://github.com/E3SM-Project/zppy-interfaces/commits/main
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n zi-main-20250926-rerun
conda activate zi-main-20250926-rerun
python -m pip install .
pytest tests/unit/global_time_series/test_*.py
# 10 passed in 24.96s
cd ~/ez/zppy
git status
# Check for uncommitted changes
# Branch: zppy_pcmdi_202505_rev => Need to change branches
git fetch upstream main
git checkout -b test-zppy-main-20250926-rerun upstream/main
git log
# Last commit: Unit and integration test fixes (#736)
# Good, matches https://github.com/E3SM-Project/zppy/commits/main
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n zppy-main-20250926-rerun
conda activate zppy-main-20250926-rerun
pytest tests/test_*.py
# 25 passed in 0.49s
# Edit tests/integration/utils.py:
# TEST_SPECIFICS: Dict[str, Any] = {
# "diags_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate e3sm-diags-main-20250926-rerun",
# "global_time_series_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate zi-main-20250926-rerun",
# "cfgs_to_run": [
# "weekly_bundles",
# "weekly_comprehensive_v2",
# "weekly_comprehensive_v3",
# "weekly_legacy_3.0.0_bundles",
# "weekly_legacy_3.0.0_comprehensive_v2",
# "weekly_legacy_3.0.0_comprehensive_v3",
# ],
# "tasks_to_run": ["e3sm_diags", "mpas_analysis", "global_time_series", "ilamb"],
# "unique_id": "weekly_test_20250926_rerun",
# }
python tests/integration/utils.py
pre-commit run --all-files
python -m pip install .
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v3_chrysalis.cfg
# ~2 hours to run
# Check on bundles status
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/weekly_test_20250926_rerun/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/weekly_test_20250926_rerun/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
# Now, run bundles part 2
cd ~/ez/zppy
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg
# Review complete runs
### v2 ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/weekly_test_20250926_rerun/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/weekly_test_20250926_rerun/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# Good, no errors
### v3 ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/weekly_test_20250926_rerun/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/weekly_test_20250926_rerun/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
### bundles ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/weekly_test_20250926_rerun/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/weekly_test_20250926_rerun/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
# Good, no errors in any of the output directories!
# Now, the tests themselves:
cd ~/ez/zppy
ls tests/integration/test_*.py
pytest tests/integration/test_bash_generation.py
# 1 failed in 6.51s, diffs appear to be expected based on recent changes on `main`
pytest tests/integration/test_campaign.py
# 6 failed in 4.07s, diffs appear to be expected based on recent changes on `main`
pytest tests/integration/test_defaults.py
# 1 failed in 2.80s, diffs appear to be expected based on recent changes on `main`
pytest tests/integration/test_last_year.py
# 1 failed in 0.72s, diff is because the prov file is always unique
pytest tests/integration/test_bundles.py
# 2 passed in 0.46s
hostname
# chrlogin1.lcrc.anl.gov
screen # Activate screen
lcrc_conda
conda activate zppy-main-20250926-rerun
pytest tests/integration/test_images.py 2>&1 | tee out.txt
# CTRL A D
# [detached from 3331906.pts-26.chrlogin1]
screen -ls
# There is a screen on:
# 3331906.pts-26.chrlogin1 (Detached)
# 1 Socket in /run/screen/S-ac.forsyth2.
tail out.txt
# Good, it's writing to the output file.
###############################################################################
# Picking up 9/30
hostname
# chrlogin1.lcrc.anl.gov
screen -ls
# There is a screen on:
# 3331906.pts-26.chrlogin1 (Detached)
# 1 Socket in /run/screen/S-ac.forsyth2.
screen -R
# Number of missing images: 0
# Number of mismatched images: 0
# Number of correct images: 3
# web_portal_base_path: /lcrc/group/e3sm/public_html/diagnostic_output
# web_portal_base_url: https://web.lcrc.anl.gov/public/e3sm/diagnostic_output
# Making image diff grid for /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_www/weekly_test_20250926_rerun/v3.LR.historical_0051/image_check_failures_legacy_3.0.0_bundles/global_time_series
# Saving to:
# /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_www/weekly_test_20250926_rerun/v3.LR.historical_0051/image_check_failures_legacy_3.0.0_bundles/global_time_series/image_diff_grid.pdf
# Web page will be at:
# https://web.lcrc.anl.gov/public/e3sm/diagnostic_output//ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_www/weekly_test_20250926_rerun/v3.LR.historical_0051/image_check_failures_legacy_3.0.0_bundles/global_time_series/image_diff_grid.pdf
# Opening expected images file /lcrc/group/e3sm/public_html/zppy_test_resources/image_list_expected_legacy_3.0.0_bundles.txt
# Reading expected images file /lcrc/group/e3sm/public_html/zppy_test_resources/image_list_expected_legacy_3.0.0_bundles.txt
# On line # 250
# Total: 388
# Number of missing images: 0
# Number of mismatched images: 0
# Number of correct images: 388
# web_portal_base_path: /lcrc/group/e3sm/public_html/diagnostic_output
# web_portal_base_url: https://web.lcrc.anl.gov/public/e3sm/diagnostic_output
# Making image diff grid for /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_www/weekly_test_20250926_rerun/v3.LR.historical_0051/image_check_failures_legacy_3.0.0_bundles/ilamb
# Saving to:
# /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_www/weekly_test_20250926_rerun/v3.LR.historical_0051/image_check_failures_legacy_3.0.0_bundles/ilamb/image_diff_grid.pdf
# Web page will be at:
# https://web.lcrc.anl.gov/public/e3sm/diagnostic_output//ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_www/weekly_test_20250926_rerun/v3.LR.historical_0051/image_check_failures_legacy_3.0.0_bundles/ilamb/image_diff_grid.pdf
# Copy the output of test_images_summary.md to a Pull Request comment
# =========================== short test summary info ============================
# FAILED tests/integration/test_images.py::test_images - assert 3031 == 2745
# ======================== 1 failed in 3351.13s (0:55:51) ========================
exit # Exit screen
cd ~/ez/zppy
tail -n 4 out.txt
# Copy the output of test_images_summary.md to a Pull Request comment
# =========================== short test summary info ============================
# FAILED tests/integration/test_images.py::test_images - assert 3031 == 2745
# ======================== 1 failed in 3351.13s (0:55:51) ========================
cat test_images_summary.mdComplete testing summaryExpandDiff subdir is where to find the lists of missing/mismatched images, the image diff grid, and the individual diffs.
Concise testing summaryDiff subdir is where to find the lists of missing/mismatched images, the image diff grid, and the individual diffs.
NOTE: These links no longer work because I've since updated the expected results and rerun the tests. That is, there are no longer any errors reported. AnalysisComparing these results with the original run, we see:
That is, these results are all good. We are now able to update the expected results! Updating expected resultsExpand# First, let's copy over the old expected results
cd /lcrc/group/e3sm/public_html/
# Using 20250926 because we're testing off code as if it was from that day.
# (Recall that e3sm_diags had a new commit on 9/29)
cp -r zppy_test_resources/ zppy_test_resources_previous/expected_results_until_20250926 # ~14 min to run
# Check old expected results were copied over correctly, so we have a record:
ls zppy_test_resources_previous/expected_results_until_20250926
# Now, make new expected results for all the tests
cd ~/ez/zppy
git status
# Only one of the update scripts has a diff:
# tests/integration/generated/update_weekly_expected_files_chrysalis.sh
# That file uses multiple `#expand` parameters.
# The others only use `#expand` for the expected dir, which is consistent.
# template_update_{bash_generation, campaign, defaults}_expected_files.sh
# Let's update the simpler tests' results first:
ls tests/integration/test_*.py
# tests/integration/test_bash_generation.py tests/integration/test_defaults.py
# tests/integration/test_bundles.py tests/integration/test_images.py
# tests/integration/test_campaign.py tests/integration/test_last_year.py
lcrc_conda
conda activate zppy-main-20250926-rerun
./tests/integration/generated/update_bash_generation_expected_files_chrysalis.sh
# Only in test_bash_generation_output/post/scripts: provenance.20250930_164307_367483.cfg
# 1 failed in 1.60s
# As noted in https://github.com/E3SM-Project/zppy/discussions/634#discussioncomment-13726657,
# `zppy -c tests/integration/test_bash_generation.cfg` is going to create a new provenance each time!
# https://github.com/E3SM-Project/zppy/issues/728 was created to address this at some point.
./tests/integration/generated/update_campaign_expected_files_chrysalis.sh
# Same issue as above
# 6 failed in 5.45s
./tests/integration/generated/update_defaults_expected_files_chrysalis.sh
# Same issue as above
# 1 failed in 0.84s
# test_last_year doesn't need an update script because its cfg is stand-alone:
pytest -vv tests/integration/test_last_year.py
# Actual has provenance.20250617_010036_322732.cfg, whereas Expected doesn't have a prov cfg at all.
# That appears to be because the directory `zppy/test_last_year_output/` was never being deleted.
# Made some changes to the files
# 1 passed in 1.09s
# Let's edit the other test files accordingly.
./tests/integration/generated/update_defaults_expected_files_chrysalis.sh
# 1 passed in 1.22s
./tests/integration/generated/update_campaign_expected_files_chrysalis.sh
# 6 passed in 1.50s
./tests/integration/generated/update_bash_generation_expected_files_chrysalis.sh
# For this one, we needed to remove the prov cfg in the expected results as well.
# Default & campaign only copied over specific files,
# but this one copies over the entire directory.
# 1 passed in 1.01s
# Also updated test_images.py to make sure the tasks_to_run copy is always defined.
# Now update the expected files for the major tests:
./tests/integration/generated/update_weekly_expected_files_chrysalis.sh # 10:29
# For these, we rerun the tests in distinct commands:
pytest tests/integration/test_bundles.py
# 2 passed in 0.88s
pytest tests/integration/test_images.py # 10:55
# 1 passed in 3785.53s (1:03:05)
# That's ~20 minutes longer than before
cat test_images_summary.md
# No errors to reportAll expected results have been updated. The previous expected results have been moved to an archive. All integraiton & unit tests now pass. Some fixes to mergeExpandgit status
# Reset TEST_SPECIFICS in tests/integration/utils.py:
git diff tests/integration/utils.py
# All diffs were part of TEST_SPECIFICS, so we can just restore that file
git restore tests/integration/utils.py
git diff tests/integration/utils.py
# Now, no diffs
# Reset the generated files:
python tests/integration/utils.py
git status
rm -rf out.txt www/
git diff
# All remaining diffs are the test fixes
git add -A
# Change the branch name
# We'reresolving https://github.com/E3SM-Project/zppy/issues/728
git branch -m fix-issue-728
pre-commit run --all-files
git add -A
git commit -m "Integration test fixes"
git push upstream fix-issue-728I added some fixes on the testing side: #737 |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
@chengzhuzhang Please let me know if you want to review the testing fixes (#737) or if I can go ahead and merge that. Now that the tests are passing on |
Beta Was this translation helpful? Give feedback.
-
2025-10-03 ThreadWhat changes are we testing?
Testing processExpandlcrc_conda
cd ~/ez/e3sm_diags
git status
# branch: main
# nothing to commit
git fetch upstream main
git reset --hard upstream/main
git log
# Last commit: Add ARs and ETCs detection script (#997)
# Good, matches https://github.com/E3SM-Project/e3sm_diags/commits/main
rm -rf build
conda clean --all --y
conda env create -f conda-env/dev.yml -n e3sm-diags-main-20251003
conda activate e3sm-diags-main-20251003
python -m pip install .
cd ~/ez/zppy-interfaces
git status
# branch: add-pcmdi
# nothing to commit
git fetch upstream main
git checkout main
git reset --hard upstream/main
git log
# Last commit: Comment out docs build (#36)
# Good, matches https://github.com/E3SM-Project/zppy-interfaces/commits/main
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n zi-main-20251003
conda activate zi-main-20251003
python -m pip install .
pytest tests/unit/global_time_series/test_*.py
# 10 passed in 24.03s
cd ~/ez/zppy
# branch: zppy_pcmdi_202505_rev
# Uncommitted changes
git add -A
git commit -m "Testing" --no-verify
git fetch upstream main
git checkout -b test-zppy-main-20251003 upstream/main
git log
# Last commit: Switch packaging to pyproject.toml (#739)
# Good, matches https://github.com/E3SM-Project/zppy/commits/main
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n zppy-main-20251003
conda activate zppy-main-20251003
pytest tests/test_*.py
# 25 passed in 1.39s
# Edit tests/integration/utils.py:
# TEST_SPECIFICS: Dict[str, Any] = {
# "diags_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate e3sm-diags-main-20251003",
# "global_time_series_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate zi-main-20251003",
# "cfgs_to_run": [
# "weekly_bundles",
# "weekly_comprehensive_v2",
# "weekly_comprehensive_v3",
# "weekly_legacy_3.0.0_bundles",
# "weekly_legacy_3.0.0_comprehensive_v2",
# "weekly_legacy_3.0.0_comprehensive_v3",
# ],
# "tasks_to_run": ["e3sm_diags", "mpas_analysis", "global_time_series", "ilamb"],
# "unique_id": "weekly_test_20251003",
# }
python tests/integration/utils.py
pre-commit run --all-files
python -m pip install .
# Add the commit from https://github.com/E3SM-Project/zppy/pull/738
git fetch upstream further-test-fixes
git cherry-pick b0132f0600cb6f867e6f45b37f2acf2d32a49c59
python tests/integration/utils.py
pre-commit run --all-files
python -m pip install .
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v3_chrysalis.cfg
# Check on bundles status
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/weekly_test_20251003/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/weekly_test_20251003/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
# Now, run bundles part 2
cd ~/ez/zppy
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg
# Review finished runs
### v2 ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/weekly_test_20251003/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/weekly_test_20251003/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# Good, no errors
### v3 ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/weekly_test_20251003/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/weekly_test_20251003/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
### bundles ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/weekly_test_20251003/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/weekly_test_20251003/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd ~/ez/zppy
ls tests/integration/test_*.py
pytest tests/integration/test_bash_generation.py
# 1 failed in 2.27s, diffs appear to be expected based on recent changes on `main`
pytest tests/integration/test_campaign.py
# 6 passed in 2.09s
pytest tests/integration/test_defaults.py
# 1 passed in 0.44s
pytest tests/integration/test_last_year.py
# 1 passed in 0.27s
pytest tests/integration/test_bundles.py
# 2 passed in 1.05s
pytest tests/integration/test_images.py
# 1 failed in 3367.01s (0:56:07)
# Copy the output of test_images_summary.md to a Pull Request comment
# =================================================================== short test summary info ===================================================================
# FAILED tests/integration/test_images.py::test_images - assert 3031 == 2745
# =============================================================== 1 failed in 2803.42s (0:46:43) ================================================================
cat test_images_summary.mdComplete resultsExpandDiff subdir is where to find the lists of missing/mismatched images, the image diff grid, and the individual diffs.
Concise results
This test fails because we changed the v3 test cases, so this is expected. Let's take a look manually:
All these look good. Updating expected resultsExpand# First, let's copy over the old expected results
cd /lcrc/group/e3sm/public_html/
cp -r zppy_test_resources/ zppy_test_resources_previous/expected_results_until_20251003
# Check old expected results were copied over correctly, so we have a record:
ls zppy_test_resources_previous/expected_results_until_20251003
# Now, make new expected results for all the tests
cd ~/ez/zppy
git status
# Only one of the update scripts has a diff:
# tests/integration/generated/update_weekly_expected_files_chrysalis.sh
# That file uses multiple `#expand` parameters.
# The others only use `#expand` for the expected dir, which is consistent.
# template_update_{bash_generation, campaign, defaults}_expected_files.sh
# Let's update the simpler tests' results first:
ls tests/integration/test_*.py
# tests/integration/test_bash_generation.py tests/integration/test_defaults.py
# tests/integration/test_bundles.py tests/integration/test_images.py
# tests/integration/test_campaign.py tests/integration/test_last_year.py
./tests/integration/generated/update_bash_generation_expected_files_chrysalis.sh
# 1 passed in 1.72s
./tests/integration/generated/update_campaign_expected_files_chrysalis.sh
# 6 failed in 1.86s
# Just rerun it; it was expecting files to exist...
# (The test output directories get removed if the test was successful,
# so there was nothing for the update script to take as new results).
./tests/integration/generated/update_campaign_expected_files_chrysalis.sh
# 6 passed in 2.07s
./tests/integration/generated/update_defaults_expected_files_chrysalis.sh
# 1 failed in 0.92s
# Same as above, just rerun:
./tests/integration/generated/update_defaults_expected_files_chrysalis.sh
# 1 passed in 0.43s
# test_last_year doesn't need an update script because its cfg is stand-alone:
pytest -vv tests/integration/test_last_year.py
# 1 passed in 0.36s
# Now update the expected files for the major tests:
./tests/integration/generated/update_weekly_expected_files_chrysalis.sh # < 34 minutes
# For these, we rerun the tests in distinct commands:
pytest tests/integration/test_bundles.py
# 2 passed in 1.17s
pytest tests/integration/test_images.py
# 1 passed in 3460.87s (0:57:40)
cat test_images_summary.md
# No errors to reportAll expected results have been updated. The previous expected results have been moved to an archive. All integraiton & unit tests now pass. This also served as another test of the testing fix commit at #738. |
Beta Was this translation helpful? Give feedback.
0 replies
-
2025-10-13 ThreadWe're testing the What changes are we testing?
Testing processExpandlcrc_conda
cd ~/ez/e3sm_diags
git status
# branch: main
# nothing to commit
git fetch upstream main
git reset --hard upstream/main
git log
# Last commit: Bump to 3.1.0rc1 (#1010)
# Good, matches https://github.com/E3SM-Project/e3sm_diags/commits/main
rm -rf build
conda clean --all --y
conda env create -f conda-env/dev.yml -n e3sm-diags-main-20251013
conda activate e3sm-diags-main-20251013
python -m pip install .
cd ~/ez/zppy-interfaces
git status
# branch: pcmdi-updates
# nothing to commit
git fetch upstream main
git checkout main
git reset --hard upstream/main
git log
# Last commit: Further pcmdi_diags updates (#41)
# Good, matches https://github.com/E3SM-Project/zppy-interfaces/commits/main
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n zi-main-20251013
conda activate zi-main-20251013
python -m pip install .
pytest tests/unit/global_time_series/test_*.py
# 10 passed in 27.72s
pytest tests/unit/pcmdi_diags/test_*.py
# 7 passed, 2 warnings in 16.56s
cd ~/ez/zppy
git status
# branch: pcmdi-updates
# nothing to commit
git fetch upstream main
git checkout -b test-zppy-main-20251013 upstream/main
git log
# Last commit: Merge pull request #742 from E3SM-Project/pcmdi-updates
# Good, matches https://github.com/E3SM-Project/zppy/commits/main
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n zppy-main-20251013
conda activate zppy-main-20251013
pytest tests/test_*.py
# 35 passed in 1.27s
# Edit tests/integration/utils.py:
# TEST_SPECIFICS: Dict[str, Any] = {
# "diags_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate e3sm-diags-main-20251013",
# "global_time_series_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate zi-main-20251013",
# "pcmdi_diags_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate zi-main-20251013",
# "cfgs_to_run": [
# "weekly_bundles",
# "weekly_comprehensive_v2",
# "weekly_comprehensive_v3",
# "weekly_legacy_3.0.0_bundles",
# "weekly_legacy_3.0.0_comprehensive_v2",
# "weekly_legacy_3.0.0_comprehensive_v3",
# ],
# "tasks_to_run": [
# "e3sm_diags",
# "mpas_analysis",
# "global_time_series",
# "ilamb",
# "pcmdi_diags",
# ],
# "unique_id": "weekly_test_20251013",
# }
python tests/integration/utils.py
pre-commit run --all-files
python -m pip install .
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v3_chrysalis.cfg
# zppy.utils.ParameterNotProvidedError: cmip_plevdata was not provided, and inferring is turned off. Turn on inferring by setting infer_path_parameters to True.
# The turned off Parameter inference breaks backwards compatibility.
# Check on bundles status
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/weekly_test_20251013/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/weekly_test_20251013/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
# Now, run bundles part 2
cd ~/ez/zppy
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg
# Review finished runs
### v2 ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/weekly_test_20251013/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/weekly_test_20251013/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# Good, no errors
### v3 ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/weekly_test_20251013/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/weekly_test_20251013/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
# However:
ls *.status
# Only the climo and ts tasks ran because of the launch problem re: path inference
### bundles ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/weekly_test_20251013/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/weekly_test_20251013/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd ~/ez/zppy
ls tests/integration/test_*.py
pytest tests/integration/test_bash_generation.py
# 1 failed in 1.92s, diffs appear to be expected based on recent changes on `main`
pytest tests/integration/test_campaign.py
# 6 passed in 2.09s, same as above
pytest tests/integration/test_defaults.py
# 1 failed in 0.50s, same as above
pytest tests/integration/test_last_year.py
# 1 passed in 0.32s
pytest tests/integration/test_bundles.py
# 2 passed in 0.28s
emacs tests/integration/utils.py
# Comment out legacy test that failed to launch
# Comment out pcmdi_diags since we don't have expected results yet
# # "weekly_legacy_3.0.0_comprehensive_v3",
# #"pcmdi_diags",
pytest tests/integration/test_images.py
# 1 passed in 2500.69s (0:41:40)
cat test_images_summary.mdComplete resultsExpandDiff subdir is where to find the lists of missing/mismatched images, the image diff grid, and the individual diffs.
Concise resultsNo explicit test failures. Two things to note:
|
Beta Was this translation helpful? Give feedback.
3 replies
-
|
@chengzhuzhang ^ FYI latest |
Beta Was this translation helpful? Give feedback.
-
|
hey @forsyth2 thanks for the explanation! That helps, but I’m still having a hard time wrapping my head around this part: In PR 732 e3sm_to_cmip.py, This says: If cmip_plevdata is not set in the dictionary c, then set it to the value of default_cmip_plevdata. Otherwise, leave it as-is. It impact the legacy v3 cfg is because it change behavior of the results? Is it true that the cfg includes ilamb, but not pmp? Unless I can understand better, I'm not useful to help provide recommendation on this... |
Beta Was this translation helpful? Give feedback.
-
|
@chengzhuzhang It might help to dive into the function's definition in utils.py: def set_value_of_parameter_if_undefined(
c: Dict[str, Any],
parameter: str,
backup_option: str,
inference_type: ParameterInferenceType,
) -> None:
# Determine which type of inference to use.
inference_type_parameter: str = get_inference_type_parameter(inference_type)
# Define a value, if possible.
if (parameter in c.keys()) and (c[parameter] == ""):
if c[inference_type_parameter]:
c[parameter] = backup_option
else:
raise ParameterNotProvidedError(
f"{parameter} was not provided, and inferring is turned off. Turn on inferring by setting {inference_type_parameter} to True."
)
# If parameter is already defined, do nothing.
This is the block most users will find themselves in. They haven't set a value in the cfg, and so
This is the block causing problems on a legacy cfg. As part of the This is why my thinking is to only support backward compatibility for the default case ( The biggest benefit to me of being able to turn off parameter inference is it makes it obvious where
Yes, the legacy cfg template has |
Beta Was this translation helpful? Give feedback.
-
2025-12-16 ThreadWe're testing the What changes are we testing?The last full integration test was run using E3SM Unified
Testing processExpandlcrc_conda # Activate conda
# Set up e3sm_diags env
cd ~/ez/e3sm_diags
git status
# On branch diags-1019-xc-break-test2
# nothing to commit, working tree clean
git fetch upstream main
git checkout main
git reset --hard upstream/main
git log
# Good, last commit: Bump/v3.1.0 (#1018)
# Matches https://github.com/E3SM-Project/e3sm_diags/commits/main
rm -rf build
conda clean --all --y
conda env create -f conda-env/dev.yml -n test-diags-main-20251216
conda activate test-diags-main-20251216
python -m pip install .
# Set up zppy-interfaces env
cd ~/ez/zppy-interfaces
git status
# On branch official_results_unified_1.12.0_chrysalis
# nothing to commit, working tree clean
git fetch upstream main
git checkout main
git reset --hard upstream/main
git log
# Good, last commit: Bump to 0.2.0 (#45)
# Matches https://github.com/E3SM-Project/zppy-interfaces/commits/main
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n test-zi-main-20251216
conda activate test-zi-main-20251216
python -m pip install .
# zppy itself #################################################################
cd ~/ez/zppy
git status
# On branch zppy-diags-1019-xc-break-test2
# Handle uncommitted changes:
git add -A
git commit -m "Testing" --no-verify
git fetch upstream main
git checkout -b zppy-weekly-test-20251216 upstream/main
git log
# Good, last commit: Add legacy cfgs for zppy v3.1.0 (#762)
# Matches https://github.com/E3SM-Project/zppy/commits/main
# Get testing commit: https://github.com/E3SM-Project/zppy/pull/767
# 806dfa2e1ce947930ecf7483eeb3f35e18c41bc6
git fetch upstream issue-766-test-links
git cherry-pick 806dfa2e1ce947930ecf7483eeb3f35e18c41bc6
git log
# Good, commit is added
# Now, the diff grid PDFs should have links to higher-res diffs.
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n zppy-weekly-test-20251216-env
conda activate zppy-weekly-test-20251216-env
pre-commit run --all-files # Optional, we haven't changed anything aside from the testing commit
python -m pip install .
pytest tests/test_*.py
# 35 passed in 0.96s
# Edit tests/integration/utils.py
# There haven't been any changes to the test cfgs, so let's keep using the 3.0.0 legacy cfgs for now.
# TEST_SPECIFICS:
# TEST_SPECIFICS: Dict[str, Any] = {
# # These are custom environment_commands for specific tasks.
# # Never set these to "", because they will print the line
# # `environment_commands = ""` for the corresponding task,
# # thus overriding the value set higher up in the cfg.
# # That is, there will be no environment set.
# # (`environment_commands = ""` only redirects to Unified
# # if specified under the [default] task)
# "diags_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-diags-main-20251216",
# "global_time_series_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-zi-main-20251216",
# "pcmdi_diags_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-zi-main-20251216",
# # This is the environment setup for other tasks.
# # Leave as "" to use the latest Unified environment.
# "environment_commands": "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh",
# # For a complete test, run the set of latest cfgs and at least one set of legacy cfgs
# "cfgs_to_run": [
# "weekly_bundles",
# "weekly_comprehensive_v2",
# "weekly_comprehensive_v3",
# # "weekly_legacy_3.1.0_bundles",
# # "weekly_legacy_3.1.0_comprehensive_v2",
# # "weekly_legacy_3.1.0_comprehensive_v3",
# "weekly_legacy_3.0.0_bundles",
# "weekly_legacy_3.0.0_comprehensive_v2",
# "weekly_legacy_3.0.0_comprehensive_v3",
# ],
# "tasks_to_run": [
# "e3sm_diags",
# "mpas_analysis",
# "global_time_series",
# "ilamb",
# "pcmdi_diags",
# ],
# "unique_id": "zppy_weekly_test_20251216",
# }
git diff
# Diff looks good
python tests/integration/utils.py
# CFG FILES HAVE BEEN GENERATED FROM TEMPLATES WITH THESE SETTINGS:
# UNIQUE_ID=zppy_weekly_test_20251216
# diags_environment_commands=source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-diags-main-20251216
# global_time_series_environment_commands=source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-zi-main-20251216
# pcmdi_diags_environment_commands=source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-zi-main-20251216
# environment_commands=source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh
# Reminder: `environment_commands=''` => the latest E3SM Unified environment will be used
# For reference:
alias sq
# alias sq='sqa -u ac.forsyth2'
alias sqa
# alias sqa='squeue -o "%8u %.7a %.4D %.9P %7i %.2t %.10r %.10M %.10l %j" --sort=P,-t,-p'
sq
# No jobs currently queued
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v3_chrysalis.cfg
sq | wc -l # Tue 12/16 10:45 => 144 - header row = 143 jobs
sq | wc -l # Tue 12/16 13:09 => 1 - header row = 0 jobs
# Check on bundles status
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/zppy_weekly_test_20251216/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/zppy_weekly_test_20251216/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
# Now, run bundles part 2
cd ~/ez/zppy
git status
# On branch zppy-weekly-test-20251216
# Good, correct branch
# Current conda env: zppy-weekly-test-20251216-env
# Good, correct environment
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg
sq | wc -l # Tue 12/16 13:12 => 5 - header row = 4 jobs
sq | wc -l # Tue 12/16 13:41 => 1 - header row = 0 jobs
# Review finished runs
### v2 ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/zppy_weekly_test_20251216/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/zppy_weekly_test_20251216/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# Good, no errors
### v3 ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/zppy_weekly_test_20251216/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/zppy_weekly_test_20251216/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
### bundles ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/zppy_weekly_test_20251216/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/zppy_weekly_test_20251216/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
cd ~/ez/zppy
git status
# On branch zppy-weekly-test-20251216
# Good, correct branch
# Env: zppy-weekly-test-20251216-env
# Good, correct env
ls tests/integration/test_*.py
# tests/integration/test_bash_generation.py tests/integration/test_campaign.py tests/integration/test_images.py
# tests/integration/test_bundles.py tests/integration/test_defaults.py tests/integration/test_last_year.py
pytest tests/integration/test_bash_generation.py
# 1 failed in 1.91s, failures are expected based on changes to `main`
pytest tests/integration/test_campaign.py
# 6 failed in 2.54s, same reason
pytest tests/integration/test_defaults.py
# 1 failed in 0.57s, same reason
pytest tests/integration/test_last_year.py
# 1 passed in 0.32s
pytest tests/integration/test_bundles.py
# 2 passed in 0.43s
sinfo -p debug # Shows a TIMELIMIT of 4h
# Let's add a little buffer time.
salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm
# We need to reactivate conda now that we're on a compute node
lcrc_conda # Activate conda
conda activate zppy-weekly-test-20251216-env
pytest tests/integration/test_images.py
# 1 failed in 836.84s (0:13:56)
cat test_images_summary.mdSummary table
Results analysisNote that the image diff grid PDFs now include links to a higher-res png of the diff, meaning you no longer have to manually find that. To compare actual and expected in higher-res, replace the
Blocker for individual PR testsI would like to update expected results soon so I can have a good baseline for testing:
|
Beta Was this translation helpful? Give feedback.
4 replies
-
|
@chengzhuzhang Can you please review the "Summary table" & "Results analysis" sections above? Do those look expected based on the table in the "What changes are we testing?" section? If so, I'll go ahead and update the expected results. Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
@forsyth2 the test is based on main branch. By brief inspection I think the results are expected. |
Beta Was this translation helpful? Give feedback.
-
|
Great, and I've made an issue (#768) for the In the meantime, I'm going to update the expected results for |
Beta Was this translation helpful? Give feedback.
-
|
New expected results based on:
Setup steps# First, let's copy over the old expected results
cd /lcrc/group/e3sm/public_html/
ls zppy_test_resources_previous # Review previous expected results
# We last updated using the rc4 results (https://github.com/E3SM-Project/zppy/discussions/746#discussioncomment-14964092)
cp -r zppy_test_resources/ zppy_test_resources_previous/expected_results_until_20251113_from_unified_1.12.0rc4 # ~15 minutes
# We used the wrong date in the dir name; let's correct it:
mv zppy_test_resources_previous/expected_results_until_20251113_from_unified_1.12.0rc4 zppy_test_resources_previous/expected_results_until_20251217_from_unified_1.12.0rc4
# Check old expected results were copied over correctly, so we have a record:
ls zppy_test_resources_previous/expected_results_until_20251217_from_unified_1.12.0rc4
# Just for reference, let's see how much space the expected results are taking up:
du -sh zppy_test_resources_previous/expected_results_until_20251217_from_unified_1.12.0rc4
# 13G
ls zppy_test_resources_previous/ | grep expected_results | wc -l
# 18
# So, we can estimate 18 expected results dirs * 13G/dir = 234G of previous expected results
# Now, make new expected results for all the tests
cd ~/ez/zppy
git status
# On branch zppy-diags-1019-xc-break-test2
# Handle uncommitted changes:
git add -A
git commit -m "Testing" --no-verify
git checkout zppy-weekly-test-20251216
# On branch zppy-weekly-test-20251216
# Good, correct branch
lcrc_conda
conda activate zppy-weekly-test-20251216-env
# Good, correct env
# Reminder: only one of the update scripts has a diff:
# tests/integration/generated/update_weekly_expected_files_chrysalis.sh
# That file uses multiple `#expand` parameters.
# The others only use `#expand` for the expected dir, which is consistent.
# template_update_{bash_generation, campaign, defaults}_expected_files.sh
# Let's update the simpler tests' results first:
ls tests/integration/generated/update_*
./tests/integration/generated/update_bash_generation_expected_files_chrysalis.sh
# 1 passed in 1.58s
./tests/integration/generated/update_campaign_expected_files_chrysalis.sh
# 6 passed in 1.64s
./tests/integration/generated/update_defaults_expected_files_chrysalis.sh
# 1 passed in 0.33s
# test_last_year doesn't need an update script because its cfg is stand-alone:
pytest -vv tests/integration/test_last_year.py
# 1 passed in 0.31s
# Now update the expected files for the major tests:
./tests/integration/generated/update_weekly_expected_files_chrysalis.sh # ~20 minutes
# NOTE: Once we start actually using the 3.1.0 legacy cfgs, we'll have to update those too!
# In whatever PR starts using them
# (i.e., the first PR to modify the main test cfgs, thus diverging from the 3.1.0 cfgs),
# we'll update tests/integration/template_update_weekly_expected_files.sh, specifically:
# tests=("comprehensive_v2" "comprehensive_v3" "bundles" "legacy_3.0.0_comprehensive_v2" "legacy_3.0.0_comprehensive_v3" "legacy_3.0.0_bundles")
# For these, we rerun the tests in distinct commands:
pytest tests/integration/test_bundles.py
# 2 passed in 1.01s
salloc --nodes=1 --partition=debug --time=01:00:00 --account=e3sm
lcrc_conda
conda activate zppy-weekly-test-20251216-env
pytest tests/integration/test_images.py
# 1 passed in 722.75s (0:12:02)
cat test_images_summary.md
|
Beta Was this translation helpful? Give feedback.
-
2025-12-30 ThreadWe're testing the What changes are we testing?The last full integration test was run using a dev environment (2025-12-16) and expected results were last updated based on thar run (link).
Testing processExpandlcrc_conda # Activate conda
# Set up e3sm_diags env
cd ~/ez/e3sm_diags
git status
# On branch diags-1019-xc-break-test4
# nothing to commit, working tree clean
git fetch upstream main
git checkout main
git reset --hard upstream/main
git log
# Good, last commit: Bump/v3.1.0 (#1018)
# Matches https://github.com/E3SM-Project/e3sm_diags/commits/main
rm -rf build
conda clean --all --y
conda env create -f conda-env/dev.yml -n test-diags-main-20251230
conda activate test-diags-main-20251230
python -m pip install .
# Set up zppy-interfaces env
cd ~/ez/zppy-interfaces
git status
# On branch test-fix-missing-year-key-original-plots
# nothing to commit, working tree clean
git fetch upstream main
git checkout main
git reset --hard upstream/main
git log
# Good, last commit: fix two ocn plots when no atm data (#46)
# Matches https://github.com/E3SM-Project/zppy-interfaces/commits/main
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n test-zi-main-20251230
conda activate test-zi-main-20251230
python -m pip install .
# zppy itself #################################################################
cd ~/ez/zppy
git status
# On branch zppy-diags-1019-xc-break-test4
# Handle uncommitted changes:
git add -A
git commit -m "Testing" --no-verify
git fetch upstream main
git checkout -b test-zppy-main-20251230 upstream/main
git log
# Good, last commit: Update ilamb.cfg fixing html generation error (#770)
# Matches https://github.com/E3SM-Project/zppy/commits/main
# Get testing commit: https://github.com/E3SM-Project/zppy/pull/769
# b56a38c6ae5b24a96bbc80a2dacbc6d1b3dd730b
git fetch upstream test-fixes
git cherry-pick b56a38c6ae5b24a96bbc80a2dacbc6d1b3dd730b
git log
# Good, commit is added
# Nothing in this commit should actually affect anything for this run.
# We're not running the legacy 3.1.0 tests and we're not running ilamb without e3sm_diags.
# Nevertheless, it would be good to know this doesn't break any of the tests.
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n test-zppy-main-20251230-env
conda activate test-zppy-main-20251230-env
pre-commit run --all-files # Optional, we haven't changed anything aside from the test-fixes commit
python -m pip install .
pytest tests/test_*.py
# 35 passed in 0.38s
# Let's review diffs to the cfgs
git log
# 2025-12-16 test ran on:
# ac7dca27232d63e72c1824f6662d020e9d251698 Add legacy cfgs for zppy v3.1.0 (#762)
git diff ac7dca27232d63e72c1824f6662d020e9d251698 HEAD tests/integration/template_weekly_legacy_3.0.0*
git diff ac7dca27232d63e72c1824f6662d020e9d251698 HEAD tests/integration/template_weekly_comprehensive_*
git diff ac7dca27232d63e72c1824f6662d020e9d251698 HEAD tests/integration/template_weekly_bundles*
# We've changed settings for parameters, but have not introduced new parameters.
# That is, we can keep using the legacy 3.0.0 cfgs for now, since the main cfgs have not diverge
# in a significant way from the 3.1.0 cfgs.
# Edit tests/integration/utils.py
# TEST_SPECIFICS: Dict[str, Any] = {
# # These are custom environment_commands for specific tasks.
# # Never set these to "", because they will print the line
# # `environment_commands = ""` for the corresponding task,
# # thus overriding the value set higher up in the cfg.
# # That is, there will be no environment set.
# # (`environment_commands = ""` only redirects to Unified
# # if specified under the [default] task)
# "diags_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-diags-main-20251230",
# "mpas_analysis_environment_commands": "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh",
# "global_time_series_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-zi-main-20251230",
# "pcmdi_diags_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-zi-main-20251230",
# # This is the environment setup for other tasks.
# # Leave as "" to use the latest Unified environment.
# "environment_commands": "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh",
# # For a complete test, run the set of latest cfgs and at least one set of legacy cfgs
# "cfgs_to_run": [
# "weekly_bundles",
# "weekly_comprehensive_v2",
# "weekly_comprehensive_v3",
# # "weekly_legacy_3.1.0_bundles",
# # "weekly_legacy_3.1.0_comprehensive_v2",
# # "weekly_legacy_3.1.0_comprehensive_v3",
# "weekly_legacy_3.0.0_bundles",
# "weekly_legacy_3.0.0_comprehensive_v2",
# "weekly_legacy_3.0.0_comprehensive_v3",
# ],
# "tasks_to_run": [
# "e3sm_diags",
# "mpas_analysis",
# "global_time_series",
# "ilamb",
# "pcmdi_diags",
# ],
# "unique_id": "zppy_main_branch_test_20251230",
# }
git diff
# Diff looks good
python tests/integration/utils.py
# CFG FILES HAVE BEEN GENERATED FROM TEMPLATES WITH THESE SETTINGS:
# UNIQUE_ID=zppy_main_branch_test_20251230
# diags_environment_commands=source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-diags-main-20251230
# mpas_analysis_environment_commands=source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh
# global_time_series_environment_commands=source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-zi-main-20251230
# pcmdi_diags_environment_commands=source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-zi-main-20251230
# environment_commands=source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh
# Reminder: `environment_commands=''` => the latest E3SM Unified environment will be used
# For reference:
alias sq
# alias sq='sqa -u ac.forsyth2'
alias sqa
# alias sqa='squeue -o "%8u %.7a %.4D %.9P %7i %.2t %.10r %.10M %.10l %j" --sort=P,-t,-p'
sq
# No jobs currently queued
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v3_chrysalis.cfg
sq | wc -l # Tue 12/30 09:27 => 144 - header row = 143 jobs
sq | wc -l # Tue 12/30 11:30 => 1 - header row = 0 jobs
# Check on bundles status
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/zppy_main_branch_test_20251230/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/zppy_main_branch_test_20251230/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
# Now, run bundles part 2
cd ~/ez/zppy
git status
# On branch test-zppy-main-20251230
# Good, correct branch
# Current conda env: test-zppy-main-20251230-env
# Good, correct environment
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg
sq | wc -l # Tue 12/30 11:32 => 5 - header row = 4 jobs
sq | wc -l # Tue 12/30 11:41 => 1 - header row = 0 jobs
# Review finished runs
### v2 ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/zppy_main_branch_test_20251230/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/zppy_main_branch_test_20251230/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# Good, no errors
### v3 ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/zppy_main_branch_test_20251230/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/zppy_main_branch_test_20251230/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
### bundles ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/zppy_main_branch_test_20251230/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/zppy_main_branch_test_20251230/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
cd ~/ez/zppy
git status
# On branch test-zppy-main-20251230
# Good, correct branch
# Env: test-zppy-main-20251230-env
# Good, correct env
ls tests/integration/test_*.py
# tests/integration/test_bash_generation.py tests/integration/test_campaign.py tests/integration/test_images.py
# tests/integration/test_bundles.py tests/integration/test_defaults.py tests/integration/test_last_year.py
pytest tests/integration/test_bash_generation.py
# 1 failed in 1.22s, failures are expected based on changes to `main` (#760, #765 in particular)
pytest tests/integration/test_campaign.py
# 3 failed, 3 passed in 2.38s, failures are expected based on changes to `main` (#765 in particular)
pytest tests/integration/test_defaults.py
# 1 passed in 0.37s
pytest tests/integration/test_last_year.py
# 1 passed in 0.44s
pytest tests/integration/test_bundles.py
# 2 passed in 0.24s
sinfo -p debug # Shows a TIMELIMIT of 4h
# Let's add a little buffer time.
salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm
# We need to reactivate conda now that we're on a compute node
lcrc_conda # Activate conda
conda activate test-zppy-main-20251230-env
pytest tests/integration/test_images.py
# 1 passed in 735.51s (0:12:15)
cat test_images_summary.mdSummary tableAll pass
Results analysisAll image tests passed. Other integration test failures were specifically due to changes on pytest tests/integration/test_bash_generation.py
# 1 failed in 1.22s, failures are expected based on changes to `main` (#760, #765 in particular)
pytest tests/integration/test_campaign.py
# 3 failed, 3 passed in 2.38s, failures are expected based on changes to `main` (#765 in particular)Updating expected resultsSetup steps# First, let's copy over the old expected results
cd /lcrc/group/e3sm/public_html/
ls zppy_test_resources_previous # Review previous expected results
/usr/lpp/mmfs/bin/mmlsquota -u ac.forsyth2 --block-size T fs2
# We're doing ok on space, so let's continue copying over all expected results,
# even though not all actually need to be updated.
cp -r zppy_test_resources/ zppy_test_resources_previous/expected_results_until_20251230 # ~15 minutes
# Check old expected results were copied over correctly, so we have a record:
ls zppy_test_resources_previous/expected_results_until_20251230
# Now, make new expected results for all the tests
cd ~/ez/zppy
git status
# On branch test-zppy-main-20251230
# Good, correct branch
# Conda env: test-zppy-main-20251230-env
# Good, correct env
# For completeness, let's just update everything, as usual.
# That's probably good practice anyway -- the tests are checking that expected images aren't missing,
# but it's possible newer code could be adding images to expect in the first place.
# (Probably not the case for the recent PRs though).
# Reminder: only one of the update scripts has a diff:
# tests/integration/generated/update_weekly_expected_files_chrysalis.sh
# That file uses multiple `#expand` parameters.
# The others only use `#expand` for the expected dir, which is consistent.
# template_update_{bash_generation, campaign, defaults}_expected_files.sh
# Let's update the simpler tests' results first:
ls tests/integration/generated/update_*
./tests/integration/generated/update_bash_generation_expected_files_chrysalis.sh
# 1 passed in 1.60s
./tests/integration/generated/update_campaign_expected_files_chrysalis.sh
# Ran into some problems with the update script
# Need to stop deleting result directories at end of test_campaign.py tests
# 6 passed in 1.65s
# Made note: https://github.com/E3SM-Project/zppy/pull/769#issuecomment-3700561845
./tests/integration/generated/update_defaults_expected_files_chrysalis.sh
# 1 passed in 0.33s
# test_last_year doesn't need an update script because its cfg is stand-alone:
pytest -vv tests/integration/test_last_year.py
# 1 passed in 0.32s
# Now update the expected files for the major tests:
./tests/integration/generated/update_weekly_expected_files_chrysalis.sh # ~20 minutes
# Couldn't copy 3.1.0 results of course, because we didn't run those:
# cp: cannot stat '/lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_legacy_3.1.0_comprehensive_v2_www/zppy_main_branch_test_20251230/v2.LR.historical_0201': No such file or directory
# ./tests/integration/generated/update_weekly_expected_files_chrysalis.sh: line 56: cd: /lcrc/group/e3sm/public_html/zppy_test_resources/expected_legacy_3.1.0_comprehensive_v2: No such file or directory
# cp: cannot stat '/lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_legacy_3.1.0_comprehensive_v3_www/zppy_main_branch_test_20251230/v3.LR.historical_0051': No such file or directory
# ./tests/integration/generated/update_weekly_expected_files_chrysalis.sh: line 56: cd: /lcrc/group/e3sm/public_html/zppy_test_resources/expected_legacy_3.1.0_comprehensive_v3: No such file or directory
# cp: cannot stat '/lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_legacy_3.1.0_bundles_www/zppy_main_branch_test_20251230/v3.LR.historical_0051': No such file or directory
# ./tests/integration/generated/update_weekly_expected_files_chrysalis.sh: line 56: cd: /lcrc/group/e3sm/public_html/zppy_test_resources/expected_legacy_3.1.0_bundles: No such file or directory
# For these, we rerun the tests in distinct commands:
pytest tests/integration/test_bundles.py
# 2 passed in 0.93s
# We're still on the compute node, so let's just run this:
pytest tests/integration/test_images.py
# 1 passed in 713.88s (0:11:53)
cat test_images_summary.md
|
Beta Was this translation helpful? Give feedback.
0 replies
-
2026-01-30 ThreadWe're testing the What changes are we testing?The last full integration test was run using a dev environment (2025-12-30) and expected results were last updated based on that run (same link, "Updating expected results" section).
Testing processExpandlcrc_conda # Activate conda
# Set up e3sm_diags env
cd ~/ez/e3sm_diags
git status
# On branch main
# nothing to commit, working tree clean
git fetch upstream main
git checkout main
git reset --hard upstream/main
git log
# Good, last commit: Add new set for Precipitation PDF over specified regions (#1023)
# Matches https://github.com/E3SM-Project/e3sm_diags/commits/main
rm -rf build
conda clean --all --y
conda env create -f conda-env/dev.yml -n test-diags-main-20260130
conda activate test-diags-main-20260130
python -m pip install .
# Set up zppy-interfaces env
cd ~/ez/zppy-interfaces
git status
# On branch main
# nothing to commit, working tree clean
git fetch upstream main
git checkout main
git reset --hard upstream/main
git log
# Good, last commit: fix two ocn plots when no atm data (#46)
# Matches https://github.com/E3SM-Project/zppy-interfaces/commits/main
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n test-zi-main-20260130
conda activate test-zi-main-20260130
python -m pip install .
pytest tests/unit/global_time_series/test_*.py # 10 passed in 27.13s
pytest tests/unit/pcmdi_diags/test_*.py # 7 passed in 22.44s
# zppy itself #################################################################
cd ~/ez/zppy
git status
# On branch automate-main-branch-testing
# nothing to commit, working tree clean
git fetch upstream main
git fetch upstream test-fixes
git checkout -b test-zppy-main-20260130 upstream/test-fixes
git rebase upstream/main
git log --oneline | head -n 3
# a09e1320 Increase mpas_analysis_walltime
# f7cb09e7 Test fixes
# 2b275376 Update ilamb.cfg fixing html generation error (#770)
#
# Matches https://github.com/E3SM-Project/zppy/commits/main + https://github.com/E3SM-Project/zppy/pull/769/commits
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n test-zppy-main-20260130-env
conda activate test-zppy-main-20260130-env
python -m pip install .
pytest tests/test_*.py # 35 passed in 0.64s
# Let's review diffs to the cfgs
# ac7dca27232d63e72c1824f6662d020e9d251698 Add legacy cfgs for zppy v3.1.0 (#762)
git diff ac7dca27232d63e72c1824f6662d020e9d251698 HEAD tests/integration/template_weekly_legacy_3.0.0*
git diff ac7dca27232d63e72c1824f6662d020e9d251698 HEAD tests/integration/template_weekly_comprehensive_*
git diff ac7dca27232d63e72c1824f6662d020e9d251698 HEAD tests/integration/template_weekly_bundles*
# We've changed settings for parameters, but have not introduced new parameters.
# That is, we can keep using the legacy 3.0.0 cfgs for now, since the main cfgs have not diverged
# in a significant way from the 3.1.0 cfgs.
# Edit tests/integration/utils.py
# TEST_SPECIFICS: Dict[str, Any] = {
# # These are custom environment_commands for specific tasks.
# # Never set these to "", because they will print the line
# # `environment_commands = ""` for the corresponding task,
# # thus overriding the value set higher up in the cfg.
# # That is, there will be no environment set.
# # (`environment_commands = ""` only redirects to Unified
# # if specified under the [default] task)
# "diags_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-diags-main-20260130",
# "mpas_analysis_environment_commands": "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh",
# "global_time_series_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-zi-main-20260130",
# "pcmdi_diags_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-zi-main-20260130",
# # This is the environment setup for other tasks.
# # Leave as "" to use the latest Unified environment.
# "environment_commands": "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh",
# # For a complete test, run the set of latest cfgs and at least one set of legacy cfgs
# "cfgs_to_run": [
# "weekly_bundles",
# "weekly_comprehensive_v2",
# "weekly_comprehensive_v3",
# # "weekly_legacy_3.1.0_bundles",
# # "weekly_legacy_3.1.0_comprehensive_v2",
# # "weekly_legacy_3.1.0_comprehensive_v3",
# "weekly_legacy_3.0.0_bundles",
# "weekly_legacy_3.0.0_comprehensive_v2",
# "weekly_legacy_3.0.0_comprehensive_v3",
# ],
# "tasks_to_run": [
# "e3sm_diags",
# "mpas_analysis",
# "global_time_series",
# "ilamb",
# "pcmdi_diags",
# ],
# "unique_id": "zppy_main_branch_test_20260130",
# }
git diff
# Diff looks good
python tests/integration/utils.py
# CFG FILES HAVE BEEN GENERATED FROM TEMPLATES WITH THESE SETTINGS:
# UNIQUE_ID=zppy_main_branch_test_20260130
# diags_environment_commands=source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-diags-main-20260130
# mpas_analysis_environment_commands=source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh
# global_time_series_environment_commands=source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-zi-main-20260130
# pcmdi_diags_environment_commands=source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-zi-main-20260130
# environment_commands=source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh
# Reminder: `environment_commands=''` => the latest E3SM Unified environment will be used
# For reference:
alias sq
# alias sq='sqa -u ac.forsyth2'
alias sqa
# alias sqa='squeue -o "%8u %.7a %.4D %.9P %7i %.2t %.10r %.10M %.10l %j" --sort=P,-t,-p'
sq
# No jobs currently queued
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v3_chrysalis.cfg
sq | wc -l # Fri 1/30 16:29 => 144 - header row = 143 jobs
sq | wc -l # Fri 1/30 17:12 => 29 - header row = 28 jobs
sq | wc -l # Fri 1/30 17:59 => 7 - header row = 6 jobs
sq
# USER ACCOUNT NODE PARTITION JOBID ST REASON TIME TIME_LIMIT NAME
# ac.forsy e3sm 1 compute 1140339 R None 23:31 5:00:00 e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983
# ac.forsy e3sm 1 compute 1140340 R None 28:31 5:00:00 e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1980-1981_vs_1980-1981
# ac.forsy e3sm 1 debug 1140312 PD Dependency 0:00 2:00:00 pcmdi_diags_synthetic_plots_model_vs_obs
# A couple jobs are still running, but they aren't bundles-related, so we can proceed with running bundles part 2.
# The PCMDI diags job won't run because there are no jobs left that would satisfy the dependency.
# Check on bundles status
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/zppy_main_branch_test_20260130/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/zppy_main_branch_test_20260130/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
# Now, run bundles part 2
cd ~/ez/zppy
git status
# On branch test-zppy-main-20260130
# Good, correct branch
# Current conda env: test-zppy-main-20260130-env
# Good, correct environment
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg
sq | wc -l # Fri 1/30 18:08 => 7 - header row = 6 jobs
sq
# USER ACCOUNT NODE PARTITION JOBID ST REASON TIME TIME_LIMIT NAME
# ac.forsy e3sm 1 compute 1140339 R None 25:52 5:00:00 e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983
# ac.forsy e3sm 1 debug 1140312 PD Dependency 0:00 2:00:00 pcmdi_diags_synthetic_plots_model_vs_obs
# ac.forsy e3sm 1 compute 1140468 PD Priority 0:00 7:00:00 ilamb_1985-1986
# ac.forsy e3sm 1 compute 1140469 PD Priority 0:00 7:00:00 bundle3
# ac.forsy e3sm 1 compute 1140470 PD Priority 0:00 7:00:00 ilamb_1985-1986
# ac.forsy e3sm 1 compute 1140471 PD Priority 0:00 7:00:00 bundle3
sq
# USER ACCOUNT NODE PARTITION JOBID ST REASON TIME TIME_LIMIT NAME
# ac.forsy e3sm 1 compute 1140339 R None 34:21 5:00:00 e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983
# ac.forsy e3sm 1 debug 1140312 PD Dependency 0:00 2:00:00 pcmdi_diags_synthetic_plots_model_vs_obs
# Review finished runs
### v2 ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/zppy_main_branch_test_20260130/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/zppy_main_branch_test_20260130/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.status:RUNNING 1140339
# This is the one we have to check back in on.
### v3 ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/zppy_main_branch_test_20260130/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# pcmdi_diags_synthetic_plots_model_vs_obs.status:WAITING 1140312
# pcmdi_diags_variability_modes_atm_model_vs_obs_1985-1994.status:ERROR (14)
cat pcmdi_diags_variability_modes_atm_model_vs_obs_1985-1994.o1140311
# ValueError: assignment destination is read-only
# This actually does appear to match the error in the test automation PR
# (https://github.com/E3SM-Project/zppy/pull/774)
# Since there have been no more commits to zppy/zppy-interfaces,
# it follows that something external must have changed,
# likely with data permissions.
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/zppy_main_branch_test_20260130/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
### bundles ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/zppy_main_branch_test_20260130/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/zppy_main_branch_test_20260130/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
### Check back in
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/zppy_main_branch_test_20260130/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# Good, no errors
scancel -u ac.forsyth2 # Cancel the 1 pcmdi_diags job that will never run.
cd ~/ez/zppy
git status
# On branch test-zppy-main-20260130
# Good, correct branch
# Env: test-zppy-main-20260130-env
# Good, correct env
ls tests/integration/test_*.py
pytest tests/integration/test_bash_generation.py
# 1 passed in 1.99s
pytest tests/integration/test_campaign.py
# 6 passed in 2.33s
pytest tests/integration/test_defaults.py
# 1 passed in 0.37s
pytest tests/integration/test_last_year.py
# 1 passed in 0.31s
pytest tests/integration/test_bundles.py
# 2 passed in 0.28s
salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm
# We need to reactivate conda now that we're on a compute node
lcrc_conda # Activate conda
conda activate test-zppy-main-20260130-env
pytest tests/integration/test_images.py
cat test_images_summary.mdOutput: Complete summary table:
Summary table -- only failing tests
Results analysisMost image tests passed. All other integration tests passed. Image test failures:
|
Beta Was this translation helpful? Give feedback.
1 reply
-
See #777 |
Beta Was this translation helpful? Give feedback.
-
2026-03-05 ThreadWe're testing the What changes are we testing?The last full integration test was run using a dev environment (2026-01-30) and expected results were last updated based on that run (same link, "Updating expected results" section). Note: I added the row on Testing processSet up environmentslcrc_conda # Activate conda
# Set up e3sm_diags env
cd ~/ez/e3sm_diags
git status
# On branch main
# nothing to commit, working tree clean
git fetch upstream main
git checkout main
git reset --hard upstream/main
git log --oneline
# Good, last commit: Refactor mp_partition with new v3 simulator output (#1028)
# Matches https://github.com/E3SM-Project/e3sm_diags/commits/main
rm -rf build
conda clean --all --y
conda env create -f conda-env/dev.yml -n test-diags-main-20260305
conda activate test-diags-main-20260305
python -m pip install .
# Set up zppy-interfaces env
cd ~/ez/zppy-interfaces
git status
# On branch main
# nothing to commit, working tree clean
git fetch upstream main
git checkout main
git reset --hard upstream/main
git log --oneline
# Good, last commit: Add Python 3.14 support (#44)
# Matches https://github.com/E3SM-Project/zppy-interfaces/commits/main
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n test-zi-main-20260305
conda activate test-zi-main-20260305
python -m pip install .
pytest tests/unit/global_time_series/test_*.py # 10 passed in 25.06s
pytest tests/unit/pcmdi_diags/test_*.py # 7 passed in 12.60szppy run, try 1# zppy itself #################################################################
cd ~/ez/zppy
git status
# On branch add-mpas-analysis-model-vs-model
# nothing to commit, working tree clean
git fetch upstream main
git checkout -b test-zppy-main-20260305 upstream/main
git log --oneline
# Good, last commit: Merge pull request #772 from xylar/add-mpas-analysis-model-vs-model
# Matches https://github.com/E3SM-Project/zppy/commits/main
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n test-zppy-main-20260305
conda activate test-zppy-main-20260305
python -m pip install .
pytest tests/test_*.py # 35 passed in 0.64s
# Let's review diffs to the cfgs
# ac7dca27232d63e72c1824f6662d020e9d251698 Add legacy cfgs for zppy v3.1.0 (#762)
git diff ac7dca27232d63e72c1824f6662d020e9d251698 HEAD tests/integration/template_weekly_legacy_3.0.0*
git diff ac7dca27232d63e72c1824f6662d020e9d251698 HEAD tests/integration/template_weekly_comprehensive_*
# tests/integration/template_weekly_comprehensive_v3.cfg has a change
# that goes beyond changing values
git diff ac7dca27232d63e72c1824f6662d020e9d251698 HEAD tests/integration/template_weekly_bundles*
# Because of the change to a cfg, we'll now want to start testing the v3.1.0 legacy cfgs as well.
# For now, we'll test both the v3.1.0 and v3.0.0 legacy cfgs,
# since this is our first test of the former.
# Edit tests/integration/utils.py
# Note that we'll have to update mpas_analysis_environment_commands as well!TEST_SPECIFICS: Dict[str, Any] = {
# These are custom environment_commands for specific tasks.
# Never set these to "", because they will print the line
# `environment_commands = ""` for the corresponding task,
# thus overriding the value set higher up in the cfg.
# That is, there will be no environment set.
# (`environment_commands = ""` only redirects to Unified
# if specified under the [default] task)
"diags_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-diags-main-20260305",
"mpas_analysis_environment_commands": "source /home/ac.xylar/chrysalis/miniforge3/etc/profile.d/conda.sh && conda activate mpas_analysis_dev",
"global_time_series_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-zi-main-20260305",
"pcmdi_diags_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-zi-main-20260305",
# This is the environment setup for other tasks.
# Leave as "" to use the latest Unified environment.
"environment_commands": "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh",
# For a complete test, run the set of latest cfgs and at least one set of legacy cfgs
"cfgs_to_run": [
"weekly_bundles",
"weekly_comprehensive_v2",
"weekly_comprehensive_v3",
"weekly_legacy_3.1.0_bundles",
"weekly_legacy_3.1.0_comprehensive_v2",
"weekly_legacy_3.1.0_comprehensive_v3",
"weekly_legacy_3.0.0_bundles",
"weekly_legacy_3.0.0_comprehensive_v2",
"weekly_legacy_3.0.0_comprehensive_v3",
],
"tasks_to_run": [
"e3sm_diags",
"mpas_analysis",
"global_time_series",
"ilamb",
"pcmdi_diags",
],
"unique_id": "zppy_main_branch_test_20260305",
}git diff
# Diff looks good
python tests/integration/utils.py
# CFG FILES HAVE BEEN GENERATED FROM TEMPLATES WITH THESE SETTINGS:
# UNIQUE_ID=zppy_main_branch_test_20260305
# diags_environment_commands=source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-diags-main-20260305
# mpas_analysis_environment_commands=source /home/ac.xylar/chrysalis/miniforge3/etc/profile.d/conda.sh && conda activate mpas_analysis_dev
# global_time_series_environment_commands=source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-zi-main-20260305
# pcmdi_diags_environment_commands=source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-zi-main-20260305
# environment_commands=source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh
# Reminder: `environment_commands=''` => the latest E3SM Unified environment will be used
# For reference:
alias sq
# alias sq='sqa -u ac.forsyth2'
alias sqa
# alias sqa='squeue -o "%8u %.7a %.4D %.9P %7i %.2t %.10r %.10M %.10l %j" --sort=P,-t,-p'
sq
# No jobs currently queued
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
# Error
# zppy.utils.DependencySkipError: ...skipping because of dependency status file missing
# /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/zppy_main_branch_test_20260305/v3.LR.historical_0051/post/scripts/mpas_analysis_ts_1985-1989_climo_1985-1989.status
scancel -u ac.forsyth2zppy run, try 2# This will unfortunatley require functionality changes.
git add -A
pre-commit run --all-files
git add -A
git commit -m "Integration testing part 1"
# Now, make functionality changes
git add -A
pre-commit run --all-files # Iterate until passing
git add -A
git diff --staged # Review diff
git commit -m "Add mpas_analysis_subsection parameter to global_time_series"
python -m pip install .
pytest tests/test_*.py # 1 failed, 34 passed in 0.63s
# Make further changes
pre-commit run --all-files
python -m pip install .
pytest tests/test_*.py # 35 passed in 0.14s
git add -A
pre-commit run --all-files
git commit --amend --no-edit
git log --oneline | head -n 3
# 83bc97c6 Add mpas_analysis_subsection parameter to global_time_series
# 9d43b3dc Integration testing part 1
# c9bd1817 Merge pull request #772 from xylar/add-mpas-analysis-model-vs-model
# Edit tests/integration/utils.py
# "unique_id": "zppy_main_branch_test_20260305_try2",
python tests/integration/utils.py
# CFG FILES HAVE BEEN GENERATED FROM TEMPLATES WITH THESE SETTINGS:
# UNIQUE_ID=zppy_main_branch_test_20260305_try2
# diags_environment_commands=source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-diags-main-20260305
# mpas_analysis_environment_commands=source /home/ac.xylar/chrysalis/miniforge3/etc/profile.d/conda.sh && conda activate mpas_analysis_dev
# global_time_series_environment_commands=source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-zi-main-20260305
# pcmdi_diags_environment_commands=source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-zi-main-20260305
# environment_commands=source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh
# Reminder: `environment_commands=''` => the latest E3SM Unified environment will be used
# We'll start with the cfg that gave the error:
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v3_chrysalis.cfg
sq | wc -l # Thu 3/5 18:02 => 227 - header row = 226 jobs
sq | wc -l # Thu 3/5 18:40 => 57 - header row = 56 jobsBundles part 2# Picking back up 2026-03-06 ##################################################
cd ~/ez/zppy
git status
# On branch test-zppy-main-20260305
lcrc_conda
conda activate test-zppy-main-20260305
python -m pip install .
sq | wc -l # Fri 3/6 9:50 => 1 - header row = 0 jobs
# Check on bundles status
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/zppy_main_branch_test_20260305_try2/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# bundle1.status:ERROR
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1985-1986.status:ERROR (11)
emacs e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1985-1986.bash
# rsync -a --delete ${results_dir} ${web_dir}/
# if [ $? != 0 ]; then
# cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/zppy_main_branch_test_20260305_try2/v3\
# .LR.historical_0051/post/scripts
# echo 'ERROR (11)' > e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1985-1986.status
# exit 11
# fi
# Looks like the rsync failed.
grep -in error bundle1.o1183029 | wc -l
# 1878 lines mentioning errors, all appear related to the polar driver
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_bundles_output/zppy_main_branch_test_20260305_try2/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/zppy_main_branch_test_20260305_try2/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
# Now, run bundles part 2 for the working cfgs.
cd ~/ez/zppy
git status
# On branch test-zppy-main-20260305
# Good, correct branch
# Current conda env: test-zppy-main-20260305
# Good, correct environment
# Don't run this one, because bundle1 failed
# zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg
sq | wc -l # Fri 3/6 9:58 => 5 - header row = 4 jobs
sq | wc -l # Fri 3/6 10:08 => 1 - header row = 0 jobsReview finished runs### v2 ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/zppy_main_branch_test_20260305_try2/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_comprehensive_v2_output/zppy_main_branch_test_20260305_try2/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.status:RUNNING 1183088
emacs e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.o1183088
# Hit time limit
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/zppy_main_branch_test_20260305_try2/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.status:RUNNING 1183168
emacs e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.o1183168
# Hit time limit
### v3 ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/zppy_main_branch_test_20260305_try2/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# global_time_series_all_lnd_var_viewer_1985-1995.status:ERROR (8)
emacs global_time_series_all_lnd_var_viewer_1985-1995.o1183021
# ===== COPY OCEAN RESULTS TO CASE DIRECTORY =====
# No ocean results directory found at /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/zppy_main_branch_test_20260305_try2/v3.LR.historical_0051/post/scripts/glob\
# al_time_series_all_lnd_var_viewer_1985-1995_results/ocn
# ===== COPY FILES TO WEB SERVER =====
# file has vanished: "/lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v3_www/zppy_main_branch_test_20260305_try2/v3.LR.historical_0051/global\
# _time_series/global_time_series_all_lnd_var_viewer_1985-1995_results/.v3.LR.historical_0051_n_lnd_FPSN_WJ.png.0ZOC9K"
# rsync warning: some files vanished before they could be transferred (code 24) at main.c(1189) [sender=3.1.3]
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_comprehensive_v3_output/zppy_main_branch_test_20260305_try2/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/zppy_main_branch_test_20260305_try2/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
### bundles ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/zppy_main_branch_test_20260305_try2/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# bundle1.status:ERROR
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1985-1986.status:ERROR (11)
# (These were explored above)
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_bundles_output/zppy_main_branch_test_20260305_try2/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/zppy_main_branch_test_20260305_try2/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
# Before proceeding, let's summarize what results we were able to generate.
Python testscd ~/ez/zppy
git status
# On branch test-zppy-main-20260305
# Good, correct branch
# Env: test-zppy-main-20260305
# Good, correct env
ls tests/integration/test_*.py
pytest tests/integration/test_bash_generation.py
# 1 failed in 3.24s
# Errors look to be expected based on https://github.com/E3SM-Project/zppy/pull/772
pytest tests/integration/test_campaign.py
# 6 failed in 2.35s
# Errors look to be expected based on https://github.com/E3SM-Project/zppy/pull/772
pytest tests/integration/test_defaults.py
# 1 failed in 0.48s
# Errors look to be expected based on https://github.com/E3SM-Project/zppy/pull/772
pytest tests/integration/test_last_year.py
# 1 passed in 0.32s
pytest tests/integration/test_bundles.py
# 2 passed in 1.03s
salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm
# We need to reactivate conda now that we're on a compute node
lcrc_conda # Activate conda
conda activate test-zppy-main-20260305
pytest tests/integration/test_images.pyOutput: We should have turned off testing for 3.1.0 since we have no expected results for it. exit # Exit compute node
emacs tests/integration/utils.py
# "cfgs_to_run": [
# "weekly_bundles",
# "weekly_comprehensive_v2",
# "weekly_comprehensive_v3",
# # "weekly_legacy_3.1.0_bundles",
# # "weekly_legacy_3.1.0_comprehensive_v2",
# # "weekly_legacy_3.1.0_comprehensive_v3",
# "weekly_legacy_3.0.0_bundles",
# "weekly_legacy_3.0.0_comprehensive_v2",
# "weekly_legacy_3.0.0_comprehensive_v3",
# ],
python tests/integration/utils.py
salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm
# We need to reactivate conda now that we're on a compute node
lcrc_conda # Activate conda
conda activate test-zppy-main-20260305
pytest tests/integration/test_images.py
cat test_images_summary.mdOutput: Complete summary table:
Summary table -- only failing tests
Reviewing errors in output directoriesv2 time limitsRe: I suppose these just need the time limit increased. v3 global_time_series failureRe: cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/zppy_main_branch_test_20260305_try2/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# global_time_series_all_lnd_var_viewer_1985-1995.status:ERROR (8)
tail -n 6 global_time_series_all_lnd_var_viewer_1985-1995.o1183021
# No ocean results directory found at /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/zppy_main_branch_test_20260305_try2/v3.LR.historical_0051/post/scripts/global_time_series_all_lnd_var_viewer_1985-1995_results/ocn
# ===== COPY FILES TO WEB SERVER =====
# file has vanished: "/lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v3_www/zppy_main_branch_test_20260305_try2/v3.LR.historical_0051/global_time_series/global_time_series_all_lnd_var_viewer_1985-1995_results/.v3.LR.historical_0051_n_lnd_FPSN_WJ.png.0ZOC9K"
# rsync warning: some files vanished before they could be transferred (code 24) at main.c(1189) [sender=3.1.3]
ls /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/zppy_main_branch_test_20260305_try2/v3.LR.historical_0051/post/scripts/global_time_series_all_lnd_var_viewer_1985-1995_results/ocn
# No such file or directory
grep -in error global_time_series_all_lnd_var_viewer_1985-1995.o1183021
# 61:2026-03-05 21:53:20,571 [ERROR]: utils.py(process_variable:220) >> Failed to process variable CH4PROD: Units don't match: gC/m2/s vs gC/m^2/s
# 62:2026-03-05 21:53:20,571 [ERROR]: utils.py(set_var_parallel_with_plots:765) >> Failed CH4PROD: Units don't match: gC/m2/s vs gC/m^2/s
# 522:2026-03-05 21:55:04,486 [ERROR]: utils.py(process_variable:220) >> Failed to process variable SABG_PEN: Units don't match: watt/m^2 vs W/m^2
# 523:2026-03-05 21:55:04,486 [ERROR]: utils.py(set_var_parallel_with_plots:765) >> Failed SABG_PEN: Units don't match: watt/m^2 vs W/m^2
# 733:2026-03-05 21:55:45,749 [ERROR]: utils.py(process_variable:220) >> Failed to process variable QSNWCPICE_NODYNLNDUSE: Units don't match: mm H2O/s vs mm/s
# 734:2026-03-05 21:55:45,749 [ERROR]: utils.py(set_var_parallel_with_plots:765) >> Failed QSNWCPICE_NODYNLNDUSE: Units don't match: mm H2O/s vs mm/s
# 1240:2026-03-05 21:57:37,886 [ERROR]: utils.py(process_variable:220) >> Failed to process variable FCOV: Units don't match: unitless vs proportion
# 1241:2026-03-05 21:57:37,886 [ERROR]: utils.py(set_var_parallel_with_plots:765) >> Failed FCOV: Units don't match: unitless vs proportion
# 1462:2026-03-05 21:58:27,099 [ERROR]: plots_component.py(process_data:72) >> globalAnnual could not be computed for these variables: ['CH4PROD', 'SABG_PEN', 'QSNWCPICE_NODYNLNDUSE', 'FCOV']
grep -in ocn global_time_series_all_lnd_var_viewer_1985-1995.o1183021
# 1467:2026-03-05 21:58:27,099 [INFO]: driver.py(run:78) >> Skipping original plotting for ocn - plots already generated during processing
# 1473:2026-03-05 21:58:27,100 [INFO]: driver.py(run:78) >> Skipping original plotting for ocn - plots already generated during processing
# 1479:2026-03-05 21:58:27,100 [INFO]: driver.py(run:78) >> Skipping original plotting for ocn - plots already generated during processing
# 1506:No ocean results directory found at /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/zppy_main_branch_test_20260305_try2/v3.LR.historical_0051/post/scripts/global_time_series_all_lnd_var_viewer_1985-1995_results/ocn
grep -in ocean global_time_series_all_lnd_var_viewer_1985-1995.o1183021
# 1504:===== COPY OCEAN RESULTS TO CASE DIRECTORY =====
# 1506:No ocean results directory found at /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/zppy_main_branch_test_20260305_try2/v3.LR.historical_0051/post/scripts/global_time_series_all_lnd_var_viewer_1985-1995_results/ocn
grep -in mpas global_time_series_all_lnd_var_viewer_1985-1995.settings
# 32: 'mpas_analysis_subsection': 'original',
grep -in mpas global_time_series_all_lnd_var_viewer_1985-1995.o1183021
# No results
grep -in mpas global_time_series_all_lnd_var_viewer_1985-1995.bash
# No results
grep -in ocn global_time_series_all_lnd_var_viewer_1985-1995.bash
# 39:zi-global-time-series --use_ocn False --input /lcrc/group/e3sm2/ac.wlin//E3SMv3/v3.LR.historical_0051 --input_subdir archive/ocn/hist --moc_file None --case_dir /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/zppy_main_branch_test_20260305_try2/v3.LR.historical_0051 --experiment_name v3.LR.historical_0051 --figstr v3.LR.historical_0051 --color Blue --ts_num_years 5 --plots_original None --plots_atm None --plots_ice None --plots_lnd all --plots_ocn None --nrows 4 --ncols 2 --results_dir ${results_dir} --regions glb,n,s --make_viewer True --start_yr 1985 --end_yr 1995
# 56:if [ -d "${results_dir_absolute_path}/ocn" ]; then
# 57: mkdir -p ${case_dir}/post/ocn
# 58: rsync -av ${results_dir_absolute_path}/ocn/ ${case_dir}/post/ocn/
# 64: echo "Ocean results copied to ${case_dir}/post/ocn/"
# 66: echo "No ocean results directory found at ${results_dir_absolute_path}/ocn"
# 86:# Copy files (excluding ocn directory)
# 87:rsync -a --delete --exclude='ocn' ${results_dir_absolute_path} ${top_level}I imagine this must be related to the changes to bundles (v3) rsync failureRe: cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/zppy_main_branch_test_20260305_try2/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# bundle1.status:ERROR
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1985-1986.status:ERROR (11)
grep -in e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1985-1986 bundle1.o1183029
# 1397:=== e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1985-1986.bash ===
grep -n "=== " bundle1.o1183029 | tail -n 6
# 1397:=== e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1985-1986.bash ===
# 1399:===== RUN E3SM DIAGS =====
# 15601:===== COPY FILES TO WEB SERVER =====
# 15605:=== e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1987-1988.bash ===
# 15607:===== RUN E3SM DIAGS =====
# 29806:===== COPY FILES TO WEB SERVER =====
sed -n '1397,15605p' bundle1.o1183029 | grep -in rsync
# 14208:rsync warning: some files vanished before they could be transferred (code 24) at main.c(1338) [sender=3.4.1]
sed -n '1397,15605p' bundle1.o1183029 | grep -in error | wc -l
# 936 matches
sed -n '1397,15605p' bundle1.o1183029 | grep -in error | grep -in polar | wc -l
# 312 matches
sed -n '1397,15605p' bundle1.o1183029 | grep -in "No file found for" | wc -l
# 604 matchesThese matches are typically of the form Where Starting fixesI started fixes in #788. There will need to be more fixes added to that. Set up fixescd ~/ez/zppy
git status
# On branch test-zppy-main-20260305
git add -A # Make sure all new files are included
pre-commit run --all-files # Iterate until passing
git add -A # Make sure any changes from pre-commit make it into the commit
git commit -m "Integration testing part 2"
git log --oneline | head -n 4
# df66f616 Integration testing part 2
# 83bc97c6 Add mpas_analysis_subsection parameter to global_time_series
# 9d43b3dc Integration testing part 1
# c9bd1817 Merge pull request #772 from xylar/add-mpas-analysis-model-vs-model
git fetch upstream main
git checkout -b fix-integration-tests upstream/main
git cherry-pick 83bc97c6
git log --oneline | head -n 2
# 4f466ab0 Add mpas_analysis_subsection parameter to global_time_series
# c9bd1817 Merge pull request #772 from xylar/add-mpas-analysis-model-vs-model
git push upstream fix-integration-testsResults analysisFor now, we can hold off on reviewing the missing and mismatched images in the summary table above. I'd like to get the Question on Question on [[ original ]]
# Keep configuration from original mpas_analysis tests for global_time_series to use
ts_years = "1985-1989", "1985-1995",
climo_years = "1985-1989", "1990-1995",
enso_years = "1985-1989", "1990-1995",I then added a parameter to
|
Beta Was this translation helpful? Give feedback.
1 reply
-
@forsyth2, I don't think that's likely. The different
I don't think any of your three suggested approaches is quite right but your solution in #788 is close to what we want. I'll comment there as well. What I would do is similar to #788 but instead of adding a new Chrysalis seems to be down today. I'll see if I can implement an alternative to #788 that does this when the machine comes back. |
Beta Was this translation helpful? Give feedback.
-
|
@forsyth2 regarding the bundle |
Beta Was this translation helpful? Give feedback.
0 replies
-
2026-03-16 ThreadWe're testing the What changes are we testing?The last full integration test was run using a dev environment ([2026-03-05/06])(#634). However, expected results were not updated on that run, due to multiple errors encountered during runtime. The latest expected results are based on the (2026-01-30) run.
Note: as with the previous test, I added the row on Testing processSet up environmentslcrc_conda # Activate conda
lcrc-quota # Have enough space
# Set up e3sm_diags env
cd ~/ez/e3sm_diags
git status
# On branch main
# nothing to commit, working tree clean
git fetch upstream main
git checkout main
git reset --hard upstream/main
git log --oneline | head -n 1
# ade8e26d Add repository-level custom AI instruction files (Copilot, Agents, Claude) (#1039)
# Good, matches https://github.com/E3SM-Project/e3sm_diags/commits/main
rm -rf build
conda clean --all --y
conda env create -f conda-env/dev.yml -n test-diags-main-20260316 -vvv > conda.log 2>&1
conda activate test-diags-main-20260316
python -m pip install .
rm conda.log
# Set up zppy-interfaces env
cd ~/ez/zppy-interfaces
git status
# On branch main
# nothing to commit, working tree clean
git fetch upstream main
git checkout main
git reset --hard upstream/main
git log --oneline | head -n 1
# 02638f4 Add Python 3.14 support (#44)
# Good, matches https://github.com/E3SM-Project/zppy-interfaces/commits/main
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n test-zi-main-20260316 -vvv > conda.log 2>&1
conda activate test-zi-main-20260316
python -m pip install .
rm conda.log
pytest tests/unit/global_time_series/test_*.py # 10 passed in 22.31s
pytest tests/unit/pcmdi_diags/test_*.py # 7 passed in 10.38szppy run# zppy itself #################################################################
cd ~/ez/zppy
git status
# On branch fix-integration-tests
# nothing to commit, working tree clean
git fetch upstream main
git checkout -b test-zppy-main-20260316 upstream/main
git log --oneline | head -n 1
# 3f541bcb Updates for supporting running zppy on Dane (#789)
# Good, matches https://github.com/E3SM-Project/zppy/commits/main
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n test-zppy-main-20260316 -vvv > conda.log 2>&1
conda activate test-zppy-main-20260316
python -m pip install .
rm conda.log
# Let's get the commit from https://github.com/E3SM-Project/zppy/pull/788
git fetch upstream fix-integration-tests
git cherry-pick 4f466ab091e42f57f1e86d4a69ea6d9232a56491
git log --oneline | head -n 2
# cce31f35 Add mpas_analysis_subsection parameter to global_time_series
# 3f541bcb Updates for supporting running zppy on Dane (#789)
# We also need to update the time limits, based on the 03/05 results
# We already have the following set though:
# "diags_walltime": "5:00:00",
# That's already quite high.
# Let's rerun with the same walltime and see if the error repeats.
python -m pip install .
pytest tests/test_*.py # 35 passed in 1.21s
# Edit tests/integration/utils.py
# Note that we'll have to update mpas_analysis_environment_commands as well!TEST_SPECIFICS: Dict[str, Any] = {
# These are custom environment_commands for specific tasks.
# Never set these to "", because they will print the line
# `environment_commands = ""` for the corresponding task,
# thus overriding the value set higher up in the cfg.
# That is, there will be no environment set.
# (`environment_commands = ""` only redirects to Unified
# if specified under the [default] task)
"diags_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-diags-main-20260316",
"mpas_analysis_environment_commands": "source /home/ac.xylar/chrysalis/miniforge3/etc/profile.d/conda.sh && conda activate mpas_analysis_dev",
"global_time_series_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-zi-main-20260316",
"pcmdi_diags_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-zi-main-20260316",
# This is the environment setup for other tasks.
# Leave as "" to use the latest Unified environment.
"environment_commands": "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh",
# For a complete test, run the set of latest cfgs and at least one set of legacy cfgs
"cfgs_to_run": [
"weekly_bundles",
"weekly_comprehensive_v2",
"weekly_comprehensive_v3",
"weekly_legacy_3.1.0_bundles",
"weekly_legacy_3.1.0_comprehensive_v2",
"weekly_legacy_3.1.0_comprehensive_v3",
"weekly_legacy_3.0.0_bundles",
"weekly_legacy_3.0.0_comprehensive_v2",
"weekly_legacy_3.0.0_comprehensive_v3",
],
"tasks_to_run": [
"e3sm_diags",
"mpas_analysis",
"global_time_series",
"ilamb",
"pcmdi_diags",
],
"unique_id": "zppy_main_branch_test_20260316_try2",
}git diff
# Diff looks good
python tests/integration/utils.py
# CFG FILES HAVE BEEN GENERATED FROM TEMPLATES WITH THESE SETTINGS:
# UNIQUE_ID=zppy_main_branch_test_20260316_try2
# diags_environment_commands=source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-diags-main-20260316
# mpas_analysis_environment_commands=source /home/ac.xylar/chrysalis/miniforge3/etc/profile.d/conda.sh && conda activate mpas_analysis_dev
# global_time_series_environment_commands=source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-zi-main-20260316
# pcmdi_diags_environment_commands=source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-zi-main-20260316
# environment_commands=source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh
# Reminder: `environment_commands=''` => the latest E3SM Unified environment will be used
# For reference:
alias sq
# alias sq='sqa -u ac.forsyth2'
alias sqa
# alias sqa='squeue -o "%8u %.7a %.4D %.9P %7i %.2t %.10r %.10M %.10l %j" --sort=P,-t,-p'
sq
# No jobs currently queued
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
# Works now that we added the commit from #788
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v3_chrysalis.cfg
sq | wc -l # Mon 3/16 11:15 => 222 - header row = 221 jobs
sq | wc -l # Mon 3/16 11:45 => 58 - header row = 57 jobs
sq | wc -l # Mon 3/16 12:22 => 23 - header row = 22 jobs
sq | wc -l # Mon 3/16 12:34 => 15 - header row = 14 jobs
sq
# All remaining jobs are waiting for dependency.
# That means we had failures.Bundles part 2# Check on bundles status
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/zppy_main_branch_test_20260316_try2/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors, unlike the 3/5-6 run. No rsync failure.
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_bundles_output/zppy_main_branch_test_20260316_try2/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/zppy_main_branch_test_20260316_try2/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
# Now, run bundles part 2 for the working cfgs.
cd ~/ez/zppy
git status
# On branch test-zppy-main-20260316
# Good, correct branch
# Current conda env: test-zppy-main-20260316
# Good, correct environment
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg
sq | wc -l # Mon 3/16 12:38 => 21 - header row = 20 jobs (+6 jobs)Review finished runs### v2 ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/zppy_main_branch_test_20260316_try2/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# Error, unlike the 3/5-6 run
# global_time_series_1980-1990.status:WAITING 1195205
# mpas_analysis_ts_1980-1984_climo_1980-1984.status:ERROR (1)
# mpas_analysis_ts_1980-1990_climo_1985-1990.status:WAITING 1195204
emacs mpas_analysis_ts_1980-1984_climo_1980-1984.o1195203shows: cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_comprehensive_v2_output/zppy_main_branch_test_20260316_try2/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# Good, no errors, unlike the 3/5-6 run. No time limit failure.
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/zppy_main_branch_test_20260316_try2/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# No time limit error, unlike the 3/5-6 run.
# However, there is a new error:
# global_time_series_1980-1990.status:WAITING 1195319
# mpas_analysis_ts_1980-1984_climo_1980-1984.status:ERROR (1)
# mpas_analysis_ts_1980-1990_climo_1985-1990.status:WAITING 1195318
tail mpas_analysis_ts_1980-1984_climo_1980-1984.o1195317
# Same issue as for non-legacy v2 test.
### v3 ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/zppy_main_branch_test_20260316_try2/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# No global_time_series error, unlike the 3/5-6 run.
# However, this is because we can't even get to that point now:
# global_time_series_classic_pdf_both_1985-1995.status:WAITING 1195164
# global_time_series_classic_pdf_original_1985-1995.status:WAITING 1195165
# global_time_series_viewer_both_1985-1995.status:WAITING 1195161
# global_time_series_viewer_original_1985-1995.status:WAITING 1195162
# mpas_analysis_mvm_ts_1985-1995_climo_1990-1995_vs_ref_ts_1985-1989_climo_1985-1989.status:WAITING 1195160
# mpas_analysis_original_ts_1985-1989_climo_1985-1989.status:ERROR (1)
# mpas_analysis_original_ts_1985-1995_climo_1990-1995.status:WAITING 1195157
# mpas_analysis_reference_ts_1985-1989_climo_1985-1989.status:WAITING 1195158
# mpas_analysis_test_ts_1985-1995_climo_1990-1995.status:WAITING 1195159
tail mpas_analysis_original_ts_1985-1989_climo_1985-1989.o1195156
# Same issue as for non-legacy v2 test, legacy 3.0.0 v2 test
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_comprehensive_v3_output/zppy_main_branch_test_20260316_try2/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/zppy_main_branch_test_20260316_try2/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
# global_time_series_1985-1995.status:WAITING 1195349
# mpas_analysis_ts_1985-1989_climo_1985-1989.status:ERROR (1)
# mpas_analysis_ts_1985-1995_climo_1990-1995.status:WAITING 1195348
tail mpas_analysis_ts_1985-1989_climo_1985-1989.o1195347
# Same issue as for non-legacy v2 test, legacy 3.0.0 v2 test
### bundles ###
sq | wc -l # Mon 3/16 12:38 => 15 - header row = 14 jobs
sq
# Back to all the jobs that need dependencies still.
scancel -u ac.forsyth2
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/zppy_main_branch_test_20260316_try2/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors, unlike the 3/5-6 run.
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_bundles_output/zppy_main_branch_test_20260316_try2/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Errors appear, unlike the 3/5-6 run.
# bundle3.status:ERROR
# e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1987-1988_vs_1985-1986.status:ERROR (11)
grep -in e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1987-1988_vs_1985-1986 bundle3.o1195359
# 43:=== e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1987-1988_vs_1985-1986.bash ===
grep -n "=== " bundle3.o1195359
# 1:=== ts_rof_monthly_1985-1986-0002.bash ===
# 22:=== ts_rof_monthly_1987-1988-0002.bash ===
# 43:=== e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1987-1988_vs_1985-1986.bash ===
# 45:===== RUN E3SM DIAGS =====
# 6577:===== COPY FILES TO WEB SERVER =====
grep -in error bundle3.o1195359 | wc -l
# 260 matches
grep -in "OSError: The dataset file has no matching source variables for" bundle3.o1195359 | wc -l
# 40 matches
grep -in "KeyError: 'units'" bundle3.o1195359 | wc -l
# 70 matches
tail -n 5 bundle3.o1195359
# file has vanished: "/lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_bundles_www/zppy_main_branch_test_20260316_try2/v3.LR.historical_0051/e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/polar/model_vs_model/.v3.LR.historical_0051-SST-SON-polar_S.png.rC1gVl"
# rsync warning: some files vanished before they could be transferred (code 24) at main.c(1338) [sender=3.4.1]
# ==============================================
# Elapsed time: 311 seconds
# ==============================================
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/zppy_main_branch_test_20260316_try2/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
# There are too many errors to make it worthwhile proceeding with the Python tests.
# Let's summarize what results we were able to generate.
None of these errors make sense to me. I'm using Xylar's dev environment for MPAS-Analysis, which I also used for the 3/5-6 test, but this time it fails to load @xylar @chengzhuzhang Any thoughts? |
Beta Was this translation helpful? Give feedback.
6 replies
-
|
@forsyth2 thanks for checking. It is interesting that conda picked up 3.13 based on the dev.yaml file for the latest run 3/15. This explained why most e3sm_diags runs completed. Could you check the python version for the 3/5-6 run? |
Beta Was this translation helpful? Give feedback.
-
|
@chengzhuzhang Unfortunately, I can't check The alternative of re-creating that environment wouldn't be particularly effective because there was no functionality change since that run (only E3SM-Project/e3sm_diags#1039). In that sense, this is already a recreation of that environment. |
Beta Was this translation helpful? Give feedback.
-
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/zppy_main_branch_test_20260316_try2/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
emacs mpas_analysis_ts_1980-1984_climo_1980-1984.o1195203
source /home/ac.xylar/chrysalis/miniforge3/etc/profile.d/conda.sh && conda activate mpas_analysis_dev
pythonfrom mpas_analysis.__main__ import main@xylar Have you changed anything with your development environment for On a related note, I tried to make more progress on #788. I'm not sure if you've had a chance to look at that any further. Thanks! |
Beta Was this translation helpful? Give feedback.
-
@forsyth2, I didn't change anything. Something must have gone wrong in the testing itself. |
Beta Was this translation helpful? Give feedback.
-
|
@forsyth2, I take that back. I didn't realize you have been using a conda environment out of my personal |
Beta Was this translation helpful? Give feedback.
-
2026-03-25 ThreadWe're testing the What changes are we testing?The latest expected results are based on the (2026-01-30) run. Since then, we've also run:
Reminder: for the Testing processSet up environmentslcrc_conda # Activate conda
lcrc-quota # Have enough space
# Set up e3sm_diags env
cd ~/ez/e3sm_diags
git status
# On branch main
# nothing to commit, working tree clean
git fetch upstream main
git reset --hard upstream/main
git log --oneline | head -n 1
# 20e72b2e Fixes for numpy 2.4, eamxx var derivation, AOD vars without units attribute. (#1041)
# Good, matches https://github.com/E3SM-Project/e3sm_diags/commits/main
rm -rf build
conda clean --all --y
conda env create -f conda-env/dev.yml -n test-diags-main-20260325
conda activate test-diags-main-20260325
python -m pip install .
# Set up mpas_analysis env for the first time
cd ~/ez
# Get clone address from https://github.com/MPAS-Dev/MPAS-Analysis
git clone git@github.com:MPAS-Dev/MPAS-Analysis.git
cd MPAS-Analysis/
git status
# On branch develop
# nothing to commit, working tree clean
git remote -v
git remote add upstream git@github.com:MPAS-Dev/MPAS-Analysis.git
git fetch upstream develop
git reset --hard upstream/develop
git log --oneline | head -n 1
# 21bac98457 Merge pull request #1138 from xylar/fix-control-global-subtitle-spacing
# Good, matches https://github.com/MPAS-Dev/MPAS-Analysis/commits/develop/
rm -rf build
conda clean --all --y
head -n 2 dev-spec.txt
# This file may be used to create an environment using:
# $ conda create --name <env> --file <this file>
conda create --name test-mpas-analysis-develop-20260325 --file dev-spec.txt
conda activate test-mpas-analysis-develop-20260325
python -m pip install .
# Set up zppy-interfaces env
cd ~/ez/zppy-interfaces
git status
# On branch main
# nothing to commit, working tree clean
git fetch upstream main
git reset --hard upstream/main
git log --oneline | head -n 1
# 02638f4 Add Python 3.14 support (#44)
# Good, matches https://github.com/E3SM-Project/zppy-interfaces/commits/main
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n test-zi-main-20260325
conda activate test-zi-main-20260325
python -m pip install .
pytest tests/unit/global_time_series/test_*.py # 10 passed in 23.94s
pytest tests/unit/pcmdi_diags/test_*.py # 7 passed in 7.15szppy run# zppy itself #################################################################
cd ~/ez/zppy
git status
# On branch add-livvkit
# nothing to commit, working tree clean
git fetch upstream main
git checkout -b test-zppy-main-20260325 upstream/main
git log --oneline | head -n 1
# 0f8964f1 Add precip_pdf and mp_partition from e3sm diags (#787)
# Good, matches https://github.com/E3SM-Project/zppy/commits/main
# Let's get the commit from https://github.com/E3SM-Project/zppy/pull/788
git fetch upstream fix-integration-tests
git cherry-pick 4f466ab091e42f57f1e86d4a69ea6d9232a56491
git log --oneline | head -n 2
# 65064ef3 Add mpas_analysis_subsection parameter to global_time_series
# 0f8964f1 Add precip_pdf and mp_partition from e3sm diags (#787)
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n test-zppy-main-20260325
conda activate test-zppy-main-20260325
python -m pip install .
pytest tests/test_*.py # 35 passed in 0.45s
# Edit tests/integration/utils.pyTEST_SPECIFICS: Dict[str, Any] = {
# These are custom environment_commands for specific tasks.
# Never set these to "", because they will print the line
# `environment_commands = ""` for the corresponding task,
# thus overriding the value set higher up in the cfg.
# That is, there will be no environment set.
# (`environment_commands = ""` only redirects to Unified
# if specified under the [default] task)
"diags_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-diags-main-20260325",
"mpas_analysis_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-mpas-analysis-develop-20260325",
"global_time_series_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-zi-main-20260325",
"pcmdi_diags_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-zi-main-20260325",
# This is the environment setup for other tasks.
# Leave as "" to use the latest Unified environment.
"environment_commands": "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh",
# For a complete test, run the set of latest cfgs and at least one set of legacy cfgs
"cfgs_to_run": [
"weekly_bundles",
"weekly_comprehensive_v2",
"weekly_comprehensive_v3",
"weekly_legacy_3.1.0_bundles",
"weekly_legacy_3.1.0_comprehensive_v2",
"weekly_legacy_3.1.0_comprehensive_v3",
"weekly_legacy_3.0.0_bundles",
"weekly_legacy_3.0.0_comprehensive_v2",
"weekly_legacy_3.0.0_comprehensive_v3",
],
"tasks_to_run": [
"e3sm_diags",
"mpas_analysis",
"global_time_series",
"ilamb",
"pcmdi_diags",
],
"unique_id": "zppy_main_branch_test_20260325",
}git diff # Diff looks good
python tests/integration/utils.py
# CFG FILES HAVE BEEN GENERATED FROM TEMPLATES WITH THESE SETTINGS:
# UNIQUE_ID=zppy_main_branch_test_20260325
# diags_environment_commands=source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-diags-main-20260325
# mpas_analysis_environment_commands=source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-mpas-analysis-develop-20260325
# global_time_series_environment_commands=source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-zi-main-20260325
# pcmdi_diags_environment_commands=source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-zi-main-20260325
# environment_commands=source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh
# Reminder: `environment_commands=''` => the latest E3SM Unified environment will be used
# For reference:
alias sq
# alias sq='sqa -u ac.forsyth2'
alias sqa
# alias sqa='squeue -o "%8u %.7a %.4D %.9P %7i %.2t %.10r %.10M %.10l %j" --sort=P,-t,-p'
sq
# No jobs currently queued
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v3_chrysalis.cfg
sq | wc -l # Wed 3/25 11:09 => 227 - header row = 221 jobs
sq | wc -l # Wed 3/25 14:02 => 1 - header row = 0 jobsBundles part 2# Check on bundles status
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/zppy_main_branch_test_20260325/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_bundles_output/zppy_main_branch_test_20260325/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/zppy_main_branch_test_20260325/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors
# Now, run bundles part 2 for the working cfgs.
cd ~/ez/zppy
git status
# On branch test-zppy-main-20260325
# Good, correct branch
# Current conda env: test-zppy-main-20260325
# Good, correct environment
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg
sq | wc -l # Wed 3/25 14:03 => 7 - header row = 6 jobsReview finished runs### v2 ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/zppy_main_branch_test_20260325/v2.LR.historical_0201/post/scripts
grep -v "OK" *status # Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_comprehensive_v2_output/zppy_main_branch_test_20260325/v2.LR.historical_0201/post/scripts
grep -v "OK" *status # Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/zppy_main_branch_test_20260325/v2.LR.historical_0201/post/scripts
grep -v "OK" *status # Good, no errors
### v3 ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/zppy_main_branch_test_20260325/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_comprehensive_v3_output/zppy_main_branch_test_20260325/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/zppy_main_branch_test_20260325/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors
### bundles ###
sq | wc -l # Wed 3/25 14:13 => 1 - header row = 0 jobs
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/zppy_main_branch_test_20260325/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# bundle3.status:ERROR
# e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1987-1988_vs_1985-1986.status:ERROR (11)
grep -in e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1987-1988_vs_1985-1986 bundle3.o1204803
# 43:=== e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1987-1988_vs_1985-1986.bash ===
grep -in "=== " bundle3.o1204803
# 1:=== ts_rof_monthly_1985-1986-0002.bash ===
# 22:=== ts_rof_monthly_1987-1988-0002.bash ===
# 43:=== e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1987-1988_vs_1985-1986.bash ===
# 45:===== RUN E3SM DIAGS =====
# 5037:===== COPY FILES TO WEB SERVER =====
grep -in error bundle3.o1204803 | wc -l # 120 matches
grep -in "OSError: The dataset file has no matching source variables for" bundle3.o1204803 | wc -l # 40 matches
tail -n 7 bundle3.o1204803
# ===== COPY FILES TO WEB SERVER =====
# file has vanished: "/lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_bundles_www/zppy_main_branch_test_20260325/v3.LR.historical_0051/e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/polar/model_vs_model/.v3.LR.historical_0051-FSNTOA-MAM-polar_S.png.UXwg6r"
# rsync warning: some files vanished before they could be transferred (code 24) at main.c(1338) [sender=3.4.1]
# ==============================================
# Elapsed time: 294 seconds
# ==============================================
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_bundles_output/zppy_main_branch_test_20260325/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/zppy_main_branch_test_20260325/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors
Aside from the Python testscd ~/ez/zppy
git status
# On branch test-zppy-main-20260325
# Good, correct branch
# Env: test-zppy-main-20260325
# Good, correct env
ls tests/integration/test_*.py
pytest tests/integration/test_bash_generation.py
# 1 failed in 2.59s
# Errors look to be expected based on https://github.com/E3SM-Project/zppy/pull/772
pytest tests/integration/test_campaign.py
# 6 failed in 3.39s
# Errors look to be expected based on https://github.com/E3SM-Project/zppy/pull/772
pytest tests/integration/test_defaults.py
# 1 failed in 1.78s
# Errors look to be expected based on https://github.com/E3SM-Project/zppy/pull/772
pytest tests/integration/test_last_year.py
# 1 passed in 1.12s
pytest tests/integration/test_bundles.py
# 2 passed in 1.03s
# Now, we have to turn off testing for 3.1.0 since we have no expected results for it.
emacs tests/integration/utils.py
# "cfgs_to_run": [
# "weekly_bundles",
# "weekly_comprehensive_v2",
# "weekly_comprehensive_v3",
# # "weekly_legacy_3.1.0_bundles",
# # "weekly_legacy_3.1.0_comprehensive_v2",
# # "weekly_legacy_3.1.0_comprehensive_v3",
# "weekly_legacy_3.0.0_bundles",
# "weekly_legacy_3.0.0_comprehensive_v2",
# "weekly_legacy_3.0.0_comprehensive_v3",
# ],
python tests/integration/utils.py
salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm
# We need to reactivate conda now that we're on a compute node
lcrc_conda # Activate conda
conda activate test-zppy-main-20260325
pytest tests/integration/test_images.py
cat test_images_summary.mdOutput: Complete summary table:
Summary table -- only failing tests
Results analysis@chengzhuzhang For @xylar For If both of you are ok with these diffs, I can update the expected results for the zppy test suite. |
Beta Was this translation helpful? Give feedback.
5 replies
-
|
@forsyth2, I looked at a handful of images and in all cases the diffs can be explained by slight shifts in the plots, which almost certainly relate to either intentional formatting changes or different versions of matplotlib, cartopy, etc. I am happy to "bless" all the new images for use as the new baseline. Thank you so much for doing this testing! It goes a long way toward maintaining our confidence in the tools. |
Beta Was this translation helpful? Give feedback.
-
|
Great, thanks @xylar!! |
Beta Was this translation helpful? Give feedback.
-
|
The e3sm_digs results also look great! @forsyth2 thank you for making it very easy to review. |
Beta Was this translation helpful? Give feedback.
-
|
Updating expected results: Setup steps# First, let's copy over the old expected results
cd /lcrc/group/e3sm/public_html/
ls zppy_test_resources_previous # Review previous expected results
/usr/lpp/mmfs/bin/mmlsquota -u ac.forsyth2 --block-size T fs2
# We're doing ok on space
cp -r zppy_test_resources/ zppy_test_resources_previous/expected_results_until_20260326
# ~20 minutes
# Check old expected results were copied over correctly, so we have a record:
ls zppy_test_resources_previous/expected_results_until_20260326
# Now, make new expected results for all the tests
cd ~/ez/zppy
git status
# On branch add-livvkit
# Need to switch branches
git checkout test-zppy-main-20260325
# On branch test-zppy-main-20260325
# nothing to commit, working tree clean
lcrc_conda
conda activate test-zppy-main-20260325
# For completeness, let's just update everything, as usual.
# That's probably good practice anyway -- the tests are checking that expected images aren't missing,
# but it's possible newer code could be adding images to expect in the first place.
# (Probably not the case for the recent PRs though).
git diff --name-only HEAD^ HEAD
# tests/integration/generated/update_weekly_expected_files_chrysalis.sh
# is the only update script that has a diff.
# That file uses multiple `#expand` parameters.
# The others only use `#expand` for the expected dir, which is consistent.
# template_update_{bash_generation, campaign, defaults}_expected_files.sh
# Let's update the simpler tests' results first:
ls tests/integration/generated/update_*
./tests/integration/generated/update_bash_generation_expected_files_chrysalis.sh
# 1 passed in 1.78s
./tests/integration/generated/update_campaign_expected_files_chrysalis.sh
# 6 passed in 1.91s
./tests/integration/generated/update_defaults_expected_files_chrysalis.sh
# 1 passed in 0.36s
# test_last_year doesn't need an update script because its cfg is stand-alone:
pytest -vv tests/integration/test_last_year.py
# 1 passed in 0.33s
# Now update the expected files for the major tests:
cat tests/integration/generated/update_weekly_expected_files_chrysalis.sh
# Notice:
# tests=("comprehensive_v2" "comprehensive_v3" "bundles" "legacy_3.1.0_comprehensive_v2" "legacy_3.1.0_comprehensive_v3" "legacy_3.1.0_bundles" "legacy_3.0.0_comprehensive_v2" "legacy_3.0.0_comprehensive_v3" "legacy_3.0.0_bundles")
# So, we will be updating both the 3.0.0 and 3.1.0 legacy files.
./tests/integration/generated/update_weekly_expected_files_chrysalis.sh
# ~30 minutes
# cp: cannot stat '/lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_legacy_3.1.0_comprehensive_v2_www/zppy_main_branch_test_20260325/v2.LR.historical_0201': No such file or directory
# ./tests/integration/generated/update_weekly_expected_files_chrysalis.sh: line 56: cd: /lcrc/group/e3sm/public_html/zppy_test_resources/expected_legacy_3.1.0_comprehensive_v2: No such file or directory
# cp: cannot stat '/lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_legacy_3.1.0_comprehensive_v3_www/zppy_main_branch_test_20260325/v3.LR.historical_0051': No such file or directory
# ./tests/integration/generated/update_weekly_expected_files_chrysalis.sh: line 56: cd: /lcrc/group/e3sm/public_html/zppy_test_resources/expected_legacy_3.1.0_comprehensive_v3: No such file or directory
# cp: cannot stat '/lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_legacy_3.1.0_bundles_www/zppy_main_branch_test_20260325/v3.LR.historical_0051': No such file or directory
# ./tests/integration/generated/update_weekly_expected_files_chrysalis.sh: line 56: cd: /lcrc/group/e3sm/public_html/zppy_test_resources/expected_legacy_3.1.0_bundles: No such file or directory
ls /lcrc/group/e3sm/public_html/zppy_test_resources
# No 3.1.0 expected dirs here
ls /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2
# No 3.1.0 output dirs here
# But didn't the jobs succeed?
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_comprehensive_v2_output/zppy_main_branch_test_20260325/v2.LR.historical_0201/post/scripts
# This, the output directory, exists
# So, not the web results?
grep -in rsync global_time_series*.o* # No results
grep -in error global_time_series*.o* # No results
# So, global time series completed, had no errors and still didn't produce `www` results?
emacs global_time_series_1980-1990.o1204565
# ===== COPY FILES TO WEB SERVER =====
#
# /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v2_www/zppy_main_branch_test_20260325/v2.LR.historical_0201/global_time_series /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_comprehensive_v2_output/zppy_main_branch_test_20260325/v2.LR.historical_0201/post/scripts
# /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_comprehensive_v2_output/zppy_main_branch_test_20260325/v2.LR.historical_0201/post/scripts
ls /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v2_www/zppy_main_branch_test_20260325/v2.LR.historical_0201/global_time_series
# global_time_series_1980-1990_results
# The output is in the wrong location!!
cd ~/ez/zppy/tests/integration
grep www template_weekly_legacy_3.1.0*
# template_weekly_legacy_3.1.0_bundles.cfg:# To run this test, edit `output` and `www` in this file, along with `actual_images_dir` in test_bundles.py
# template_weekly_legacy_3.1.0_bundles.cfg:www = "#expand user_www#zppy_weekly_bundles_www/#expand unique_id#"
# template_weekly_legacy_3.1.0_comprehensive_v2.cfg:www = "#expand user_www#zppy_weekly_comprehensive_v2_www/#expand unique_id#"
# template_weekly_legacy_3.1.0_comprehensive_v3.cfg:www = "#expand user_www#zppy_weekly_comprehensive_v3_www/#expand unique_id#"
# Indeed, all of them are using the exact same www paths as the regular/main/non-legacy tests!!
ls /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v3_www/zppy_main_branch_test_20260325/v3.LR.historical_0051
# e3sm_diags global_time_series ilamb image_check_failures_comprehensive_v3 mpas_analysis mpas_analysis_mvm pcmdi_diags provenance.20260325_180738_557862.cfg
# So, this one www path must now be hosting a combination of the non-legacy cfg's output and the legacy 3.1.0 cfg's output.
# We have to imagine that some files were overwritten.
# Let's at least test what we have:
cd ~/ez/zppy
pytest tests/integration/test_bundles.py
# 2 passed in 1.67s
salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm
# We need to reactivate conda now that we're on a compute node
lcrc_conda # Activate conda
conda activate test-zppy-main-20260325
pytest tests/integration/test_images.py # 1 passed in 884.29s (0:14:44)
cat test_images_summary.mdUpdated expected results, but after I did so, I found out the |
Beta Was this translation helpful? Give feedback.
-
|
This file path collision issue would also explain the seemingly-at-random |
Beta Was this translation helpful? Give feedback.
-
2026-03-27 ThreadWe're testing the What changes are we testing?The latest expected results are based on the (2026-03-25 test) run.
Testing processSet up environmentslcrc_conda # Activate conda
lcrc-quota # Have enough space
# Set up e3sm_diags env
cd ~/ez/e3sm_diags
# git status
# nothing to commit, working tree clean
git fetch upstream main
git reset --hard upstream/main
git log --oneline | head -n 1
# 4b62dbac Preserve attrs on scalar regrid masks (#1047)
# Good, matches https://github.com/E3SM-Project/e3sm_diags/commits/main
rm -rf build
conda clean --all --y
conda env create -f conda-env/dev.yml -n test-diags-main-20260327
conda activate test-diags-main-20260327
python -m pip install .
# Set up mpas_analysis env for the first time
cd ~/ez/MPAS-Analysis/
git status
# On branch develop
# nothing to commit, working tree clean
git fetch upstream develop
git reset --hard upstream/develop
git log --oneline | head -n 1
# 1bac98457 Merge pull request #1138 from xylar/fix-control-global-subtitle-spacing
# Good, matches https://github.com/MPAS-Dev/MPAS-Analysis/commits/develop/
rm -rf build
conda clean --all --y
head -n 2 dev-spec.txt
# This file may be used to create an environment using:
# $ conda create --name <env> --file <this file>
conda create --name test-mpas-analysis-develop-20260327 --file dev-spec.txt
conda activate test-mpas-analysis-develop-20260327
python -m pip install .
# LIVVKit
# We have no instructions to create our own livvkit environment
# We will simply use the provided one:
# source /lcrc/group/e3sm/livvkit/software/load_e3sm_unified_1.12.1_lex.sh
# Set up zppy-interfaces env
cd ~/ez/zppy-interfaces
git status
# On branch main
# nothing to commit, working tree clean
git fetch upstream main
git reset --hard upstream/main
git log --oneline | head -n 1
# 02638f4 Add Python 3.14 support (#44)
# Good, matches https://github.com/E3SM-Project/zppy-interfaces/commits/main
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n test-zi-main-20260327
conda activate test-zi-main-20260327
python -m pip install .
pytest tests/unit/global_time_series/test_*.py # 10 passed in 20.80s
pytest tests/unit/pcmdi_diags/test_*.py # 7 passed in 5.30szppy run# zppy itself #################################################################
cd ~/ez/zppy
git status
# On branch fix-legacy-310-tests
# nothing to commit, working tree clean
git fetch upstream main
git checkout -b test-zppy-main-20260327 upstream/main
git log --oneline | head -n 1
# 016d220b Merge pull request #783 from mkstratos/add-livvkit
# Good, matches https://github.com/E3SM-Project/zppy/commits/main
# Now, we have to add the extra, unmerged commits.
# From https://github.com/E3SM-Project/zppy/pull/788/commits:
git fetch upstream fix-integration-tests
git cherry-pick 4f466ab091e42f57f1e86d4a69ea6d9232a56491
git cherry-pick 3f3cadc867edd1f4bb405bf6d47883784a676dbf
git cherry-pick 91d39e7225589ab7316d4250e1f4b033f2e157dd
git cherry-pick 2964b0f598f4d7d7fe0f26bfdb787f74908b3728
# From https://github.com/E3SM-Project/zppy/pull/801/commits:
git fetch upstream fix-legacy-310-tests
git cherry-pick 1416d96d1662ba6cd91983660eb6489ed1d0a978
git log --oneline | head -n 6
# 461467e2 Fix legacy 3.1.0 www paths
# d70aeaf7 Add unit tests to ensure correct status filenames
# 076a34c5 Fix mpas_analysis status file names.
# d5a72532 Allow multiple mpas_analysis subsections
# 0ca0ce11 Add mpas_analysis_subsection parameter to global_time_series
# 016d220b Merge pull request #783 from mkstratos/add-livvkit
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n test-zppy-main-20260327
conda activate test-zppy-main-20260327
pre-commit run --all-files # Confirm pre-commit still passes after adding commits
python -m pip install .
pytest tests/test_*.py # 38 passed in 0.59s
# Edit tests/integration/utils.py. Notice livvkit is included now.TEST_SPECIFICS: Dict[str, Any] = {
# These are custom environment_commands for specific tasks.
# Never set these to "", because they will print the line
# `environment_commands = ""` for the corresponding task,
# thus overriding the value set higher up in the cfg.
# That is, there will be no environment set.
# (`environment_commands = ""` only redirects to Unified
# if specified under the [default] task)
"diags_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-diags-main-20260327",
"mpas_analysis_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-mpas-analysis-develop-20260327",
"global_time_series_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-zi-main-20260327",
"livvkit_environment_commands": "source /lcrc/group/e3sm/livvkit/software/load_e3sm_unified_1.12.1_lex.sh",
"pcmdi_diags_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-zi-main-20260327",
# This is the environment setup for other tasks.
# Leave as "" to use the latest Unified environment.
"environment_commands": "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh",
# For a complete test, run the set of latest cfgs and at least one set of legacy cfgs
"cfgs_to_run": [
"weekly_bundles",
"weekly_comprehensive_v2",
"weekly_comprehensive_v3",
"weekly_legacy_3.1.0_bundles",
"weekly_legacy_3.1.0_comprehensive_v2",
"weekly_legacy_3.1.0_comprehensive_v3",
"weekly_legacy_3.0.0_bundles",
"weekly_legacy_3.0.0_comprehensive_v2",
"weekly_legacy_3.0.0_comprehensive_v3",
],
"tasks_to_run": [
"e3sm_diags",
"mpas_analysis",
"global_time_series",
"ilamb",
"livvkit",
"pcmdi_diags",
],
"unique_id": "zppy_main_branch_test_20260327",
}git diff # Diff looks good
python tests/integration/utils.py
# CFG FILES HAVE BEEN GENERATED FROM TEMPLATES WITH THESE SETTINGS:
# UNIQUE_ID=zppy_main_branch_test_20260327
# diags_environment_commands=source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-diags-main-20260327
# mpas_analysis_environment_commands=source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-mpas-analysis-develop-20260327
# global_time_series_environment_commands=source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-zi-main-20260327
# livvkit_environment_commands=source /lcrc/group/e3sm/livvkit/software/load_e3sm_unified_1.12.1_lex.sh
# pcmdi_diags_environment_commands=source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate test-zi-main-20260327
# environment_commands=source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh
# Reminder: `environment_commands=''` => the latest E3SM Unified environment will be used
# For reference:
alias sq
# alias sq='sqa -u ac.forsyth2'
alias sqa
# alias sqa='squeue -o "%8u %.7a %.4D %.9P %7i %.2t %.10r %.10M %.10l %j" --sort=P,-t,-p'
sq
# No jobs currently queued
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v3_chrysalis.cfg
sq | wc -l # Fri 3/27 12:23 => 233 - header row = 232 jobs
sq | wc -l # Fri 3/27 13:56 => 1 - header row = 0 jobsBundles part 2# Check on bundles status
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/zppy_main_branch_test_20260327/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_bundles_output/zppy_main_branch_test_20260327/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/zppy_main_branch_test_20260327/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors
# Now, run bundles part 2 for the working cfgs.
cd ~/ez/zppy
git status
# On branch test-zppy-main-20260327
# Good, correct branch
# Current conda env: test-zppy-main-20260327
# Good, correct environment
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg
sq | wc -l # Fri 3/27 14:02 => 7 - header row = 6 jobsReview finished runs### v2 ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/zppy_main_branch_test_20260327/v2.LR.historical_0201/post/scripts
grep -v "OK" *status # Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_comprehensive_v2_output/zppy_main_branch_test_20260327/v2.LR.historical_0201/post/scripts
grep -v "OK" *status # Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/zppy_main_branch_test_20260327/v2.LR.historical_0201/post/scripts
grep -v "OK" *status # Good, no errors
### v3 ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/zppy_main_branch_test_20260327/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_comprehensive_v3_output/zppy_main_branch_test_20260327/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/zppy_main_branch_test_20260327/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors
### bundles ###
sq | wc -l # Fri 3/27 14:02 => 7 - header row = 6 jobs
sq | wc -l # Fri 3/27 14:10 => 1 - header row = 0 jobs
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/zppy_main_branch_test_20260327/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.1.0_bundles_output/zppy_main_branch_test_20260327/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/zppy_main_branch_test_20260327/v3.LR.historical_0051/post/scripts
grep -v "OK" *status # Good, no errors
Python testscd ~/ez/zppy
git status
# On branch test-zppy-main-20260327
# Good, correct branch
# Env: test-zppy-main-20260327
# Good, correct env
ls tests/integration/test_*.py
pytest tests/integration/test_bash_generation.py
# 1 failed in 2.04s
# Errors look to be expected based on https://github.com/E3SM-Project/zppy/pull/783
pytest tests/integration/test_campaign.py
# 5 failed, 1 passed in 2.37s
# Errors look to be expected based on https://github.com/E3SM-Project/zppy/pull/783
pytest tests/integration/test_defaults.py
# 1 failed in 1.06s
# Errors look to be expected based on https://github.com/E3SM-Project/zppy/pull/783
pytest tests/integration/test_last_year.py
# 1 passed in 0.36s
pytest tests/integration/test_bundles.py
# 2 passed in 0.20s
# Now, we have to turn off testing for 3.1.0 since we have no expected results for it.
emacs tests/integration/utils.py
# "cfgs_to_run": [
# "weekly_bundles",
# "weekly_comprehensive_v2",
# "weekly_comprehensive_v3",
# # "weekly_legacy_3.1.0_bundles",
# # "weekly_legacy_3.1.0_comprehensive_v2",
# # "weekly_legacy_3.1.0_comprehensive_v3",
# "weekly_legacy_3.0.0_bundles",
# "weekly_legacy_3.0.0_comprehensive_v2",
# "weekly_legacy_3.0.0_comprehensive_v3",
# ],
python tests/integration/utils.py
salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm
# We need to reactivate conda now that we're on a compute node
lcrc_conda # Activate conda
conda activate test-zppy-main-20260327
pytest tests/integration/test_images.py # 1 failed in 778.87s (0:12:58)
cat test_images_summary.mdOutput: Complete summary table:
Summary table -- only failing tests
Results analysis@xylar This test included the 4 commits from https://github.com/E3SM-Project/zppy/pull/788/commits. It looks like the diffs are because the years changed from 1985-1989 to 1985-1995. Looking at the 3 commits added on #788 since the 3/25 test, we removed: [[ original ]]
# Keep configuration from original mpas_analysis tests for global_time_series to use
ts_years = "1985-1989", "1985-1995",
climo_years = "1985-1989", "1990-1995",
enso_years = "1985-1989", "1990-1995",to instead use: [[ reference ]]
ts_years = "1985-1989",
climo_years = "1985-1989",
enso_years = "1985-1989",
[[ test ]]
ts_years = "1985-1995",
climo_years = "1990-1995",
enso_years = "1990-1995",Would that change explain why the years are different? I feel like we would have expected more than 14 mismatched images though. If this change is explainable, then I can go ahead and update the expected results again, and #788 would also be good to merge (after a visual inspection). |
Beta Was this translation helpful? Give feedback.
5 replies
-
|
@forsyth2, first it seems like mvm mismatches show up in the list but not in the grid. Can we fix that? Yes, I think the "actual" are correct and the "expected" are wrong in all these diffs. The "expected" were erroneously using the full 1985-1995 time series in cases where they should have only been using the 1985-1989. Please "bless" the new results as the expected ones. |
Beta Was this translation helpful? Give feedback.
-
Ah I see the problem: ls /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v3_www/zppy_main_branch_test_20260327/v3.LR.historical_0051/
# e3sm_diags image_check_failures_comprehensive_v3 mpas_analysis_mvm
# global_time_series livvkit pcmdi_diags
# ilamb mpas_analysis provenance.20260327_192219_693811.cfgNotice The subtasks should all be under one dir. For example, in ls /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v3_www/zppy_main_branch_test_20260327/v3.LR.historical_0051/e3sm_diags
# atm_monthly_180x360_aave atm_monthly_180x360_aave_mvm lnd_monthly_mvm_lndIn any case, that's what's throwing off the image grid generation. The list generation is using the task name to determine the path prefix ( The grid generation sets ls /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v3_www/zppy_main_branch_test_20260327/v3.LR.historical_0051/image_check_failures_comprehensive_v3/
# e3sm_diags global_time_series ilamb mpas_analysis mpas_analysis_mvm pcmdi_diagsThe Before I update the expected results, could you confirm those look alright? Thanks! So two options:
if "weekly_comprehensive_v3" in expansions["cfgs_to_run"]:
available_tasks = [
"e3sm_diags",
"mpas_analysis",
"global_time_series",
"ilamb",
"pcmdi_diags",
]and also adding a trailing
|
Beta Was this translation helpful? Give feedback.
-
Yes, the "actual" look right for all images.
I think that's a fine solution. I wasn't that familiar with how this was handled elsewhere. Can you make the change so thing are laid out like you want, and ask me to review? |
Beta Was this translation helpful? Give feedback.
-
|
Sounds good, thanks. I'll get started on both updating the results and working on a directory name fix. |
Beta Was this translation helpful? Give feedback.
-
|
Updated expected results: Setup steps# First, let's copy over the old expected results
cd /lcrc/group/e3sm/public_html/
ls zppy_test_resources_previous # Review previous expected results
/usr/lpp/mmfs/bin/mmlsquota -u ac.forsyth2 --block-size T fs2
# We're doing ok on space
cp -r zppy_test_resources/ zppy_test_resources_previous/expected_results_until_20260330
# Check old expected results were copied over correctly, so we have a record:
ls zppy_test_resources_previous/expected_results_until_20260330
# Now, make new expected results for all the tests
cd ~/ez/zppy
git status
# On branch test-zppy-main-20260327
lcrc_conda
conda activate test-zppy-main-20260327
# tests/integration/generated/update_weekly_expected_files_chrysalis.sh
# is the only update script that will have a diff.
# That file uses multiple `#expand` parameters.
# The others only use `#expand` for the expected dir, which is consistent.
# template_update_{bash_generation, campaign, defaults}_expected_files.sh
# Let's update the simpler tests' results first:
ls tests/integration/generated/update_*
./tests/integration/generated/update_bash_generation_expected_files_chrysalis.sh
# 1 passed in 1.78s
./tests/integration/generated/update_campaign_expected_files_chrysalis.sh
# 1 failed, 5 passed => Need to handle high_res_v1 issue:
rm -rf /lcrc/group/e3sm/public_html/zppy_test_resources/test_campaign_high_res_v1_expected_files
mkdir -p /lcrc/group/e3sm/public_html/zppy_test_resources/test_campaign_high_res_v1_expected_files
mv test_campaign_high_res_v1_output/post/scripts/*.settings /lcrc/group/e3sm/public_html/zppy_test_resources/test_campaign_high_res_v1_expected_files
pytest tests/integration/test_campaign.py
# 6 passed in 4.29s
./tests/integration/generated/update_defaults_expected_files_chrysalis.sh
# 1 passed in 0.76s
# test_last_year doesn't need an update script because its cfg is stand-alone:
pytest -vv tests/integration/test_last_year.py
# 1 passed in 0.71s
# Now update the expected files for the major tests:
./tests/integration/generated/update_weekly_expected_files_chrysalis.sh
# ~30 minutes
# Good, no "cannot stat" errors this time
ls /lcrc/group/e3sm/public_html/zppy_test_resources
# 3.1.0 dirs are present now!
cd ~/ez/zppy
pytest tests/integration/test_bundles.py
# 2 passed in 1.00s
salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm
# We need to reactivate conda now that we're on a compute node
lcrc_conda # Activate conda
conda activate test-zppy-main-20260327
pytest tests/integration/test_images.py # 1 passed in 798.76s (0:13:18)
cat test_images_summary.md
We're still not producing 3.1.0 plots though. That's because we still need to uncomment the legacy 3.1.0 cfgs to run: "cfgs_to_run": [
"weekly_bundles",
"weekly_comprehensive_v2",
"weekly_comprehensive_v3",
"weekly_legacy_3.1.0_bundles",
"weekly_legacy_3.1.0_comprehensive_v2",
"weekly_legacy_3.1.0_comprehensive_v3",
"weekly_legacy_3.0.0_bundles",
"weekly_legacy_3.0.0_comprehensive_v2",
"weekly_legacy_3.0.0_comprehensive_v3",
],python tests/integration/utils.py
# Now rerun:
pytest tests/integration/test_images.py # 1 passed in 814.12s (0:13:34)
cat test_images_summary.md # Now 3.1.0 tests are added!
|
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Starting this page to log the process/results for the weekly runs of
zppy's integration testing: bundles, comprehensive_v2, comprehensive_v3Beta Was this translation helpful? Give feedback.
All reactions