Merge pull request #625 from MPAS-Dev/develop

Merge develop to master for v1.2.5 release

Author: xylar

README.md

@@ -30,7 +30,7 @@ All platforms:
 
 MPAS-Analysis is available as an anaconda package via the `conda-forge` channel:
 
-``` bash
+```
 conda config --add channels conda-forge
 conda create -n mpas-analysis mpas-analysis
 conda activate mpas-analysis
@@ -58,14 +58,15 @@ environment with the following packages:
  * shapely
  * cartopy
  * geometric\_features
+ * gsw
 
 These can be installed via the conda command:
-``` bash
+```
 conda config --add channels conda-forge
-conda create -n mpas-analysis numpy scipy matplotlib netCDF4 \
-    xarray dask bottleneck basemap lxml nco>=4.8.1 pyproj pillow \
-    cmocean progressbar2 requests setuptools shapely cartopy \
-    geometric_features
+conda create -n mpas-analysis python=3.7 numpy scipy "matplotlib>=3.0.2" \
+    netCDF4 "xarray>=0.10.0" dask bottleneck basemap lxml "nco>=4.8.1" pyproj \
+    pillow cmocean progressbar2 requests setuptools shapely cartopy \
+    geometric_features gsw
 conda activate mpas-analysis
 ```
 
@@ -78,13 +79,13 @@ Then, get the code from:
 If you installed the `mpas-analysis` package, download the data that is
 necessary to MPAS-Analysis by running:
 
-``` bash
+```
 download_analysis_data -o /path/to/mpas_analysis/diagnostics
 ```
 
 If you are using the git repository, run:
 
-``` bash
+```
 ./download_analysis_data.py -o /path/to/mpas_analysis/diagnostics
 ```
 
@@ -121,13 +122,13 @@ should see a warning that the data are being downloaded.
 **Note**: If you are having issues downloading the shape files (e.g., a time out error or forbidden error), follow these steps:
 
 1. Run the following in python on your local machine (i.e., one that has no trouble downloading these files):
-```python
+```
 import cartopy.io.shapereader as shpreader
 for name in ['ocean', 'coastline', 'land']:
     shpfilename = shpreader.natural_earth(resolution='110m',
                                           category='physical',
                                           name=name)
-    reader = shpreader.Reader(shpfilename)
+    shpreader.Reader(shpfilename)
 ```
 2. On your local machine, run `python -c "import cartopy; print(cartopy.config['data_dir'])"`. This will print out the directory in which the natural earth shapefiles are being placed locally.
 3. Copy these files onto the remote machine you are working on. Include folders `shapefiles/natural_earth/physical/*` where `*` is the set of shapefiles that were downloaded.
@@ -140,12 +141,12 @@ for name in ['ocean', 'coastline', 'land']:
 If you installed the `mpas-analysis` package, list the available analysis tasks
 by running:
 
-``` bash
+```
 mpas_analysis --list
 ```
 
 If using a git repository, run:
-``` bash
+```
 python -m mpas_analysis --list
 ```
 
@@ -228,13 +229,13 @@ To purge old analysis (delete the whole output directory) before running run
 the analysis, add the `--purge` flag.  If you installed `mpas-analysis` as
 a package, run:
 
-``` bash
+```
 mpas_analysis --purge <config.file>
 ```
 
 If you are running in the repo, use:
 
-``` bash
+```
 python -m mpas_analysis --purge <config.file>
 ```
 
@@ -300,7 +301,7 @@ within the `mpas_analysis/shared` directory.
    `build_analysis_list`, see below.
 
 A new analysis task can be added with:
-```python
+```
    analyses.append(<component>.MyTask(config, myArg='argValue'))
 ```
 This will add a new object of the `MyTask` class to a list of analysis tasks
@@ -314,10 +315,15 @@ to be generated and is set up properly.
 ## Generating Documentation
 
 To generate the `sphinx` documentation, run:
-```bash
+```
+conda config --add channels conda-forge
+conda remove -y --all -n mpas-analysis-docs
 conda env create -f docs/environment.yml
+conda install -y -n mpas-analysis-docs mock pillow sphinx sphinx_rtd_theme
 conda activate mpas-analysis-docs
-python setup.py install
+pip install .
+rm -rf build dist mpas_analysis.egg-info
 cd docs
+make clean
 make html
 ```

ci/requirements.yml

@@ -21,3 +21,4 @@ dependencies:
   - shapely
   - cartopy
   - geometric_features
+  - gsw

configs/compy/config.20190711.testLuke.A_WCYCL1850S.ne30_oECv3.compy

@@ -0,0 +1,138 @@
+[runs]
+## options related to the run to be analyzed and control runs to be
+## compared against
+
+# mainRunName is a name that identifies the simulation being analyzed.
+mainRunName = 20190711.testLuke.A_WCYCL1850S.ne30_oECv3.compy 
+
+# config file for a control run to which this run will be compared.  The
+# analysis should have already been run to completion once with this config
+# file, so that the relevant MPAS climatologies already exist and have been
+# remapped to the comparison grid.  Leave this option commented out if no
+# control run is desired.
+# controlRunConfigFile = /path/to/config/file
+
+# config file for a main run on which the analysis was already run to
+# completion.  The relevant MPAS climatologies already exist and have been
+# remapped to the comparison grid and time series have been extracted.
+# Leave this option commented out if the analysis for the main run should be
+# performed.
+mainRunConfigFile = config.GM_20190605_GMtest_Kappa3d_vary_c2_intel.compy 
+
+[execute]
+## options related to executing parallel tasks
+
+# the number of parallel tasks (1 means tasks run in serial, the default)
+parallelTaskCount = 6
+
+# the parallelism mode in ncclimo ("serial" or "bck")
+# Set this to "bck" (background parallelism) if running on a machine that can
+# handle 12 simultaneous processes, one for each monthly climatology.
+ncclimoParallelMode = bck
+
+[diagnostics]
+## config options related to observations, mapping files and region files used
+## by MPAS-Analysis in diagnostics computations.
+
+# The base path to the diagnostics directory.  Typically, this will be a shared
+# directory on each E3SM supported machine (see the example config files for
+# its location).  For other machines, this would be the directory pointed to
+# when running "download_analysis_data.py" to get the public observations,
+# mapping files and region files.
+baseDirectory = /compyfs/diagnostics
+
+[input]
+## options related to reading in the results to be analyzed
+
+# directory containing model results
+baseDirectory = /compyfs/wolf966/e3sm_scratch/20190711.testLuke.A_WCYCL1850S.ne30_oECv3.compy/run 
+
+# names of ocean and sea ice meshes (e.g. oEC60to30v3, oQU240v3, oRRS30to10v3, etc.)
+mpasMeshName = oEC60to30v3
+
+# names of namelist and streams files, either a path relative to baseDirectory
+# or an absolute path.
+#oceanNamelistFileName = mpas-o_in
+#oceanStreamsFileName = streams.ocean
+#seaIceNamelistFileName = mpas-cice_in
+#seaIceStreamsFileName = streams.cice
+
+[output]
+## options related to writing out plots, intermediate cached data sets, logs,
+## etc.
+
+# directory where analysis should be written
+baseDirectory = /path/to/output/directory
+
+# Anvil doesn't have direct access to a web portal, so output will need
+# to be copied elsewhere (e.g. NERSC web portal)
+htmlSubdirectory = html
+
+# a list of analyses to generate.  Valid names can be seen by running:
+#   mpas_analysis --list
+# This command also lists tags for each analysis.
+# Shortcuts exist to generate (or not generate) several types of analysis.
+# These include:
+#   'all' -- all analyses will be run
+#   'all_<tag>' -- all analysis with a particular tag will be run
+#   'all_<component>' -- all analyses from a given component (either 'ocean'
+#                        or 'seaIce') will be run
+#   'only_<component>', 'only_<tag>' -- all analysis from this component or
+#                                       with this tag will be run, and all
+#                                       analysis for other components or
+#                                       without the tag will be skipped
+#   'no_<task_name>' -- skip the given task
+#   'no_<component>', 'no_<tag>' -- in analogy to 'all_*', skip all analysis
+#                                   tasks from the given compoonent or with
+#                                   the given tag.  Do
+#                                      mpas_analysis --list
+#                                   to list all task names and their tags
+# an equivalent syntax can be used on the command line to override this
+# option:
+#    mpas_analysis config.analysis --generate \
+#         all,no_ocean,all_timeSeries
+# All tasks with tag "landIceCavities" are disabled because this run did not
+# include land-ice cavities.
+generate = ['all', 'no_landIceCavities', 'no_eke', 'no_BGC', 'no_icebergs',
+            'no_min', 'no_max']
+
+[climatology]
+## options related to producing climatologies, typically to compare against
+## observations and previous runs
+
+# the first year over which to average climatalogies
+startYear = 31
+# the last year over which to average climatalogies
+endYear = 40
+
+[timeSeries]
+## options related to producing time series plots, often to compare against
+## observations and previous runs
+
+# start and end years for timeseries analysis. Using out-of-bounds values
+#   like start_year = 1 and end_year = 9999 will be clipped to the valid range
+#   of years, and is a good way of insuring that all values are used.
+startYear = 1
+endYear = 40
+
+[index]
+## options related to producing nino index.
+
+# start and end years for the nino 3.4 analysis.  Using out-of-bounds values
+#   like start_year = 1 and end_year = 9999 will be clipped to the valid range
+#   of years, and is a good way of insuring that all values are used.
+# For valid statistics, index times should include at least 30 years
+startYear = 1
+endYear = 9999
+
+[streamfunctionMOC]
+## options related to plotting the streamfunction of the meridional overturning
+## circulation (MOC)
+
+# Use postprocessing script to compute the MOC? You want this to be True
+# for low-resolution simulations that use GM to parameterize eddies, because
+# the online MOC analysis member currently does not include the bolus velocity
+# in its calculation, whereas the postprocessing script does.
+# NOTE: this is a temporary option that will be removed once the online
+# MOC takes into account the bolus velocity when GM is on.
+usePostprocessingScript = True

configs/compy/job_script.compy.bash

@@ -0,0 +1,39 @@
+#!/bin/bash -l
+# This software is open source software available under the BSD-3 license.
+#
+# Copyright (c) 2019 Triad National Security, LLC. All rights reserved.
+# Copyright (c) 2019 Lawrence Livermore National Security, LLC. All rights
+# reserved.
+# Copyright (c) 2019 UT-Battelle, LLC. All rights reserved.
+#
+# Additional copyright and license information can be found in the LICENSE file
+# distributed with this code, or at
+# https://raw.githubusercontent.com/MPAS-Dev/MPAS-Analysis/master/LICENSE
+
+#SBATCH --nodes=1
+#SBATCH --time=1:00:00
+#SBATCH --account=e3sm
+#SBATCH --job-name=mpas_analysis
+#SBATCH --output=mpas_analysis.o%j
+#SBATCH --error=mpas_analysis.e%j
+
+cd $SLURM_SUBMIT_DIR
+export OMP_NUM_THREADS=1
+
+source /compyfs/software/e3sm-unified/load_latest_e3sm_unified.sh 
+export HDF5_USE_FILE_LOCKING=FALSE
+
+# MPAS/ACME job to be analyzed, including paths to simulation data and
+# observations. Change this name and path as needed
+run_config_file="config.run_name_here"
+
+if [ ! -f $run_config_file ]; then
+    echo "File $run_config_file not found!"
+    exit 1
+fi
+
+# if using the mpas_analysis conda package instead of the git repo, remove
+# "python -m"
+
+srun -N 1 -n 1 python -m mpas_analysis $run_config_file
+

docs/analysis_tasks.rst

@@ -15,6 +15,7 @@ Analysis Tasks
    tasks/climatologyMapBGC
    tasks/climatologyMapArgoTemperature
    tasks/climatologyMapArgoSalinity
+   tasks/climatologyMapWoa
    tasks/climatologyMapSchmidtko
    tasks/climatologyMapAntarcticMelt
 
@@ -31,6 +32,7 @@ Analysis Tasks
    tasks/soseTransects
    tasks/geojsonTransects
    tasks/oceanRegionalProfiles
+   tasks/regionalTSDiagrams
 
    tasks/climatologyMapSeaIceConcNH
    tasks/climatologyMapSeaIceThickNH

docs/api.rst

@@ -7,7 +7,7 @@ more details and examples, refer to the relevant chapters in the main part of
 the documentation.
 
 Top-level script: mpas_analysis
-===================================
+===============================
 
 .. currentmodule:: mpas_analysis.__main__
 
@@ -21,8 +21,17 @@ Top-level script: mpas_analysis
    update_generate
    run_analysis
    wait_for_task
-   download_analysis_data
+ 
+Downloading data
+================
+
+.. currentmodule:: mpas_analysis.download_data
+
+.. autosummary::
+   :toctree: generated/
 
+   download_analysis_data
+   download_natural_earth_110m
 
 Analysis tasks
 ==============

docs/conf.py

@@ -76,9 +76,9 @@
 # built documents.
 #
 # The short X.Y version.
-version = u'1.2.4'
+version = u'1.2.5'
 # The full version, including alpha/beta/rc tags.
-release = u'1.2.4'
+release = u'1.2.5'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

docs/config/diagnostics.rst

@@ -18,12 +18,19 @@ interpolate MPAS data and observations to common reference grids::
   # mapping files and region files.
   baseDirectory = /path/to/diagnostics
 
+  # A second directory where custom diagonstics data such as mapping files and
+  # regions masks for unsupported grids can be found.  The structure of
+  # subdirectories in this directory must be the same as baseDirectory
+  customDirectory = none
+
   # Directory for mapping files (if they have been generated already). If mapping
   # files needed by the analysis are not found here, they will be generated and
-  # placed in the output mappingSubdirectory
+  # placed in the output mappingSubdirectory.  The user can supply an absolute
+  # path here to point to a path that is not within the baseDirectory above.
   mappingSubdirectory = mpas_analysis/maps
 
-  # Directory for region mask files
+  # Directory for region mask files. The user can supply an absolute path here to
+  # point to a path that is not within the baseDirectory above.
   regionMaskSubdirectory = mpas_analysis/region_masks
 
 Diagnostics Directories
@@ -35,6 +42,10 @@ this data has already been downloaded to a shared location (see example config
 files in the subdirectories of the ``configs`` directory in the MPAS-Analysis
 repository).
 
+The ``customDirectory``, if it is not ``none`` is another directory where
+observations, mapping files and region masks may be stored.  This is useful for
+runs on non-standard grids or for testing out new observations.
+
 The remaining options point to the subdirectories for mapping files (see
 below) and region masks (see :ref:`config_colormaps`), respectively.
 Typically, there is no reason to change ``mappingSubdirectory`` or