Merge pull request #93 from cropsinsilico/topic/mamba

Topic/mamba

Author: langmm

.github/workflows/test-install.yml

@@ -1,3 +1,4 @@
+repos:
 -   repo: https://github.com/pycqa/flake8
     rev: '4.0.1'
     hooks:

.pre-commit-config.yaml

@@ -2,7 +2,31 @@
 History
 =======
 
-1.8.2 (2021-03-18) Migrate tests out of package into pytest fixtures & various bug fixes
+1.8.4 (2023-01-27) Update CI to use mamba, improve dev utilities, & various bug fixes
+------------------
+* Update utility scripts so that requirements can be maintained in a single file with pip requirements.txt file and conda recipe generated via utils/manage_requirements.py
+* Consolidate CLI utilities for creating environments
+* Add cron job to build docker images periodically to ensure that there is always one available
+* Migrate CI to use mamba instead of conda for improved performance
+* Update documents to encourage use of mamba and reflect updates to development utilities
+* Manage optional dependencies through pip extras and additional conda builds
+* Disable flaky tests by default on CI so that tests that fail in teardown are re-run in the second attempt. They can be enabled via the pytest plugin flag "--rerun-flaky"
+* Add utility for tracking memory usage of multitasking processes
+* Update remote integration tests to use render.io instance and only run on one CI job to prevent overtaxing it
+* Handle failure on CI where MPI was not being installed
+* Various updates to comply with updated PEP8 standards
+* Add option to track memory usage to integration services manager
+* Expand output of ygginfo to include python configuration variables, common executables, and environment variables as well as providing more detailed information when a language is marked as not configured
+* Limit SBML testing to single CI job
+* Use sysconfig options for compiled languages if they match selected compilation tools
+* Fix bug where loading a yaml file with "model_only=True" yielded a result with default inputs and outputs
+
+1.8.3 (2022-07-18) Minor bug fixes for the R interface and CI
+------------------
+* Fix bug in R interface resulting from calling "is.na" on vectors
+* Stop using specific installations of R from apt on linux CI
+
+1.8.2 (2022-03-18) Migrate tests out of package into pytest fixtures & various bug fixes
 ------------------
 
 * Move tests out of package to take advantage of pytest conftest.py structure and reduce the size of the package

HISTORY.rst

@@ -46,3 +46,7 @@ include versioneer.py
 include yggdrasil/_version.py
 include create_coveragerc.py
 include utils/*.py
+include utils/requirements/*.txt
+include utils/requirements/*.yaml
+include utils/requirements/*.json
+include utils/requirements/*.ini

MANIFEST.in

@@ -10,8 +10,7 @@ Development of |yggdrasil| should be done on branches and/or forks that
 are then merged in via pull request after passing linting (via
 `flake8 <http://flake8.pycqa.org/en/latest/>`_), testing (via
 continuous integration on
-`travis <https://travis-ci.org/cropsinsilico/yggdrasil>`_ and
-`appveyor <https://ci.appveyor.com/project/langmm/yggdrasil>`_),
+`Github Actions <https://github.com/cropsinsilico/yggdrasil/actions>`_),
 and code review. Prior to beginning new development,
 developers should open a Github issue on the repository that describes
 the proposed changes and/or features so that discussion can help identify
@@ -36,36 +35,40 @@ The following is only one method for setting up a development environment. You a
 
      $ conda config --add channels conda-forge
 
+#. [OPTIONAL] Install mamba. Mamba is a drop in replacement for conda that is often much faster. If you don't install mamba, replace mamba with conda in the following commands.::
+
+     $ conda install mamba
+
 #. Create a fork of the |yggdrasil| Github repository for you Github account (``Fork`` button located in the upper right corner of the |yggdrasil| `Github repository <https://github.com/cropsinsilico/yggdrasil>`_).
-#. Clone your fork of the |yggdrasil| repository using git and then change directory into the cloned repository. NOTE: If you do not have ``git`` you can either `install it yourself <https://git-scm.com/book/en/v2/Getting-Started-Installing-Git>`_ or using conda (``conda install git``).::
+#. Clone your fork of the |yggdrasil| repository using git and then change directory into the cloned repository. NOTE: If you do not have ``git`` you can either `install it yourself <https://git-scm.com/book/en/v2/Getting-Started-Installing-Git>`_ or using mamba/conda (``mamba install git``).::
 
      $ git clone --recurse-submodules https://github.com/<your Github username>/yggdrasil.git
      $ cd yggdrasil
 
 #. [OPTIONAL] Create a conda environment for development and activate that environment. You can create an environment using any of the supported versions of Python and may need more than one for testing.::
 
-     $ conda create -n ygg python=3.6
+     $ mamba create -n ygg python=3.7
      $ conda activate ygg
      
-   |yggdrasil| provides a python script, ``utils/create_envs.py`` for creating development environments and installing yggdrasil in develoment mode for a combination of dependency installation methods (pip vs. conda) and versions of Python. For example, to create a conda environment with Python 3.6 (as above) and install |yggdrasil|'s dependencies using conda, the following commands can
+   |yggdrasil| provides a python script, ``utils/setup_test_env.py`` for creating development environments and installing yggdrasil in develoment mode for a combination of dependency installation methods (pip vs. conda) and versions of Python. For example, to create a conda environment with Python 3.7 (as above) and install |yggdrasil|'s dependencies using mamba, the following commands can
    be executed from the root directory of your local |yggdrasil| repository.::
 
-     $ python utils/create_envs.py --name=ygg
+     $ python utils/setup_test_env.py devenv mamba 3.7 --name=ygg
      $ conda activate ygg
 
    .. note::
       **If you use ``utils/create_envs.py`` to create your dev environment, you can skip to the last step.**
-#. Install the requirements using conda via the helper script ``utils/install_from_requirements.py``::
+#. Install the requirements using mamba/conda via the helper script ``utils/install_from_requirements.py``::
 
-     $ python utils/install_from_requirements.py conda requirements.txt requirements_condaonly.txt requirements_testing.txt
+     $ python utils/manage_requirements.py install mamba --for-development
 
-   Alternatively, if you did not install conda, you can install the Python dependencies using ``pip`` via the same script.::
+   Alternatively, if you did not install mamba or conda, you can install the Python dependencies using ``pip`` via the same script. This script will attempt to install non-python dependencies via a package manager like ``apt``, ``brew``, or ``vcpkg`` unless ``--python-only`` is passed.::
 
-     $ python utils/install_from_requirements.py pip requirements.txt requirements_testing.txt
+     $ python utils/manage_requirements.py install pip --for-development
 
-   However, if you use this method, you will need to manually install the non-Python dependencies as described :ref:`here <manual_install_rst>`. This should be done BEFORE installing |yggdrasil| in the next step.
+   If you use this method with the ``--python-only`` flag, you will need to manually install the non-Python dependencies as described :ref:`here <manual_install_rst>`. This should be done BEFORE installing |yggdrasil| in the next step.
 
-   There are also two additional requirements files, ``requirements_documentation.txt`` and ``requirements_optional.txt``, that can optionally be added to the end of either of these commands. ``requirements_documentation.txt`` includes packages required for building the documentation and ``requirements_optional.txt`` includes packages required for optional |yggdrasil| features (e.g. using ``astropy`` table parsing or ``pika`` for RabbitMQ communication).
+   There are additional optional requirements that can be enabled via various command line flags. To see what these options are, take a look at the help information for the ``utils/manage_requirements.py install`` script by passing ``-h`` or ``--help``.
 #. Run the |yggdrasil| installation script in development mode.::
 
      $ python setup.py develop
@@ -98,4 +101,4 @@ the module directory containing the class being tested. In some cases
 attributes and/or methods are defined (e.g. serialization, communication,
 and connection driver classes).
 
-Tests can be run using the ``yggtest`` command. If no arguments are provided, ``yggtest`` will run all of the tests (excluding the examples). If you only want to run some tests, you can provide the path to file or directory containing the tests you would like to run; these paths can be absolute, relative to the current directory, or relative to the top level directory of the |yggdrasil| source tree. To also run example tests, include the ``--with-examples`` flag.
+Tests can be run by passing the tests directory to the ``pytest`` command. If no other arguments are provided, ``pytest`` will run all of the tests (excluding the examples). If you only want to run some tests, you can provide the path to file or directory containing the tests you would like to run; these paths can be absolute, relative to the current directory, or relative to the top level directory of the |yggdrasil| source tree. To also run example tests, include the ``--with-examples`` flag.

docs/source/development/general.rst

@@ -14,19 +14,25 @@ Installation
     * "Desktop development with C++" - Workload under "Windows" section
     * "MSVC v140 - VS 2015 C++ build tools (v14.00)" - Individual component under "Compilers, build tools, and runtimes" section.
 
-   If you *do not use conda* to install |yggdrasil|, you will also need to initialize the command line build tools in any prompt you will be calling |yggdrasil| from. This can be done by calling |yggdrasil| from a "VS2015 x64 Native Tools Developer Command Prompt", or by locating the ``vsvarsall.bat`` script that comes with Visual Studio. Information on the developer prompt and how to locate the ``vsvarsall.bat`` script can be found `here <https://docs.microsoft.com/en-us/cpp/build/building-on-the-command-line?view=vs-2019>`_. The prompt/script used must enable the Visual Studio 2015 build tools (the specific year) to be compatible with the Python C library (see the discussion `here <https://wiki.python.org/moin/WindowsCompilers>`_). The VS 2015 tools prompt/script will be installed from within Visual Studio 2019 by selecting the components indicated above. On a 64bit Windows machine (assuming you will be using 64 bit Python), the command to initialize these tools within a regular command prompt will look something like this, but the exact path will vary with your particular installation::
+   If you *do not use conda/mamba* to install |yggdrasil|, you will also need to initialize the command line build tools in any prompt you will be calling |yggdrasil| from. This can be done by calling |yggdrasil| from a "VS2015 x64 Native Tools Developer Command Prompt", or by locating the ``vsvarsall.bat`` script that comes with Visual Studio. Information on the developer prompt and how to locate the ``vsvarsall.bat`` script can be found `here <https://docs.microsoft.com/en-us/cpp/build/building-on-the-command-line?view=vs-2019>`_. The prompt/script used must enable the Visual Studio 2015 build tools (the specific year) to be compatible with the Python C library (see the discussion `here <https://wiki.python.org/moin/WindowsCompilers>`_). The VS 2015 tools prompt/script will be installed from within Visual Studio 2019 by selecting the components indicated above. On a 64bit Windows machine (assuming you will be using 64 bit Python), the command to initialize these tools within a regular command prompt will look something like this, but the exact path will vary with your particular installation::
 
      $ call "C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\vcvarsall.bat" amd64
 
 .. note::
 
    **Mac Users**
 
-   If you will be running C/C++/Fortran models, you will need to install the command line developer tools on Mac which provides several utilities for compilation and code management (e.g. clang, gcc, make, git). `This <https://cloudtechpoint.medium.com/how-to-install-command-line-tools-homebrew-in-macos-without-xcode-1b910742d923>`_ article outlines how to check if they are installed and install them if they are not. It also describes how to install the Homebrew package manager, which will be useful for installing non-Python dependencies if you do not plan on using conda to install |yggdrasil|.
+   If you will be running C/C++/Fortran models, you will need to install the command line developer tools on Mac which provides several utilities for compilation and code management (e.g. clang, gcc, make, git). `This <https://cloudtechpoint.medium.com/how-to-install-command-line-tools-homebrew-in-macos-without-xcode-1b910742d923>`_ article outlines how to check if they are installed and install them if they are not. It also describes how to install the Homebrew package manager, which will be useful for installing non-Python dependencies if you do not plan on using conda/mamba to install |yggdrasil|.
 
 
-Conda Installation (recommended)
---------------------------------
+Conda/Mamba Installation (recommended)
+--------------------------------------
+
+.. note::
+
+   **Mamba Installation**
+
+   Mamba is a drop in replacement for conda that is often faster and provides clearer error messages when there are conflicts. You can install mamba by following the instructions `here <https://mamba.readthedocs.io/en/latest/installation.html>`_ (including by using conda). Then you may just use ``mamba`` in place of any of the commands below.
 
 Download and install Miniconda from `here <https://docs.conda.io/en/latest/miniconda.html>`_ (or Anaconda from `here <https://www.anaconda.com/download/>`_ if you would like additional Python libraries installed by default). There are conda distributions available for |yggdrasil| from `conda-forge <https://github.com/conda-forge/yggdrasil-feedstock>`_. You can install |yggdrasil| from conda-forge by calling::
 
@@ -54,7 +60,7 @@ the terminal (or Anaconda Prompt on Windows).::
    Then run ``yggconfig`` to finish the installation process for C and C++.
 
 .. warning::
-   If conda takes a very long time to install |yggdrasil| (>5 min spent solving the environment) or fails with an error about conflicts and you are using a version of conda older than 4.7.2, try either updating conda and/or adding the conda-forge channel (if you havn't already)::
+   If conda takes a very long time to install |yggdrasil| (>5 min spent solving the environment) or fails with an error about conflicts and you are using a version of conda older than 4.7.2, try either updating conda, using mamba (see note above), and/or adding the conda-forge channel (if you havn't already)::
 
      conda config --add channels conda-forge