Merge branch 'release/0.9.1'

Author: simonvh

.codeclimate.yml

@@ -7,6 +7,10 @@ plugins:
       count_threshold: 3
 
 checks:
+  method-complexity:
+    enabled: true
+    config:
+      threshold: 10
   argument-count:
     enabled: false
     config:

.travis.yml

@@ -40,7 +40,9 @@ before_script:
 script:
   - black --check setup.py genomepy/ tests/
   - flake8 setup.py genomepy/ tests/
-  - pytest -vv --disable-pytest-warnings --cov=genomepy --cov-config=tests/.coveragerc --cov-report=xml tests/*
+  - pytest -vv --disable-pytest-warnings tests/*
+    --reruns 1 --reruns-delay 10
+    --cov=genomepy --cov-config=tests/.coveragerc --cov-report=xml
 
 after_script:
   # send the coverage data to Code Climate

CHANGELOG.md

@@ -6,51 +6,87 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 
 ## [Unreleased]
 
+### Added
+
+### Changed
+
+### Fixed
+
 ## [0.9.0] - 2020-09-01
 
 ### Added
+- `genomepy install` flag `-k/--keep-alt` to keep alternative regions
+- argparse custom type for a genome command line argument
+
+### Changed
+
+- added retries to UCSC and NCBI
+- added retries to Travis tests
+- Bucketcache improvements
+- `genomepy search` keeps searching after an exact match is found
+- `genomepy install` removes alternative regions by default
+
+### Fixed
+
+- `genomepy clean` wont complain when there is nothing to clean
+- properly gzip the annotation.gtf if it was unzipped during sanitizing
+- `genomepy install` can use the URL provider again
+- `genomepy install` with `-f/--force` will overwrite previouse sizes and gaps files
+
+## [0.9.0] - 2020-09-01
+
+### Added
+
 - check to see if providers are online + error message if not
 - automatic provider selection for `genomepy install`
   - optional provider flag for `genomepy install` (`-p/--provider`)
   - if no provider is passed to `genomepy install`, the first provider with the genome is used (order: Ensembl > UCSC > NCBI).
 - `genomepy clean` removes local caches. Will be reloaded when required.
 
 ### Changed
+
 - Ensembl genomes always download over ftp (http was too unstable)
 - Ensembl release versions obtained via REST API (http was too unstable)
 - `genomepy search` and `genomepy providers` only check online providers
 - Online function now have a timeout and a retry system
 - API changes to `download_genome` and `download_annotation` for consistency
 
 ### Fixed
+
 - Ensembl status check uses lighter url (more stable)
 - `search` and `install` now consistently use safe search terms (no spaces)
 - `search` now uses UTF-8, no longer crashing for \u2019 (some quotation mark).
 - `search` case insensitivity fixed for assembly names.
 - Bucketcache now stores less data, increasing responsiveness.
 
-
 ## [0.8.4] - 2020-07-29
+
+### Fixed
+
 - Fix bug where Genome.sizes dict contains str instead of int (#110).
 - Fix bug with UTF-8 in README (#109).
 - Fix bug where BED files with chr:start-end in 4th column are not recognized as BED files.
 
 ## [0.8.3] - 2020-06-03
 
 ### Fixed
+
 - Fixed bug introduced by fixing a bug: Provider-specific options for `genomepy install` on command line work again
 - UCSC annotations can now once again be obtained from knownGene.txt
 
 ### Added
+
 - UCSC gene annotations will now be downloaded in GTF format where possible
 - Desired UCSC gene annotation type can now be specified in the `genomepy install` command using `--ucsc-annotation`
 
 ### Changed
+
 - Added the NCBI RefSeq gene annotation to the list of potential UCSC gene annotations for download
 
 ## [0.8.2] - 2020-05-25
 
 ### Fixed
+
 - `Genome.sizes` and `Genome.gaps` are now populated automatically.
 - backwards compatibility with old configuration files (with `genome_dir` instead of `genomes_dir`)
 - updating the README.txt will only happen if you have write permission
@@ -64,6 +100,7 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 ## [0.8.1] - 2020-05-11
 
 ### Added
+
 - Now using the UCSC REST API
 - `genomepy search` now accepts taxonomy IDs
 - `genomepy search` will now return taxonomy IDs and Accession numbers
@@ -80,6 +117,7 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
     - STAR and HISAT2 will now generate splice-aware indexes if annotation files are available.
 
 ### Changed
+
 - `Genome.props` has been renamed to `Genome.plugin`
 - sizes no longer a plugin, but always gets executed
 - `genomepy FUNCTION --help` texts expanded
@@ -88,18 +126,21 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 - more descriptive feedback to installing & many errors
 
 ### Removed
+
 - Sizes plugin
 - Old tests
 - Removed outdated dependency `xmltodict`
 
 ### Fixed
+
 - `genomepy config` options made more robust
 - README.txt will no longer:
   - update 3x for each command
   - drop regex info
   - have duplicate lines
 
-### Refactoring
+### Changed
+
 - Genome class moved to `genome.py`
 - Many functions moved to `utils.py`
 - Many other functions made static methods of a class
@@ -123,17 +164,20 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 ## [0.7.2] - 2019-03-31
 
 ### Fixed
+
 - Fix minor issue with hg19 wrong blacklist url
 - Ensembl downloads over http instead of https (release 99 no longer has https)
 
 ## [0.7.1] - 2019-11-20
 
 ### Fixed
+
 - STAR is not longer enabled by default
 
 ## [0.7.0] - 2019-11-18
 
 ### Added
+
 - Direct downloading from url through url provider.
 - Added `--force` flag. Files will no longer be overwritten by default.
 - Provider specific options:
@@ -143,12 +187,14 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 - Added STAR index plugin
 
 ### Changed
+
 - Providers are now case-insensitive.
 - Extended testing.
 - Increased minimal Python version to 3.6.
 - Removed gaps from plugins, added gaps to core functionality.
 
 ### Fixed
+
 - bugfix: NCBI will show all versions of an assembly (will no longer filter on BioSample ID, instead filters on asm_name).
 - fix: gaps file will be generated when needed.
 
@@ -222,6 +268,7 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 - Added `-r` and `--match/--no-match` option to select sequences by regex.
 
 [Unreleased]: https://github.com/vanheeringen-lab/genomepy/compare/master...develop
+[0.9.0]: https://github.com/vanheeringen-lab/genomepy/compare/0.8.4...0.9.0
 [0.8.4]: https://github.com/vanheeringen-lab/genomepy/compare/0.8.3...0.8.4
 [0.8.3]: https://github.com/vanheeringen-lab/genomepy/compare/0.8.2...0.8.3
 [0.8.2]: https://github.com/vanheeringen-lab/genomepy/compare/0.8.1...0.8.2

README.md

@@ -22,18 +22,18 @@ Currently, genomepy supports UCSC, Ensembl and NCBI.
 **Pssst, hey there!** Is genomepy not doing what you want? Does it fail? Is it clunky? Is the documentation unclear? Have any other ideas on how to improve it? Don't be shy and [let us know](https://github.com/vanheeringen-lab/genomepy/issues)!
 
 ## Table of Contents
-1.  [Installation](#Installation)
-2.  [Quick usage](#Quick-usage)
-3.  [Plugins and indexing](#Plugins-and-indexing)
-4.  [Configuration](#Configuration)
-5.  [Usage](#Usage)
-    * [Command line](#Command-line)
-    * [Python](#Python)
-6.  [Known Issues](#Known-Issues)
-7.  [Citation](#Citation)
-8.  [Getting help](#Getting-help)
-9.  [Contributing](#Contributing)
-10.  [License](#License)
+1.  [Installation](#installation)
+2.  [Quick usage](#quick-usage)
+3.  [Plugins and indexing](#plugins-and-indexing)
+4.  [Configuration](#configuration)
+5.  [Usage](#usage)
+    * [Command line](#command-line)
+    * [Python](#python)
+6.  [Known issues](#known-issues)
+7.  [Citation](#citation)
+8.  [Getting help](#getting-help)
+9.  [Contributing](#contributing)
+10.  [License](#license)
 
 
 ## Installation
@@ -86,7 +86,7 @@ You can find the binaries [here](http://hgdownload.cse.ucsc.edu/admin/exe/).
 
 ## Plugins and indexing
 
-By default genomepy generates a file with chromosome sizes and a BED file with
+By default genomepy generates an index, a file with chromosome sizes and a BED file with
 gap locations (Ns in the sequence). 
 
 For some genomes genomepy can download blacklist files (generated by the Kundaje lab). 
@@ -172,9 +172,10 @@ Options:
   -h, --help  Show this message and exit.
 
 Commands:
+  clean      remove provider data
   config     manage configuration
   genomes    list available genomes
-  install    install genome
+  install    install a genome & run active plugins
   plugin     manage plugins
   providers  list available providers
   search     search for genomes
@@ -434,36 +435,44 @@ Note that the first time you run `genomepy search` or `list` the command will ta
 as the genome lists have to be downloaded. 
 The lists are cached locally, which will save time later. The cached files are stored in 
 `~/.cache/genomepy` and expire after 7 days. You can also delete this directory to clean the 
-cache.
+cache using `genomepy clean`.
 
 ### Python
 
-```python
+```
 >>> import genomepy
 >>> for row in genomepy.search("GRCh38"):
-...     print("\t".join(row))
-...
-UCSC	hg38	Human Dec. 2013 (GRCh38/hg38) Genome at UCSC
-NCBI	GRCh38.p10	Homo sapiens; Genome Reference Consortium
-NCBI	GRCh38	Homo sapiens; Genome Reference Consortium
-NCBI	GRCh38.p1	Homo sapiens; Genome Reference Consortium
-NCBI	GRCh38.p2	Homo sapiens; Genome Reference Consortium
-NCBI	GRCh38.p3	Homo sapiens; Genome Reference Consortium
-NCBI	GRCh38.p4	Homo sapiens; Genome Reference Consortium
-NCBI	GRCh38.p5	Homo sapiens; Genome Reference Consortium
-NCBI	GRCh38.p6	Homo sapiens; Genome Reference Consortium
-NCBI	GRCh38.p7	Homo sapiens; Genome Reference Consortium
-NCBI	GRCh38.p8	Homo sapiens; Genome Reference Consortium
-NCBI	GRCh38.p9	Homo sapiens; Genome Reference Consortium
-Ensembl	GRCh38.p10	Human
->>> genomepy.install_genome("hg38", "UCSC", genomes_dir="/data/genomes")
-downloading...
-done...
+...    print("\t".join([x.decode('utf-8') for x in row]))
+...    
+GRCh38.p13	Ensembl	GCA_000001405.28	Homo sapiens	9606	2014-01-Ensembl/2020-03
+hg38	UCSC	GCA_000001405.27	Homo sapiens	9606	Dec. 2013 (GRCh38/hg38)
+GRCh38	NCBI	GCA_000001405.15	Homo sapiens	9606	Genome Reference Consortium
+GRCh38.p1	NCBI	GCA_000001405.16	Homo sapiens	9606	Genome Reference Consortium
+GRCh38.p2	NCBI	GCA_000001405.17	Homo sapiens	9606	Genome Reference Consortium
+GRCh38.p3	NCBI	GCA_000001405.18	Homo sapiens	9606	Genome Reference Consortium
+GRCh38.p4	NCBI	GCA_000001405.19	Homo sapiens	9606	Genome Reference Consortium
+GRCh38.p5	NCBI	GCA_000001405.20	Homo sapiens	9606	Genome Reference Consortium
+GRCh38.p6	NCBI	GCA_000001405.21	Homo sapiens	9606	Genome Reference Consortium
+GRCh38.p7	NCBI	GCA_000001405.22	Homo sapiens	9606	Genome Reference Consortium
+GRCh38.p8	NCBI	GCA_000001405.23	Homo sapiens	9606	Genome Reference Consortium
+GRCh38.p9	NCBI	GCA_000001405.24	Homo sapiens	9606	Genome Reference Consortium
+GRCh38.p10	NCBI	GCA_000001405.25	Homo sapiens	9606	Genome Reference Consortium
+GRCh38.p11	NCBI	GCA_000001405.26	Homo sapiens	9606	Genome Reference Consortium
+GRCh38.p12	NCBI	GCA_000001405.27	Homo sapiens	9606	Genome Reference Consortium
+GRCh38.p13	NCBI	GCA_000001405.28	Homo sapiens	9606	Genome Reference Consortium
+
+>>> genomepy.install_genome("hg38", "UCSC", genomes_dir="./data/genomes")
+Downloading genome from UCSC.
+Target URL: http://hgdownload.soe.ucsc.edu/goldenPath/hg38/bigZips/hg38.fa.gz...
+Genome download successful, starting post processing...
 name: hg38
-fasta: /data/genomes/hg38/hg38.fa
->>> g = genomepy.Genome("hg38", genomes_dir="/data/genomes")
->>> g["chr6"][166502000:166503000]
-tgtatggtccctagaggggccagagtcacagagatggaaagtggatggcgggtgccgggggctggggagctactgtgcagggggacagagctttagttctgcaagatgaaacagttctggagatggacggtggggatgggggcccagcaatgggaacgtgcttaatgccactgaactgggcacttaaacgtggtgaaaactgtaaaagtcatgtgtatttttctacaattaaaaaaaATCTGCCACAGAGTTAAAAAAATAACCACTATTTTCTGGAAATGGGAAGGAAAAGTTACAGCATGTAATTAAGATGACAATTTATAATGAACAAGGCAAATCTTTTCATCTTTGCCTTTTGGGCATATTCAATCTTTGCCCAGAATTAAGCACCTTTCAAGATTAATTCTCTAATAATTCTAGTTGAACAACACAACCTTTTCCTTCAAGCTTGCAATTAAATAAGGCTATTTTTAGCTGTAAGGATCACGCTGACCTTCAGGAGCAATGAGAACCGGCACTCCCGGCCTGAGTGGATGCACGGGGAGTGTGTCTAACACACAGGCGTCAACAGCCAGGGCCGCACGAGGAGGAGGAGTGGCAACGTCCACACAGACTCACAACACGGCACTCCGACTTGGAGGGTAATTAATACCAGGTTAACTTCTGGGATGACCTTGGCAACGACCCAAGGTGACAGGCCAGGCTCTGCAATCACCTCCCAATTAAGGAGAGGCGAAAGGGGACTCCCAGGGCTCAGAGCACCACGGGGTTCTAGGTCAGACCCACTTTGAAATGGAAATCTGGCCTTGTGCTGCTGCTCTTGTGGGGAGACAGCAGCTGCGGAGGCTGCTCTCTTCATGGGATTACTCTGGATAAAGTCTTTTTTGATTCTACgttgagcatcccttatctgaaatgcctgaaaccggaagtgtttaggatttggggattttgcaatatttacttatatataatgagatatcttggagatgggccacaa
+local name: hg38
+fasta: ./data/genomes/hg38/hg38.fa
+
+>>> g = genomepy.Genome("hg38", genomes_dir="./data/genomes")
+>>> g["chr6"][166502000:166502100]
+>chr6:166502001-166502100
+tgtatggtccctagaggggccagagtcacagagatggaaagtggatggcgggtgccgggggctggggagctactgtgcagggggacagagctttagttct
 ```
 
 The `genomepy.Genome()` method returns a Genome object. This has all the
@@ -472,9 +481,29 @@ see the [documentation](https://github.com/mdshw5/pyfaidx) for more examples on
 
 ## Known issues
 
-There might be issues with specific genome sequences.
-Sadly, not everything (naming, structure, filenames) is always consistent on the provider end. 
-Let me know if you encounter issues with certain downloads.
+Genomepy utilizes external databases to obtain your files. Unfortunately this sometimes causes issues.
+Here are some of the more common issues with solutions.
+
+Let us know if you encounter issues you cannot solve!
+
+#### Provider is offline/URL is broken
+Occasionally one of the providers experience connection issues, which can last anywhere between seconds to hours.
+When this happens genomepy will warn that the provider is offline, or that the URL is broken.
+
+Connection issues are usually resolved in minutes.
+
+#### A genome is missing from `genomepy search`
+Genomepy stores provider data on your computer to rerun it faster later.
+If a provider was offline during this time, it may miss (parts of) the data.
+
+To re-download the data, remove the local data with `genomepy clean`, then `search` for your genome again.
+
+#### URL is still broken
+Sadly, not everything (naming, structure, filenames) is always consistent on the provider end. Contact the provider to get it fixed!
+One notable group are Ensembl fungi, which seems to be mostly mislabelled.
+
+In the meantime, you can still use the power of genomepy by manually retrieving the URLs,
+and downloading the files with `genomepy install GENOME_URL -p url --url-to-annotation ANNOTATION_URL`.
 
 ## Citation
 

environment.yml

@@ -41,4 +41,5 @@ dependencies:
   - flake8
   - flake8-bugbear
   - pytest
+  - pytest-rerunfailures
   - pytest-cov

genomepy/__about__.py

@@ -1,3 +1,3 @@
 """Metadata"""
-__version__ = "0.9.0"
+__version__ = "0.9.1"
 __author__ = "Simon van Heeringen"

genomepy/argparse_support.py

@@ -0,0 +1,33 @@
+import sys
+import argparse
+from genomepy.functions import Genome, install_genome
+
+
+def parse_genome(auto_install=False, genomes_dir=None):
+    """argparse action for command-line genome option.
+
+    Parameters
+    ----------
+        auto_install : bool, optional
+            Install a genome if it's not found locally.
+
+        genomes_dir : str, optional
+            Directory to look for and/or insall genomes.
+    """
+
+    class GenomeAction(argparse.Action):
+        def __call__(self, parser, args, name, option_string=None):
+            try:
+                genome = Genome(name, genomes_dir=genomes_dir)
+            except FileNotFoundError:
+                print(f"Genome {name} not found!")
+                if auto_install:
+                    print("Trying to install it automatically using genomepy...")
+                    install_genome(name, annotation=True, genomes_dir=genomes_dir)
+                    genome = Genome(name, genomes_dir=genomes_dir)
+                else:
+                    print("You can install it using `genomepy install`.")
+                    sys.exit(1)
+            setattr(args, self.dest, genome)
+
+    return GenomeAction