v2.1.32.1

bugfixes:

• workflow sarscov2_nextstrain and sarscov2_nextstrain_aligned_input: bug fix to DAG -- ensure that treetime and ancestral inference are using masked alignments, not unmasked alignments [#358]
• task crsp_meta_etl: add more possible values to the controlled vocabulary options for body_part [#357]

v2.1.32.0

new features:

• most task runtime blocks now support cromwell auto memory scaling/retry
• automated data release and delivery sarscov2_data_release
• batch recalling of pango/nextclade lineages sarscov2_batch_relineage
• improved automated BioSample registration and metadata handling from Broad CRSP samples and external non-Broad samples via GP pipeline
• add sarscov2_biosample_load as optional subworkflow call at the beginning of sarscov2_illumina_full for fully automated use by Terra workflow launcher
• updated/improved Picard-based illumina demux
• move state public health reporting from sarscov2_illumina_full to sarscov2_data_release

bug fixes:

minor updates to docker images and vm shapes:

• pangolin 3.1.11 / pangolearn 2021-08-09
• nextclade 1.2.3
• vadr 1.3
• nextstrain 20210413T201712Z
• sc2-rmd, viral-core

build changes:

• GitHub Actions CI added, now primary. Travis CI still active at the moment

v2.1.28.0

new features:

• new workflow sarscov2_nextstrain [#204, #208, #219]
• updates to genbank submission [#201, #207]
• update vadr alert criteria based on NCBI recommendations [#234, #249]
• add nextclade tree outputs to sarscov2_illumina_full [#233]
• add sequencing reports via rmarkdown (sarscov2_illumina_full) [#222, #226, #228, #235, #236, #244, #245, #248, #265]
• ivar trim updates: emit ivar trim stats (assemble_refbased) and compute summary stats (sarscov2_illumina_full) [#237]
• terra table upload and download [#206, #241]
• add picard wgs metrics, alignment metrics, and insert size metrics to assemble_refbased and sarscov2_illumina_full [#239, #282]
• add bucket delivery of data for CDC, SRA, and GP reporting to sarscov2_illumina_full [#258, #263, #278]
• add tasks and workflows for NCBI BioSample registration and metadata retrieval [#279]
• automated filtering of libraries from failed NTC controls [#266]

bug fixes:

• bugfix whitespace handling in gzcat task [#230]
• deduplicate output rows from sra_meta_prep [#220]
• GISAID metadata output should be CSV not TSV [#273]
• derive Illumina run ID from XML instead of tarball filename [#275]

minor updates to docker images and vm shapes:

• vm shape updates on augur steps [#205, #224, #225, #229, #232]
• bump viral-core docker [#242, #243, #268, #270, #274, #276, #281, #284]
• bump ivar docker [#209]
• bump pangolin/pangolearn [#203, #205, #210, #213, #214, #215, #216, #217, #218, #240, #250, #254, #267, #271, #285]
• bump vadr docker [#216, #264]
• bump nextstrain/base [#238]
• bump sc2-rmd docker [#269, #277]
• update nextmeta tsv output behavior to match new nextmeta spellings [#231]

build changes:

• bump cromwell and womtool 54 to 61 [#272]
• temporarily drop dnanexus builds from Travis until we clean up the dnanexus CI project [#280, #283]

v2.1.19.0

Added new workflow: sarscov2_sra_to_genbank -- this takes sequencing reads from INSDC (via NCBI SRA), assembles, annotates, and QCs genomes, and produces Genbank and GISAID submission bundles based on the metadata in NCBI (SRA and BioSample). The Genbank submission will be tied to the same source BioProject and BioSamples that the reads were linked to in SRA. This workflow is able to merge together multiple read sets (SRA records) from the same BioSample and produce one assembly per BioSample. It will automatically detect sequencing platform (only Illumina and Oxford Nanopore currently supported) as well as amplicon vs metagenomic library designs based on the SRA metadata, and assemble appropriately. This has been tested on Illumina reads, ONT reads, amplicon libraries, metagenomic libraries, reads submitted to NCBI SRA, and reads originally submitted to ENA and synced with NCBI. [#197, #200]

Minor changes and fixes to sarscov2_illumina_full:

• filter genbank/gisaid submission packages to only sequences present in biosample attributes file [#200]
• relax minimum genome unambig bp cutoff from 20kb to 15kb [#200]
• allow for merging multiple biosample attributes tsvs together in sarscov2_illumina_full [#200]
• add "Sequencing Technology" column to both genbank and gisaid submission packages [#200]
• greatly simplify the final assembly metrics metadata output from both workflows (single tsv instead of compound array structures) [#200]
• makes filename outputs a bit more organized [#200]
• exposes cleaned_bam_uris text file output for easy SRA submission [#200]
• replace the first several steps with an invocation of demux_deplete as a subworkflow to reduce code duplication [#197]

Other minor changes:

• sarscov2_lineages and sarscov2_illumina_full: rename output variable pangolin_clade to pango_lineage to stay in line with the nomenclature of the PANGOLIN authors. [#197]
• increase default RAM for GATK UG consensus calling in assemble_refbased from 7GB to 15GB. [#200]
• bump nextclade image and pangoLEARN database to latest [#198]. nextclade update improves deletion variant naming. pangolin update keeps up with latest lineage assignments.
• bump viral-core docker 2.1.18 to 2.1.19 to fix demux scenario with single-index/paired-reads [#199]

v2.1.18.0

New general workflows:

• new workflow demux_deplete. This sits between demux_only (demux and fastqc) and demux_plus (which adds kraken, spades, etc) and just does demux, fastqc, and depletion. If optionally given "augmented" samplesheets and NCBI BioSample mappings, it will produce SRA submission bundles as well. [#191]
• new workflow mafft_and_snp_annotated, which adds snpEff annotation to the snp-sites output [#194]

New SARS-CoV-2 specific workflows. Up until this release, all included workflows were generally applicable to most viral taxa. This release includes a number of single-taxon workflows exclusively for SARS-CoV-2 in order to increase efficiency for high throughput work on this one virus.

• new workflow sarscov2_illumina_full is a full end-to-end workflow from Illumina BCL tarball through Genbank, SRA, and GISAID submission bundles. It wraps together demux_deplete, assemble_refbased, sarscov2_lineages, sarscov2_genbank. It requires the user to pre-register NCBI BioSample entries and to provide an "augmented" samplesheet for demux. [#191, #196]

• new workflow sarscov2_genbank. Prepares single-segmented genome assemblies for submission to NCBI Genbank using their new SARS-CoV-2 submission mechanism (which may become more mainstream for other viruses as well). Incorporates the new VADR (Viral Annotation DefineR) tool from NCBI to annotate (produce tbl files) and QC (flag frameshift and other problems) using the same settings that Genbank uses for QC -- this filters out genomes from submission that fail VADR QC and should result in Genbank submissions with no rejections. [#191]

• new workflow sarscov2_lineages and sarscov2_nextclade_multi. Runs Nextclade and Pangolin to do lineage/clade classification on SARS-CoV-2 genomes. [#184, #185, #186]

• nextstrain/augur workflow improvements and bugfixes to allow for merging of multiple metadata tsv files. This simplifies the process of regular builds where some data is changing frequently [#189, #181, #191]

• docker image updates: [#195, #190, #187, #182, #191, #193]

• VM shape updates [#188]

• README update with diagram [#183, @llangit-broad]

v2.1.12.0

Fixes:

• remove recursion limit for finding the RunInfo.xml file when unpacking sequencing run tarballs in illumina_demux [#179]

Updated:

• bump viral docker base layers to images based on viral-core 2.1.12 [#180]

v2.1.10.0

New or changed WDL workflows:

• new workflow: subsample_by_metadata_with_focal [#161]

Changes:

• rename workflow augur_from_newick to augur_export_only, add new workflow augur_from_mltree [#151]
• drop support for trinity (pinned version) assembler [#168]
• bump upstream docker images [#169, #163]
• README changes [#162]

Fixes:

• bugfix: isnvs_per_sample when specifying optional parameters [#160]
• bugfixes for use of set -o pipefail [#175, #173]
• bugfix: optional input handling in filter_bam_to_taxa [#171]

VM shape parameterization or changes to defaults:

• mafft parameterization [#170]
• memory increase to task filter_sequences_to_list [#159]
• parameterize BEAST GPU settings [#158]
• memory increase to assemble_refbased (specifically to task align_reads) [#157]
• memory increase to task multi_align_mafft [#155]
• memory increase & cpu decrease to task refine_augur_tree and ancestral_tree (the timetree invocations) [#153]
• parameterize CPU count for task draft_augur_tree (iqtree) [#150]

DNAnexus:

• update DNAnexus demux_launcher to v2 instance types [#156]
• update instance types to v2 [#167]

v2.1.8.0

Update to Dockstore 1.9 API

• direct github apps based integration for all branch commits and tagged releases to viral-pipelines to Dockstore (skips viral-ngs-staging intermediary) using their latest API released a few days ago [#139, #140]

Workflow updates:

• fixes to NCBI Fetch_SRA_to_BAM [#138]
• new workflow classify_single (a single-sample version of classify_multi) [#133]
• new mafft_and_snp [#132, #133]
• replace all augur align calls with our own mafft invocation that is much faster [#132]
• augur_from_msa can take multiple keep_list inputs [#134]
• allow kraken2 task to take fasta input, not just bam (to allow for running on contigs) [#127]

Misc:

• various upstream docker image version updates [#126, #136, #137, #138]
• VM shape tuning [#128, #129]
• dxWDL update [#131]

v2.1.4.1

• assemble_refbased: additional stats output (num_read_groups, num_libraries, dist_to_ref_snps, dist_to_ref_indels) and more stringent default coverage requirements for run_discordance stats [#125]
• correct semantic versioning to match viral-core base version 2.1.4

v2.1.0.4

assemble_refbased workflow:

• default to minimap2 aligner instead of novoalign [#123, #110]
• add run_discordance task and output metrics for measuring discordance between sequencing runs within a sample [#124]
• disable ivar trim quality trimming due to unidirectional trim behavior (only run ivar trim for PCR primers) [#122]
• remove MultiQC from refbased workflow [#82]

metagenomics:

• bugfixes for kraken2 build [#120, #117]
• bugfixes for kaiju task [#113]
• remove blastx (optional) tasks from classify_multi [#102, #106]

BEAST (GPU):

• increase dnanexus runtime limit and specify dx_instance type [#116, #118]
• update docker image [#111]

nextstrain / phylogenetics:

• add snp-sites task to emit vcf files [#115]
• add augur mask step [#99]
• VM shape tuning [#112]
• rename and reorganize workflows a bit so they are all called augur_from_xxx [#121, #101]

miscellaneous:

• demux workflows: append lane number to bam filenames [#98, #100]
• update viral-core [#105, #124]
• update dxWDL [#103]