-
Notifications
You must be signed in to change notification settings - Fork 157
GloSED database incorporation #1022
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 13 commits
Commits
Show all changes
26 commits
Select commit
Hold shift + click to select a range
2723d4c
Merge pull request #1014 from nf-core/dev
d4straub 84c38dd
adding glosed to nf-core/ampliseq
tom-brekke 5ed586f
Merge branch 'nf-core:master' into GloSED_2
tom-brekke 5c28e3e
bumped up the memory for glosed
tom-brekke 036b0c1
bumped up the memory for glosed
tom-brekke ef61940
trying a slightly different taxonomy format
tom-brekke e62fa6c
updated citations to include GloSED database
tom-brekke 99f8390
updated usage doc to include the glosed database
tom-brekke b18e57b
updated changelog for GloSED addition
tom-brekke 9b8afb4
added glosed.nf.test.snap
tom-brekke 98604f3
added a versioned glosed as well as a most-up-to-date one into the re…
tom-brekke 4b4e760
updated changelog
tom-brekke 7090485
Merge branch 'dev' into GloSED_2
tom-brekke fc4609b
Update nextflow_schema.json
tom-brekke 39630b2
Update conf/ref_databases.config
tom-brekke 68c013d
[automated] Fix code linting
nf-core-bot e522174
Merge branch 'dev' into GloSED_2
tom-brekke 4fa332f
Merge branch 'dev' into GloSED_2
tom-brekke a4a5f11
updated test_glosed.config to use the scaled-down glosed database hos…
tom-brekke efd305e
moved the includeConfig conf/ref_databases.config line up above the p…
tom-brekke 8edf453
updated test_glosed.config to specify the test-databases
tom-brekke ee35794
updated paths and reformatting for the test dataset of glosed
tom-brekke 01ccdb9
[automated] Fix code linting
nf-core-bot aa6d88a
Fix typo in test for glosed 1k link
d4straub 72aea64
Re-add appropriate resource limits
d4straub 6fc8371
updated glosed test.snap file
tom-brekke File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,82 @@ | ||
| #!/bin/sh | ||
|
|
||
| # Reformat GloSED references for assignTaxonomy and addSpecies. | ||
| # Inputs (in current directory by default): | ||
| # - GloSED__OTU_sequences.fasta.gz | ||
| # - GloSED__Taxonomy.tsv.zip | ||
| # Outputs: | ||
| # - assignTaxonomy.fna | ||
| # - addSpecies.fna | ||
|
|
||
| set -eu | ||
|
|
||
| SEQ_GZ="${1:-GloSED__OTU_sequences.fasta.gz}" | ||
| TAX_ZIP="${2:-GloSED__Taxonomy.tsv.zip}" | ||
|
|
||
| [ -f "$SEQ_GZ" ] || { echo "ERROR: missing $SEQ_GZ" >&2; exit 1; } | ||
| [ -f "$TAX_ZIP" ] || { echo "ERROR: missing $TAX_ZIP" >&2; exit 1; } | ||
|
|
||
| TMPDIR_LOCAL="$(mktemp -d glosed_reformat.XXXXXX)" | ||
| trap 'rm -rf "$TMPDIR_LOCAL"' EXIT INT TERM | ||
|
|
||
| TAX_TSV="${TMPDIR_LOCAL}/glosed_taxonomy.tsv" | ||
| META_TSV="${TMPDIR_LOCAL}/glosed_meta.tsv" | ||
|
|
||
| unzip -p "$TAX_ZIP" > "$TAX_TSV" | ||
|
|
||
| awk -F '\t' 'BEGIN { OFS="\t" } | ||
| NR==1 { | ||
| for (i = 1; i <= NF; i++) { | ||
| if ($i == "OTU") otu = i | ||
| else if ($i == "Kingdom") kingdom = i | ||
| else if ($i == "Phylum") phylum = i | ||
| else if ($i == "Class") classcol = i | ||
| else if ($i == "Order") ordercol = i | ||
| else if ($i == "Family") family = i | ||
| else if ($i == "Genus") genus = i | ||
| else if ($i == "Species") species = i | ||
| } | ||
| next | ||
| } | ||
| { | ||
| id = $otu | ||
| k = $kingdom; p = $phylum; c = $classcol; o = $ordercol; f = $family; g = $genus; s = $species | ||
| gsub(/ /, "_", k); gsub(/ /, "_", p); gsub(/ /, "_", c); gsub(/ /, "_", o); gsub(/ /, "_", f); gsub(/ /, "_", g); gsub(/ /, "_", s) | ||
|
|
||
| tax = "" | ||
| if (k != "." && k != "") { | ||
| tax = k ";" | ||
| if (p != "." && p != "") { | ||
| tax = tax p ";" | ||
| if (c != "." && c != "") { | ||
| tax = tax c ";" | ||
| if (o != "." && o != "") { | ||
| tax = tax o ";" | ||
| if (f != "." && f != "") { | ||
| tax = tax f ";" | ||
| if (g != "." && g != "") { | ||
| tax = tax g ";" | ||
| if (s != "." && s != "") { | ||
| tax = tax s ";" | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
| print id, tax, g, s | ||
| } | ||
| ' "$TAX_TSV" > "$META_TSV" | ||
|
|
||
| gzip -dc "$SEQ_GZ" | awk -F '\t' 'NR==FNR { tax[$1] = $2; next } | ||
| /^>/ { id = substr($0,2); if (id in tax && tax[id] != "") print ">" tax[id]; else print ">" id; next } | ||
| { print } | ||
| ' "$META_TSV" - > assignTaxonomy.fna | ||
|
|
||
| gzip -dc "$SEQ_GZ" | awk -F '\t' 'NR==FNR { if ($3 != "." && $3 != "" && $3 != "NA" && $4 != "." && $4 != "" && $4 != "NA") addsp[$1] = $3 " " $4; next } | ||
| /^>/ { id = substr($0,2); keep = (id in addsp); if (keep) print ">" id " " addsp[id]; next } | ||
| { if (keep) print } | ||
| ' "$META_TSV" - > addSpecies.fna | ||
|
|
||
| echo "Created files: assignTaxonomy.fna addSpecies.fna" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| /* | ||
| ======================================================================================== | ||
| Nextflow config file for running minimal tests with GloSED database | ||
| ======================================================================================== | ||
| Defines input files and everything required to run a fast and simple pipeline test | ||
| using the GloSED (Global Standardised Soil Eukaryome Dataset) reference database. | ||
|
|
||
| Use as follows: | ||
| nextflow run nf-core/ampliseq -profile test_glosed,<docker/singularity> --outdir <OUTDIR> | ||
|
|
||
| ---------------------------------------------------------------------------------------- | ||
| */ | ||
|
|
||
|
d4straub marked this conversation as resolved.
|
||
| process { | ||
| resourceLimits = [ | ||
| cpus: 4, | ||
| memory: '256.GB', | ||
| time: '2.h' | ||
| ] | ||
| } | ||
|
|
||
| params { | ||
| config_profile_name = 'Test profile for GloSED database' | ||
| config_profile_description = 'Minimal test dataset to verify pipeline function with GloSED reference database' | ||
|
|
||
| // Input data | ||
| FW_primer = "GTGYCAGCMGCCGCGGTAA" | ||
| RV_primer = "GGACTACNVGGGTWTCTAAT" | ||
| input = params.pipelines_testdata_base_path + "ampliseq/samplesheets/Samplesheet.tsv" | ||
| dada_ref_taxonomy = "glosed" | ||
|
|
||
| skip_qiime = true | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| nextflow_pipeline { | ||
|
|
||
| name "Test pipeline with GloSED" | ||
| script "../main.nf" | ||
| tag "pipeline" | ||
| profile "test_glosed" | ||
|
|
||
| test("-profile test_glosed") { | ||
|
|
||
| when { | ||
| params { | ||
| outdir = "$outputDir" | ||
| } | ||
| } | ||
|
|
||
| then { | ||
| // stable_name: All files + folders in ${params.outdir}/ with a stable name | ||
| def stable_name = getAllFilesFromDir(params.outdir, relative: true, includeDir: true, ignore: ['pipeline_info/*.{html,json,txt}']) | ||
|
|
||
| assert workflow.success | ||
| assertAll( | ||
| { assert snapshot( | ||
| // pipeline versions.yml file for multiqc from which Nextflow version is removed because we test pipelines on multiple Nextflow versions | ||
| removeNextflowVersion("$outputDir/pipeline_info/software_versions.yml"), | ||
| // All stable path name, with a relative path | ||
| stable_name, | ||
| // Manually chosen files for content checks | ||
| path("$outputDir/overall_summary.tsv"), | ||
| path("$outputDir/barrnap/rrna.arc.gff"), | ||
| path("$outputDir/barrnap/rrna.bac.gff"), | ||
| path("$outputDir/barrnap/rrna.euk.gff"), | ||
| path("$outputDir/barrnap/rrna.mito.gff"), | ||
| path("$outputDir/cutadapt/cutadapt_summary.tsv"), | ||
| path("$outputDir/dada2/ASV_seqs.fasta"), | ||
| path("$outputDir/dada2/ASV_table.tsv"), | ||
| path("$outputDir/dada2/ref_taxonomy.glosed.txt"), | ||
| path("$outputDir/dada2/DADA2_stats.tsv"), | ||
| path("$outputDir/dada2/DADA2_table.tsv"), | ||
| path("$outputDir/input/Samplesheet.tsv"), | ||
| path("$outputDir/multiqc/multiqc_data/multiqc_fastqc.txt"), | ||
| path("$outputDir/multiqc/multiqc_data/multiqc_general_stats.txt"), | ||
| path("$outputDir/multiqc/multiqc_data/multiqc_cutadapt.txt") | ||
| ).match() } | ||
| ) | ||
| } | ||
| } | ||
| } |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.