The format is based on Keep a Changelog and this project adheres to Semantic Versioning.
Our 15th release for sanger-tol/treeval.
- Update to documentation.
- Correct
RAPID-TOLtoRAPID_TOL
- Correct
- Better defaults for various paramters if not needed by user.
- Remove the
kmer_profarguments from the yaml. This wasn't in use by the pipeline. - Replace the above with the param
kmer_length. - Added an example
params-filehere.
| Old Parameter | New Parameter |
|---|---|
| - | --kmer_length |
Our 14th release for sanger-tol/treeval.
- Bug fix, .sizes file was no longer being output, due to being published by subworkflow.
Our 13th release for sanger-tol/treeval.
- Update to pretext map, adds option to generate Ultra HiRes map.
- Changes to confirm with nf-core strict syntax.
| Old Parameter | New Parameter |
|---|---|
| - | --run_ultra |
| Module | Old Version | New Versions |
|---|---|---|
| GET_LARGEST_SCAFFOLD (coreutils) | 9.1 | REMOVED |
| busco/busco | 5.7.1 | 6.0.0 |
| bwamem2/index (bwa-mem2) | 2.2.1 | 2.3+htslib=1.22.1+samtools=1.22.1 |
| cat/cat (pigz) | 2.3.4 | 2.8 |
| cooler/cload (cooler) | 0.9.2 | 0.10.4 |
| cooler/zoomify (cooler) | 0.9.2 | 0.10.4+numpy=1.26.4 |
| custom/dumpsoftwareversions | multiqc=1.20 | REMOVED |
| custom/getchromsizes (htslib+samtools) | 1.21+1.22.1 | REMOVED |
| fastk/fastk | 1.0.2 | 1.2 |
| gnu/sort (coreutils) | 9.3 | coreutils=9.5 |
| juicertools (juicertools) | 1.8.9+java-jdk=8.0.112 | 2.20.00+openjdk=23.0.2 |
| merquryfk/merquryfk | fastk=1.0.2 | 1.2 |
| minimap2/align (minimap2+samtools) | 2.28+1.20+htslib=1.20 | 2.29+1.21 |
| minimap2/index (minimap2) | 2.28 | 2.29 |
| pretextmap (pretextmap+samtools) | 0.1.9+1.17 | 0.2.4+1.23+htslib=1.23 |
| pretextsnapshot | 0.0.4 | 0.0.7 |
| samtools/faidx (htslib+samtools) | 1.21+1.21 | 1.22.1+1.22.1 |
| samtools/index (htslib+samtools) | 1.21+1.21 | REMOVED |
| samtools/markdup (htslib+samtools) | 1.21+1.21 | 1.22.1+1.22.1 |
| samtools/merge (htslib+samtools) | 1.21+1.21 | 1.22.1+1.22.1 |
| samtools/sort (htslib+samtools) | 1.21+1.21 | REMOVED |
| samtools/view (htslib+samtools) | 1.21+1.21 | REMOVED |
| tabix/bgziptabix | 1.20+tabix=1.11 | 1.21 |
| windowmasker/mkcounts (blast) | 2.15.0 | 2.17.0 |
| windowmasker/ustat (blast) | 2.15.0 | 2.17.0 |
| yahs | 1.2a.2 | 1.2.2 |
Our 12th release for sanger-tol/treeval.
- Bug fix, pretext files weren't being output due to the use of a closure rather than a list in modules.config.
- Updated to BEDTOOLS INTERSECT resources
Our 11th release for sanger-tol/treeval.
- Replaced the TELO_FINDER subworkflow with the SANGER-TOL/TELO_FINDER
- By request, added split telomere ingestion to have the 5 and 3 prime directions separately displayable in Pretext. This, by default is false.
- Updated the code in PRETEXT_GRAPH to handle the new telo input which maybe split, whole or both.
- In split mode, the whole telomere will also be ingested.
- Reference input can now be gzipped.
- Moved the GAWK_UPPER_SEQUENCE module into the workflow main.nf file from the telomere subworkflow.
- Updated list for modes that run synteny.
- Removed redundant groovy.
- Fix for params not correctly running various components of the pipeline.
GAWK_SPLIT_TELOMEREhas been updated to use the nf-core module rather than a slightly custom local version.
| Old Parameter | New Parameter |
|---|---|
| - | --split_telomere |
| Module | Old Version | New Versions |
|---|---|---|
| gunzip | NA | 1.13 |
Our 10th release for sanger-tol/treeval, containing bug fixes.
- Bug fix fixing mode selecting subworkflows.
Our 9th release for sanger-tol/treeval, containing bug fixes.
- Bug fix preventing production of bin files.
- Update handling of input params.
Our 8th release for sanger-tol/treeval adding a new mode.
- Add FULL_COMBINED mode - running all subworkflows aside from gene_alignment and selfcomp - as not worth running on combined maps.
- Update resource allocation based on usage.
Our 7th release for sanger-tol/treeval.
- NF-Core template upgrade to 3.3.1.
- Pipeline now uses NF-Schema for input validation.
- Pipeline now uses nf-test for local and CI testing.
- Updating the structure of modules and subworkflows to follow new standards.
- Updating Channel names to follow new standards.
- Addition of
run_hiresflag (boolean) to control use of the hires pretext modules. - Removal of unused modules such as
avgcov, the average coverage module. - Addition of the
modeparameter, to replace the depreciating-entryflag.- Mode lists are 'include' lists, listing the processes needed per run.
- Removing entry points has significantly simplified the pipeline;
modenow controls a set ofstepsfor pipeline execution. - Addition of specified reads.
- Longreads and HiC are now specified in the input yaml file. Details in the usage document.
- Longread and HiC data can optionally take a fofn (file of file names) where each line contains one file, see
YAML_INPUTfunctionfn_get_validated_channel. - NOTE: in cases where the same file is in both the fofn and a supplied list, the pipeline will exit with an error which will be detailed in the nextflow.log file.
- Corrected input values.
- SummaryStats Code has been removed, this will be replaced by a plugin in the future!
- Replace 5 modules with GAWK instead of cat | sed pattern modules.
- Simplification of the
YAML_INPUTsubworkflow. - Removal of
GrabFilesand replacement with the newer.resolve(). - Cleaning up of
it -> it[0]into the more correct_meta, file -> filepattern. - Moved GAWK commands into their own awk script files, this cleans up modules.config.
- Added
my_absfunction to theGAWK_REFORMAT_INTERSECTto caluclate difference between start and end.
- Added
- Added
[TreeVal:Error]and[TreeVal:Info]into intentional print statements to make it easier to see on the CLI and to search for in logs.
| Old Parameter | New Parameter |
|---|---|
| - | --mode |
| - | --run_hires |
| Module | Old Version | New Versions |
|---|---|---|
| generate_cram_csv | 1.0 | 1.1 |
| GAWK | - | 5.3.0 |
| GET_BUSCO_GENES | coreutils=9.1 | - |
| REPLACE_DOTS | coreutils=9.1 | - |
| GAP LENGTH | coreutils=9.1 | - |
| REFORMAT INTERSECT | coreutils=9.1 | - |
| RENAME IDS | coreutils=9.1 | - |
- Bug fix for pretext_graph
- Tracks were not being named correctly, ending up being called
null.
- Tracks were not being named correctly, ending up being called
- Updated CI so a list of all output from the pipeline is printed.
Our 6th release for sanger-tol/treeval, a mix of bug fixes and standardising modules.
- Updated resources for larger assemblies to avoid hugemem and teramem queues at sanger.
- Update
PRETEXT_INGESTIONto usePRETEXT_GRAPHinstead. - Update
SELFCOMPsubworkflow to not require motif_len to be specified. - Add
GAWK_UPPER_SEQUENCEto force the assembly into uppercase to make telo_finding easier. - Add
GAWK_CLEAN_TELOto clean the telomere file when lowercase motif is specified / or telomotif not found in sequence, otherwise this may have "you screwed up" (legacy internal error message will be changed to something more professional) errors which will break processing inFIND_TELOMERE_WINDOWS. - Add
GAWK_MAP_TELOwhich will act as a fallback option when the original windows output is empty. - Professionalise
FIND_TELOMERE_REGIONSand update module. - Update PRETEXT_GRAPH to latest release, which fixes a bug in file ingestion.
- Update resource requirements for
BAMTOBED_SORTto reduce waste.
Current minimum version of nextflow does not support arity so this has been temporarily removed until for 1.4.0.
| Module | Old Version | New Versions |
|---|---|---|
GAWK as GAWK_UPPER_SEQUENCE |
- | 5.3.0 |
GAWK as GAWK_CLEAN_TELO |
- | 5.3.0 |
PRETEXT_GRAPH |
0.0.8-yy5-c1 | 0.0.9-yy5-c2 |
FIND_TELOMERE_REGIONS |
0.0.1-c1 | 0.0.9-yy5-c2 |
Our 5th release for sanger-tol/treeval, correcting a software bug inside PretextGraph.
- Correction to the PRETEXT_GRAPH module, remade pretextgraph container with newest version 0.0.8.
- Converted shell block modules into script block modules
- Update to documentation.
- Read data (longread and hic) are now checked for extension (this will look for fa(sta).gz or fofn---containing fa(asta).gz files ).
| Module | Old Version | New Versions |
|---|---|---|
pretextmap + samtools |
0.0.2-c4 + 1.17 | 0.0.8-c1 + 1.17 |
Our 4th release for sanger-tol/treeval, focusing on refining methods.
- Spelling mistake for the steps parameter:
--steps gene_alignmentrather than--steps gene_alignments - Remove extra characters
- Correction to the PRETEXT_GRAPH module.
| Module | Old Version | New Versions |
|---|---|---|
pretextmap + samtools |
0.0.3 + 1.17 | 0.0.2-c4 + 1.17 |
Our 3rd release for sanger-tol/treeval.
- Togglable subworkflows
- Adds a JBrowse Only workflow (this will lead to an update to the FULL workflow which can now call JBROWSE_ONLY and RAPID).
- Updates to containers (local modules) to remove Anaconda dependencies following policy changes.
- Updates to modules to remove Anaconda dependencies following policy changes
- The majority of these updates only remove the
defaultchannel from the environment.yml
- The majority of these updates only remove the
- CONDA warnings for modules which cannot use CONDA.
- Removable of a liberal use of spaces.
- reformat_intersect was previously not outputing version data.
- Adding arch specification to Pretext GitHub actions runner. Hopefully this will stop the spurious errors we see on there.
- Addition of steps into schema.
- Adds *ktab as an output.
- Adds *bin as an output for faster downsteam map generation.
- Updated singularity containers
- Added
--metaeukto BUSCO_BUSCO, default was causing pipeline errors on Actions -- Needs more investigation. - Replaced Pyfasta split (depreciated 6 years ago) with Seqkit split which is frequently updated and very fast.
- Allocated resource review
| Old Parameter | New Parameter |
|---|---|
| - | --steps |
Note, since the pipeline is using Nextflow DSL2, each process will be run with its own Biocontainer. This means that on occasion it is entirely possible for the pipeline to be using different versions of the same tool. However, the overall software dependency changes compared to the last release have been listed below for reference.
| Module | Old Version | New Versions |
|---|---|---|
| bamtobed_sort ( bedtools + samtools ) | 2.31.0 + 1.17 | |
| bedtools | 2.31.1 | - |
| busco | 5.5.0 | - |
| bwa-mem2 | 2.2.1 | |
| cat | 2.3.4 | |
| chunk_fasta ( pyfasta ) | 0.5.2-1 | REMOVED |
| cooler | 0.9.2 | |
| cram_filter_align_bwamem2_fixmate_sort | - | |
| ^ ( samtools + bwamem2 ) ^ | 1.17 + 2.2.1 | |
| coreutils | 9.1 | |
| fastk | 1.0.1 | |
| gcc | 10.4.0 | |
| find_telomere_windows ( java-jdk ) | 8.0.112 | |
| generate_cram_csv ( samtools ) | 1.17 | |
| gnu-sort | 8.25 | 9.3 |
| juicer_tools_pre ( java-jdk ) | 8.0.112 | |
| perl | 5.26.2 | |
| merquryfk | 1.0.1 | |
| minimap2 + samtools | 2.24 + 1.14 | |
| minimap2_index | 2.24 | 2.28 |
| miniprot | 0.11--he4a0461_2 | |
| mummer | 3.23 | |
| paftools ( minimap2 + samtools ) | 2.24 + 1.14 | |
| pretextmap + samtools | 0.0.2 + 1.17 | 0.0.3 + 1.17 |
| python | 3.9 | - |
| - pandas | 1.5.2 | - |
| samtools | 1.18 | 1.21 |
| selfcomp_splitfasta ( perl-bioperl ) | 1.7.8-1 | |
| seqtk | 1.4 | |
| seqkit | ADDED | 2.9.0--h9ee0642_0 |
| tabix | 1.11 | |
| ucsc | 377 | 447 |
| windowmasker (blast) | 2.14.0 | 2.15.0 |
- busco is currently pinned to v5.5.0 - Upgrading v5.7.1 would cause github actions to crash. Further investigation needed.
- Generate CRAM CSV fix to allow for multi-readgroup cram files
- Removing KMER_READCOV
- tmp directory was being used
- Output file adjustment (names and location)
The second release for sanger-tol, created with the nf-core template.
This builds on the initial release by adding subworkflows which generate kmer based coverage tracks and a kmer spectra graph. There are also a number of updates to logic used throughout the pipeline, as well as to the resources required by a significant number of modules.
- Updates to the resource allocation methods used by a number of modules in the base.config.
- Added a flag to stop the usage of Juicer.
- Subworkflow to generate a kmer based coverage track.
- Subworkflow to generate/update a kmer spectra graph.
- Subworkflow to use minimap2 for HiC mapping, if selected.
- Subworkflow to use BWAmem2 for HiC mapping, if selected.
- Subworkflow to ingest Pretext accessory files into the Pretext file, simplifying post-TreeVal data manipulation.
- Updated the logic in use throughout the pipeline.
- Updated the modules.config to include some of the logic, cleaning the code.
- Updated the HiC subworkflow to include subsampling the HiC data for Juicer due to resource requirements with large amounts of data.
- Updated the YAML_INPUT subworkflow, this now contains "flags" to change some software options.
- Updated the data names in the input YAML to reduce confusion.
- Updated software (Pretext{View,Snapshot,Graph}) to allow for use on large genomes with big data.
- Added associated patch files and cpu architecture files.
- Updated the minimap2 align module to remove samtools view in preference of paftools for our usecase.
- Updated the test.yml inline with the above changes.
- Updated the SELFCOMP subworkflow to allow for the parallelisation of the work on large genomes.
- Updated the READ_COVERAGE subworkflow to produce the scaffold based AVG coverage and STND coverage
- Updated Modules from NF-Core - mostly relates to module structure rather than software.
- Updated the SummaryStats output to include HiC container counts.
- Added -T / -t flags where possible to minimise the use of the /tmp directory.
- Replaced CONCAT_MUMMER with CATCAT for simplicity.
- Removed JUICER from the RAPID entrypoint.
- Removed the csi or tbi logic. CSI is now used by default, this simplified the workflow and enlarges the capacity to handle much larger genomes. The logic block previously required was then moved.
- Added NF-DOWNLOAD to the CI-CD due to an error that causes incomplete downloaded when downloading a number of images at the same time.
- Added the RAPID_TOL entry point which is more geared towards the requirements of Sanger.
- Fix a bug in build_alignment_blocks.py to avoid indexing errors happening in large genomes.
- Change output BEDGRAPH from EXTRACT_TELO module.
- Generate CRAM CSV fix to allow for multi-readgroup cram files
- Removing KMER_READCOV
- tmp directory was being used
- Output file adjustment (names and location)
| Old Parameter | New Parameter |
|---|---|
| - | --juicer |
Note, since the pipeline is using Nextflow DSL2, each process will be run with its own Biocontainer. This means that on occasion it is entirely possible for the pipeline to be using different versions of the same tool. However, the overall software dependency changes compared to the last release have been listed below for reference.
| Module | Old Version | New Versions |
|---|---|---|
| bamtobed_sort ( bedtools + samtools ) | - | 2.31.0 + 1.17 |
| bedtools | 2.31.0 | 2.31.1 |
| busco | 5.4.3 | 5.5.0 |
| bwa-mem2 | - | 2.2.1 |
| cat | - | 2.3.4 |
| chunk_fasta ( pyfasta ) | - | 0.5.2-1 |
| cooler | - | 0.9.2 |
| cram_filter_align_bwamem2_fixmate_sort | - | |
| ^ ( samtools + bwamem2 ) ^ | - | 1.17 + 2.2.1 |
| coreutils | - | 9.1 |
| fastk | - | 1.0.1 |
| gcc | 7.1.0 | 10.4.0 |
| find_telomere_windows ( java-jdk ) | - | 8.0.112 |
| generate_cram_csv ( samtools ) | - | 1.17 |
| gnu-sort | - | 8.25 |
| juicer_tools_pre ( java-jdk ) | - | 8.0.112 |
| perl | - | 5.26.2 |
| merquryfk | - | 1.0.1 |
| minimap2 + samtools | - | 2.24 + 1.14 |
| miniprot | - | 0.11--he4a0461_2 |
| mummer | - | 3.23 |
| paftools ( minimap2 + samtools ) | - | 2.24 + 1.14 |
| pretextmap + samtools | 0.1.9 + 1.17 | 0.0.2 + 1.17 |
| python | 3.9 | - |
| - pandas | 1.5.2 | - |
| samtools | 1.17 | 1.18 |
| selfcomp_splitfasta ( perl-bioperl ) | - | 1.7.8-1 |
| seqtk | - | 1.4 |
| tabix | - | 1.11 |
| ucsc | - | 377 |
| windowmasker (blast) | - | 2.14.0 |
- Resource allocations being calculated incorrectly.
- Pretext bugs related to large data.
Initial release of sanger-tol/treeval, created with the nf-core template.
The essential pathways of the gEVAL pipeline have now been converted to Nextflow DSL2 from vr-runner, snakemake and wr. Of the original pipeline there is only Bionano left to implement.
- Updated to nf-core/tools template v2.8.0.
- Subworkflow to generate channels from input yaml.
- Subworkflow to generate genome summary file using samtools
- Subworkflow to generate busco gene tracks and ancestral busco mapping.
- Subworkflow to generate HiC maps with cooler, juicebox and pretext.
- Subworkflow to generate gene alignments using miniprot and minimap2.
- Subworkflow to generate insilico digest tracks.
- Subworkflow to generate longread coverage tracks from pacbio data.
- Subworkflow to generate punchlists detailing regions of interest in the genome.
- Subworkflow to generate repeat density tracks.
- Subworkflow to generate tracks detailing self complementary regions.
- Subworkflow to generate syntenic alignments to high quality genomes.
- Subworkflow to generate tracks containing telomeric sites.
- Custom Groovy for reporting to provide file metrics and resource usage.
- Citations and all docs (including walkthroughs).
- Added gitpod.yml for running in the cloud. This is the tutorial written for BGA23.
| Old Parameter | New Parameter |
|---|---|
| - | --input |
Note, since the pipeline is using Nextflow DSL2, each process will be run with its own Biocontainer. This means that on occasion it is entirely possible for the pipeline to be using different versions of the same tool. However, the overall software dependency changes compared to the last release have been listed below for reference.
| Module | Old Version | New Versions |
|---|---|---|
| assign_ancestal ( pandas + Python ) | - | 1.5.2 + 3.9 |
| bamtobed_sort ( bedtools + samtools ) | - | 2.31.0 + 1.17 |
| bedtools | - | 2.31.0 |
| busco | - | 5.4.3 |
| bwa-mem2 | - | 2.2.1 |
| cat | - | 2.3.4 |
| chunk_fasta ( pyfasta ) | - | 0.5.2-1 |
| cooler | - | 0.9.2 |
| concat_block ( coreutils ) | - | 9.1 |
| concat_mummer ( coreutils ) | - | 9.1 |
| cram_filter_align_bwamem2_fixmate_sort | - | |
| ^ ( samtools + bwamem2 ) ^ | - | 1.16.1 + 2.2.1 |
| extract_ancestral ( python ) | - | 3.9 |
| extract_buscogene ( coreutils ) | - | 9.1 |
| extract_cov_id ( coreutils ) | - | 9.1 |
| extract_repeat ( perl ) | - | 5.26.2 |
| extract_telo ( coreutils ) | - | 9.1 |
| find_telomere_regions ( gcc ) | - | 7.1.0 |
| find_telomere_windows ( java-jdk ) | - | 8.0.112 |
| findhalfcoverage ( python ) | - | 3.9 |
| gap_length ( coreutils ) | - | 9.1 |
| generate_cram_csv ( samtools ) | - | 1.17 |
| get_largest_scaff ( coreutils ) | - | 9.1 |
| get_paired_contact_bed ( coreutils ) | - | 9.1 |
| get_synteny_genomes ( coreutils ) | - | 9.1 |
| getminmaxpunches ( coreutils ) | - | 9.1 |
| graphoverallcoverage ( perl ) | - | 5.26.2 |
| gnu-sort | - | 8.25 |
| juicer_tools_pre ( java-jdk ) | - | 8.0.112 |
| makecmap_cmap2bed ( python ) | - | 3.9 |
| makecmap_fa2cmapmulticolor ( perl ) | - | 5.26.2 |
| makecmap_renamecmapids ( perl ) | - | 5.26.2 |
| minimap2 + samtools | - | 2.24 + 1.14 |
| miniprot | - | 0.11--he4a0461_2 |
| mummer | - | 3.23 |
| paf_to_bed ( coreutils ) | - | 9.1 |
| paftools ( minimap2 + samtools ) | - | 2.24 + 1.14 |
| pretextmap + samtools | - | 0.1.9 + 1.17 |
| reformat_intersect ( coreutils ) | - | 9.1 |
| reformat_ids ( coreutils ) | - | 9.1 |
| replace_dots ( coreutils ) | - | 9.1 |
| samtools | - | 1.17 |
| selfcomp_alignmentblocks ( python ) | - | 3.9 |
| selfcomp_mapids ( python ) | - | 3.9 |
| selfcomp_mummer2bed ( python ) | - | 3.9 |
| selfcomp_splitfasta ( perl-bioperl ) | - | 1.7.8-1 |
| seqtk | - | 1.4 |
| tabix | - | 1.11 |
| ucsc | - | 377 |
| windowmasker (blast) | - | 2.14.0 |