Skip to content

Hacked out some code to allow for multi odb lineage runs#234

Draft
DLBPointon wants to merge 20 commits into
devfrom
multi_odb_runs
Draft

Hacked out some code to allow for multi odb lineage runs#234
DLBPointon wants to merge 20 commits into
devfrom
multi_odb_runs

Conversation

@DLBPointon

@DLBPointon DLBPointon commented Apr 29, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

VERY MUCH A DRAFT

My midnight romp through the blobtoolkit code to add multi-odb runs.
I've tried to do this with as minimal changes as possible. So it effects 1 function and adds another.

Edit: my test run has just completed too, looks fine. Although I should load it into a viewer first.

Excuse the over-zealous linter making changes to the python

required creating a merged odb mapping, so we don't miss out on any mappings:
cat assets/mapping*.txt | uniq > mixed.txt

and this modified raw_test.nf.config:

/*
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    Nextflow config file for running minimal tests
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    Defines input files and everything required to run a fast and simple pipeline test.

    Use as follows:
        nextflow run sanger-tol/blobtoolkit -profile test_raw,<docker/singularity> --outdir <OUTDIR>

----------------------------------------------------------------------------------------
*/

process {
    resourceLimits = [
        cpus: 2,
        memory: '6.GB',
        time: '6.h'
    ]
}

params {
    config_profile_name        = 'Raw test profile'
    config_profile_description = 'Minimal raw test dataset to check pipeline function'

    // Unaligned reads in a fetchngs samplesheet
    fetchngs_samplesheet = true
    input     = "${projectDir}/assets/samplesheet_raw.csv"
    align     = true

    // Fasta reference, unmasked
    fasta     = "https://tolit.cog.sanger.ac.uk/test-data/Meles_meles/assembly/release/mMelMel3.1_paternal_haplotype/GCA_922984935.2.subset.unmasked.fasta.gz"
    accession = "GCA_922984935.2"
    taxon     = "Meles meles"
    mask      = true

    // Databases
    // taxdump = "https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/new_taxdump/new_taxdump.tar.gz"
    taxdump = "https://tolit.cog.sanger.ac.uk/test-data/resources/new_taxdump.tar.gz"
    busco   = "/lustre/scratch122/tol/resources/busco/latest/"
    blastp  = "https://tolit.cog.sanger.ac.uk/test-data/Meles_meles/resources/mMelMel3.1.buscogenes.dmnd.tar.gz"
    blastx  = "https://tolit.cog.sanger.ac.uk/test-data/Meles_meles/resources/mMelMel3.1.buscoregions.dmnd.tar.gz"
    blastn  = "https://tolit.cog.sanger.ac.uk/test-data/Meles_meles/resources/nt_mMelMel3.1.tar.gz"
    lineage_tax_ids = "${projectDir}/assets/mixed.txt"

    // Only certain BUSCO, to speed things up
    busco_lineages = "carnivora_odb10,tetrapoda_odb10,carnivora_odb12"

    // Need to be set to avoid overfilling /tmp
    use_work_dir_as_temp = true
}

@DLBPointon
DLBPointon marked this pull request as draft April 29, 2026 22:54
@sanger-tolsoft

sanger-tolsoft commented Apr 29, 2026

Copy link
Copy Markdown
Contributor

Warning

Newer version of the nf-core template is available.

Your pipeline is using an old version of the nf-core template: 3.5.2.
Please update your pipeline to the latest version.

For more documentation on how to update your pipeline, please see the nf-core documentation and Synchronisation documentation.

@github-actions

github-actions Bot commented Apr 29, 2026 •
edited
Loading

Copy link
Copy Markdown

nf-core pipelines lint overall result: Passed ✅ ⚠️

Posted for pipeline commit ecfec54

+| ✅ 203 tests passed       |+
#| ❔  30 tests were ignored |#
!| ❗  18 tests had warnings |!
Details

❗ Test warnings:

❔ Tests ignored:

  • files_exist - File is ignored: CODE_OF_CONDUCT.md
  • files_exist - File is ignored: assets/nf-core-blobtoolkit_logo_light.png
  • files_exist - File is ignored: docs/images/nf-core-blobtoolkit_logo_light.png
  • files_exist - File is ignored: docs/images/nf-core-blobtoolkit_logo_dark.png
  • files_exist - File is ignored: .github/ISSUE_TEMPLATE/config.yml
  • files_exist - File is ignored: .github/workflows/awstest.yml
  • files_exist - File is ignored: .github/workflows/awsfulltest.yml
  • files_exist - File is ignored: conf/igenomes.config
  • nextflow_config - Config variable ignored: manifest.name
  • nextflow_config - Config variable ignored: manifest.homePage
  • files_unchanged - File ignored due to lint config: .gitattributes
  • files_unchanged - File ignored due to lint config: CODE_OF_CONDUCT.md
  • files_unchanged - File ignored due to lint config: LICENSE or LICENSE.md or LICENCE or LICENCE.md
  • files_unchanged - File ignored due to lint config: .github/CONTRIBUTING.md
  • files_unchanged - File ignored due to lint config: .github/ISSUE_TEMPLATE/bug_report.yml
  • files_unchanged - File does not exist: .github/ISSUE_TEMPLATE/config.yml
  • files_unchanged - File ignored due to lint config: .github/PULL_REQUEST_TEMPLATE.md
  • files_unchanged - File ignored due to lint config: .github/workflows/branch.yml
  • files_unchanged - File ignored due to lint config: .github/workflows/linting_comment.yml
  • files_unchanged - File ignored due to lint config: .github/workflows/linting.yml
  • files_unchanged - File ignored due to lint config: assets/email_template.html
  • files_unchanged - File ignored due to lint config: assets/email_template.txt
  • files_unchanged - File ignored due to lint config: assets/sendmail_template.txt
  • files_unchanged - File ignored due to lint config: assets/nf-core-blobtoolkit_logo_light.png
  • files_unchanged - File ignored due to lint config: docs/images/nf-core-blobtoolkit_logo_light.png
  • files_unchanged - File ignored due to lint config: docs/images/nf-core-blobtoolkit_logo_dark.png
  • files_unchanged - File ignored due to lint config: .gitignore or .prettierignore
  • actions_awstest - 'awstest.yml' workflow not found: /home/runner/work/blobtoolkit/blobtoolkit/.github/workflows/awstest.yml
  • template_strings - template_strings
  • merge_markers - merge_markers

✅ Tests passed:

Run details

  • nf-core/tools version 3.5.2
  • Run at 2026-05-19 15:50:42

@DLBPointon

DLBPointon commented Apr 30, 2026

Copy link
Copy Markdown
Contributor Author

Needs a bit of cleaning - some other thoughts:

  • I think logic for the basal lineages should be moved to generate_config <-- working on this at the minute
    • This way we can just generate a list of lineages and pass that through the pipeline rather than have this parsing odf the config file and then
  • is .flatMap { lineages -> lineages.withIndex() } needed considering it is thrown away in the next bit of processing?

@DLBPointon

DLBPointon commented May 5, 2026

Copy link
Copy Markdown
Contributor Author

@muffato I've made the changes you suggested in #235

It's now a csv string, defaulting to "eukaryota,bacteria,archaea"

@DLBPointon
DLBPointon marked this pull request as ready for review May 12, 2026 13:25
@DLBPointon
DLBPointon requested a review from muffato May 13, 2026 16:29
@DLBPointon

DLBPointon commented May 13, 2026

Copy link
Copy Markdown
Contributor Author

@muffato I've added a test config for multi_odb runs, it's not in use at the minute though. It will just need carnivora_odb12 adding to be and then nf-test run.

But do you think that can be merged into plots branch

@DLBPointon DLBPointon linked an issue May 14, 2026 that may be closed by this pull request
Basal odb groups should be modifiable #235
Open
@DLBPointon
DLBPointon marked this pull request as draft May 16, 2026 08:58
@DLBPointon

DLBPointon commented May 16, 2026

Copy link
Copy Markdown
Contributor Author

Move this across to the busco pipeline

Base automatically changed from plots to dev May 17, 2026 09:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@muffato muffato Awaiting requested review from muffato

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Basal odb groups should be modifiable

2 participants

@DLBPointon @sanger-tolsoft