Using the date in output folders is causing issues for us

[TomHarrop]

Hello from down under! 🦘

@amytims and I are having some problems with the 0.50.0 release, because we can no longer override the date in the output folders.

• We can't predict where to find output for different runs of the pipeline for the same data. We are versioning our config files, but right now we can't definitively match our assembly version with pipeline output.
• Jobs get re-run when we try to resume the pipeline and the date has changed.

In the past we were overriding the date by passing e.g. --timestamp v0 from our config but this doesn't work in 0.50.0.

I know you're planning to remove this but I just wanted to flag that it's broken for us in the current release.

Thanks!
Tom

[prototaxites]

Hi Tom, nice to hear from you!

In this case, I think you might already be better off running the dev version of genomeassembly, which adds the parameterisable assembly I was telling you about. There's no documentation for it yet, but hopefully it's not too difficult to figure out:

1. Create an input YAML file describing your genomic datasets, e.g. https://github.com/sanger-tol/genomeassembly/blob/dev/assets/test_genomic.yaml - the schema describing the format is here: https://github.com/sanger-tol/genomeassembly/blob/dev/assets/schema_genomic.json

2. Create an input assembly specification YAML file, e.g. https://github.com/sanger-tol/genomeassembly/blob/dev/assets/test_spec.yaml. The id field of each specification will be the name of the output directory for that assembly. The schema describing all the options is here, though the example is pretty comprehensive for basic purposes: https://github.com/sanger-tol/genomeassembly/blob/dev/assets/schema_assembly.json

Run the pipeline with:

nextflow run sanger-tol/genomeassembly -r dev --genomic_data <path to genomic YAML> --assembly-specs <path to specs YAML>

Most of the parameters of the pipeline have otherwise been stripped out, leaving just a few that tune overall pipeline behaviour with sensible defaults.

[prototaxites]

Hi Tom,

Usage documentation is now updated in the docs branch. Output documentation is still to be updated!

https://github.com/sanger-tol/genomeassembly/blob/docs/docs/usage.md

If you have any comments regarding the documentation, they would be more than welcome.

[TomHarrop]

Thanks for the replies. The new docs look useful, from a quick skim I can already see answers to a couple of things we had been wondering about.

We are currently working through some datasets and will get back to you with our findings!

[prototaxites]

Hi Tom, that's great - please do let me know if you have any questions.

I have now also updated the output documentation: https://pipelines.tol.sanger.ac.uk/genomeassembly/dev/output