Fix missing bowtie index

[piplus2]

PR checklist

• This comment contains a description of changes (with reason).
• If you've fixed a bug or added code that should be tested, add tests!
• If you've added a new tool - have you followed the pipeline conventions in the contribution docs
• If necessary, also make a PR on the nf-core/smrnaseq branch on the nf-core/test-datasets repository.
• Make sure your code lints (nf-core pipelines lint).
• Ensure the test suite passes (nextflow run . -profile test,docker --outdir <OUTDIR>).
• Check for unexpected warnings in debug mode (nextflow run . -profile debug,test,docker --outdir <OUTDIR>).
• Usage Documentation in docs/usage.md is updated.
• Output Documentation in docs/output.md is updated.
• CHANGELOG.md is updated.
• README.md is updated (including new tool citations and authors/contributors).

When the argument --bowtie_index is passed with no value, the pipeline assigns the value true which then fails when val_bowtie_index.endsWith(".tar.gz") is called.
Forcing a cast to String before checking the extension fixes this issue.

[nf-core-bot]

Warning

Newer version of the nf-core template is available.

Your pipeline is using an old version of the nf-core template: 3.5.1.
Please update your pipeline to the latest version.

For more documentation on how to update your pipeline, please see the nf-core documentation and Synchronisation documentation.

[lpantano]

it seems seqcluster module in conda is broken. I will check on tuesday, and try to fix it. we can ask slack if ok to merge this since it is a different issue.

[piplus2]

it seems seqcluster module in conda is broken. I will check on tuesday, and try to fix it. we can ask slack if ok to merge this since it is a different issue.

Thanks, if you want I'm also on Slack.

[apeltzer]

I would prefer fixing this in the seqcluster module first, then we could consider having this fixed in the pipeline.

[lpantano]

sounds good. I will try to fix the recipe. it is because python 3.14 needs setuptools as dependency now. maybe there are more errors. I will try to check asap.

[lpantano]

ok, i got to update the two modules that have issues with newer version of python. they were already updated in the modules repo. Now it is failing due to diff in snapshot difference in the pipeline, since I updated the modules. In case one of you can run it and fix the snapshots. Prob I wont have time again until end of the week.

[piplus2]

The nf-test fails at DATATABLE_MERGE. The input file joined_samples_mirtop.tsv is being generated without a header row.

When I ran head -n1, I got:

iso-17-11K865H  AGGCCCCCGGGGCACAG  hsa-miR-5008-5p ...

The R script is crashing with object 'miRNA' not found:

Test Process DATATABLE_MERGE

  Test [f7590508] 'Contains hsa-miR-365b-3p, hsa-miR-7-5p, hsa-miR-103a-3p' PASSED (13.894s)
  Test [eee8aa71] 'Does not contain hsa-miR-107, hsa-miR-365a-3p' PASSED (13.292s)

Test Process EDGER_QC

  Test [6b1d52a8] 'Should not produce MDS plot' PASSED (12.276s)
  Test [1fea66bb] 'Should produce MDS plot' PASSED (14.391s)

Test Workflow CONTAMINANT_FILTER

  Test [14aa5929] 'Should remove other contaminants' PASSED (46.636s)
  Test [3a0103a3] 'Should remove tRNA contaminants' PASSED (22.14s)
  Test [9c6da9a0] 'Should remove ncRNA contaminants' PASSED (44.461s)

Test pipeline

  Test [48f63baa] '-profile test' Assertion failed:

assert workflow.success
       |        |
       workflow false

FAILED (179.14s)

  Assertion failed:

  1 of 2 assertions failed

  Nextflow stdout:

  ERROR ~ Error executing process > 'NFCORE_SMRNASEQ:MIRNA_QUANT:DATATABLE_MERGE'

  Caused by:
    Process `NFCORE_SMRNASEQ:MIRNA_QUANT:DATATABLE_MERGE` terminated with an error exit status (1)


  Command executed:

    collapse_mirtop.r joined_samples_mirtop.tsv

    cat <<-END_VERSIONS > versions.yml
    "NFCORE_SMRNASEQ:MIRNA_QUANT:DATATABLE_MERGE":
        r-base: $(echo $(R --version 2>&1) | sed 's/^.*R version //; s/ .*$//')
    END_VERSIONS

  Command exit status:
    1

  Command output:
    (empty)

  Command error:
    Error in eval(bysub, parent.frame(), parent.frame()) :
      object 'miRNA' not found
    Calls: [ -> [.data.table -> eval -> eval
    Execution halted

  Work dir:
    /home/pinglese-iit.local/GitHub/smrnaseq/.nf-test/tests/48f63baa49f37881cf28c8d7e8b28d62/work/d3/4024c29ceeaba4cfc80eb16ffadb08

  Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

   -- Check '/home/pinglese-iit.local/GitHub/smrnaseq/.nf-test/tests/48f63baa49f37881cf28c8d7e8b28d62/meta/nextflow.log' file for details
  ERROR ~ Pipeline failed. Please refer to troubleshooting docs: https://nf-co.re/docs/usage/troubleshooting

   -- Check '/home/pinglese-iit.local/GitHub/smrnaseq/.nf-test/tests/48f63baa49f37881cf28c8d7e8b28d62/meta/nextflow.log' file for details
  Nextflow stderr:

We either need to fix the upstream module to include the header or update the R script to handle headerless files.

[piplus2]

Found the issue. The header is treated as any other line in the PIVOT_WIDER process, then it ends in the middle of the csv file after sorting.

The file contains the header:

> grep miRNA joined_samples_mirtop.tsv
UID     Read    miRNA   Variant iso_5p  iso_3p  iso_add3p       iso_snp iso_5p_nt       iso_3p_nt       iso_add3p_nt    iso_snp_nt    Clone9_N3       Clone1_N3       Control_N2      Clone9_N2       Control_N1      Clone9_N1       Clone1_N1       Control_N3

but the first line is incorrect:

> head -n 1 joined_samples_mirtop.tsv
iso-17-11K865H  AGGCCCCCGGGGCACAG       hsa-miR-5008-5p iso_3p:-3,iso_5p:+2,iso_snv_seed        +2      -3      0       1    tg       tgg     0       7CT8CT  0       0       0       1       0       0       3       0

The solution is to first pass the header into joined_samples_mirtop.tsv and then sort:

head -n 1 ${meta.id}_concatenated_temp.csv > joined_samples_mirtop.tsv
tail -n +2 ${meta.id}_concatenated_temp.csv | sort -t\$'\t' -k1,1 >> joined_samples_mirtop.tsv

Now all tests pass:

🚀 nf-test 0.9.5
https://www.nf-test.com
Please cite: https://doi.org/10.1093/gigascience/giaf130
(c) 2021 - 2026 Lukas Forer and Sebastian Schoenherr

Load .nf-test/plugins/nft-bam/0.6.0/nft-bam-0.6.0.jar
Load .nf-test/plugins/nft-utils/0.0.7/nft-utils-0.0.7.jar
Load .nf-test/plugins/nft-vcf/1.0.7/nft-vcf-1.0.7.jar
Warning: every snapshot that fails during this test run is re-recorded.

Test Process DATATABLE_MERGE

  Test [f7590508] 'Contains hsa-miR-365b-3p, hsa-miR-7-5p, hsa-miR-103a-3p' PASSED (22.045s)
  Test [eee8aa71] 'Does not contain hsa-miR-107, hsa-miR-365a-3p' PASSED (5.163s)

Test Process EDGER_QC

  Test [6b1d52a8] 'Should not produce MDS plot' PASSED (21.476s)
  Test [1fea66bb] 'Should produce MDS plot' PASSED (8.858s)

Test Workflow CONTAMINANT_FILTER

  Test [14aa5929] 'Should remove other contaminants' PASSED (60.022s)
  Test [3a0103a3] 'Should remove tRNA contaminants' PASSED (8.472s)
  Test [9c6da9a0] 'Should remove ncRNA contaminants' PASSED (24.043s)

Test pipeline

  Test [48f63baa] '-profile test' PASSED (270.956s)
  Snapshots:
    1 updated [-profile test]

Test Workflow main.nf - test_contamination_tech_reps

  Test [b085b6d2] '-profile test_contamination_tech_reps' PASSED (172.187s)
  Snapshots:
    1 updated [-profile test_contamination_tech_reps]

Test Workflow main.nf - test_mirgenedb

  Test [5fd8884c] '-profile test_mirgenedb' PASSED (175.669s)
  Snapshots:
    1 updated [-profile test_mirgenedb]

Test Workflow main.nf - test_nextflex

  Test [5ba802da] '-profile test_nextflex' PASSED (176.423s)
  Snapshots:
    1 updated [-profile test_nextflex]

Test Workflow main.nf - test_skipfastp

  Test [3aff2f37] '-profile test_skipfastp' PASSED (181.635s)
  Snapshots:
    1 updated [-profile test_skipfastp]

Test Workflow main.nf - test_umi

  Test [6a2b5617] '-profile test_umi' PASSED (168.16s)
  Snapshots:
    1 updated [-profile test_umi]


Snapshot Summary:
  6 updated

SUCCESS: Executed 13 tests in 1299.628s

[piplus2]

It seems there's still some issues with Python 3.14, and things like MultiQC which make the checks fail.
If you want I can have a look at them.

EDIT: what about keeping the compatibility with python <= 3.13?

EDIT2: seems that actually MultiQC with Python 3.14.1 is broken MultiQC/MultiQC#3414

EDIT3: this is the MultiQC bug that was fixed 5 days ago MultiQC/MultiQC@292e765

[lpantano]

I am little confused about what is the latest error, but if it is multqic, it seems we need a module update first to get the latest fixed version of multiqc, or we do a patch now in the pipeline's module env.yml to keep it being < 3.13.

[piplus2]

I am little confused about what is the latest error, but if it is multqic, it seems we need a module update first to get the latest fixed version of multiqc, or we do a patch now in the pipeline's module env.yml to keep it being < 3.13.

Hi, it seems that rich > 14.2.0 forces the from rich.panel import Panel thing, while before it was possible to call rich.panel.Panel. For this reason, I've set rich==14.2.0 in the environment.yml of multiqc and for safety python <= 3.13 as it seems that multiqc has got some issues with some iterations of py3.14.

At the moment, it's running the checks. Hopefully conda will work.

this describes the issue:
MultiQC/MultiQC#3536

[piplus2]

It seems everything is fine now.
Basically, until the patched multiqc is not pushed upstream, we can set rich==14.2.0 (which is not ideal, but it's the only workaround at the moment).
There was a bug in DATATABLE_MERGE which broke the header of the output csv.
Sorry for the many commits for the updated snapshots, but I struggled to understand why it passed the tests locally but not here. It was just because multiqc must be 1.33 in both conda and docker (I forgot to reset the version after testing 1.32 locally) and I had to use the +docker profile in the local run of nf-test.