the bin filename ${bin} does not match the expected format 'MEGAHIT-MetaBinner-group-612.([_[:alnum:]]+).fa'!

[jorondo1]

Description of the bug

I'm not sure what's going on here! I can reproduce this error working from dev (PR 1066, commit 56a0eb3) but it doesn't happen at the same jobs; if I resume, I pick up the cache, get to do about a thousand more RENAME_PREDASTOOL jobs, then it happens again. See the attached zip for the full work dir.

Any idea what's going on ? Also attaching params, config, let me know if you need anything else to help me with this. Currently testing from dev commit bed815e (PR 1070) to see if it still happens.

Command used and terminal output

nextflow run nf-core-mag-dev/dev -profile singularity \    
-resume -c narval.config \                                                         
-params-file params/params_coassembly.yaml

Relevant files

40ceb28cd3fbbb296ddb39fe2f2b51.zip

narval.config.txt

params_coasssembly.yaml

System information

NF version 26.04.4
HPC with AlmaLinux 9.8: https://docs.alliancecan.ca/wiki/Narval/en
Slurm, singularity

[dialvarezs]

Hi @jorondo1!

This looks like an infrastructure issue because, as you describe, it seems to be happening randomly with no clear pattern.

My guess is that when the job fires, the singularity binary isn't available on the execution node, likely a transient I/O error (Singularity is probably available on Narval via a network-mounted filesystem). This should explain the message env: ‘singularity’: Input/output error.

You can check whether it's always the same node: grab the SLURM job ID from .nextflow.log and inspect it with sacct:

sacct -j <job_id> -o JobID,NodeList,State,ExitCode

(or just filter by exit code 126 with grep instead of specifying a job id)
If the failures are associated with a particular node, it would be worth notifying the HPC administrators.

Meanwhile, add this to your config to auto-retry on exit code 126:

process {
    errorStrategy = { task.exitStatus in ((130..145) + 104 + 126) ? 'retry' : 'finish' }
    maxRetries = 3
}

I made it global since the I/O error can hit any containerized task, not just RENAME_PREDASTOOL.

[dialvarezs]

By the way there are a few issues with your config:

• SEMIBIN_SINGLEEASYBIN:BINNING_METABINNER is not real, maybe you meant NFCORE_MAG:MAG:BINNING:BINNING_METABINNER:METABINNER_METABINNER?
• NFCORE_MAG:MAG:BINNING:COMEBIN_RUNCOMEBIN has a leading space, so it won't match the process.

[jorondo1]

I'll monitor and report, and maybe wait for their response to close this issue.

Thanks for the diagnosis and for examining my config!