Snakemake Tips

Running in cluster mode

Each step of the pipeline is broken up into a number of different Snakemake rules. Each rule can be submitted as a job on a cluster using commands for the appropriate scheduler. This is accomplished by creating a cluster configuration file (see Sample Cluster Config) that outlines the resources that should be applied to each rule. There can also be a default setting which will be the fall-back position for rules that are not provided a specific configuration. Any of the variables defined in the config can be accessed when defining the cluster script.

$ head cluster.cfg

__default__:
  cpus: '{threads}'
  mem: 5000
  log: 'logs/cluster_{log}'
  jobname: '{rule}'
  time: "30:00"

fastp:
  jobname: "fastp"

readprep:
  jobname: "readprep"

binning:
  jobname: "binning"
  mem: 30000
  time: "2:00:00"

The values in the cluster configuration file can be accessed using the cluster keyword when passed to the --cluster argument. For example, using slurm:

mtsv binning -c mtsv.cfg --cluster-config cluster.cfg 
--cluster "sbatch --mem {cluster.mem} --time {cluster.time}
--job-name {cluster.jobname} --output {cluster.log}
--cpus-per-task {threads}" --jobs 100

For the binning jobs, cluster.mem, cluster.time, and cluster.job-name values will be pulled from the cluster.cfg file under "binning". the --cpus-per-task will be pulled from the __default__ value for cpus. In this example it is set to {threads}. The value in curly braces means that it will retrieve the value defined in the rule. For each rule, a max number of threads is defined and will be inserted as the number of cpus. If you which to restrain this value, you can set cpus to a different value in the cluster config (increasing the value will not increase the number of threads over the maximum provided in the rule definition).

The default value for cluster.log is also defined using the {log} value set in the rule. Any of these values can be modified in the cluster config.

The --jobs/--cores argument must be provided and this will determine how many jobs will be scheduled at once.

Snakemake scheduler

If you are not running the jobs individually on a cluster, you will be relying on Snakemake to schedule jobs. In this case you will need to provide maximum resources for Snakemake to use. Here the --jobs/--cores will determine how cores are available to run. This defaults to 1, which means that only one job will run at a time and it will be restricted to one thread even if the rule has a higher thread defined. If you have 2 jobs that both have a maximum thread number set to 8 and you provide 16 cores, then both jobs will run at the same time with the 8 threads. If you provided --cores 10, then only one job will run at a time with 8 threads. Maximum resource contstraints can be added using --resources. Run snakemake --help for more information

Passing config values from the command line.

To pass config values from the command line you can add the Snakemake --config argument followed by param1=value1 param2=value2. For list parameters run parameter=1,2,3. Note: Snakemake does not recognize targets that follow the --config argument so make sure that the config values are passed after any targets.

Troubleshooting

Locked directory

Snakemake locks the working directory when it is running and, in the event that the process does not exit gracefully, a stale lock may remain until the directory is manually unlocked. Snakemake will provide a message letting the user know that the directory is locked. When this occurs, rerun the same command with --unlock appended as an argument. Then rerun the original command.

Incomplete Files Exception

When an error occurs during execution, Snakemake recognizes the error and deletes incomplete files. If the process is terminated unexpectedly, such that Snakemake cannot delete the files, they will be marked in the metadata as incomplete. When the process is restarted, the incomplete files are detected and must be handled before restarting. The two options are --rerun-incomplete, which rerun the rule to generate the incomplete files, and --cleanup-metadata, which will reset the metadata to ignore the incomplete status (the latter can be dangerous if the files are really not complete). Either of these options just need to be appended to the original MTSv command that produced the error.

Nothing to be done

The most likely cause of this output at the beginning of the MTSv pipeline, is a incorrect fastq_pattern. If there are no fastq files that match the provided pattern, there will be no sample targets generated. Ensure that the correct path is provided and {sample} is included in the pattern. Also ensure that the correct extension (e.g. .fq vs .fastq) is in your pattern.

If there is not an issue with the fastq pattern, it is likely that this step has already been completed. To trigger a rerun you can pass --forcerun followed by the rule that you want to repeat. --forceall will force all rules to be rerun.

always catch when parameters change and may therefore claim that nothing needs to be done for certain rules. If this occurs, rerun the same command but add --force or --forceall to ensure the rules are rerun with the new parameters.

JSONDecodeError

If Snakemake throws an unexpected JSONDecodeError like the one below, it is likely that the snakemake metadata has been corrupted. In this case, it is recommended that you remove the .snakemake/metadata directory from your working directory and run again. See Issue.

raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Other Helpful Options

• Use -k or --keep-going to continue processing independent jobs when one job fails.
• Use -t or --touch to touch output files (mark them up to date without really changing them) instead of running their commands. This is used to pretend that the rules were executed, in order to fool future invocations of snakemake. This is helpful if you modified an stream file but do not want to rerun all down stream rules.
• Use --dry-run to see which rules will be called with the provided command.
• Use --directory DIR, or -d DIR to run commands in a different directory.
• Use --show-failed-logs to automatically display logs of failed jobs.
• Use --notemp, or --nt to keep temporary files that are usually deleted when no longer required by downstream rules (e.g. individual binning files before merge).
• Use --list, or -l to show all available rules.