How to best recover from a "too many erroneous kmers" error in an automated way

[lfaller-zymergen]

Hello!

I have a pipeline that runs spades 3.15.2 with -k 21,33,55,77,99. Sometimes, one of these values of k leads to the too many erroneous kmers error and then spades crashes (most recently it reported error code 21).

If a smaller k did run successfully, I would like to avoid crashing the whole pipeline.

For example, recently the pipeline crashed because k=99 produced too many erroneous kmers.

My output directory looked as follows:

$> ls -lah
total 204K
drwxr-sr-x 12 root zusers 6.0K Sep  2 17:18 ./
drwxr-sr-x  5 root zusers 6.0K Sep  2 17:16 ../
drwxr-sr-x  2 root zusers 6.0K Sep  2 17:17 .bin_reads/
drwxr-sr-x  6 root zusers 6.0K Sep  2 17:16 configs/
-rw-r--r--  1 root zusers  133 Sep  2 17:16 dataset.info
-rw-r--r--  1 root zusers  400 Sep  2 17:16 input_dataset.yaml
drwxr-sr-x  4 root zusers 6.0K Sep  2 17:17 K21/
drwxr-sr-x  4 root zusers 6.0K Sep  2 17:17 K33/
drwxr-sr-x  4 root zusers 6.0K Sep  2 17:17 K55/
drwxr-sr-x  4 root zusers 6.0K Sep  2 17:17 K77/
drwxr-sr-x  3 root zusers 6.0K Sep  2 17:16 K99/
drwxr-sr-x  2 root zusers 6.0K Sep  2 17:16 misc/
-rw-r--r--  1 root zusers 1.8K Sep  2 17:16 params.txt
drwxr-sr-x  2 root zusers 6.0K Sep  2 17:17 pipeline_state/
-rw-r--r--  1 root zusers 4.6K Sep  2 17:16 run_spades.sh
-rw-r--r--  1 root zusers 5.9K Sep  2 17:16 run_spades.yaml
-rw-r--r--  1 root zusers 124K Sep  2 17:18 spades.log
drwxr-sr-x  2 root zusers 6.0K Sep  2 17:18 tmp/
-rw-r--r--  1 root zusers  352 Sep  2 17:18 warnings.log

Is there any file that I can use to continue running the downstream analysis that I could select programmatically?

Or is the only way to rescue this run by completely rerunning spades but this time dropping the problematic kmer size (99) and running spades with -k 21,33,55,77 ?

Thanks for any advice!

[asl]

How does it "crash"? Can you attach your spades.log? There is no automagical way to recover as there might be many reasons for this, e.g,. low / uneven coverage, contamination, excessive error rate, etc.

[lfaller-zymergen]

Here is the log: spades.log

[asl]

So, everything is quite clear. The log reads:

  0:00:11.534     3M / 45M   ERROR   General                 (kmer_coverage_model.cpp   : 187)   Invalid kmer coverage histogram, make sure that the coverage is indeed uniform

And indeed, you're running k-mer length 99, however:

  0:00:00.113     1M / 11M   INFO    General                 (construction.cpp          : 153)   Max read length 150
  0:00:00.113     1M / 11M   INFO    General                 (construction.cpp          : 159)   Average read length 72.1767

There is easy solution: just stick to the default options. And ensure your data is of enough coverage

[lfaller-zymergen]

Ah, I see. So the k-mer length of 99 would likely not have worked because the average read length is too short?

[asl]

Sure. We do not suggest changing settings (include the set of k-mers) unless you 100% sure what you're doing and what might be an outcome. Just stick to the defaults.

[lfaller-zymergen]

Ok sounds good, thank you!