umi_tools dedup is really slow

[Nitin123-4]

Hello UMI-tools team,

I am evaluating umi_tools dedup for bulk RNA-seq libraries. While the deduplication results appear reasonable, I am observing very long runtimes.

Command used:

bash
umi_tools dedup
--paired
--output-stats sample
-I sample.Aligned.sortedByCoord.out.bam
-S sample.Aligned.sortedByCoord.out.dedup.bam
-L sample.dedup.log

Dataset statistics:

Input reads: 76,872,910 read pairs
Read 2 unmapped: 2,205,461
Positions deduplicated: 40,512,150
Mean unique UMIs per position: 1.51
Maximum unique UMIs at a single position: 42,881

Resource usage:

Runtime: ~8 hours 14 minutes
Peak memory: ~24 GB RSS
CPU utilization: ~99% (single-core bound)

I also tested --method=unique and observed nearly identical runtime (~8 hours 18 minutes), suggesting that the bottleneck may not be the UMI grouping strategy itself.

Are there any recommended options, optimizations, or best practices for improving performance on large RNA-seq datasets? In particular, we are interested in understanding whether the observed runtime is expected for datasets of this size and whether there are options that can reduce processing time without substantially affecting deduplication accuracy.

Thank you for your help.

[IanSudbery]

Hi Nitin,

This was in our FAQ, although I've just realised that recent builds of the documentation are missing the FAQ!

Your problem is probably mostly the stats generation. Part of generating the stats is generating an empirical null distribution of UMI distances, and this is slooooow and memory intensive. Our recommendation is to not run stats generation in production. If you do want an idea of the full stats, including the UMI distance null distribution, we recommend you run just a sample of your file, e.g. --chrom=chr22 for the purposes os stats generation, separately from the production deduplication.