How to Generate a UMI Whitelist from a String List Using umitools?

[DC-Jade]

I’m trying to generate a UMI whitelist using umitools, but my input is a list of strings (e.g., a text file with one UMI per line). I’m not sure how to proceed with this type of input.

For example, my input file (umis.txt) looks like this:

ATGCTAAAA
GCTAGCTTT
TAGCTAAAA
.
I want to use umi_tools whitelist to process this list and generate a whitelist. However, most of the documentation seems to focus on FASTQ files as input.

Could someone guide me on:

Whether it’s possible to directly use a string list with umi_tools whitelist?
If not, what would be the best way to convert this list into a compatible format for umitools?
Any help or examples would be greatly appreciated!

Thanks in advance.

[TomSmithCGAT]

umi_tools whitelist is designed to handle cell barcodes not UMIs and comes with the major caveat that the accuracy of the whitelisting approach has not been fully appraised and it has not been compared to alternative approaches.

It's not common to want to a 'whitelist' of UMIs. A 'whitelist' is a set of values which are acceptable from a wider set of possible values. For example, in the case of cell barcodes and droplet-based scRNA-seq, this would be the cell barcodes which represent the droplets containing a cell.

With UMIs, we label each initial DNA molecule with a sufficiently large pool of random sequences, such that we can assume the combinations of DNA sequence and UMI are unique. I'll refer to these UMIs as 'real'. Following amplification and sequencing, the UMIs associated with each different DNA sequence will include both the true UMI sequence and those containing errors (error UMIs), with each UMI sequence possibly being present multiple times. The process of reducing the observed UMIs for a given DNA sequence to the likely set of true UMIs, we refer to as 'deduplication', with the UMI-tools algorithms enabling deduplication in a error-aware manner. For more details see publication and blog post.

If your list of strings are really UMIs, and assuming you want to 'deduplicate' these, you can use the UMI-tools algorithms outside of the command line tool as documented here: https://umi-tools.readthedocs.io/en/latest/API.html.

If you actually want to a whitelist of UMIs, can you please explain the use-case so we can advise better.

[IanSudbery]

I think they're are some kits now that provide predetermined UMIs. I'm not sure whitelisting is necessary though.

…

On Mon, 14 Apr 2025, 09:08 Tom Smith, ***@***.***> wrote: umi_tools whitelist is designed to handle *cell barcode* not UMIs and comes with the major caveat that the accuracy of the whitelisting approach has not been fully appraised and it has been compared to alternative approaches. It's not common to want to a 'whitelist' of UMIs though. A 'whitelist' is a set of values which are acceptable from a wider set of possible values. For example, in the case of cell barcodes and droplet-based scRNA-seq, this would be the cell barcodes which represent the droplets containing a cell. With UMIs, we label each initial DNA molecule with a sufficiently large pool of random sequences, such that we can assume the combinations of DNA sequence and UMI are unique. I'll refer to these UMIs as 'real'. Following amplification and sequencing, the UMIs associated with each different DNA sequence will include both the true UMI sequence and those containing errors (error UMIs), with each UMI sequence possibly being present multiple times. The process of reducing the observed UMIs for a given DNA sequence to the likely set of true UMIs, we refer to as 'deduplication', with the UMI-tools algorithms enabling deduplication in a error-aware manner. For more details see publication <https://genome.cshlp.org/content/early/2017/01/18/gr.209601.116.abstract> and blog post <https://cgatoxford.wordpress.com/2015/08/14/unique-molecular-identifiers-the-problem-the-solution-and-the-proof/> . If your list of strings are really UMIs, and assuming you want to 'deduplicate' these, you can use the UMI-tools algorithms outside of the command line tool as documented here: https://umi-tools.readthedocs.io/en/latest/API.html. If you actually want to a whitelist of UMIs, can you please explain the use-case so we can advise better. — Reply to this email directly, view it on GitHub <#681 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABJELDS5NEWAPTVC64BO3GL2ZNUJFAVCNFSM6AAAAAB3CDRNLCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQMBQHAYTMMRYGA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***> *TomSmithCGAT* left a comment (CGATOxford/UMI-tools#681) <#681 (comment)> umi_tools whitelist is designed to handle *cell barcode* not UMIs and comes with the major caveat that the accuracy of the whitelisting approach has not been fully appraised and it has been compared to alternative approaches. It's not common to want to a 'whitelist' of UMIs though. A 'whitelist' is a set of values which are acceptable from a wider set of possible values. For example, in the case of cell barcodes and droplet-based scRNA-seq, this would be the cell barcodes which represent the droplets containing a cell. With UMIs, we label each initial DNA molecule with a sufficiently large pool of random sequences, such that we can assume the combinations of DNA sequence and UMI are unique. I'll refer to these UMIs as 'real'. Following amplification and sequencing, the UMIs associated with each different DNA sequence will include both the true UMI sequence and those containing errors (error UMIs), with each UMI sequence possibly being present multiple times. The process of reducing the observed UMIs for a given DNA sequence to the likely set of true UMIs, we refer to as 'deduplication', with the UMI-tools algorithms enabling deduplication in a error-aware manner. For more details see publication <https://genome.cshlp.org/content/early/2017/01/18/gr.209601.116.abstract> and blog post <https://cgatoxford.wordpress.com/2015/08/14/unique-molecular-identifiers-the-problem-the-solution-and-the-proof/> . If your list of strings are really UMIs, and assuming you want to 'deduplicate' these, you can use the UMI-tools algorithms outside of the command line tool as documented here: https://umi-tools.readthedocs.io/en/latest/API.html. If you actually want to a whitelist of UMIs, can you please explain the use-case so we can advise better. — Reply to this email directly, view it on GitHub <#681 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABJELDS5NEWAPTVC64BO3GL2ZNUJFAVCNFSM6AAAAAB3CDRNLCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQMBQHAYTMMRYGA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[TomSmithCGAT]

Yep. I've used one before (thought it was from Lexogen but can't find it easily now). There are hidden options in both umi_tools extract and dedup to handle UMI whitelists at either point which may be of use, but need details before making any recommendations...

UMI-tools/umi_tools/extract.py

Lines 236 to 243 in ee39b77

	group.add_option("--umi-whitelist", dest="umi_whitelist",
	type="string", default=None,
	#help="A whitelist of accepted UMIs [default=%default]"
	help=optparse.SUPPRESS_HELP)
	group.add_option("--umi-whitelist-paired", dest="umi_whitelist_paired",
	type="string", default=None,
	#help="A whitelist of accepted UMIs for read2[default=%default]"
	help=optparse.SUPPRESS_HELP)

UMI-tools/umi_tools/Utilities.py

Lines 762 to 772 in ee39b77

	group.add_option("--umi-whitelist", dest="umi_whitelist",
	type="string", default=None,
	#help="A whitelist of accepted UMIs"
	#"[default=%default]"
	help=optparse.SUPPRESS_HELP)
	group.add_option("--umi-whitelist-paired", dest="umi_whitelist_paired",
	type="string", default=None,
	#help="A whitelist of accepted UMIs for "
	#"read2[default=%default]"
	help=optparse.SUPPRESS_HELP)

[Y-Pei]

Yep. I've used one before (thought it was from Lexogen but can't find it easily now). There are hidden options in both umi_tools extract and dedup to handle UMI whitelists at either point which may be of use, but need details before making any recommendations...

We currently use the Twist UMI Adapter System which is non-random/predetermined/predefined/known with 32 available UMIs.

From dedup, we can see that about 2-3% of the observed UMIs are NOT part of the predetermined list, presumably due to PCR/sequencing errors. Rather than filtering them out, I think most of these 2-3% can be corrected into the predetermined ones.

Any comments on if this is possible with UMI-tools or any preprocessing tools that can account for this before running UMI-tools?

Thanks!!!

[fangling0913]

Yep. I've used one before (thought it was from Lexogen but can't find it easily now). There are hidden options in both umi_tools extract and dedup to handle UMI whitelists at either point which may be of use, but need details before making any recommendations...

UMI-tools/umi_tools/extract.py

Lines 236 to 243 in ee39b77

group.add_option("--umi-whitelist", dest="umi_whitelist",
type="string", default=None,
#help="A whitelist of accepted UMIs [default=%default]"
help=optparse.SUPPRESS_HELP)
group.add_option("--umi-whitelist-paired", dest="umi_whitelist_paired",
type="string", default=None,
#help="A whitelist of accepted UMIs for read2[default=%default]"
help=optparse.SUPPRESS_HELP)
UMI-tools/umi_tools/Utilities.py

Lines 762 to 772 in ee39b77

group.add_option("--umi-whitelist", dest="umi_whitelist",
type="string", default=None,
#help="A whitelist of accepted UMIs"
#"[default=%default]"
help=optparse.SUPPRESS_HELP)

group.add_option("--umi-whitelist-paired", dest="umi_whitelist_paired",
type="string", default=None,
#help="A whitelist of accepted UMIs for "
#"read2[default=%default]"
help=optparse.SUPPRESS_HELP)

When using the extract tool as the following command, I encountered the error "error: no such option: --umi_whitelist" when trying to specify a UMI whitelist. How can I provide a whitelist of known UMIs for error correction during the extraction step?

umi_tools extract --stdin=../R014V1_raw.R1.fastq.gz --read2-in=../R014V1_raw.R2.fastq.gz --stdout=R014V1_umi.R1.fastq.gz --read2-out=R014V1_umi.R2.fastq.gz --bc-pattern=NNNNNN --bc-pattern2=NNNNNN --extract-method=string --filter-umi --umi_whitelist=../white1.txt --umi_whitelist_paired=../white1.txt

[IanSudbery]

Hi FangLing

The option is --umi-whitelist not --umi_whitelist

[fangling0913]

Hi FangLing

The option is --umi-whitelist not --umi_whitelist

Thanks for your quick reply!