When using large gene catalogues with Meteor, generation of CRAM alignment files fails due to header size limitations.
For instance, this issue occurs with the gg_13_6_caecal catalogue.
Error from HTSlib:
[E::cram_write_SAM_hdr] Header is too long for CRAM format
HTSlib sets a hard limit on header size for BAM and CRAM files (~2^31 characters).
This limit is more likely to be exceeded with CRAM files due to the additional M5 and UR fields in the header, as shown below:
Example CRAM header:
@SQ SN:19 LN:28320 M5:3d5be069b8cfc143f4ca40bce9c4cfb1 UR:/biodatabank/meteor2/v2018/hs_10_4_gut/fasta/hs_10_4_gut.fasta.gz
@SQ SN:20 LN:27477 M5:9c0b369c5bd0cb3e1174b55123ab635a UR:/biodatabank/meteor2/v2018/hs_10_4_gut/fasta/hs_10_4_gut.fasta.gz
@SQ SN:21 LN:27459 M5:155baa057d6c420a40760ae77a16d5b4 UR:/biodatabank/meteor2/v2018/hs_10_4_gut/fasta/hs_10_4_gut.fasta.gz
@SQ SN:22 LN:27141 M5:c11f06f5623c51bcf48f64b379798f7c UR:/biodatabank/meteor2/v2018/hs_10_4_gut/fasta/hs_10_4_gut.fasta.gz
Equivalent BAM header:
@SQ SN:19 LN:28320
@SQ SN:20 LN:27477
@SQ SN:21 LN:27459
@SQ SN:22 LN:27141
The CRAM header can therefore be 3–4× larger than the BAM header.
Empirical limits on catalogue size:
CRAM: ~11–12 million genes
BAM: ~30–40 million genes
Potential fixes:
Drop CRAM support entirely and use BAM files in all cases
Use CRAM by default, but automatically switch to BAM for large catalogues
When using large gene catalogues with Meteor, generation of CRAM alignment files fails due to header size limitations.
For instance, this issue occurs with the gg_13_6_caecal catalogue.
Error from HTSlib:
[E::cram_write_SAM_hdr] Header is too long for CRAM formatHTSlib sets a hard limit on header size for BAM and CRAM files (~2^31 characters).
This limit is more likely to be exceeded with CRAM files due to the additional M5 and UR fields in the header, as shown below:
Example CRAM header:
Equivalent BAM header:
The CRAM header can therefore be 3–4× larger than the BAM header.
Empirical limits on catalogue size:
CRAM: ~11–12 million genes
BAM: ~30–40 million genes
Potential fixes:
Drop CRAM support entirely and use BAM files in all cases
Use CRAM by default, but automatically switch to BAM for large catalogues