nt index incomplete build

[igordot]

I was trying to build my own nt index with make THREADS=16 nt. It looked like it completed without errors, but there are only two nt.*.cf files and nt.2.cf is empty.

This is centrifuge-build-nt.log:

Settings:
  Output files: "tmp_nt/nt.*.cf"
  Line rate: 7 (line is 128 bytes)
  Lines per side: 1 (side is 128 bytes)
  Offset rate: 4 (one in 16)
  FTable chars: 14
  Strings: unpacked
  Local offset rate: 3 (one in 8)
  Local fTable chars: 6
  Max bucket size: default
  Max bucket size, sqrt multiplier: default
  Max bucket size, len divisor: 4
  Difference-cover sample period: 1024
  Endianness: little
  Actual local endianness: little
  Sanity checking: disabled
  Assertions: disabled
  Random seed: 0
  Sizeofs: void*:8, int:4, long:8, size_t:8
Input files DNA, FASTA:
  nt-dusted.fna
Reading reference sizes
Warning: Encountered reference sequence with only gaps
Warning: Encountered reference sequence with only gaps
Warning: Encountered reference sequence with only gaps
...
Warning: Encountered reference sequence with only gaps
Warning: Encountered reference sequence with only gaps
Warning: Encountered reference sequence with only gaps
  Time reading reference sizes: 01:37:54
Calculating joined length
Writing header
Reserving space for joined string
Joining reference sequences

What went wrong?

[mourisl]

How much memory do you have on your machine? I guess the job got killed due to the out-of-memory issue.

[igordot]

I used 150G. What would be a reasonable amount?

It did not give any kind of memory error.

[mourisl]

150G is not enough for nt database. I don't remember the total size of the current nt, but I think you may need a machine with more than 1TB memory to build the nt index.

[igordot]

nt.fna is 1.5TB. Should I provide more memory than that?

[mourisl]

You may need about 3TB memory for that then..

[igordot]

If I don't have that much, are there any alternatives?

[mourisl]

How much memory do you have?

[igordot]

Theoretically 1.5TB, but it's a shared environment and unclear how long I would need to wait to actually get that.

[mourisl]

If you have the files ready, you may try centrifuger (https://github.com/mourisl/centrifuger). It has the option "--build-mem" in "centrifuger-build". Perhaps you can try something like "--build-mem 1200G" or slightly more, which will try to find appropriate parameters so the memory usage is roughly within the given range. If you want to try centrifuger, please use the "git clone" to get the package. I recently accelerated the index building time efficiency, and the updated code will be in the next formal release. You can use "-t 16" for parallelization too.

[igordot]

Is the Centrifuger index compatible with Centrifuge?

[mourisl]

No, the underlying data structure is quite different, so the index is not compatible.

[igordot]

It was not able to finish in 5 days with 500G. Does that seem reasonable?

[mourisl]

500G might not be enough in the end. For 1.5T sequence, storing the raw sequences takes about 400G space, and representing the BWT can take another 400G, which is well over the memory allocation. I think much of the time will be spent on memory page swapping. I would still recommend to allocate memory as much as possible.

[igordot]

I tried Centrifuger 1.0.2 with 1400G mem and 16 threads. In 15 days, it was able to extract 368/86745 chunks, so still far from complete. Probably a lot more memory is needed for this to be feasible in a reasonable amount of time.

[mourisl]

Thank you for the updates. Indeed, this is too slow. Speeding this up is one of the next goals..How long does it take to process one batch (16 chunks?). What are the inferred "Esimtated block size" and "dcv"?