NnUNetTrainerUMambaEncNoAMP mode displays CUDA out of memory

[cjf128]

After pre training with nnUNetv2_plan_and_preprocessing - d 101-- verify_daset_integrity, I was able to train nnUNetTrainerUMambaBot normally, but when I changed it to nnUNetTrainerUMambaEncNoAMP, the following error occurred:

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1000.00 MiB. GPU 0 has a total capacty of 23.46 GiB of which 406.88 MiB is free. Including non-PyTorch memory, this process has 23.06 GiB memory in use. Of the allocated memory 22.78 GiB is allocated by PyTorch, and 82.27 MiB is reserved by PyTorch but unallocated.

[fuyuao000]

Disabling AMP will indeed cause an increase in memory consumption. If you have reasons that require disabling AMP and encounter OOM errors, it becomes necessary to reduce memory usage from other aspects, such as gradient accumulation, gradient checkpointing, or disabling deep supervision.

[cjf128]

Disabling AMP will indeed cause an increase in memory consumption. If you have reasons that require disabling AMP and encounter OOM errors, it becomes necessary to reduce memory usage from other aspects, such as gradient accumulation, gradient checkpointing, or disabling deep supervision.

Thank you for your reply. Since using AMP would lead to a dice score of 0 during training, I opted for NOAMP. I have now resolved this issue and am currently using nnUNetTrainerUMambaEnc instead of nnUNetTrainerUMambaEncNoAMP. However, I have reduced the learning rate to 0.001, as I found that using 0.001 seems to perform better on my dataset (I'm not familiar with the reply function on GitHub, sorry for the inconvenience)