How to accelerate the training of the state model?

[Yan-Jack-jie]

GPU: NVIDIA H200
Memory: 128 GB
CPU: Intel® Xeon® Platinum 8558 Processor (260 MB Cache, 2.10 GHz)

Each training run of my state model takes an entire day. Is there any way to speed up the training? Increasing num_workers from 12 to 24 didn’t provide much improvement.

[abhinadduri]

hi @Yan-Jack-jie , can you share more information on the data you are training on? most likely it is due to data loader bottlenecks.