Can ROCM be supported？

[skzyy]

I hope my 24G video memory can be put to good use @

[Mega4alik]

@skzyy the reason why Nvidia is the only supported GPU at moment is kvikio. I will make it optional in the next release so there's no additional restrictions beyond Huggingface Transformers and pytorch.

[Mega4alik]

@skzyy kvikio and flash-attn are optional now and should work on any device supported by HF transformers. Kindly update the ollm and try it!

[Aravindnaiduv]

specific technologies like CUDA You might want to check for recent updates to the library.