馃専 Briefly describe the feature
Add rocm to ollama
馃摑 Detailed description
Currently, the Ollama LXC installation script primarily caters to CPU or NVIDIA (CUDA) environments. I am requesting an update to the script to include support for AMD GPUs by integrating the ROCm backend (specifically ROCm 7.2 or the latest compatible version).
Ideally, the script could include an interactive prompt during setup (e.g., "Would you like to install AMD ROCm support?") or auto-detect an AMD GPU that has been passed through to the container. The script would then need to handle the installation of the necessary AMD GPU drivers and ROCm libraries required by Ollama for hardware acceleration.
馃挕 Why is this useful?
While CPU inference is great, hardware acceleration is essential for running larger LLMs with usable generation speeds. Expanding on "AMD GPUs":
Massive Performance Boost: ROCm support allows Ollama to utilize AMD GPUs, drastically increasing token-per-second (t/s) inference speeds compared to CPU-only setups.
Homelab Hardware Diversity: Many Proxmox users and homelabbers utilize AMD GPUs due to their price-to-performance ratio or availability.
Native Ollama Support: Ollama already officially supports ROCm for AMD GPUs. Adding this to the Proxmox helper script bridges the gap, saving users from having to manually configure complex ROCm environments and driver dependencies inside their LXC containers.
馃専 Briefly describe the feature
Add rocm to ollama
馃摑 Detailed description
Currently, the Ollama LXC installation script primarily caters to CPU or NVIDIA (CUDA) environments. I am requesting an update to the script to include support for AMD GPUs by integrating the ROCm backend (specifically ROCm 7.2 or the latest compatible version).
Ideally, the script could include an interactive prompt during setup (e.g., "Would you like to install AMD ROCm support?") or auto-detect an AMD GPU that has been passed through to the container. The script would then need to handle the installation of the necessary AMD GPU drivers and ROCm libraries required by Ollama for hardware acceleration.
馃挕 Why is this useful?
While CPU inference is great, hardware acceleration is essential for running larger LLMs with usable generation speeds. Expanding on "AMD GPUs":
Massive Performance Boost: ROCm support allows Ollama to utilize AMD GPUs, drastically increasing token-per-second (t/s) inference speeds compared to CPU-only setups.
Homelab Hardware Diversity: Many Proxmox users and homelabbers utilize AMD GPUs due to their price-to-performance ratio or availability.
Native Ollama Support: Ollama already officially supports ROCm for AMD GPUs. Adding this to the Proxmox helper script bridges the gap, saving users from having to manually configure complex ROCm environments and driver dependencies inside their LXC containers.