adding rocm7.2 to ollama

[BillyOutlast]

🌟 Briefly describe the feature

Add rocm to ollama

📝 Detailed description

Currently, the Ollama LXC installation script primarily caters to CPU or NVIDIA (CUDA) environments. I am requesting an update to the script to include support for AMD GPUs by integrating the ROCm backend (specifically ROCm 7.2 or the latest compatible version).

Ideally, the script could include an interactive prompt during setup (e.g., "Would you like to install AMD ROCm support?") or auto-detect an AMD GPU that has been passed through to the container. The script would then need to handle the installation of the necessary AMD GPU drivers and ROCm libraries required by Ollama for hardware acceleration.

💡 Why is this useful?

While CPU inference is great, hardware acceleration is essential for running larger LLMs with usable generation speeds. Expanding on "AMD GPUs":

Massive Performance Boost: ROCm support allows Ollama to utilize AMD GPUs, drastically increasing token-per-second (t/s) inference speeds compared to CPU-only setups.

Homelab Hardware Diversity: Many Proxmox users and homelabbers utilize AMD GPUs due to their price-to-performance ratio or availability.

Native Ollama Support: Ollama already officially supports ROCm for AMD GPUs. Adding this to the Proxmox helper script bridges the gap, saving users from having to manually configure complex ROCm environments and driver dependencies inside their LXC containers.

[lumbarthought]

so youve got a working config with ollama running on your gpu via rocm? you installed with the script and then took the steps outlined above?

[lumbarthought]

I've been able to get my own custom install working with vulkan. i kept running into roadblocks with rocm. i suspect the alignment between proxmox version and LXC version, matching drivers, etc, might make a ROCM script too finicky to make work consistently for all users. But maybe vulkan could be doable since it's more forgiving. That said, helper scripts jellyfin version doesn't support AMD gpus so I think AMD gpus are second class citizens in general

[jamiej]

Using the PVE scripts and some tweaking, I have ollama running with rocm. The PVE ollama script installs rocm 7.2.0 in it's default location under /opt, however ollama's rocm (https://docs.ollama.com/linux#amd-gpu-install) expects to untar those overlaying them in /usr. If you do that, then rocm ought work.

You may wish to customize ollama.service too. My setup is dual R9700, and I'm able to run gemma4-31b using rocm w/ 64k context window. Still tuning, but this works for me:

Environment=HSA_OVERRIDE_GFX_VERSION=12.0.1
Environment=ROCR_VISIBLE_DEVICES=GPU-c888cf3dcec8ebdf,GPU-686364bdf23a9124
Environment=ROCM_PATH=/opt/rocm
Environment=PYTORCH_ROCM_ARCH=gfx1201
Environment=LD_LIBRARY_PATH=/opt/rocm/lib:/usr/local/lib

Custom Gemma4 (Gemma4.Modelfile):

FROM ./google_gemma-4-31B-it-Q4_K_M.gguf
PARAMETER num_ctx 65536

[NeilS1966]

Jamie, are you able to elaborate further on the changes you made to the script?

[jamiej]

I didn't change the script, I went in after the fact. Ollama expect the following,

curl -fsSL https://ollama.com/download/ollama-linux-amd64-rocm.tar.zst \ | sudo tar x -C /usr

Per the docs here, AMD GPU install

If targeting AMD, then I'd recommend checking out Lemonade. Super easy to use, with management through web-interface. Including downloading models, and backends like Vulkan and ROCm. Fetch the docker image, add GPU's to the resulting LXC.

Here's a script I wrote to add 2xR9700 to an LXC.

#!/bin/bash

# Usage: ./add-gpus.sh <VMID>
VMID=${1}

if [[ -z "$VMID" ]]; then
    echo "Error: Please provide a VMID (e.g., ./add-gpus.sh 100)"
    exit 1
fi

# Verify the container exists
if ! pct status "$VMID" >/dev/null 2>&1; then
    echo "Error: Container $VMID not found."
    exit 1
fi

echo "Adding AMD GPU devices to LXC $VMID..."

# Map the DRI cards (Video Group: 44)
pct set "$VMID" --dev0 /dev/dri/card1,gid=44
pct set "$VMID" --dev1 /dev/dri/card2,gid=44

# Map the Render nodes (Render Group: 109)
pct set "$VMID" --dev2 /dev/dri/renderD128,gid=109
pct set "$VMID" --dev3 /dev/dri/renderD129,gid=109

# Map the Kernel Fusion Driver (Crucial for ROCm/KFD)
pct set "$VMID" --dev4 /dev/kfd,gid=109

echo "Done. Please restart container $VMID for changes to take effect."

[NeilS1966]

Thank you, much appreciated.

[DiSHTiX]

I didn't change the script, I went in after the fact. Ollama expect the following,

curl -fsSL https://ollama.com/download/ollama-linux-amd64-rocm.tar.zst \ | sudo tar x -C /usr

Per the docs here, AMD GPU install

Should be
curl -fsSL https://ollama.com/download/ollama-linux-amd64.tar.zst \ | sudo tar x -C /usr
curl -fsSL https://ollama.com/download/ollama-linux-amd64-rocm.tar.zst \ | sudo tar x -C /usr

Ollama-rocm, is an addition to the base package.

Thanks for sharing Lemonade, i hadnt seen this before

[jamiej]

@DiSHTiX

Thanks for sharing Lemonade, i hadnt seen this before

Assuming my patience holds, community-scripts/ProxmoxVED#1949.