Docker Desktop on Windows with WSL2 and Nvidia card 5090

[Javdu10]

Hello! Seems like the cookbook is falling short somewhere and could not identify what yet, I had nvidia-smi showing my card

So I went in the container and built it manually, starting from the start :

Go in the docker container from odysseus folder in wsl

docker compose exec odysseus bash

1. Add the correct NVIDIA CUDA keyring for Debian 13

apt install -y wget ca-certificates gnupg

wget -O /tmp/cuda-keyring.deb \
  https://developer.download.nvidia.com/compute/cuda/repos/debian13/x86_64/cuda-keyring_1.1-1_all.deb

dpkg -i /tmp/cuda-keyring.deb

apt update

2. Install CUDA toolkit / dependencies

Full toolkit:

apt install -y cuda-toolkit-13-3

3. Export CUDA paths

export CUDA_HOME=/usr/local/cuda-13.3
export CUDA_PATH=/usr/local/cuda-13.3
export PATH=/usr/local/cuda-13.3/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda-13.3/lib64:${LD_LIBRARY_PATH:-}
export CMAKE_CUDA_ARCHITECTURES=120

Optional but useful:

ln -sfn /usr/local/cuda-13.3 /usr/local/cuda

echo "/usr/local/cuda-13.3/lib64" > /etc/ld.so.conf.d/cuda-13-3.conf
ldconfig

Verify:

nvcc --version
ldconfig -p | grep 'libcudart\.so'

4. Build llama-server with CUDA for RTX 5090

cd /app/llama.cpp

rm -rf build
mkdir -p ~/bin

cmake -B build \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES=120 \
  -DCMAKE_CUDA_COMPILER=/usr/local/cuda-13.3/bin/nvcc

cmake --build build -j"$(nproc)" --target llama-server

5. Put llama-server where Odysseus expects it

ln -sf /app/llama.cpp/build/bin/llama-server ~/bin/llama-server

6. Verify the binary

~/bin/llama-server --version

ldd ~/bin/llama-server | grep -E 'cudart|cublas|cuda'

7. Then re-serve the model from Odysseus

Odysseus should now find the built llama-server and just use it

[Spaction]

In the odysseus folder there is a docker-compose.gpu-nvidia.yml. you need to change your docker compose command to use that file instead, which exposes your GPU to the docker images. The direction are missing that.

the command i used is
docker compose -f docker-compose.gpu-nvidia.yml up -d --build

The extra piece of config is
deploy: resources: reservations: devices: - driver: nvidia count: all capabilities: [gpu]

[Javdu10]

Hello, you might have missed the first few sentences :)

I've said that nvidia-smi was working in the container which already implies this step about the compose file (which was adjusted with the .env file)