[Script request]: EXO

[Data-PhiIe]

Application Name

EXO

Official Website

https://exolabs.net/

Source Repository

https://github.com/exo-explore/exo

Repository Stars

47,1000

Minimum Adoption Requirement

• The official source repository has at least 1,000 stars.

Description

Run frontier AI locally.

Why should this be added?

It's a good way to distribute an LLM across multiple nodes. I currently have this set up manually, with both GPU and RDMA passthrough.

Installation Notes

I aplologize for this very long section I'm going to copy paste the notes from when I set mine up originally, hopefully it helps take the steps I took and automate them with a helper script! :)

Installation Steps for Exo Labs on Debian

Prerequisites

• Before starting the installation, ensure you have the following:

  • Python 3.12+: Required for running Exo.
  • Git: Needed to clone the repository.

Step-by-Step Installation

Follow these steps to install Exo Labs on your Debian system:

1. Update Package List
Open your terminal and run the following command to update your package list:

apt update

2. Install Node.js and npm
Install Node.js and npm using the following command:

sudo apt install -y build-essential nodejs npm python3 python3-pip python3.13-dev python3.13-venv pkg-config libssl-dev git curl
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
source $HOME/.cargo/env
python3.13 -m venv venv && source venv/bin/activate
python -m pip install --upgrade pip setuptools wheel

3, Clone the Exo Repository
Clone the Exo repository from GitHub:

git clone https://github.com/exo-explore/exo.git

4. Install Required Dependencies
Navigate to the cloned directory and install the required Python dependencies:

cd exo
python -m pip install --no-binary :all: . -v

Additional Notes

• Ensure that you have the necessary permissions to install packages and run commands.
• If you encounter any issues, check the Exo documentation for troubleshooting tips.

By following these steps, you will have Exo Labs installed and ready to use on your Debian system.

---

Generative Tool usage disclosure.

This solution was what worked in my configuration. MD formatting done myself with LLM assistance (not just a 1:1 copy paste) + generated with DuckDuckGo Search Assist, and/or DuckAI web sources referenced by LLM below:
Markdown formatting done by myself + LLM assisted sugestions to achieve the desired outcome, instructions generated with DuckDuckGo Search Assist, sources the LLM referenced below:
Github Sonit Medium

Enabling CUDA

To enable CUDA on ExoLabs, you need to install the NVIDIA CUDA toolkit and ensure your GPU drivers are up to date. After that, you can run the Exo application with the appropriate commands to utilize CUDA for distributed inference.

Steps to Enable CUDA on ExoLabs

To enable CUDA on ExoLabs, follow these steps:

1. Install NVIDIA CUDA Toolkit

Download the latest version of the NVIDIA CUDA toolkit from the official NVIDIA website.

Ensure that you select the correct version compatible with your operating system.

2. Update GPU Drivers

Make sure your NVIDIA GPU drivers are up to date. You can check your current driver version using the command:

nvidia-smi

If your drivers are outdated, download and install the latest drivers from the NVIDIA website.

3. Verify CUDA Installation

After installing the CUDA toolkit, verify the installation by running the following command in your terminal:

nvcc --version

This command should display the version of the installed CUDA toolkit.
If nvcc --version returns a value skip to step [] if not continue from here
install the CUDA toolkit and configure the environment so applications (and ExoLabs) can use CUDA.

Steps (assume Ubuntu/Debian; adapt if different):

4. Install CUDA toolkit

• Add NVIDIA package repo and install toolkit matching your driver/CUDA compatibility. For CUDA 13.3 (your driver shows CUDA UMD 13.3) use the CUDA 13.x toolkit.
• Fetch and install the NVIDIA apt repo package from developer.nvidia.com (or use the network repo instructions for CUDA 13.x).
• Then:

apt update && upgrade
apt install -y cuda-toolkit-13-3

Alternatively, use the runfile installer from NVIDIA if you prefer.

5. Verify nvcc and PATH

After install, ensure nvcc is on PATH:
Typical locations: /usr/local/cuda-13.3/bin

• Add to shell profile:

nano .bashrc

• add cuda path to the file

# cuda path
export PATH=/usr/local/cuda-13.3/bin:$PATH export LD_LIBRARY_PATH=/usr/local/cuda-13.3/lib64:$LD_LIBRARY_PATH

• type ctrl + X followed by y then enter
• Reload:

source ~/.bashrc

• Check:

nvcc --version

6. Verify driver & CUDA runtime compatibility

If you installed a mismatched toolkit, reinstall a compatible one.
Install cuDNN / NCCL if needed

If ExoLabs or your workloads need cuDNN or NCCL, download matching versions for CUDA 13.x from NVIDIA (or apt packages where available) and install.

ExoLabs-specific settings (common patterns)

Ensure ExoLabs process sees GPUs:

• run a test program (nvidia-smi, nvidia-smi topo --matrix, or a simple CUDA sample like deviceQuery).
• If ExoLabs has config for device selection or backend, set backend to CUDA and point to CUDA toolkit path if required.
• If ExoLabs uses Python, install CUDA-aware packages (torch with cuda support, tensorflow cuda builds) matching CUDA 13.x or use wheel/cuda toolkit compat wheels.

---

Troubleshooting

nvcc still missing after toolkit install:

• verify package installed and PATH points to correct /usr/local/cuda-/bin.
• Permission issues: ensure drivers and toolkit installed as root or via sudo.
• Library mismatches: check dmesg and nvidia-smi for driver errors.

Quick verification commands

nvidia-smi
nvcc --version

(if using PyTorch) /usr/local/cuda-13.3/extras/demo_suite/deviceQuery (CUDA samples)

python -c "import torch; print(torch.cuda.is_available()); print(torch.cuda.get_device_name(0))"

7. Set Up Exo Application

• Create a virtual environment and activate it:

python3 -m venv exo/.venv/
source exo/.venv/bin/activate

• Install the Exo application with the following command:

pip install -e .

• Upgrade tinygrad to the latest version to fix any CUDA issues:

pip install --upgrade git+https://github.com/tinygrad/tinygrad.git

8. Run Exo with CUDA

Start the Exo application using the CUDA backend with the following command:

exo --inference-engine tinygrad --chatgpt-api-port 8001 --disable-tui

9. Test CUDA Functionality

To confirm that CUDA is functioning correctly, run the minimal smoke test commands:

nvidia-smi
python3 -c "from tinygrad import Device; print(Device.DEFAULT)"

If everything is set up correctly, you should see "CUDA" as the output.

By following these steps, you will successfully enable CUDA on ExoLabs for distributed inference.

Generative Tool usage disclosure.

This solution was what worked in my configuration. Markdown formatting done by myself + LLM assisted sugestions to achieve the desired outcome (not just a 1:1 copy paste) + generated with DuckDuckGo Search Assist, and/or DuckAI web sources referenced by LLM below:

---

modal.com Medium

Service File

sudo tee /etc/default/exo > /dev/null <<'EOF'
# /etc/default/exo - environment for exo systemd service
# Adjust CUDA version/path if different

PATH=/usr/local/cuda-13.3/bin:/root/exo/.venv/bin:/root/.local/bin:/root/bin:/sbin:/bin:/usr/sbin:/usr/bin
LD_LIBRARY_PATH=/usr/local/cuda-13.3/lib64:/usr/lib/x86_64-linux-gnu
# Optional: set CUDA_HOME
CUDA_HOME=/usr/local/cuda-13.3

EOF

# reload systemd and restart service
sudo systemctl daemon-reload
sudo systemctl restart exo.service

nano /etc/systemd/system/exo.service

[Unit]
Description=exo-explore service
After=network.target

[Service]
Type=simple
User=root
WorkingDirectory=/root/exo
Environment="PATH=/usr/local/cuda-13.3/bin:/root/exo/.venv/bin:/root/.venv/bin:/root/.local/bin:/root/bin:/sbin:/bin:/usr/sbin:/usr/bin"
Environment="LD_LIBRARY_PATH=/usr/local/cuda-13.3/lib64:/usr/local/cuda/lib64:/usr/lib/x86_64-linux-gnu:$LD_LIBRARY_PATH"
ExecStart=/root/exo/.venv/bin/exo
Restart=on-failure
RestartSec=3
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

Enable RoCE on Mellanox 100GbE cards (Proxmox / Linux)

Assumptions: Mellanox cards support RoCE, firmware and drivers are up to date, host OS is Debian/Proxmox (based on Debian/Ubuntu), and your network switches support DCB/PFC or you will use RoCEv2 (UDP/IP) without lossless fabric.

1. Install/verify drivers and tools

• Install Mellanox OFED or ensure the kernel mlx5 driver is present:
• For Mellanox OFED (recommended for latest features): download and install the appropriate MLNX_OFED package for your kernel.
• Install utilities:

apt install -y rdma-core libibverbs1 libibverbs-dev ibverbs-utils perftest mstflint

• Reboot if drivers/firmware were updated.

2. Verify device and driver

• List devices:

ip link show

lspci | grep -i mellanox

• Check driver:

ethtool -i <iface>

• rdma-core tools: ibv_devinfo (for RDMA devices)

3. Choose RoCE version

• RoCEv1: link-layer (requires DCB, lossless Ethernet) — limited; generally avoid.
• RoCEv2: routable, UDP encapsulation — common choice. Use RoCEv2 unless you have a lossless DCB fabric.

4. Configure firmware/driver for RoCE

• On mlx5 devices, enable RoCEv2 in driver params if needed. Example sysfs settings:

echo 2 > /sys/class/net/<iface>/device/roce_mode

• 0 = RoCE disabled, 1 = RoCEv1, 2 = RoCEv2 (device-dependent)
  • If that sysfs node is absent, use mstconfig/mstflint or driver options; check Mellanox docs for your NIC model.
• Alternatively use mlxconfig (or mstconfig) to set RoCE:

mlxconfig -d /dev/mst/<mst_device> set LINK_TYPE_P1=ETH

(example; consult model-specific flags)

• for flags on your device:

mlxconfig --help 

5. Configure IP and MTU

• Assign IPs to interfaces (RoCEv2 uses IP/UDP).
• Set jumbo frames to reduce CPU/overhead: ip link set dev mtu 9216

6. Ensure kernel modules and services

• Load modules:

modprobe mlx5_core mlx5_ib

• Start RDMA services (if present): systemctl start rdma

7. Switch/fabric configuration

• For RoCEv2: ensure switches allow large MTU, no specific DCB required, but QoS/queueing and ECN configuration may improve performance.
• For RoCEv1/DCB: configure Priority Flow Control (PFC) and Enhanced Transmission Selection (ETS) on switches to make Ethernet lossless for RDMA.

8. Verify RDMA interface

• Run:

ibv_devices

ibv_devinfo -v

rdma link show

• Use perftest for checks:

ib_write_bw, ib_read_bw

Or for RoCE UDP tests use rping/rping2 or perftest with IPoIB/RoCE options.

9. Troubleshooting

• If ibv_devinfo shows no devices or errors: check mlx5_ib loaded, firmware compatible, and that RoCE mode is enabled.
• Check dmesg for mlx5/ib errors.
• Verify MTU and IP addressing, and firewall rules do not block UDP destination ports used by RoCE (RoCEv2 uses UDP; specific ports depend on implementation).
• If using virtualized guests: expose PF/VF via SR-IOV (vfio-pci or macvtap) rather than a Linux bridge to avoid losing RDMA capabilities. Configure SR-IOV on NIC and pass VF to VM.

10. Enabling in VM and LXC containers

In Proxmox VM context

• Enable SR-IOV on host NIC and create VFs:
  • echo > /sys/class/net/<physical_iface>/device/sriov_numvfs
• Bind VFs to vfio-pci and add to VM (PCI passthrough) or use virtio-ibmv (not common). RDMA in guests usually requires PCI passthrough or SR-IOV VFs passed through.
• Alternatively use host-based RDMA with network block storage or iSCSI offload if you cannot passthrough.

In Proxmox LXC context

Unprivileged LXCs cannot get PCI passthrough; you must keep the Mellanox driver on the host and expose the RDMA character devices and required sysfs paths to the container. Below are exact, minimal steps and the exact Proxmox LXC config lines.

Assumptions

• Host uses mlx5 driver (mlx5_core / mlx5_ib) and rdma-core installed.
• Host interface is mlx5_0 with /dev/infiniband and /dev/uverbs* present.
• LXC id is 101. Adjust paths, device names, and major:minor if they differ.

Host: check devices and nodes

1. Confirm devices:

ls -ld /dev/infiniband/* /dev/uverbs* /dev/mlx5*

2. Note device majors/minors (example):

ls -l /dev/uverbs0
ls -l /dev/infiniband/uverbs0

Bind-mount device nodes and sysfs into unprivileged LXC

1. Create a folder on host to expose devices (optional, clearer):

mkdir -p /var/lib/lxc-rdma-devs/<ct-id>/dev-infiniband
cp -a --parents /dev/infiniband/* /var/lib/lxc-rdma-devs/<ct-id>/dev-infiniband/

(use cp -a to preserve nodes)

# if present
cp -a --parents /dev/uverbs* /var/lib/lxc-rdma-devs/101/dev-infiniband/

2. Adjust permissions so container's unprivileged UID/GID mapping can access nodes:

• Determine container's first mapped uid/gid (usually 100000). Use: cat /etc/subuid | grep ^root

chown -R 100000:100000 /var/lib/lxc-rdma-devs/<ct-id>/dev-infiniband
chmod -R 0660 /var/lib/lxc-rdma-devs/<ct-id>/dev-infiniband/*

• Alternatively:

uid=$(awk -F: '/^root:/{print $2;exit}' /etc/subuid)
gid=$(awk -F: '/^root:/{print $2;exit}' /etc/subgid)
chown -R ${uid}:${gid} /var/lib/lxc-rdma-devs/<ct-id>
find /var/lib/lxc-rdma-devs/<ct-id> -type d -exec chmod 0755 {} \;
find /var/lib/lxc-rdma-devs/<ct-id> -type f -exec chmod 0660 {} \;

3. Bind-mount into LXC config (/etc/pve/lxc/.conf):

nano /etc/pve/lxc/<ct-id>.conf

Add these exact lines:

dev5: /dev/infiniband/by-ibdev/uverbs-mlx5_0
lxc.mount.entry: /var/lib/lxc-rdma-devs/<ct-id>/dev-infiniband dev/infiniband none bind,create=dir
mp2: /var/lib/lxc-rdma-devs/<ct-id>/dev-infiniband,mp=/dev/infiniband
lxc.cgroup2.devices.allow: c 10:263 rwm
lxc.cgroup2.devices.allow: c 231:0 rwm
lxc.cgroup2.devices.allow: c 231:192 rwm
#(Adjust major:minor numbers as reported by ls -l; example majors: /dev/infiniband/* use major 10:59, /dev/uverbs* often major 243)
# Also note these 3 cgroups are what I needed for my config, there may be more in a different configuration.

Exact device-major lookup command (run on host) and substitute results:

• for f in /dev/infiniband/* /dev/uverbs*; do ls -l $f; done

Expose sysfs and net device info (bind-mounts)

Add these mp entries to /etc/pve/lxc/101.conf:

• mp1: /sys/class/infiniband,mp=/sys/class/infiniband
• mp2: /sys/class/net/<host_iface>/device,mp=/sys/class/net/<host_iface>/device

Replace <host_iface> with your Mellanox interface name (ip link show to find it).

Allow needed capabilities (already limited for unprivileged; add only what’s necessary)

Add to /etc/pve/lxc/101.conf:

• lxc.cap.drop:
• (Do not add CAP_SYS_ADMIN. Unprivileged containers lack privilege to create device nodes; we used bind-mounts instead.)

Inside the container (after starting it)

1. Install userspace RDMA libs:

apt update && apt install -y rdma-core ibverbs-utils

2. Verify devices visible:

ls -l /dev/infiniband /dev/uverbs*

ibv_devinfo

rdma link show

Notes and troubleshooting

• If ibv_devinfo returns "No IB devices found": check device node ownership/permissions and cgroup device allow entries (major:minor must match).
• If udev recreates nodes with different minor numbers after reboot, recreate the host folder and repopulate bind-mounts, update ownership.
• Performance: this exposes user-space verbs to container; kernel driver remains on host so control operations (reset, firmware) stay host-managed.
• Security: exposing device nodes to unprivileged container still gives hardware access; ensure the container is trusted.

Prompts to run for AI assistance:

• If you paste ls -l output for /dev/infiniband and /dev/uverbs* and your LXC id, I'll return the exact lxc.cgroup2.devices.allow lines with correct major:minor numbers.
  Run:

ls -l /dev/infiniband /dev/uverbs*

If you want, tell me your NIC model (output of lspci -nn | grep -i mellanox) and whether you plan host-only or VM passthrough, give exact commands for your hardware and Proxmox version.
Run:

lspci -nn | grep -i mellanox

Use these exact lines in /etc/pve/lxc/101.conf (adjust mp0 path if your LXC id differs):

mp0: /var/lib/lxc-rdma-devs/101/dev-infiniband,mp=/dev/infiniband
lxc.cgroup2.devices.allow: c 10:263 rwm
lxc.cgroup2.devices.allow: c 231:0 rwm
lxc.cgroup2.devices.allow: c 231:192 rwm

Explanation of mapping (from your ls -l):

• /dev/infiniband/rdma_cm = major:minor 10:263
• /dev/infiniband/umad0 = major:minor 231:0
• /dev/infiniband/uverbs0 = major:minor 231:192

After updating the config:

1. Ensure the host folder contains the nodes with same ownership:
   • cp -a --parents /dev/infiniband/* /var/lib/lxc-rdma-devs/101/dev-infiniband/
   • chown -R 100000:100000 /var/lib/lxc-rdma-devs/101/dev-infiniband
2. Restart the container:
   • pct stop 101 && pct start 101
3. Inside container verify:
   • ls -l /dev/infiniband
   • ibv_devinfo

If you get permission denied inside the container, paste the output of ls -l /var/lib/lxc-rdma-devs/101/dev-infiniband and cat /etc/subuid for adjustments.

iwpmd.service is a disabled or a static unit, not starting it.
rdma-hw.target is a disabled or a static unit, not starting it.
rdma-ndd.service is a disabled or a static unit, not starting it.
rocep59s0: Failed to write 'change' to '/sys/devices/pci0000:3a/0000:3a:00.0/0000:3b:00.0/infiniband/rocep59s0/uevent': Permission denied
umad0: Failed to write 'change' to '/sys/devices/pci0000:3a/0000:3a:00.0/0000:3b:00.0/infiniband_mad/umad0/uevent': Permission denied

Due Diligence

• I have searched existing scripts and found no duplicate.
• I have searched existing discussions and found no duplicate request.
• The application has an official public source repository.
• The application is self-hosted and suitable for Proxmox VE Helper-Scripts.
• The application appears to be actively maintained and provides official releases, tags, or release tarballs.
• I understand that requests outside the project scope may be closed without an extensive explanation.