Look into `NVIDIA_VISIBLE_DEVICES`

[russellb]

canhazgpu run automatically sets CUDA_VISIBLE_DEVICES. reserve includes a hint in the output about setting that same variable.

There is also NVIDIA_VISIBLE_DEVICES when working with containers.

https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/docker-specialized.html

We should consider changes to make working with containers and GPU reservations easier.

[russellb]

The simplest first change would be to extend the output from reserve to include a hint on how to properly set this variable when running docker.

Today we have:

Using config file: /home/rbryant/.canhazgpu.yaml
Reserved 1 GPU(s): [2] for 8h 0m 0s

Run the following command to run only on these GPUs:
export CUDA_VISIBLE_DEVICES=2

We could extend it to be:

Using config file: /home/rbryant/.canhazgpu.yaml
Reserved 1 GPU(s): [2] for 8h 0m 0s

Run the following command to run only on these GPUs:
export CUDA_VISIBLE_DEVICES=2

For the NVIDIA container toolkit:
docker run -e NVIDIA_VISIBLE_DEVICES=2 ...

[russellb]

For run, we could support some string substitution in the command. If someone picks this up, I'd make this a separate PR from the simple change to the reserve output described above.

An idea would be ...

canhazgpu run -- docker run -e NVIDIA_VISIBLE_DEVICES=CANHAZGPU_DEVICE_IDS ...

where we automatically replace CANHAZGPU_DEVICE_IDS in the command with the device IDs allocated by run. Some would probably prefer CHG_DEVICE_IDS as a shorter name.