[Bug]: k8s-device-plugin fails to detect Jetson platform

[shahizat]

Hello

Environment:

• Hardware: NVIDIA Jetson AGX Thor Developer Kit (aarch64, iGPU "NVIDIA Thor")
• JetPack 7.2, driver 595.78, CUDA 13.2
• MicroK8s 1.35 (containerd 2.1.6), Helm chart nvidia-device-plugin 0.9.0
• Device-plugin image: nvcr.io/nvidia/k8s-device-plugin:v0.19.3

Installed via:

helm upgrade -i nvidia-device-plugin \
  nvidia/nvidia-device-plugin \
  -n gpu-operator \
  --version 0.9.0 \
  --set image.tag=v0.19.3 \
  --set deviceIDStrategy=index

On Nvidia Jetson Thor the v0.19.3 device-plugin pod starts and immediately dies with Exit Code: 1 in a crash loop. The pod log shows the plugin no longer detects the Jetson platform and falls back to NVML discovery, which does not exist in the Jetson container image:

I0814 09:35:39.851565       1 main.go:369] Retrieving plugins.
I0814 09:35:39.851780       1 plugin-manager.go:100] Detected platform: unknown
W0814 09:35:39.851796       1 plugin-manager.go:107] Unsupported platform detected; defaulting to nvml
I0814 09:35:39.851804       1 plugin-manager.go:50] Using device discovery strategy: nvml
E0814 09:35:39.851839       1 factory.go:99] Failed to initialize NVML: ERROR_LIBRARY_NOT_FOUND.
E0814 09:35:39.851845       1 factory.go:100] If this is a GPU node, did you set the docker default runtime to `nvidia`?
E0814 09:35:39.851849       1 factory.go:101] You can check the prerequisites at: https://github.com/NVIDIA/k8s-device-plugin#prerequisites
E0814 09:35:39.851853       1 factory.go:102] You can learn how to set the runtime at: https://github.com/NVIDIA/k8s-device-plugin#quick-start
E0814 09:35:39.851857       1 factory.go:103] If this is not a GPU node, you should set up a toleration or nodeSelector to only deploy this plugin on GPU nodes
E0814 09:35:39.852018       1 main.go:185] error starting plugins: error getting plugins: unable to create plugins: failed to construct resource managers: nvml init failed: ERROR_LIBRARY_NOT_FOUND

The same Helm chart with image.tag=v0.16.2 (and deviceIDStrategy=index) runs fine on this board, registers successfully, and is allocatable:

nvidia.com/gpu: 1
Registered device plugin for 'nvidia.com/gpu' with Kubelet

[shahizat]

k8s device plugin needs to support MIG in order to provide full MIG functionality on the NVIDIA Jetson Thor.

[abrarshivani]

@shahizat Thanks for the details. For the first issue, this looks like a chart and image version mismatch.

You're using Helm chart 0.9.0 with device-plugin image v0.19.3. The current chart and plugin release are both 0.19.3:

Release:
https://github.com/NVIDIA/k8s-device-plugin/releases/tag/v0.19.3

Helm chart:
https://github.com/NVIDIA/k8s-device-plugin/tree/v0.19.3/deployments/helm/nvidia-device-plugin

The important behavior change is in platform detection. With v0.16.2, platform detection uses /, so it can see the Tegra files directly and select the Tegra path. With v0.19.3, detection uses ContainerDriverRoot (/driver-root). The old 0.9.0 chart does not mount the host driver root there, so Thor/Tegra detection returns unknown. The plugin then falls back to NVML and fails with ERROR_LIBRARY_NOT_FOUND

I recommend testing with matching chart and image versions:

helm repo add nvdp https://nvidia.github.io/k8s-device-plugin
helm repo update

helm upgrade -i nvidia-device-plugin \
  nvdp/nvidia-device-plugin \
  -n gpu-operator \
  --version 0.19.3 \
  --set image.tag=v0.19.3 \
  --set deviceIDStrategy=index \
  --set nvidiaDriverRoot=/

[shahizat]

Hello, @abrarshivani . Yes, it works, but the question remains the same: Jetson Thor exposes a single fused integrated GPU (NVIDIA Thor iGPU), not NVML-managed discrete GPU with MIG partitioning, right? If full MIG support will be added, I assume people would need to use it through Kubernetes. I have some doubts that people would actually use the kernel-level MIG partitioning directly.

[abrarshivani]

@shahizat Thanks for confirming the platform-detection issue is resolved. Agreed that exposing MIG functionality through Kubernetes would be useful for Jetson Thor, but this isn't supported by the device plugin today. Since this issue currently covers both the chart/image version mismatch and the separate MIG support request, I'll open a dedicated GitHub RFE for Thor MIG support so it can be tracked independently.

[abrarshivani]

Closing this as resolved. As, the chart/image version mismatch issue is resolved. Jetson Thor MIG support is a separate feature request and is now being tracked in #1975.