Talos 1.13.9 and Lenovo PGX workstation

[RA-JfA]

Hi,

I have Talos 1.13.9 installed on a GPU workstation (Lenovo PGX)
Installation of Talos itself works fine and i have added a bunch of extensions :

iscsi-tools v0.2.0
kata-containers 3.32 (1.13.9)
mdadm 4.6 (1.13.9)
nfs-utils 0.1.1
nvidia-container-toolkit-production 595.91.07-v1.19.1 (1.13.9)
nvidia-open-gpu-kernel-modules-production 595.91.07-v1.13.9 (1.13.9)
nvme-cli v2.14
util-linux-tools 2.41.1 (1.11.6) 2.42.2 (1.13.9) 595.91.07-v1.13.9 (1.1

The problem is enabling the Nvidia modules

talosctl get extensions --nodes <IP> NODE NAMESPACE TYPE ID VERSION NAME VERSION 10.16.51.22 runtime ExtensionStatus 0 1 iscsi-tools v0.2.0 10.16.51.22 runtime ExtensionStatus 1 1 kata-containers 3.32.0 10.16.51.22 runtime ExtensionStatus 2 1 mdadm v4.6 10.16.51.22 runtime ExtensionStatus 3 1 nfs-utils v0.1.1 10.16.51.22 runtime ExtensionStatus 4 1 nvidia-container-toolkit-production 595.91.07-v1.19.1 10.16.51.22 runtime ExtensionStatus 5 1 nvidia-open-gpu-kernel-modules-production 595.91.07-v1.13.9 10.16.51.22 runtime ExtensionStatus 6 1 nvme-cli v2.14 10.16.51.22 runtime ExtensionStatus 7 1 util-linux-tools 2.42.2 10.16.51.22 runtime ExtensionStatus 8 1 schematic 733e9958c6757ea6ee75451475f5109c5a8a34d6ab4cec6c0e5203b54ec0031f 10.16.51.22 runtime ExtensionStatus modules.dep 1 modules.dep 6.18.44-talos

$> talosctl -n <IP> service NODE SERVICE STATE HEALTH LAST CHANGE LAST EVENT 10.16.51.22 apid Running OK 19m14s ago Health check successful 10.16.51.22 auditd Running OK 19m26s ago Health check successful 10.16.51.22 containerd Running OK 19m26s ago Health check successful 10.16.51.22 cri Running OK 19m19s ago Health check successful 10.16.51.22 dashboard Running ? 19m23s ago Process Process(["/sbin/dashboard"]) started with PID 6174 10.16.51.22 ext-iscsid Running ? 19m19s ago Started task ext-iscsid (PID 6369) for container ext-iscsid 10.16.51.22 ext-nvidia-cdi-gen Waiting ? 19m25s ago Waiting for service "ext-nvidia-persistenced" to be "up" 10.16.51.22 ext-nvidia-persistenced Waiting ? 19m23s ago Waiting for file "/sys/bus/pci/drivers/nvidia" to exist 10.16.51.22 ext-rpc-statd Running ? 19m19s ago Started task ext-rpc-statd (PID 6425) for container ext-rpc-statd 10.16.51.22 ext-rpcbind Running ? 19m19s ago Started task ext-rpcbind (PID 6368) for container ext-rpcbind 10.16.51.22 kubelet Running OK 19m14s ago Health check successful 10.16.51.22 machined Running OK 19m26s ago Health check successful 10.16.51.22 syslogd Running OK 19m25s ago Health check successful 10.16.51.22 udevd Running OK 19m24s ago Health check successful

│ Nyare Talos (1.13+) upptäckt. Söker efter drivrutiner i monterade kataloger... │ Hittade bibliotek på hosten under: /talos-usr/local/lib │ Startar officiell NVIDIA Device Plugin... │ I0903 12:45:08.185106 1 main.go:250] "Starting NVIDIA Device Plugin" version=< │ 1ae8e5f0-arm64 │ commit: 1ae8e5f02a47ad80e2b4fcc6c35a757c61ddd81f │ > │ I0903 12:45:08.185183 1 main.go:253] Starting FS watcher for /var/lib/kubelet/device-plugins │ I0903 12:45:08.185248 1 main.go:260] Starting OS watcher. │ I0903 12:45:08.185743 1 main.go:275] Starting Plugins. │ I0903 12:45:08.185769 1 main.go:332] Loading configuration. │ I0903 12:45:08.186698 1 main.go:358] Updating config with default resource matching patterns. │ I0903 12:45:08.186905 1 main.go:369] │ Running with config: │ { │ "version": "v1", │ "flags": { │ "migStrategy": "none", │ "failOnInitError": true, │ "mpsRoot": "", │ "nvidiaDriverRoot": "/", │ "nvidiaDevRoot": "/", │ "gdrcopyEnabled": true, │ "gdsEnabled": true, │ "mofedEnabled": true, │ "useNodeFeatureAPI": null, │ "deviceDiscoveryStrategy": "auto", │ "plugin": { │ "passDeviceSpecs": false, │ "deviceListStrategy": [ │ "envvar" │ ], │ "deviceIDStrategy": "uuid", │ "cdiAnnotationPrefix": "cdi.k8s.io/", │ "nvidiaCTKPath": "/usr/bin/nvidia-ctk", │ "containerDriverRoot": "/driver-root" │ } │ }, │ │ "resources": { │ "gpus": [ │ { │ "pattern": "*", │ "name": "nvidia.com/gpu" │ } │ ] │ }, │ "sharing": { │ "timeSlicing": {} │ }, │ "imex": {} │ } │ I0903 12:45:08.186915 1 main.go:372] Retrieving plugins. │ E0903 12:45:08.198236 1 factory.go:94] Failed to initialize NVML: Driver Not Loaded. │ E0903 12:45:08.198248 1 factory.go:95] If this is a GPU node, did you set the docker default runtime to nvidia? │ E0903 12:45:08.198252 1 factory.go:96] You can check the prerequisites at: https://github.com/NVIDIA/k8s-device-plugin#prerequisites │ E0903 12:45:08.198256 1 factory.go:97] You can learn how to set the runtime at: https://github.com/NVIDIA/k8s-device-plugin#quick-start │ │ E0903 12:45:08.198260 1 factory.go:98] If this is not a GPU node, you should set up a toleration or nodeSelector to only deploy this plugin on GPU nodes │ │ E0903 12:45:08.198507 1 main.go:188] error starting plugins: error getting plugins: unable to create plugins: failed to construct resource managers: nvml init failed: Driver Not Loaded

│ Autoscroll:On ColumnLock:Off FullScreen:Off Timestamps:Off Wrap:On │ │ Nyare Talos-struktur (1.13+) detekterad. │ Söker brett efter NVML-biblioteket i system- och app-kataloger... │ Hittade äntligen NVIDIA-bibliotek på hosten under: /talos-usr/local/lib │ 2026/09/03 12:45:05 maxprocs: Leaving GOMAXPROCS=20: CPU quota undefined │ time="2026-09-03T12:45:05Z" level=info msg="Starting dcgm-exporter" │ Error: Failed to initialize NVML │ time="2026-09-03T12:45:05Z" level=error msg="Encountered a failure." stacktrace="goroutine 1 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x64\ngithub.qkg1.top/NVIDIA/dcgm-exporter/pkg/cmd.action.func1.1()\n\t/go/src/github. │ │ com/NVIDIA/dcgm-exporter/pkg/cmd/app.go:269 +0x3c\npanic({0x16ef400?, 0x27e9390?})\n\t/usr/local/go/src/runtime/panic.go:914 +0x218\ngithub.qkg1.top/NVIDIA/dcgm-exporter/pkg/cmd.initDCGM(0x4000602b60)\n\t/go/src/github.com/NVIDIA/dcgm-exporter/pkg/cmd/app.go:50 │ │ 9 +0x8c\ngithub.qkg1.top/NVIDIA/dcgm-exporter/pkg/cmd.startDCGMExporter(0x400051da88?, 0x40006a7650)\n\t/go/src/github.com/NVIDIA/dcgm-exporter/pkg/cmd/app.go:289 +0x80\ngithub.qkg1.top/NVIDIA/dcgm-exporter/pkg/cmd.action.func1()\n\t/go/src/github.com/NVIDIA/dcgm-ex │ │ porter/pkg/cmd/app.go:273 +0x58\ngithub.qkg1.top/NVIDIA/dcgm-exporter/pkg/stdout.Capture({0x1bd0fd0?, 0x400043dea0}, 0x400051db68)\n\t/go/src/github.com/NVIDIA/dcgm-exporter/pkg/stdout/capture.go:77 +0x1c4\ngithub.qkg1.top/NVIDIA/dcgm-exporter/pkg/cmd.action(0x40001 │ │ 291c0)\n\t/go/src/github.com/NVIDIA/dcgm-exporter/pkg/cmd/app.go:264 +0x64\ngithub.qkg1.top/NVIDIA/dcgm-exporter/pkg/cmd.NewApp.func1(0x40006af1e0?)\n\t/go/src/github.com/NVIDIA/dcgm-exporter/pkg/cmd/app.go:249 +0x1c\ngithub.qkg1.top/urfave/cli/v2.(*Command).Run(0x4 │ │ 0006af1e0, 0x40001291c0, {0x400003c0a0, 0x1, 0x1})\n\t/go/pkg/mod/github.com/urfave/cli/v2@v2.27.1/command.go:279 +0x754\ngithub.qkg1.top/urfave/cli/v2.(*App).RunContext(0x40001d5000, {0x1bd0e80?, 0x28b5420}, {0x400003c0a0, 0x1, 0x1})\n\t/go/pkg/mod/github.com/ │ │ urfave/cli/v2@v2.27.1/app.go:337 +0x534\ngithub.qkg1.top/urfave/cli/v2.(*App).Run(0x400051df18?, {0x400003c0a0?, 0x1?, 0x0?})\n\t/go/pkg/mod/github.com/urfave/cli/v2@v2.27.1/app.go:311 +0x3c\nmain.main()\n\t/go/src/github.com/NVIDIA/dcgm-exporter/cmd/dcgm-expor │ │ ter/main.go:35 +0x60\n"

Any suggestions on how to fix this ?

[smira]

Please format your message properly.

Probably you forgot to load the modules (?), but it's impossible to say.

https://docs.siderolabs.com/talos/v1.12/configure-your-talos-cluster/hardware-and-drivers/nvidia-gpu

[rothgar]

For now, the GB10 devices need to use NVIDIA proprietary drivers

[RA-JfA]

Thanks @rothgar
Do you have any tips or hints on how to fix the machineconfig to load the modules ?

[rothgar]

It's really hard to tell what your machine config is because the formatting isn't correct. I would follow the documentation which has tips for arm64 nvidia devices https://docs.siderolabs.com/talos/v1.14/configure-your-talos-cluster/hardware-and-drivers/nvidia-gpu-proprietary#nvidia-gpu-proprietary-drivers

[RA-JfA]

Thanks. I have seen that page.
Also found this: #13019, i had the same output on my screen

[RA-JfA]

I added arm64.nobti to the kernel commandline and that seems to have solved the issue with kernel module loading