Memory exhaustion and GPU passthrough crash the system on instance start

[neitsab]

Trying for the first time to start a VM with a GPU passed through resulted in the complete lock up of the server. I had to go and hard power it, it wouldn't even respond to pressing the power button anymore.

Instance config and info:

incus config show gpu --expanded 
architecture: x86_64
config:
  image.architecture: amd64
  image.description: Ubuntu resolute amd64 (20260529_07:42)
  image.os: Ubuntu
  image.release: resolute
  image.serial: "20260529_07:42"
  image.type: disk-kvm.img
  image.variant: cloud
  limits.cpu: "12"
  limits.memory: 32GiB
  volatile.base_image: e35808ce3430b24cd80d424135bef9431f053588ef60dbf7468f500d2090f5c1
  volatile.cloud-init.instance-id: dcda8634-1507-432b-ae2b-e9511fcf9db8
  volatile.eth0.host_name: tap1fd63d0b
  volatile.eth0.hwaddr: 10:66:6a:a3:e4:29
  volatile.nvidia.last_state.pci.driver: nouveau
  volatile.nvidia.last_state.pci.slot.name: "0000:01:00.0"
  volatile.uuid: 626b4612-c9bc-44f5-b9af-ad4467317741
  volatile.uuid.generation: 626b4612-c9bc-44f5-b9af-ad4467317741
  volatile.vm.needs_reset: "true"
  volatile.vm.rtc_adjustment: "0"
  volatile.vm.rtc_offset: "0"
  volatile.vsock_id: "4066275680"
devices:
  eth0:
    name: eth0
    network: incusbr0
    type: nic
  nvidia:
    gputype: physical
    pci: "0000:01:00.0"
    type: gpu
  root:
    path: /
    pool: local
    size: 50GiB
    type: disk
ephemeral: false
profiles:
  - default
stateful: false
description: Ubuntu 26.04 with GPU passthrough

% incus info --show-log gpu              
Name: gpu
Description: Ubuntu 26.04 with GPU passthrough
Status: STOPPED
Type: virtual-machine
Architecture: x86_64
Created: 2026/05/30 17:33 CEST
Last Used: 1970/01/01 01:00 CET

Snapshots:
+------------+-----------------------+------------+----------+
|    Name    |       Taken at        | Expires at | Stateful |
+------------+-----------------------+------------+----------+
| pre-launch | 2026/05/30 17:39 CEST |            | NO       |
+------------+-----------------------+------------+----------+

Log (qemu.log):

Here's the beginning of relevant system log:

[2026/06/02 10:04:39 CEST] incus-osd: 2026-06-02 10:04:39 INFO Skipping update check outside of maintenance window(s)
[2026/06/02 11:42:13 CEST] 55-scsi-sg3_id.rules: WARNING: SCSI device zd0 has no device ID, consider changing .SCSI_ID_SERIAL_SRC in 00-scsi-sg3_config.rules
[2026/06/02 11:42:14 CEST] systemd-networkd: tap1fd63d0b: Link UP
[2026/06/02 11:42:14 CEST] kernel: incusbr0: port 1(tap1fd63d0b) entered blocking state
[2026/06/02 11:42:14 CEST] kernel: incusbr0: port 1(tap1fd63d0b) entered disabled state
[2026/06/02 11:42:14 CEST] kernel: tap1fd63d0b: entered allmulticast mode
[2026/06/02 11:42:14 CEST] kernel: tap1fd63d0b: entered promiscuous mode
[2026/06/02 11:42:14 CEST] kernel: kauditd_printk_skb: 2 callbacks suppressed
[2026/06/02 11:42:14 CEST] kernel: audit: type=1700 audit(1780393334.305:490): dev=tap1fd63d0b prom=256 old_prom=0 auid=4294967295 uid=0 gid=0 ses=4294967295
[2026/06/02 11:42:14 CEST] kernel: audit: type=1300 audit(1780393334.305:490): arch=c000003e syscall=44 success=yes exit=40 a0=22 a1=64095a0a4e0 a2=28 a3=0 items=0 ppid=1 pid=1597 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="incusd" exe="/opt/incus/bin/incusd" subj=unconfined key=(null)
[2026/06/02 11:42:14 CEST] kernel: audit: type=1327 audit(1780393334.305:490): proctitle=696E63757364002D2D67726F757000696E6375732D61646D696E002D2D6C6F6766696C65
[2026/06/02 11:42:15 CEST] kernel: nouveau 0000:01:00.0: fb: VPR locked, but no scrubber binary!
[2026/06/02 11:42:15 CEST] kernel: vga_switcheroo: disabled
[2026/06/02 11:42:15 CEST] kernel: audit: type=1400 audit(1780393335.547:491): apparmor="STATUS" operation="profile_load" profile="unconfined" name="incus-gpu_</var/lib/incus>" pid=22599 comm="apparmor_parser"
[2026/06/02 11:42:15 CEST] kernel: audit: type=1300 audit(1780393335.547:491): arch=c000003e syscall=1 success=yes exit=315065 a0=6 a1=55605c460b10 a2=4ceb9 a3=0 items=0 ppid=22598 pid=22599 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="apparmor_parser" exe="/usr/sbin/apparmor_parser" subj=unconfined key=(null)
[2026/06/02 11:42:15 CEST] kernel: audit: type=1327 audit(1780393335.547:491): proctitle=61707061726D6F725F706172736572002D72574C002F7661722F6C69622F696E6375732F73656375726974792F61707061726D6F722F6361636865002F7661722F6C69622F696E6375732F73656375726974792F61707061726D6F722F70726F66696C65732F696E6375732D677075
[2026/06/02 11:42:32 CEST] kernel: Purging GPU memory, 0 pages freed, 0 pages still pinned, 1 pages left available.
[2026/06/02 11:42:34 CEST] systemd-journald: Under memory pressure, flushing caches.
[2026/06/02 11:42:35 CEST] kernel: Purging GPU memory, 0 pages freed, 0 pages still pinned, 1 pages left available.
[2026/06/02 11:42:35 CEST] kernel: Purging GPU memory, 0 pages freed, 0 pages still pinned, 1 pages left available.
[2026/06/02 11:42:35 CEST] kernel: Purging GPU memory, 0 pages freed, 0 pages still pinned, 1 pages left available.
[2026/06/02 11:42:35 CEST] kernel: Purging GPU memory, 0 pages freed, 0 pages still pinned, 1 pages left available.
[2026/06/02 11:42:35 CEST] kernel: Purging GPU memory, 0 pages freed, 0 pages still pinned, 1 pages left available.

This message repeats hundreds of time every second for the next hour, so here's the filtered output without it:

[2026/06/02 11:42:44 CEST] systemd-journald: Under memory pressure, flushing caches.
[2026/06/02 11:43:06 CEST] systemd-journald: Missed 178 kernel messages
[2026/06/02 11:43:52 CEST] systemd-journald: Missed 8632 kernel messages
[2026/06/02 11:44:13 CEST] systemd-journald: /dev/kmsg buffer overrun, some messages lost.
[2026/06/02 11:45:30 CEST] systemd-journald: Journal started
[2026/06/02 11:45:30 CEST] systemd-journald: System Journal (/var/log/journal/79041ad5454a4f1196997f29ef36d275) is 215.2M, max 2.4G, 2.2G free.
[2026/06/02 11:45:30 CEST] systemd-journald: Missed 2880 kernel messages
[2026/06/02 11:45:30 CEST] systemd: systemd-journald.service: Failed with result 'watchdog'.
[2026/06/02 11:45:30 CEST] kernel: audit: type=1131 audit(1780393511.907:493): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=systemd-journald comm="systemd" exe="/usr/l
ib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
[2026/06/02 11:45:30 CEST] kernel: audit: type=1334 audit(1780393511.974:494): prog-id=61 op=UNLOAD
[2026/06/02 11:45:30 CEST] systemd: systemd-journald.service: Consumed 42.889s CPU time, 16.6M memory peak, 1.2M memory swap peak.
[2026/06/02 11:45:30 CEST] kernel: tpm tpm0: invalid TPM_STS.x 0xa1, dumping stack for forensics
[2026/06/02 11:45:30 CEST] kernel: CPU: 11 UID: 0 PID: 127 Comm: hwrng Tainted: P           O        7.0.10-zabbly+ #debian13 PREEMPT(full) 
[2026/06/02 11:45:30 CEST] kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE
[2026/06/02 11:45:30 CEST] kernel: Hardware name: Dell Inc. XPS 15 9500/0RHXRG, BIOS 1.28.0 03/06/2024
[2026/06/02 11:45:30 CEST] kernel: Call Trace:
[2026/06/02 11:45:30 CEST] kernel:  <TASK>
[2026/06/02 11:45:30 CEST] kernel:  dump_stack_lvl+0x5f/0x90
[2026/06/02 11:45:30 CEST] kernel:  dump_stack+0x10/0x18
[2026/06/02 11:45:30 CEST] kernel:  tpm_tis_status.cold+0x18/0x1d
[2026/06/02 11:45:30 CEST] kernel:  tpm_tis_send_data+0x43/0x2d0
[2026/06/02 11:45:30 CEST] kernel:  tpm_tis_send_main+0x77/0x180
[2026/06/02 11:45:30 CEST] kernel:  tpm_tis_send+0x8e/0x110
[2026/06/02 11:45:30 CEST] kernel:  tpm_transmit+0xcc/0x310
[2026/06/02 11:45:30 CEST] kernel:  tpm_transmit_cmd+0x2f/0xc0
[2026/06/02 11:45:30 CEST] kernel:  tpm2_get_random+0x12b/0x260
[2026/06/02 11:45:30 CEST] kernel:  tpm_get_random+0x75/0x90
[2026/06/02 11:45:30 CEST] kernel:  tpm_hwrng_read+0x15/0x30
[2026/06/02 11:45:30 CEST] kernel:  hwrng_fillfn+0x7c/0x3f0
[2026/06/02 11:45:30 CEST] kernel:  ? __pfx_hwrng_fillfn+0x10/0x10
[2026/06/02 11:45:30 CEST] kernel:  kthread+0xf7/0x130
[2026/06/02 11:45:30 CEST] kernel:  ? __pfx_kthread+0x10/0x10
[2026/06/02 11:45:30 CEST] kernel:  ret_from_fork+0x2e8/0x370
[2026/06/02 11:45:30 CEST] kernel:  ? __pfx_kthread+0x10/0x10
[2026/06/02 11:45:30 CEST] kernel:  ret_from_fork_asm+0x1a/0x30
[2026/06/02 11:45:30 CEST] kernel:  </TASK>
[2026/06/02 11:45:30 CEST] systemd: systemd-journald.service: Scheduled restart job, restart counter is at 2.
[2026/06/02 11:45:30 CEST] kernel: audit: type=1334 audit(1780393524.764:495): prog-id=63 op=UNLOAD
[2026/06/02 11:45:30 CEST] kernel: audit: type=1334 audit(1780393524.764:496): prog-id=62 op=UNLOAD
[2026/06/02 11:45:30 CEST] kernel: audit: type=1334 audit(1780393527.911:497): prog-id=83 op=LOAD
[2026/06/02 11:45:30 CEST] kernel: audit: type=1334 audit(1780393527.911:498): prog-id=84 op=LOAD
[2026/06/02 11:45:30 CEST] kernel: audit: type=1334 audit(1780393527.911:499): prog-id=85 op=LOAD
[2026/06/02 11:45:30 CEST] systemd: Starting systemd-journald.service - Journal Service...
[2026/06/02 11:45:30 CEST] systemd-journald: Collecting audit messages is disabled.
[2026/06/02 11:45:30 CEST] systemd-journald: File /var/log/journal/79041ad5454a4f1196997f29ef36d275/system.journal corrupted or uncleanly shut down, renaming and replacing.
[2026/06/02 11:45:30 CEST] systemd: Started systemd-journald.service - Journal Service.
[2026/06/02 11:45:30 CEST] systemd: Started systemd-journald.service - Journal Service.
[2026/06/02 11:45:30 CEST] kernel: audit: type=1130 audit(1780393530.980:500): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=systemd-journald comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[2026/06/02 11:45:30 CEST] systemd: systemd-journald.service: Watchdog timeout (limit 3min)!
[2026/06/02 11:45:30 CEST] systemd: systemd-journald.service: Killing process 488 (systemd-journal) with signal SIGABRT.
[2026/06/02 11:45:32 CEST] systemd-journald: Under memory pressure, flushing caches.
[2026/06/02 11:45:49 CEST] systemd-journald: Under memory pressure, flushing caches.
[2026/06/02 11:45:52 CEST] systemd-journald: Under memory pressure, flushing caches.
...
[2026/06/02 12:05:21 CEST] systemd: systemd-logind.service: Failed with result 'timeout'.
[2026/06/02 12:05:21 CEST] systemd: Failed to start systemd-logind.service - User Login Management.
[2026/06/02 12:05:21 CEST] kernel: kauditd_printk_skb: 3 callbacks suppressed
[2026/06/02 12:05:21 CEST] kernel: audit: type=1130 audit(1780394699.175:582): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=systemd-logind comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
[2026/06/02 12:05:21 CEST] systemd: systemd-logind.service: Consumed 27.515s CPU time, 4.4M memory peak, 1.1M memory swap peak.
[2026/06/02 12:05:21 CEST] systemd: systemd-logind.service: Scheduled restart job, restart counter is at 2.
[2026/06/02 12:05:21 CEST] systemd-journald: /dev/kmsg buffer overrun, some messages lost.
[2026/06/02 12:05:21 CEST] systemd: Starting modprobe@drm.service - Load Kernel Module drm...
[2026/06/02 12:05:21 CEST] systemd: Started systemd-timesyncd.service - Network Time Synchronization.
[2026/06/02 12:05:21 CEST] kernel: audit: type=1130 audit(1780394705.832:583): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=systemd-timesyncd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[2026/06/02 12:05:55 CEST] systemd-journald: Missed 1415 kernel messages
[2026/06/02 12:06:18 CEST] systemd-journald: Journal stopped
[2026/06/02 12:09:23 CEST] systemd-journald: Journal started
[2026/06/02 12:09:23 CEST] systemd-journald: System Journal (/var/log/journal/79041ad5454a4f1196997f29ef36d275) is 231.3M, max 2.4G, 2.2G free.
[2026/06/02 12:09:23 CEST] systemd-journald: Missed 14721 kernel messages
[2026/06/02 12:09:34 CEST] systemd-journald: Collecting audit messages is disabled.
[2026/06/02 12:09:34 CEST] systemd: systemd-resolved.service: State 'stop-sigterm' timed out. Killing.
[2026/06/02 12:09:34 CEST] systemd: systemd-resolved.service: Failed with result 'timeout'.
[2026/06/02 12:09:34 CEST] systemd: Failed to start systemd-resolved.service - Network Name Resolution.
[2026/06/02 12:09:34 CEST] kernel: audit: type=1130 audit(1780394968.173:615): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=systemd-resolved comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
[2026/06/02 12:09:34 CEST] systemd: systemd-resolved.service: Consumed 14.243s CPU time, 6.8M memory peak, 72K memory swap peak.
[2026/06/02 12:09:34 CEST] systemd: modprobe@drm.service: Deactivated successfully.
[2026/06/02 12:09:34 CEST] systemd: Finished modprobe@drm.service - Load Kernel Module drm.
[2026/06/02 12:09:34 CEST] kernel: audit: type=1130 audit(1780394968.175:616): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=modprobe@drm comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[2026/06/02 12:09:34 CEST] kernel: audit: type=1131 audit(1780394968.175:617): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=modprobe@drm comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[2026/06/02 12:09:34 CEST] systemd: modprobe@drm.service: Consumed 5.758s CPU time, 6.1M memory peak, 820K memory swap peak.
[2026/06/02 12:09:34 CEST] systemd: systemd-networkd.service: start operation timed out. Terminating.
[2026/06/02 12:09:34 CEST] systemd: systemd-journald.service: start operation timed out. Terminating.
[2026/06/02 12:09:34 CEST] systemd-journald: Received SIGTERM from PID 1 (systemd).
[2026/06/02 12:09:34 CEST] kernel: audit: type=1334 audit(1780394968.183:618): prog-id=126 op=UNLOAD
[2026/06/02 12:09:34 CEST] systemd: systemd-resolved.service: Scheduled restart job, restart counter is at 5.
[2026/06/02 12:09:34 CEST] kernel: audit: type=1334 audit(1780394973.242:619): prog-id=131 op=LOAD
[2026/06/02 12:09:34 CEST] kernel: audit: type=1334 audit(1780394973.248:620): prog-id=132 op=LOAD
[2026/06/02 12:09:34 CEST] kernel: audit: type=1334 audit(1780394974.893:621): prog-id=133 op=LOAD
[2026/06/02 12:09:34 CEST] dbus-daemon: [system] Connection has not authenticated soon enough, closing it (auth_timeout=30000ms, elapsed: 83836ms)
[2026/06/02 12:09:34 CEST] systemd-journald: Under memory pressure, flushing caches.
[2026/06/02 12:09:34 CEST] systemd: systemd-journald.service: Watchdog timeout (limit 3min)!
[2026/06/02 12:09:34 CEST] dbus-daemon: [system] Connection has not authenticated soon enough, closing it (auth_timeout=30000ms, elapsed: 30042ms)
[2026/06/02 12:09:34 CEST] systemd: systemd-journald.service: Killing process 22717 (systemd-journal) with signal SIGABRT.
...
[2026/06/02 12:49:18 CEST] systemd-journald: Journal stopped

I can attach the full log if required.

About the system here's the abbreviated incus info, tell me if you need more:

% incus info | grep kernel_version -A5
  kernel_version: 7.0.10-zabbly+
  lxc_features: {}
  os_name: IncusOS
  os_version: "202605311846"
  project: immich
  server: incus

I have the gpu-support application enabled:

% incus admin os application show gpu-support
WARNING: The IncusOS API and configuration is subject to change

config: {}
state:
  available_versions:
    - "202605281810"
    - "202605311846"
  friendly_version: 20260519 [202605311846]
  initialized: true
  version: "202605311846"

Hardware-wise:

System:
  UUID: ...
  Vendor: Dell Inc.
  Product: XPS 15 9500
  Family: XPS
  SKU: 097D
  Serial number: ...
  Type: physical
  Chassis:
      Vendor: Dell Inc.
      Type: Notebook
      Serial: ...
  Motherboard:
      Vendor: Dell Inc.
      Product: 0RHXRG
      Serial: ...
      Version: A03
  Firmware:
      Vendor: Dell Inc.
      Version: 1.28.0
      Date: 03/06/2024

Load:
  Processes: 407
  Average: 0.01 0.03 0.01

CPU:
  Architecture: x86_64
  Vendor: GenuineIntel
  Name: Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz
  Caches:
    - Level 1 (type: Data): 32KiB
    - Level 1 (type: Instruction): 32KiB
    - Level 2 (type: Unified): 256KiB
    - Level 3 (type: Unified): 12MiB
<snip>

Memory:
  Free: 26.63GiB
  Used: 5.25GiB
  Total: 31.88GiB

GPUs:
  Card 0:
    NUMA node: 0
    Vendor: Intel Corporation (8086)
    Product: CometLake-H GT2 [UHD Graphics] (9bc4)
    PCI address: 0000:00:02.0
    Driver: i915 (7.0.10-zabbly+)
    DRM:
      ID: 0
      Card: card0 (226:0)
      Control: controlD64 (226:0)
      Render: renderD128 (226:128)
  Card 1:
    NUMA node: 0
    Vendor: NVIDIA Corporation (10de)
    Product: TU117M [GeForce GTX 1650 Ti Mobile] (1f95)
    PCI address: 0000:01:00.0
    Driver: nouveau (7.0.10-zabbly+)
    DRM:
      ID: 1
      Card: card1 (226:1)
      Control: controlD65 (226:1)
      Render: renderD129 (226:129)

My wild guesses:

• should I blacklist nouveau before passing through the Nvidia GPU?
• I overcommitted the instance RAM (32 GB vs 31.88GiB available) and this made the thing explode? Although the repeating kernel message pertains to the GPU, so...

What can I do next to troubleshoot this?

Thanks for your help!

[neitsab]

I forgot to mention: while IncusOS was unreachable via the API or Tailscale, it kept answering to ping and a port scan on 8443 showed that it was open.

[neitsab]

Ok, further testing confirmed it was the memory allocation that brought upon the problem, not the GPU. When I dialed it back to 12 GiB the VM booted without issues.

I had quickly looked in the docs before setting up this VM to see whether assigning more memory than the host has was problematic, but it hadn't turned up anything. What's the rule about memory overcommitment on Incus? I believe QEMU authorizes it, but e.g. Proxmox doesn't allow a single VM to be assigned more than the host's total.

If this is not an isolated incident, maybe a warning could be displayed or the configuration fail during validation?

[neitsab]

Also FWIW the webUI displays somewhat contradictory info regarding memory usage:

[stgraber]

stgraber@castiana:~$ free -m
               total        used        free      shared  buff/cache   available
Mem:           31397       17514       10941         289        3700       13883
Swap:          36223       14491       21732
stgraber@castiana:~$ incus create --empty --vm test-memory -c limits.memory=64GiB -c limits.cpu=4
Creating test-memory
stgraber@castiana:~$ incus start test-memory
stgraber@castiana:~$ free -m
               total        used        free      shared  buff/cache   available
Mem:           31397       17704       10682         339        3819       13693
Swap:          36223       14489       21734
stgraber@castiana:~$ 

So memory overcommit should generally work just fine, until you actually use the memory that is ;)

[stgraber]

should I blacklist nouveau before passing through the Nvidia GPU?

Yes, you really really should as otherwise the host system may take it very badly to have its GPU yanked out of the PCIe bus, leading to a full kernel panic/lockup.

[stgraber]

Ok, further testing confirmed it was the memory allocation that brought upon the problem, not the GPU. When I dialed it back to 12 GiB the VM booted without issues.

I had quickly looked in the docs before setting up this VM to see whether assigning more memory than the host has was problematic, but it hadn't turned up anything. What's the rule about memory overcommitment on Incus? I believe QEMU authorizes it, but e.g. Proxmox doesn't allow a single VM to be assigned more than the host's total.

If this is not an isolated incident, maybe a warning could be displayed or the configuration fail during validation?

So that's interesting and likely somewhat specific to this system then given that I've been unable to reproduce the issue here.

Any chance you can try the memory overcommit without a GPU?
I wonder if it's the combination of the two that caused some issues here.

[neitsab]

Yes, you really really should as otherwise the host system may take it very badly to have its GPU yanked out of the PCIe bus, leading to a full kernel panic/lockup.

Good to know this hunch was correct, thanks! Something to be queued for addition in the docs maybe?

[neitsab]

Any chance you can try the memory overcommit without a GPU?

Indeed, without the GPU even with 32 GB the VM boots fine:

% incus config show gpu --expanded  
zsh: correct 'config' to '.config' [nyae]? n
architecture: x86_64
config:
  cloud-init.vendor-data: |-
    #cloud-config

    user:
      name: cloud

    keyboard:
      layout: fr
    locale: fr_FR.UTF-8
    timezone: Europe/Paris

    ssh_genkeytypes: [ed25519]
    ssh_quiet_keygen: true

    packages:
      - micro
      - tree
      - less
  image.architecture: amd64
  image.description: Ubuntu resolute amd64 (20260529_07:42)
  image.os: Ubuntu
  image.release: resolute
  image.serial: "20260529_07:42"
  image.type: disk-kvm.img
  image.variant: cloud
  limits.cpu: "12"
  limits.memory: 32GiB
  volatile.apply_template: create
  volatile.base_image: e35808ce3430b24cd80d424135bef9431f053588ef60dbf7468f500d2090f5c1
  volatile.cloud-init.instance-id: dcda8634-1507-432b-ae2b-e9511fcf9db8
  volatile.eth0.hwaddr: 10:66:6a:a3:e4:29
  volatile.uuid: 626b4612-c9bc-44f5-b9af-ad4467317741
  volatile.uuid.generation: 83419daf-1d29-43c1-91e5-a275550f2013
devices:
  eth0:
    name: eth0
    network: incusbr0
    type: nic
  root:
    path: /
    pool: local
    size: 50GiB
    type: disk
ephemeral: false
profiles:
  - default
  - ci-default
stateful: false
description: Ubuntu 26.04 with GPU passthrough

% incus info gpu                   
Name: gpu
Description: Ubuntu 26.04 with GPU passthrough
Status: RUNNING
Type: virtual-machine
Architecture: x86_64
PID: 4484
Created: 2026/05/30 17:33 CEST
Last Used: 2026/06/02 18:27 CEST
Started: 2026/06/02 18:27 CEST

Operating System:
  OS: Ubuntu
  OS Version: 26.04
  Kernel Version: 7.0.0-22-generic
  Hostname: gpu
  FQDN: localhost

Resources:
  Processes: 218
  Disk usage:
    root: 104.36MiB
  CPU usage:
    CPU usage (in seconds): 15
  Memory usage:
    Memory (current): 774.44MiB
  Network usage:
    enp5s0:
      Type: broadcast
      State: UP
      Host interface: tap3bfafea9
      MAC address: 10:66:6a:a3:e4:29
      MTU: 1500
      Bytes received: 6.69MB
      Bytes sent: 86.42kB
      Packets received: 1107
      Packets sent: 1065
      IP addresses:
        inet:  10.162.207.110/24 (global)
        inet6: fd42:8606:8645:5df1:1266:6aff:fea3:e429/64 (global)
        inet6: fe80::1266:6aff:fea3:e429/64 (link)
    lo:
      Type: loopback
      State: UP
      MTU: 65536
      Bytes received: 10.93kB
      Bytes sent: 10.93kB
      Packets received: 122
      Packets sent: 122
      IP addresses:
        inet:  127.0.0.1/8 (local)
        inet6: ::1/128 (local)

Snapshots:
+------------+-----------------------+------------+----------+
|    Name    |       Taken at        | Expires at | Stateful |
+------------+-----------------------+------------+----------+
| pre-launch | 2026/05/30 17:39 CEST |            | NO       |
+------------+-----------------------+------------+----------+

I don't know what to name this issue anymore 😅

[stgraber]

Good to know this hunch was correct, thanks! Something to be queued for addition in the docs maybe?

Yeah, we probably could do with a howto on GPU passthrough.
You can do the module blacklisting through the kernel endpoint. Full blacklisting will require a system reboot though.

Can you try black listing the nouveau driver, reboot the system and see if you still get the same behavior?

[neitsab]

Sorry I don't know which format to pass to incus admin os system kernel edit, I tried

config:
  - blacklist_modules: [nouveau]
state: {}

but get

Config parsing error: json: cannot unmarshal array into Go struct field SystemKernel.config of type api.SystemKernelConfig

I don't think I have edited a JSON struct on IncusOS yet... Should I something along the lines of -d {blacklist_modules: [nouveau]}?

[neitsab]

I'm looking through #754 and #968 but don't really see the correct format described 😅

[neitsab]

All right, I got it with

config:
  blacklist_modules:
    - nouveau

Currently rebooting...

[neitsab]

Hmm, the machine booted and looks fine, but I can't access it via the API anymore...

Error: websocket: close 1006 (abnormal closure): unexpected EOF

I'll have to troubleshoot this further tonight.

[neitsab]

Okay, it was not accessible because it had crashed again... So blacklisting nouveau didn't change anything: when I start the VM with both 32 GiB and a GPU passthrough, it locks up.

% incus admin os debug log -b -1 | grep nouveau | wc
WARNING: The IncusOS API and configuration is subject to change

      0       0       0

% incus admin os system kernel show
WARNING: The IncusOS API and configuration is subject to change

config:
  blacklist_modules:
    - nouveau
state: {}

[2026/06/03 01:30:18 CEST] incus-osd: 2026-06-03 01:30:18 INFO System is ready version=202605311846
[2026/06/03 01:30:28 CEST] systemd: systemd-timedated.service: Deactivated successfully.
[2026/06/03 01:30:28 CEST] kernel: kauditd_printk_skb: 15 callbacks suppressed
[2026/06/03 01:30:28 CEST] kernel: audit: type=1131 audit(1780443028.584:467): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=systemd-timedated comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[2026/06/03 01:30:28 CEST] kernel: audit: type=1334 audit(1780443028.597:468): prog-id=77 op=UNLOAD
[2026/06/03 01:30:28 CEST] kernel: audit: type=1334 audit(1780443028.597:469): prog-id=76 op=UNLOAD
[2026/06/03 01:30:28 CEST] kernel: audit: type=1334 audit(1780443028.597:470): prog-id=75 op=UNLOAD
[2026/06/03 01:30:28 CEST] systemd: systemd-hostnamed.service: Deactivated successfully.
[2026/06/03 01:30:28 CEST] kernel: audit: type=1131 audit(1780443028.633:471): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[2026/06/03 01:30:28 CEST] kernel: audit: type=1334 audit(1780443028.675:472): prog-id=66 op=UNLOAD
[2026/06/03 01:35:13 CEST] systemd-networkd: tap326e8b4a: Link UP
[2026/06/03 01:35:13 CEST] kernel: incusbr0: port 1(tap326e8b4a) entered blocking state
[2026/06/03 01:35:13 CEST] kernel: incusbr0: port 1(tap326e8b4a) entered disabled state
[2026/06/03 01:35:13 CEST] kernel: tap326e8b4a: entered allmulticast mode
[2026/06/03 01:35:13 CEST] kernel: tap326e8b4a: entered promiscuous mode
[2026/06/03 01:35:13 CEST] kernel: audit: type=1700 audit(1780443313.269:473): dev=tap326e8b4a prom=256 old_prom=0 auid=4294967295 uid=0 gid=0 ses=4294967295
[2026/06/03 01:35:13 CEST] kernel: audit: type=1300 audit(1780443313.269:473): arch=c000003e syscall=44 success=yes exit=40 a0=20 a1=1f173463f440 a2=28 a3=0 items=0 ppid=1 pid=1560 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="incusd" exe="/opt/incus/bin/incusd" subj=unconfined key=(null)
[2026/06/03 01:35:13 CEST] kernel: audit: type=1327 audit(1780443313.269:473): proctitle=696E63757364002D2D67726F757000696E6375732D61646D696E002D2D6C6F6766696C65
[2026/06/03 01:35:13 CEST] kernel: audit: type=1400 audit(1780443313.337:474): apparmor="STATUS" operation="profile_load" profile="unconfined" name="incus-gpu_</var/lib/incus>" pid=2695 comm="apparmor_parser"
[2026/06/03 01:35:13 CEST] kernel: audit: type=1300 audit(1780443313.337:474): arch=c000003e syscall=1 success=yes exit=315065 a0=6 a1=7fbc4ff22010 a2=4ceb9 a3=0 items=0 ppid=2694 pid=2695 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="apparmor_parser" exe="/usr/sbin/apparmor_parser" subj=unconfined key=(null)
[2026/06/03 01:35:13 CEST] kernel: audit: type=1327 audit(1780443313.337:474): proctitle=61707061726D6F725F706172736572002D72574C002F7661722F6C69622F696E6375732F73656375726974792F61707061726D6F722F6361636865002F7661722F6C69622F696E6375732F73656375726974792F61707061726D6F722F70726F66696C65732F696E6375732D677075
[2026/06/03 01:35:28 CEST] kernel: Purging GPU memory, 0 pages freed, 0 pages still pinned, 1 pages left available.
[2026/06/03 01:35:28 CEST] kernel: Purging GPU memory, 0 pages freed, 0 pages still pinned, 1 pages left available.
[2026/06/03 01:35:28 CEST] kernel: Purging GPU memory, 0 pages freed, 0 pages still pinned, 1 pages left available.
[2026/06/03 01:35:28 CEST] kernel: Purging GPU memory, 0 pages freed, 0 pages still pinned, 1 pages left available.
[2026/06/03 01:35:28 CEST] kernel: Purging GPU memory, 0 pages freed, 0 pages still pinned, 1 pages left available.
...