Skip to content

Fix bug in qubesvm.py incorrectly treating "False" config values as True#829

Merged
marmarek merged 1 commit into
QubesOS:mainfrom
andersonarc:main
Jun 18, 2026
Merged

Fix bug in qubesvm.py incorrectly treating "False" config values as True#829
marmarek merged 1 commit into
QubesOS:mainfrom
andersonarc:main

Conversation

@andersonarc

Copy link
Copy Markdown
Contributor

The code in qubesvm.py on lines ~1128-1150 incorrectly treats configuration values which are set to "False" in the XML. The "True" case is explicitly handled correctly,

if value == "True"
    value = True

but the "False" case falls through to

self.volume_config[name][key] = value

and since bool("False") == True, this will result in incorrect behavior. I discovered this when analyzing suspiciously no-op behavior of rw="False" in the qubes.xml configuration. The fix is simple, but it might have unintended side effects if someone relies on the buggy behavior. It is, of course, up to your discretion whether this is a problem.

Specifically, the behavioral changes are:

Volume snap_on_start save_on_stop rw Before After Confirmed?
root (AppVM/DispVM) True False False Uses dm-snapshot root-cow.img Uses /dev/xvdc volatile.img via qubes_cow_setup.sh. root-cow.img is still created but is rudimentary. Yes, tested on sys-firewall and sys-net.1
private (DispVM) True False False Accumulates writes into private-cow.img, discards on stop Crashes or experiences degraded functionality since qubes_cow_setup.sh does not handle readonly /rw.2 private-cow.img becomes rudimentary. Yes, tested on sys-firewall. It starts, but I cannot even open the terminal. No functionality.
private (AppVM) False True False Accumulates writes into private-cow.img, saves on stop Crashes or experiences degraded functionality since qubes_cow_setup.sh does not handle readonly /rw. private-cow.img becomes rudimentary. Yes, tested on sys-net. Same results as above.
volatile False False False Written directly Crashes, since it would attempt to set up swap on a readonly partition. Confirmed with both sys-net and sys-firewall: Cannot connect to qrexec agent for 60 seconds, see /var/log/xen/console/guest-sys-firewall.log for details.3

With all of this in mind, this seems to be a much bigger change than anticipated. However, as far as I understand rw="False" is NOT used anywhere in Qubes by default. Therefore, even if this change is merged we would not be altering any default setups, and given that nobody has fixed this yet I don't think many people are using the rw functionality in general. If you think this is worth a proper discussion, I'm ready to continue in qubes-devel.

Footnotes

  1. Causes a FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/qubes/vm-templates/TEMPLATE_NAME/root-cow.img' upon attempting to kill the qube IF after starting it, qubesd is restarted. Full sequence to reproduce: qvm-start sys-firewall && sudo systemctl restart qubesd && qvm-kill sys-firewall. Not unique to sys-firewall, similar issue occurs with sys-net.

  2. I propose that this could be resolved by extending qubes_cow_setup.sh to handle /dev/xvdb (private /rw).

  3. I propose that this could be resolved by falling back to a RAM device backed by domU memory.

@codecov

codecov Bot commented Jun 14, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 50.00000% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 70.37%. Comparing base (b70518c) to head (ce22ff4).

Files with missing lines Patch % Lines
qubes/vm/qubesvm.py 50.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #829      +/-   ##
==========================================
- Coverage   70.37%   70.37%   -0.01%     
==========================================
  Files          61       61              
  Lines       14098    14100       +2     
==========================================
+ Hits         9922     9923       +1     
- Misses       4176     4177       +1     
Flag Coverage Δ
unittests 70.37% <50.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@qubesos-bot

qubesos-bot commented Jun 15, 2026

Copy link
Copy Markdown

OpenQA test summary

Complete test suite and dependencies: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2026061720-devel&flavor=pull-requests

Test run included the following:

New failures, excluding unstable

Compared to: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2026050504-devel&flavor=update

  • system_tests_guivm_gui_interactive

  • system_tests_network_updates

    • TC_00_Dom0Upgrade_whonix-gateway-18: test_000_update_salt (failure)
      ?[0;31mDuration: 156508.411 ms?[0;0m... ?[0;31m Changes: ?[0;0m

    • TC_11_QvmTemplateMgmtVM_fedora-43-xfce: test_010_template_install (failure)
      AssertionError: qvm-template failed: [Qrexec] ERROR: dnf command is...

  • system_tests_guivm_vnc_gui_interactive

  • system_tests_guivm_gpu_gui_interactive@hw13

    • simple_gui_apps: unnamed test (unknown)

    • simple_gui_apps: Failed (test died)
      # Test died: no candidate needle with tag(s) 'menu-vm-work' matched...

    • simple_gui_apps: unnamed test (unknown)

  • system_tests_extra

    • TC_00_Direct_whonix-workstation-18: test_050_generate (failure)
      AssertionError: 'user2@localhost' not found in '/home/user/.gnupg/p...
  • system_tests_splitgpg

    • TC_00_Direct_whonix-workstation-18: test_030_sign_verify (failure)
      ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^... AssertionError: key generation failed

Failed tests

20 failures
  • system_tests_whonix

    • [unstable] whonixcheck: fail (unknown)
      Whonixcheck for anon-whonix failed...

    • [unstable] whonixcheck: fail (unknown)
      Whonixcheck for whonix-workstation-18 failed...

    • [unstable] whonixcheck: Failed (test died)
      # Test died: systemcheck failed at qubesos/tests/whonixcheck.pm lin...

  • system_tests_guivm_gui_interactive

    • guivm_manager: unnamed test (unknown)
    • [unstable] guivm_manager: Failed (test died)
      # Test died: no candidate needle with tag(s) 'menu-qubes-tools-subm...
  • system_tests_network_ipv6

  • system_tests_network_updates

    • TC_00_Dom0Upgrade_whonix-gateway-18: test_000_update_salt (failure)
      ?[0;31mDuration: 156508.411 ms?[0;0m... ?[0;31m Changes: ?[0;0m

    • TC_11_QvmTemplateMgmtVM_fedora-43-xfce: test_010_template_install (failure)
      AssertionError: qvm-template failed: [Qrexec] ERROR: dnf command is...

  • system_tests_audio

  • system_tests_whonix@hw1

    • [unstable] whonixcheck: fail (unknown)
      Whonixcheck for whonix-workstation-18 failed...

    • [unstable] whonixcheck: Failed (test died)
      # Test died: systemcheck failed at qubesos/tests/whonixcheck.pm lin...

  • system_tests_guivm_vnc_gui_interactive

    • guivm_manager: unnamed test (unknown)
    • [unstable] guivm_manager: Failed (test died)
      # Test died: no candidate needle with tag(s) 'vm-settings-ok' match...
  • system_tests_guivm_gpu_gui_interactive@hw13

    • simple_gui_apps: unnamed test (unknown)

    • simple_gui_apps: Failed (test died)
      # Test died: no candidate needle with tag(s) 'menu-vm-work' matched...

    • simple_gui_apps: unnamed test (unknown)

  • system_tests_extra

    • TC_00_Direct_whonix-workstation-18: test_050_generate (failure)
      AssertionError: 'user2@localhost' not found in '/home/user/.gnupg/p...
  • system_tests_splitgpg

    • TC_00_Direct_whonix-workstation-18: test_030_sign_verify (failure)
      ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^... AssertionError: key generation failed

Fixed failures

Compared to: https://openqa.qubes-os.org/tests/176874#dependencies

27 fixed
  • system_tests_pvgrub_salt_storage

    • system_tests: Fail (unknown)
      Tests qubes.tests.integ.grub failed (exit code 1), details reported...

    • system_tests: Failed (test died)
      # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...

    • TC_41_HVMGrub_debian-13-xfce: test_000_standalone_vm (error)
      qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...

    • TC_41_HVMGrub_debian-13-xfce: test_001_standalone_vm_dracut (error)
      qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...

    • TC_41_HVMGrub_debian-13-xfce: test_010_template_based_vm (error)
      qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...

    • TC_41_HVMGrub_debian-13-xfce: test_011_template_based_vm_dracut (error)
      qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...

    • TC_41_HVMGrub_fedora-43-xfce: test_010_template_based_vm (error)
      qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...

  • system_tests_usbproxy

    • system_tests: Fail (unknown)
      Tests qubes.tests.extra failed (exit code 1), details reported sepa...

    • system_tests: Failed (test died)
      # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...

    • system_tests: wait_serial (wait serial expected)
      # wait_serial expected: qr/h3uXO-\d+-/...

    • TC_20_USBProxy_core3_fedora-43-xfce: test_090_attach_stubdom (error)
      qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...

  • system_tests_audio

  • system_tests_audio@hw1

    • system_tests: Fail (unknown)
      Tests qubes.tests.integ.audio failed (exit code 1), details reporte...

    • system_tests: Failed (test died)
      # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...

    • TC_20_AudioVM_Pulse_debian-13-xfce: test_222_audio_rec_unmuted_pulseaudio (failure)
      AssertionError: only silence detected, no useful audio data

  • system_tests_guivm_gpu_gui_interactive@hw13

    • shutdown: unnamed test (unknown)
    • shutdown: Failed (test died)
      # Test died: no candidate needle with tag(s) 'text-logged-in-root' ...
  • system_tests_extra

    • TC_01_InputProxyExclude_debian-13-xfce: test_000_qemu_tablet (error)
      qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...

    • TC_01_InputProxyExclude_fedora-43-xfce: test_000_qemu_tablet (error)
      qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...

    • TC_00_QVCTest_fedora-43-xfce: test_010_screenshare (failure + cleanup)
      ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^... AssertionError: 1179648 != 0

    • TC_00_QVCTest_whonix-gateway-18: test_010_screenshare (failure + cleanup)
      AssertionError: 2.3156185715769593 not less than 2.0

    • TC_00_PDFConverter_fedora-43-xfce: test_004_cancel_stops_conversion (failure)
      AssertionError: DispVM not cleaned up 10s after cancel: {<DispVM at...

Unstable tests

Details
  • system_tests_whonix

    whonixcheck/Failed (1/5 times with errors)
    • job 182677 # Test died: systemcheck failed at qubesos/tests/whonixcheck.pm lin...
    whonixcheck/Failed (2/5 times with errors)
    • job 181249 # Test died: systemcheck failed at qubesos/tests/whonixcheck.pm lin...
    • job 182411 # Test died: systemcheck failed at qubesos/tests/whonixcheck.pm lin...
    whonixcheck/fail (2/5 times with errors)
    whonixcheck/fail (1/5 times with errors)
    whonixcheck/fail (2/5 times with errors)
  • system_tests_suspend

    suspend/Failed (1/5 times with errors)
    • job 179098 # Test died: command 'qvm-run -p sys-net true' timed out at /usr/li...
    suspend/Failed (3/5 times with errors)
    • job 179097 # Test died: command '! qvm-check sys-usb || qvm-run -p sys-usb tru...
    • job 179101 # Test died: command '! qvm-check sys-usb || qvm-run -p sys-usb tru...
    • job 179957 # Test died: command '! qvm-check sys-usb || qvm-run -p sys-usb tru...
    suspend/wait_serial (1/5 times with errors)
    • job 179098 # Command: qvm-run -p sys-net true...
    suspend/wait_serial (3/5 times with errors)
    • job 179097 # Command: ! qvm-check sys-usb || qvm-run -p sys-usb true...
    • job 179101 # Command: ! qvm-check sys-usb || qvm-run -p sys-usb true...
    • job 179957 # Command: ! qvm-check sys-usb || qvm-run -p sys-usb true...
    suspend/wait_serial (1/5 times with errors)
    suspend/wait_serial (3/5 times with errors)
    • job 179097 # wait_serial expected: "lspci; echo 2E8vz-\$?-"...
    • job 179101 # wait_serial expected: "lspci; echo 2E8vz-\$?-"...
    • job 179957 # wait_serial expected: "lspci; echo 2E8vz-\$?-"...
  • system_tests_basic_vm_qrexec_gui

    system_tests/Fail (1/5 times with errors)
    • job 182429 Tests qubes.tests.integ.basic failed (exit code 1), details reporte...
    system_tests/Fail (1/5 times with errors)
    • job 181256 Tests qubes.tests.integ.vm_qrexec_gui failed (exit code 1), details...
    system_tests/Fail (1/5 times with errors)
    • job 182429 Tests qubes.tests.integ.vm_qrexec_gui failed (exit code 1), details...
    system_tests/Failed (1/5 times with errors)
    • job 183745 # Test died: command 'curl --form upload=@tests-qubes.tests.integ.b...
    system_tests/Failed (1/5 times with errors)
    • job 181256 # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...
    system_tests/Failed (1/5 times with errors)
    • job 182429 # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...
    TC_10_Generic/test_000_anyvm_deny_dom0 (1/5 times with errors)
    • job 182429 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    TC_03_QvmRevertTemplateChanges/test_000_revert_linux (1/5 times with errors)
    • job 182429 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    TC_20_NonAudio_fedora-43-xfce/test_101_qrexec_filecopy_with_autostart (1/5 times with errors)
    • job 182429 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    TC_00_Basic/test_211_bootmode_standalone (1/5 times with errors)
    • job 182429 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    TC_20_NonAudio_fedora-43-xfce/test_401_long_window_title_utf8 (1/5 times with errors)
    • job 181256 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
  • system_tests_network

    system_tests/Fail (4/5 times with errors)
    • job 181267 Tests qubes.tests.integ.network failed (exit code 1), details repor...
    • job 182440 Tests qubes.tests.integ.network failed (exit code 1), details repor...
    • job 182706 Tests qubes.tests.integ.network failed (exit code 1), details repor...
    • job 183756 Tests qubes.tests.integ.network failed (exit code 1), details repor...
    system_tests/Failed (2/5 times with errors)
    • job 181267 # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...
    • job 183756 # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...
    system_tests/Failed (2/5 times with errors)
    • job 182440 # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...
    • job 182706 # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...
    VmNetworking_fedora-43-xfce/test_001_simple_networking_paused_restart_netvm (1/5 times with errors)
    • job 182706 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    VmNetworking_fedora-43-xfce/test_001_simple_networking_paused_shutdown_netvm (1/5 times with errors)
    • job 182706 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    VmNetworking_fedora-43-xfce/test_010_simple_proxyvm (1/5 times with errors)
    VmNetworking_fedora-43-xfce/test_020_simple_proxyvm_nm (2/5 times with errors)
    • job 182440 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    • job 182706 ^^^^^^^^^^... AssertionError
    VmNetworking_fedora-43-xfce/test_030_firewallvm_firewall (2/5 times with errors)
    VmNetworking_fedora-43-xfce/test_031_firewall_dynamic_block (2/5 times with errors)
    VmNetworking_fedora-43-xfce/test_040_inter_vm (2/5 times with errors)
    VmNetworking_fedora-43-xfce/test_050_spoof_ip (2/5 times with errors)
    • job 182440 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    • job 182706 ^^^^^^^^^^... AssertionError
    VmNetworking_fedora-43-xfce/test_100_late_xldevd_startup (2/5 times with errors)
    VmNetworking_debian-13-xfce/test_110_dynamic_attach (1/5 times with errors)
    • job 182706 ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^... AssertionError: 125 != 0
    VmNetworking_fedora-43-xfce/test_110_dynamic_attach (2/5 times with errors)
    • job 182440 ^^^^^^^^^^... AssertionError
    • job 182706 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    VmNetworking_fedora-44-xfce/test_110_dynamic_attach (1/5 times with errors)
    • job 183756 ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^... AssertionError: 125 != 0
    VmNetworking_fedora-43-xfce/test_111_dynamic_detach_attach (2/5 times with errors)
    VmNetworking_fedora-43-xfce/test_112_reattach_after_provider_shutdown (2/5 times with errors)
    VmNetworking_fedora-43-xfce/test_113_reattach_after_provider_kill (2/5 times with errors)
    • job 182440 ^^^^^^^^^^... AssertionError
    • job 182706 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    VmNetworking_fedora-43-xfce/test_114_reattach_after_provider_crash (2/5 times with errors)
    VmNetworking_fedora-43-xfce/test_200_fake_ip_simple (3/5 times with errors)
    • job 181267 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    • job 182440 ^^^^^^^^^^... AssertionError
    • job 182706 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    VmNetworking_fedora-43-xfce/test_201_fake_ip_without_gw (2/5 times with errors)
    • job 182440 ^^^^^^^^^^... AssertionError
    • job 182706 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    VmNetworking_fedora-43-xfce/test_202_fake_ip_firewall (2/5 times with errors)
    • job 182440 ^^^^^^^^^^... AssertionError
    • job 182706 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    VmNetworking_fedora-43-xfce/test_203_fake_ip_inter_vm_allow (2/5 times with errors)
    VmNetworking_fedora-43-xfce/test_204_fake_ip_proxy (2/5 times with errors)
    • job 182440 ^^^^^^^^^^... AssertionError
    • job 182706 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    VmNetworking_fedora-43-xfce/test_210_custom_ip_simple (2/5 times with errors)
    VmNetworking_fedora-43-xfce/test_211_custom_ip_proxy (2/5 times with errors)
    • job 182440 ^^^^^^^^^^... AssertionError
    • job 182706 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    VmNetworking_fedora-43-xfce/test_212_custom_ip_firewall (3/5 times with errors)
    • job 181267 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    • job 182440 ^^^^^^^^^^... AssertionError
    • job 182706 ^^^^^^^^^^... AssertionError
  • system_tests_pvgrub_salt_storage

    system_tests/Fail (3/5 times with errors)
    • job 182709 Tests qubes.tests.integ.grub failed (exit code 1), details reported...
    • job 183539 Tests qubes.tests.integ.grub failed (exit code 1), details reported...
    • job 183759 Tests qubes.tests.integ.grub failed (exit code 1), details reported...
    system_tests/Failed (3/5 times with errors)
    • job 182709 # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...
    • job 183539 # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...
    • job 183759 # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...
    TC_41_HVMGrub_debian-13-xfce/test_000_standalone_vm (1/5 times with errors)
    • job 182709 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    TC_41_HVMGrub_fedora-43-xfce/test_000_standalone_vm (1/5 times with errors)
    • job 182709 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    TC_41_HVMGrub_debian-13-xfce/test_001_standalone_vm_dracut (3/5 times with errors)
    • job 182709 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    • job 183539 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    • job 183759 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    TC_41_HVMGrub_debian-13-xfce/test_010_template_based_vm (3/5 times with errors)
    • job 182709 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    • job 183539 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    • job 183759 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    TC_41_HVMGrub_fedora-43-xfce/test_010_template_based_vm (1/5 times with errors)
    • job 182709 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    TC_41_HVMGrub_fedora-44-xfce/test_010_template_based_vm (2/5 times with errors)
    • job 183539 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    • job 183759 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    TC_41_HVMGrub_debian-13-xfce/test_011_template_based_vm_dracut (3/5 times with errors)
    • job 182709 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    • job 183539 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    • job 183759 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
  • system_tests_gui_interactive

    screenlocker_lock/ (1/5 times with errors)
    screenlocker_lock/Failed (1/5 times with errors)
    • job 181264 # Test died: no candidate needle with tag(s) 'screenlocker-blank' m...
  • system_tests_guivm_gui_interactive

    guivm_manager/ (1/5 times with errors)
    guivm_manager/Failed (1/5 times with errors)
    • job 183755 # Test died: no candidate needle with tag(s) 'menu-qubes-manager' m...
    update_guivm/Failed (2/5 times with errors)
    • job 183535 # Test died: command '(set -o pipefail; qubesctl --all --show-outpu...
    • job 183568 # Test died: command '(set -o pipefail; qubesctl --all --show-outpu...
  • system_tests_usbproxy

    system_tests/Fail (2/5 times with errors)
    • job 181248 Tests qubes.tests.extra failed (exit code 1), details reported sepa...
    • job 182676 Tests qubes.tests.extra failed (exit code 1), details reported sepa...
    system_tests/Failed (2/5 times with errors)
    • job 181248 # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...
    • job 182676 # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...
    TC_00_USBProxy_fedora-43-xfce/test_000_attach_detach (1/5 times with errors)
    • job 181248 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    TC_20_USBProxy_core3_fedora-43-xfce/test_000_list (1/5 times with errors)
    • job 181248 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    TC_20_USBProxy_core3_fedora-43-xfce/test_080_attach_existing_policy (1/5 times with errors)
    • job 181248 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    TC_20_USBProxy_core3_fedora-43-xfce/test_090_attach_stubdom (1/5 times with errors)
    • job 182676 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
  • system_tests_qrexec

    system_tests/Fail (1/5 times with errors)
    • job 183576 Tests qubes.tests.integ.qrexec failed (exit code 1), details report...
    system_tests/Failed (1/5 times with errors)
    • job 183540 # Test died: command 'curl --form upload=@tests-qubes.tests.integ.q...
    system_tests/Failed (1/5 times with errors)
    • job 183576 # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...
    TC_00_Qrexec_whonix-gateway-18/test_090_qrexec_service_socket_dom0 (1/5 times with errors)
    • job 183576 AssertionError: service timeout, probably EOF wasn't transferred to...
  • system_tests_network_ipv6

    system_tests/Fail (2/5 times with errors)
    • job 181268 Tests qubes.tests.integ.network_ipv6 failed (exit code 1), details ...
    • job 182441 Tests qubes.tests.integ.network_ipv6 failed (exit code 1), details ...
    system_tests/Failed (2/5 times with errors)
    • job 181268 # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...
    • job 182441 # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...
    VmIPv6Networking_fedora-43-xfce/test_001_simple_networking_paused_restart_netvm (1/5 times with errors)
    • job 181268 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    VmIPv6Networking_fedora-43-xfce/test_010_simple_proxyvm (1/5 times with errors)
    • job 182441 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    VmIPv6Networking_fedora-43-xfce/test_020_simple_proxyvm_nm (1/5 times with errors)
    VmIPv6Networking_fedora-43-xfce/test_030_firewallvm_firewall (1/5 times with errors)
    VmIPv6Networking_fedora-43-xfce/test_031_firewall_dynamic_block (1/5 times with errors)
    VmIPv6Networking_fedora-43-xfce/test_040_inter_vm (1/5 times with errors)
    VmIPv6Networking_fedora-43-xfce/test_050_spoof_ip (1/5 times with errors)
    VmIPv6Networking_fedora-43-xfce/test_100_late_xldevd_startup (1/5 times with errors)
    VmIPv6Networking_fedora-43-xfce/test_110_dynamic_attach (1/5 times with errors)
    VmIPv6Networking_fedora-43-xfce/test_111_dynamic_detach_attach (1/5 times with errors)
    VmIPv6Networking_fedora-43-xfce/test_112_reattach_after_provider_shutdown (2/5 times with errors)
    • job 181268 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    • job 182441 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    VmIPv6Networking_fedora-43-xfce/test_113_reattach_after_provider_kill (2/5 times with errors)
    VmIPv6Networking_fedora-43-xfce/test_114_reattach_after_provider_crash (2/5 times with errors)
    VmIPv6Networking_fedora-43-xfce/test_200_fake_ip_simple (2/5 times with errors)
    VmIPv6Networking_fedora-43-xfce/test_201_fake_ip_without_gw (2/5 times with errors)
    VmIPv6Networking_fedora-43-xfce/test_202_fake_ip_firewall (2/5 times with errors)
    VmIPv6Networking_fedora-43-xfce/test_203_fake_ip_inter_vm_allow (2/5 times with errors)
    VmIPv6Networking_fedora-43-xfce/test_204_fake_ip_proxy (2/5 times with errors)
    VmIPv6Networking_fedora-43-xfce/test_210_custom_ip_simple (2/5 times with errors)
    VmIPv6Networking_fedora-43-xfce/test_211_custom_ip_proxy (2/5 times with errors)
    VmIPv6Networking_fedora-43-xfce/test_212_custom_ip_firewall (2/5 times with errors)
    • job 181268 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    • job 182441 ^^^^^^^^^^... AssertionError
    VmIPv6Networking_fedora-43-xfce/test_500_ipv6_simple_networking (2/5 times with errors)
    VmIPv6Networking_fedora-43-xfce/test_501_simple_networking_paused_change_purge_old (2/5 times with errors)
    VmIPv6Networking_fedora-43-xfce/test_501_simple_networking_paused_change_shutdown_old (2/5 times with errors)
    VmIPv6Networking_fedora-43-xfce/test_501_simple_networking_paused_from_existent_to_none (2/5 times with errors)
    VmIPv6Networking_debian-13-xfce/test_501_simple_networking_paused_from_none_to_existent (1/5 times with errors)
    • job 181268 subprocess.CalledProcessError: Command 'qubes.WaitForNetworkUplink'...
    VmIPv6Networking_fedora-43-xfce/test_501_simple_networking_paused_from_none_to_existent (2/5 times with errors)
    • job 181268 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    • job 182441 ^^^^^^^^^^... AssertionError
    VmIPv6Networking_fedora-43-xfce/test_501_simple_networking_paused_restart_netvm (2/5 times with errors)
    VmIPv6Networking_fedora-43-xfce/test_501_simple_networking_paused_shutdown_netvm (2/5 times with errors)
    • job 181268 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    • job 182441 ^^^^^^^^^^... AssertionError
    VmIPv6Networking_fedora-43-xfce/test_510_ipv6_simple_proxyvm (2/5 times with errors)
    VmIPv6Networking_fedora-43-xfce/test_520_ipv6_simple_proxyvm_nm (2/5 times with errors)
    VmIPv6Networking_fedora-43-xfce/test_530_ipv6_firewallvm_firewall (2/5 times with errors)
    VmIPv6Networking_fedora-43-xfce/test_540_ipv6_inter_vm (2/5 times with errors)
    VmIPv6Networking_fedora-43-xfce/test_550_ipv6_spoof_ip (2/5 times with errors)
    VmIPv6Networking_fedora-43-xfce/test_710_ipv6_custom_ip_simple (2/5 times with errors)
    VmIPv6Networking_fedora-43-xfce/test_711_ipv6_custom_ip_proxy (2/5 times with errors)
    VmIPv6Networking_fedora-43-xfce/test_712_ipv6_custom_ip_firewall (2/5 times with errors)
  • system_tests_network_updates

    system_tests/Fail (1/5 times with errors)
    • job 181269 Tests qubes.tests.integ.dom0_update failed (exit code 1), details r...
    system_tests/Failed (1/5 times with errors)
    • job 181269 # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...
    TC_11_QvmTemplateMgmtVM_fedora-43-xfce/test_000_template_list (1/5 times with errors)
    • job 181269 AssertionError: qvm-template failed: [Qrexec] ERROR: dnf command is...
    TC_00_Dom0Upgrade_whonix-gateway-18/test_010_instal (1/5 times with errors)
    • job 181269 subprocess.CalledProcessError: Command 'timeout=120; while ! tor-ci...
  • system_tests_dispvm

    system_tests/Fail (4/5 times with errors)
    • job 181262 Tests qubes.tests.integ.dispvm failed (exit code 1), details report...
    • job 182435 Tests qubes.tests.integ.dispvm failed (exit code 1), details report...
    • job 182701 Tests qubes.tests.integ.dispvm failed (exit code 1), details report...
    • job 183751 Tests qubes.tests.integ.dispvm failed (exit code 1), details report...
    system_tests/Failed (4/5 times with errors)
    • job 181262 # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...
    • job 182435 # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...
    • job 182701 # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...
    • job 183751 # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...
    TC_21_DispVM_Preload/test_015_preload_race_more (1/5 times with errors)
    • job 183751 raise TimeoutError from exc_val... TimeoutError
    TC_20_DispVM_debian-13-xfce/test_080_gui_app (1/5 times with errors)
    • job 183751 qubes.exc.QubesVMShutdownTimeoutError: Domain shutdown timed out: '...
    TC_20_DispVM_fedora-43-xfce/test_100_open_in_dispvm (3/5 times with errors)
    • job 181262 subprocess.CalledProcessError: Command 'cat > /etc/thunderbird/pref...
    • job 182435 subprocess.CalledProcessError: Command 'cat > /etc/thunderbird/pref...
    • job 182701 subprocess.CalledProcessError: Command 'cat > /etc/thunderbird/pref...
    TC_20_DispVM_fedora-44-xfce/test_100_open_in_dispvm (1/5 times with errors)
    • job 183751 subprocess.CalledProcessError: Command 'cat > /etc/thunderbird/pref...
    system_tests/wait_serial (1/5 times with errors)
    • job 181262 # Command: curl --form upload=@/var/log/libvirt/libxl/libxl-driver....
  • system_tests_kde_gui_interactive

    gui_keyboard_layout/Failed (1/5 times with errors)
    • job 183511 # Test died: command 'qvm-shutdown --wait work' timed out at /usr/l...
    gui_keyboard_layout/Failed (1/5 times with errors)
    • job 182415 # Test died: command 'qvm-shutdown --wait work' failed at /usr/lib/...
    gui_keyboard_layout/wait_serial (1/5 times with errors)
    • job 183511 # Command: qvm-shutdown --wait work...
    gui_keyboard_layout/wait_serial (1/5 times with errors)
    • job 183511 # Command: curl --form upload=@/var/log/libvirt/libxl/libxl-driver....
  • system_tests_basic_vm_qrexec_gui_zfs

    system_tests/Fail (1/5 times with errors)
    • job 182433 Tests qubes.tests.integ.vm_qrexec_gui failed (exit code 1), details...
    system_tests/Failed (1/5 times with errors)
    • job 182433 # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...
    TC_20_NonAudio_fedora-43-xfce-pool/test_200_timezone (1/5 times with errors)
    • job 182433 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    TC_20_NonAudio_fedora-43-xfce-pool/test_400_long_window_title (1/5 times with errors)
    • job 182433 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
  • system_tests_audio

    system_tests/Failed (2/5 times with errors)
    • job 183510 # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...
    • job 183730 # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...
    system_tests/Failed (3/5 times with errors)
    • job 181250 # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...
    • job 182414 # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...
    • job 182680 # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...
    TC_20_AudioVM_Pulse_fedora-44-xfce/test_223_audio_play_hvm (2/5 times with errors)
    • job 183510 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    • job 183730 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    TC_20_AudioVM_Pulse_whonix-workstation-18/test_223_audio_play_hvm (1/5 times with errors)
    • job 183510 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    TC_20_AudioVM_Pulse_fedora-44-xfce/test_224_audio_rec_muted_hvm (1/5 times with errors)
    • job 183510 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    TC_20_AudioVM_Pulse_fedora-43-xfce/test_225_audio_rec_unmuted_hvm (1/5 times with errors)
    • job 182414 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    TC_20_AudioVM_Pulse_fedora-44-xfce/test_225_audio_rec_unmuted_hvm (2/5 times with errors)
    • job 183510 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    • job 183730 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    TC_20_AudioVM_Pulse_whonix-workstation-18/test_225_audio_rec_unmuted_hvm (5/5 times with errors)
    • job 181250 AssertionError: too short audio, expected 10s, got 6.18662131519274...
    • job 182414 AssertionError: too short audio, expected 10s, got 4.07111111111111...
    • job 182680 AssertionError: too short audio, expected 10s, got 7.11995464852607...
    • job 183510 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    • job 183730 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    TC_20_AudioVM_PipeWire_debian-13-xfce/test_251_audio_playback_audiovm_pipewire_late_start (1/5 times with errors)
    • job 182414 AssertionError: pacat for test-inst-vm1 (xid 48) running(False) in ...
    TC_20_AudioVM_PipeWire_fedora-43-xfce/test_251_audio_playback_audiovm_pipewire_late_start (1/5 times with errors)
    • job 181250 AssertionError: too short audio, expected 10s, got 9.35857142857142...
    TC_20_AudioVM_PipeWire_whonix-workstation-18/test_251_audio_playback_audiovm_pipewire_late_start (1/5 times with errors)
    • job 182680 AssertionError: too short audio, expected 10s, got 9.36630385487528...
    TC_20_AudioVM_Pulse_debian-13-xfce/test_252_audio_playback_audiovm_switch_hvm (1/5 times with errors)
    • job 183730 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    TC_20_AudioVM_Pulse_fedora-43-xfce/test_252_audio_playback_audiovm_switch_hvm (2/5 times with errors)
    • job 181250 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    • job 182414 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    TC_20_AudioVM_Pulse_fedora-44-xfce/test_252_audio_playback_audiovm_switch_hvm (2/5 times with errors)
    • job 183510 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    • job 183730 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    TC_20_AudioVM_Pulse_whonix-workstation-18/test_252_audio_playback_audiovm_switch_hvm (2/5 times with errors)
    • job 181250 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    • job 183510 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    TC_20_AudioVM_PipeWire_debian-13-xfce/test_260_audio_mic_enabled_switch_audiovm (1/5 times with errors)
    • job 181250 AssertionError: pacat for test-inst-vm1 (xid 50) running(False) in ...
    TC_20_AudioVM_PipeWire_whonix-workstation-18/test_260_audio_mic_enabled_switch_audiovm (1/5 times with errors)
    • job 182414 AssertionError: 0 not greater than 0 : Failed to move-source-output
  • system_tests_audio@hw1

    system_tests/Failed (2/5 times with errors)
    • job 183510 # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...
    • job 183730 # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...
    system_tests/Failed (3/5 times with errors)
    • job 181250 # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...
    • job 182414 # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...
    • job 182680 # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...
    TC_20_AudioVM_Pulse_fedora-44-xfce/test_223_audio_play_hvm (2/5 times with errors)
    • job 183510 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    • job 183730 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    TC_20_AudioVM_Pulse_whonix-workstation-18/test_223_audio_play_hvm (1/5 times with errors)
    • job 183510 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    TC_20_AudioVM_Pulse_fedora-44-xfce/test_224_audio_rec_muted_hvm (1/5 times with errors)
    • job 183510 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    TC_20_AudioVM_Pulse_fedora-43-xfce/test_225_audio_rec_unmuted_hvm (1/5 times with errors)
    • job 182414 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    TC_20_AudioVM_Pulse_fedora-44-xfce/test_225_audio_rec_unmuted_hvm (2/5 times with errors)
    • job 183510 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    • job 183730 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    TC_20_AudioVM_Pulse_whonix-workstation-18/test_225_audio_rec_unmuted_hvm (5/5 times with errors)
    • job 181250 AssertionError: too short audio, expected 10s, got 6.18662131519274...
    • job 182414 AssertionError: too short audio, expected 10s, got 4.07111111111111...
    • job 182680 AssertionError: too short audio, expected 10s, got 7.11995464852607...
    • job 183510 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    • job 183730 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    TC_20_AudioVM_PipeWire_debian-13-xfce/test_251_audio_playback_audiovm_pipewire_late_start (1/5 times with errors)
    • job 182414 AssertionError: pacat for test-inst-vm1 (xid 48) running(False) in ...
    TC_20_AudioVM_PipeWire_fedora-43-xfce/test_251_audio_playback_audiovm_pipewire_late_start (1/5 times with errors)
    • job 181250 AssertionError: too short audio, expected 10s, got 9.35857142857142...
    TC_20_AudioVM_PipeWire_whonix-workstation-18/test_251_audio_playback_audiovm_pipewire_late_start (1/5 times with errors)
    • job 182680 AssertionError: too short audio, expected 10s, got 9.36630385487528...
    TC_20_AudioVM_Pulse_debian-13-xfce/test_252_audio_playback_audiovm_switch_hvm (1/5 times with errors)
    • job 183730 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    TC_20_AudioVM_Pulse_fedora-43-xfce/test_252_audio_playback_audiovm_switch_hvm (2/5 times with errors)
    • job 181250 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    • job 182414 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    TC_20_AudioVM_Pulse_fedora-44-xfce/test_252_audio_playback_audiovm_switch_hvm (2/5 times with errors)
    • job 183510 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    • job 183730 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    TC_20_AudioVM_Pulse_whonix-workstation-18/test_252_audio_playback_audiovm_switch_hvm (2/5 times with errors)
    • job 181250 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    • job 183510 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    TC_20_AudioVM_PipeWire_debian-13-xfce/test_260_audio_mic_enabled_switch_audiovm (1/5 times with errors)
    • job 181250 AssertionError: pacat for test-inst-vm1 (xid 50) running(False) in ...
    TC_20_AudioVM_PipeWire_whonix-workstation-18/test_260_audio_mic_enabled_switch_audiovm (1/5 times with errors)
    • job 182414 AssertionError: 0 not greater than 0 : Failed to move-source-output
  • system_tests_whonix@hw1

    whonixcheck/Failed (1/5 times with errors)
    • job 182677 # Test died: systemcheck failed at qubesos/tests/whonixcheck.pm lin...
    whonixcheck/Failed (2/5 times with errors)
    • job 181249 # Test died: systemcheck failed at qubesos/tests/whonixcheck.pm lin...
    • job 182411 # Test died: systemcheck failed at qubesos/tests/whonixcheck.pm lin...
    whonixcheck/fail (2/5 times with errors)
    whonixcheck/fail (1/5 times with errors)
    whonixcheck/fail (2/5 times with errors)
  • system_tests_basic_vm_qrexec_gui_ext4

    system_tests/ (2/5 times with errors)
    system_tests/Failed (2/5 times with errors)
    • job 183527 # Test died: no candidate needle with tag(s) 'x11' matched...
    • job 183569 # Test died: no candidate needle with tag(s) 'x11' matched...
  • system_tests_suspend@hw1

    suspend/Failed (1/5 times with errors)
    • job 179098 # Test died: command 'qvm-run -p sys-net true' timed out at /usr/li...
    suspend/Failed (3/5 times with errors)
    • job 179097 # Test died: command '! qvm-check sys-usb || qvm-run -p sys-usb tru...
    • job 179101 # Test died: command '! qvm-check sys-usb || qvm-run -p sys-usb tru...
    • job 179957 # Test died: command '! qvm-check sys-usb || qvm-run -p sys-usb tru...
    suspend/wait_serial (1/5 times with errors)
    • job 179098 # Command: qvm-run -p sys-net true...
    suspend/wait_serial (3/5 times with errors)
    • job 179097 # Command: ! qvm-check sys-usb || qvm-run -p sys-usb true...
    • job 179101 # Command: ! qvm-check sys-usb || qvm-run -p sys-usb true...
    • job 179957 # Command: ! qvm-check sys-usb || qvm-run -p sys-usb true...
    suspend/wait_serial (1/5 times with errors)
    suspend/wait_serial (3/5 times with errors)
    • job 179097 # wait_serial expected: "lspci; echo 2E8vz-\$?-"...
    • job 179101 # wait_serial expected: "lspci; echo 2E8vz-\$?-"...
    • job 179957 # wait_serial expected: "lspci; echo 2E8vz-\$?-"...
  • system_tests_gui_interactive@hw7

    screenlocker_lock/ (1/5 times with errors)
    screenlocker_lock/Failed (1/5 times with errors)
    • job 181264 # Test died: no candidate needle with tag(s) 'screenlocker-blank' m...
  • system_tests_basic_vm_qrexec_gui@hw7

    system_tests/Fail (1/5 times with errors)
    • job 182429 Tests qubes.tests.integ.basic failed (exit code 1), details reporte...
    system_tests/Fail (1/5 times with errors)
    • job 181256 Tests qubes.tests.integ.vm_qrexec_gui failed (exit code 1), details...
    system_tests/Fail (1/5 times with errors)
    • job 182429 Tests qubes.tests.integ.vm_qrexec_gui failed (exit code 1), details...
    system_tests/Failed (1/5 times with errors)
    • job 183745 # Test died: command 'curl --form upload=@tests-qubes.tests.integ.b...
    system_tests/Failed (1/5 times with errors)
    • job 181256 # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...
    system_tests/Failed (1/5 times with errors)
    • job 182429 # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...
    TC_10_Generic/test_000_anyvm_deny_dom0 (1/5 times with errors)
    • job 182429 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    TC_03_QvmRevertTemplateChanges/test_000_revert_linux (1/5 times with errors)
    • job 182429 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    TC_20_NonAudio_fedora-43-xfce/test_101_qrexec_filecopy_with_autostart (1/5 times with errors)
    • job 182429 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    TC_00_Basic/test_211_bootmode_standalone (1/5 times with errors)
    • job 182429 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    TC_20_NonAudio_fedora-43-xfce/test_401_long_window_title_utf8 (1/5 times with errors)
    • job 181256 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
  • system_tests_basic_vm_qrexec_gui_xfs

    switch_pool/Failed (2/5 times with errors)
    • job 182432 # Test died: command 'qvm-start sys-firewall sys-usb' failed at /us...
    • job 183748 # Test died: command 'qvm-start sys-firewall sys-usb' failed at /us...
    system_tests/Failed (1/5 times with errors)
    • job 183811 # Test died: command 'curl --form upload=@nose2-junit-qubes.tests.i...
    switch_pool/wait_serial (1/5 times with errors)
    • job 183748 # Command: qvm-run --no-gui -p -u root sys-firewall "cat /var/log/x...
  • system_tests_basic_vm_qrexec_gui_btrfs

    system_tests/ (1/5 times with errors)
    system_tests/Fail (1/5 times with errors)
    • job 182430 Tests qubes.tests.integ.basic failed (exit code 1), details reporte...
    system_tests/Failed (1/5 times with errors)
    • job 183526 # Test died: no candidate needle with tag(s) 'x11' matched...
    system_tests/Failed (1/5 times with errors)
    • job 183570 # Test died: command 'curl --form upload=@nose2-junit.xml --form up...
    system_tests/Failed (1/5 times with errors)
    • job 182430 # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...
    TC_00_Basic/test_210_bootmode_template (1/5 times with errors)
    • job 182430 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
  • system_tests_guivm_vnc_gui_interactive

    guivm_manager/ (1/5 times with errors)
    guivm_manager/Failed (1/5 times with errors)
    • job 183735 # Test died: no candidate needle with tag(s) 'menu-qubes-manager' m...
    update_guivm/Failed (1/5 times with errors)
    • job 183515 # Test died: command '(set -o pipefail; qubesctl --all --show-outpu...
  • system_tests_extra

    system_tests/Fail (3/5 times with errors)
    • job 181263 Tests qubes.tests.extra failed (exit code 1), details reported sepa...
    • job 183532 Tests qubes.tests.extra failed (exit code 1), details reported sepa...
    • job 183752 Tests qubes.tests.extra failed (exit code 1), details reported sepa...
    system_tests/Failed (3/5 times with errors)
    • job 181263 # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...
    • job 183532 # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...
    • job 183752 # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...
    TC_20_Evolution_fedora-43-xfce/test_000_send_receive_signed_encrypted (1/5 times with errors)
    • job 181263 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    TC_00_PDFConverter_fedora-44-xfce/test_002_500_pages (2/5 times with errors)
    • job 183532 AssertionError: 1 != 0 : qvm-convert-pdf failed: b'Sending file to ...
    • job 183752 AssertionError: 1 != 0 : qvm-convert-pdf failed: b'Sending file to ...
    TC_00_PDFConverter_fedora-43-xfce/test_004_cancel_stops_conversion (1/5 times with errors)
    • job 181263 AssertionError: DispVM not cleaned up 10s after cancel: {<DispVM at...
    TC_00_PDFConverter_fedora-44-xfce/test_004_cancel_stops_conversion (1/5 times with errors)
    • job 183532 AssertionError: DispVM not cleaned up 20s after cancel: ['disp4799']
    TC_20_Evolution_fedora-43-xfce/test_010_send_receive_signed_only (1/5 times with errors)
    • job 181263 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    TC_10_Thunderbird_fedora-43-xfce/test_020_send_receive_inline_with_attachment (1/5 times with errors)
    • job 181263 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
  • system_tests_splitgpg

    system_tests/Failed (1/5 times with errors)
    • job 182711 # Test died: command 'testfunc qubes.tests.extra' timed out at qube...
    system_tests/Failed (1/5 times with errors)
    • job 183761 # Test died: command 'curl --form upload=@tests-qubes.tests.extra.l...
    system_tests/wait_serial (1/5 times with errors)
    • job 182711 # Command: testfunc qubes.tests.extra...

Performance Tests

Performance degradation:

26 performance degradations
  • debian-13-xfce_socket: 9.35 🔻 ( previous job: 8.37, degradation: 111.76%)
  • debian-13-xfce_socket-data-duplex: 95.02 🔻 ( previous job: 85.67, degradation: 110.92%)
  • whonix-gateway-18_socket-data-duplex: 123.47 🔻 ( previous job: 90.03, degradation: 137.14%)
  • whonix-workstation-18_socket: 9.16 🔻 ( previous job: 7.66, degradation: 119.54%)
  • dom0_root_seq1m_q1t1_write 3:write_bandwidth_kb: 154192.00 🔻 ( previous job: 199974.00, degradation: 77.11%)
  • dom0_root_rnd4k_q1t1_write 3:write_bandwidth_kb: 3650.00 🔻 ( previous job: 5055.00, degradation: 72.21%)
  • dom0_varlibqubes_rnd4k_q32t1_read 3:read_bandwidth_kb: 14324.00 🔻 ( previous job: 102155.00, degradation: 14.02%)
  • dom0_varlibqubes_rnd4k_q32t1_write 3:write_bandwidth_kb: 3165.00 🔻 ( previous job: 10623.00, degradation: 29.79%)
  • fedora-43-xfce_root_seq1m_q1t1_read 3:read_bandwidth_kb: 204513.00 🔻 ( previous job: 310781.00, degradation: 65.81%)
  • fedora-43-xfce_root_seq1m_q1t1_write 3:write_bandwidth_kb: 42742.00 🔻 ( previous job: 78270.00, degradation: 54.61%)
  • fedora-43-xfce_root_rnd4k_q32t1_read 3:read_bandwidth_kb: 78404.00 🔻 ( previous job: 90877.00, degradation: 86.27%)
  • fedora-43-xfce_private_seq1m_q1t1_read 3:read_bandwidth_kb: 260969.00 🔻 ( previous job: 294295.00, degradation: 88.68%)
  • fedora-43-xfce_private_seq1m_q1t1_write 3:write_bandwidth_kb: 55003.00 🔻 ( previous job: 71992.00, degradation: 76.40%)
  • fedora-43-xfce_private_rnd4k_q32t1_write 3:write_bandwidth_kb: 2447.00 🔻 ( previous job: 3674.00, degradation: 66.60%)
  • fedora-43-xfce_dom0-dispvm-api (mean:19.458): 233.50 🔻 ( previous job: 89.33, degradation: 261.38%)
  • fedora-43-xfce_dom0-dispvm-preload-2-api (mean:7.441): 89.30 🔻 ( previous job: 44.03, degradation: 202.83%)
  • fedora-43-xfce_dom0-dispvm-preload-2-delay-0-api (mean:6.544): 78.52 🔻 ( previous job: 41.41, degradation: 189.61%)
  • fedora-43-xfce_dom0-dispvm-preload-2-delay-minus-1d2-api (mean:7.428): 89.13 🔻 ( previous job: 46.26, degradation: 192.68%)
  • fedora-43-xfce_dom0-dispvm-preload-4-api (mean:4.205): 50.45 🔻 ( previous job: 36.18, degradation: 139.44%)
  • fedora-43-xfce_dom0-dispvm-preload-4-delay-0-api (mean:3.914): 46.96 🔻 ( previous job: 36.15, degradation: 129.91%)
  • fedora-43-xfce_dom0-dispvm-preload-4-delay-minus-1d2-api (mean:4.467): 53.60 🔻 ( previous job: 36.13, degradation: 148.35%)
  • whonix-workstation-18_dom0-dispvm-api (mean:10.198): 122.38 🔻 ( previous job: 97.14, degradation: 125.99%)
  • whonix-workstation-18_dom0-dispvm-gui-api (mean:11.927): 143.12 🔻 ( previous job: 121.86, degradation: 117.45%)
  • whonix-workstation-18_dom0-dispvm-preload-2-api (mean:4.586): 55.03 🔻 ( previous job: 48.39, degradation: 113.72%)
  • whonix-workstation-18_dom0-dispvm-preload-2-delay-0-api (mean:4.72): 56.64 🔻 ( previous job: 48.26, degradation: 117.35%)
  • whonix-workstation-18_dom0-dispvm-preload-4-gui-api (mean:4.875): 58.50 🔻 ( previous job: 52.81, degradation: 110.76%)

Remaining performance tests:

85 tests
  • debian-13-xfce_exec: 8.38 🔻 ( previous job: 7.85, degradation: 106.63%)
  • debian-13-xfce_exec-root: 28.18 🔻 ( previous job: 26.74, degradation: 105.36%)
  • debian-13-xfce_socket-root: 9.38 🔻 ( previous job: 9.13, degradation: 102.73%)
  • debian-13-xfce_exec-data-simplex: 64.30 🔻 ( previous job: 63.05, degradation: 101.98%)
  • debian-13-xfce_exec-data-duplex: 63.34 🔻 ( previous job: 62.45, degradation: 101.42%)
  • debian-13-xfce_exec-data-duplex-root: 77.57 🟢 ( previous job: 78.04, improvement: 99.39%)
  • fedora-43-xfce_exec: 8.89 🟢 ( previous job: 9.10, improvement: 97.66%)
  • fedora-43-xfce_exec-root: 70.40 🔻 ( previous job: 69.86, degradation: 100.78%)
  • fedora-43-xfce_socket: 9.28 🔻 ( previous job: 8.52, degradation: 108.94%)
  • fedora-43-xfce_socket-root: 8.74 🔻 ( previous job: 7.96, degradation: 109.90%)
  • fedora-43-xfce_exec-data-simplex: 66.50 🔻 ( previous job: 60.82, degradation: 109.33%)
  • fedora-43-xfce_exec-data-duplex: 64.65 🟢 ( previous job: 68.31, improvement: 94.64%)
  • fedora-43-xfce_exec-data-duplex-root: 99.51 🔻 ( previous job: 98.74, degradation: 100.78%)
  • fedora-43-xfce_socket-data-duplex: 87.27 🟢 ( previous job: 91.81, improvement: 95.06%)
  • whonix-gateway-18_exec: 7.35 🟢 ( previous job: 8.01, improvement: 91.82%)
  • whonix-gateway-18_exec-root: 127.83 🔻 ( previous job: 126.14, degradation: 101.34%)
  • whonix-gateway-18_socket: 8.50 🟢 ( previous job: 8.63, improvement: 98.49%)
  • whonix-gateway-18_socket-root: 8.78 🔻 ( previous job: 8.37, degradation: 104.90%)
  • whonix-gateway-18_exec-data-simplex: 64.35 🔻 ( previous job: 62.50, degradation: 102.96%)
  • whonix-gateway-18_exec-data-duplex: 66.92 🟢 ( previous job: 68.01, improvement: 98.39%)
  • whonix-gateway-18_exec-data-duplex-root: 128.04 🟢 ( previous job: 137.28, improvement: 93.26%)
  • whonix-workstation-18_exec: 8.94 🔻 ( previous job: 8.14, degradation: 109.90%)
  • whonix-workstation-18_exec-root: 144.36 🔻 ( previous job: 144.02, degradation: 100.23%)
  • whonix-workstation-18_socket-root: 9.24 🔻 ( previous job: 8.87, degradation: 104.23%)
  • whonix-workstation-18_exec-data-simplex: 62.63 🟢 ( previous job: 66.36, improvement: 94.38%)
  • whonix-workstation-18_exec-data-duplex: 67.64 🔻 ( previous job: 65.56, degradation: 103.17%)
  • whonix-workstation-18_exec-data-duplex-root: 131.53 🔻 ( previous job: 130.67, degradation: 100.66%)
  • whonix-workstation-18_socket-data-duplex: 96.12 🔻 ( previous job: 92.97, degradation: 103.39%)
  • dom0_root_seq1m_q8t1_read 3:read_bandwidth_kb: 375564.00 🔻 ( previous job: 388793.00, degradation: 96.60%)
  • dom0_root_seq1m_q8t1_write 3:write_bandwidth_kb: 173245.00 🟢 ( previous job: 119690.00, improvement: 144.74%)
  • dom0_root_seq1m_q1t1_read 3:read_bandwidth_kb: 438001.00 🟢 ( previous job: 245223.00, improvement: 178.61%)
  • dom0_root_rnd4k_q32t1_read 3:read_bandwidth_kb: 106265.00 🟢 ( previous job: 13948.00, improvement: 761.87%)
  • dom0_root_rnd4k_q32t1_write 3:write_bandwidth_kb: 9936.00 🟢 ( previous job: 6899.00, improvement: 144.02%)
  • dom0_root_rnd4k_q1t1_read 3:read_bandwidth_kb: 8196.00 🔻 ( previous job: 8963.00, degradation: 91.44%)
  • dom0_varlibqubes_seq1m_q8t1_read 3:read_bandwidth_kb: 489302.00 🟢 ( previous job: 429040.00, improvement: 114.05%)
  • dom0_varlibqubes_seq1m_q8t1_write 3:write_bandwidth_kb: 252486.00 🔻 ( previous job: 257066.00, degradation: 98.22%)
  • dom0_varlibqubes_seq1m_q1t1_read 3:read_bandwidth_kb: 455111.00 🟢 ( previous job: 446202.00, improvement: 102.00%)
  • dom0_varlibqubes_seq1m_q1t1_write 3:write_bandwidth_kb: 185104.00 🔻 ( previous job: 201978.00, degradation: 91.65%)
  • dom0_varlibqubes_rnd4k_q1t1_read 3:read_bandwidth_kb: 10717.00 🟢 ( previous job: 8179.00, improvement: 131.03%)
  • dom0_varlibqubes_rnd4k_q1t1_write 3:write_bandwidth_kb: 4396.00 🔻 ( previous job: 4404.00, degradation: 99.82%)
  • fedora-43-xfce_root_seq1m_q8t1_read 3:read_bandwidth_kb: 383672.00 🔻 ( previous job: 415113.00, degradation: 92.43%)
  • fedora-43-xfce_root_seq1m_q8t1_write 3:write_bandwidth_kb: 140716.00 🟢 ( previous job: 138144.00, improvement: 101.86%)
  • fedora-43-xfce_root_rnd4k_q32t1_write 3:write_bandwidth_kb: 1791.00 🟢 ( previous job: 1519.00, improvement: 117.91%)
  • fedora-43-xfce_root_rnd4k_q1t1_read 3:read_bandwidth_kb: 8247.00 🔻 ( previous job: 8769.00, degradation: 94.05%)
  • fedora-43-xfce_root_rnd4k_q1t1_write 3:write_bandwidth_kb: 521.00 🟢 ( previous job: 262.00, improvement: 198.85%)
  • fedora-43-xfce_private_seq1m_q8t1_read 3:read_bandwidth_kb: 366506.00 🟢 ( previous job: 355690.00, improvement: 103.04%)
  • fedora-43-xfce_private_seq1m_q8t1_write 3:write_bandwidth_kb: 131949.00 🟢 ( previous job: 130275.00, improvement: 101.28%)
  • fedora-43-xfce_private_rnd4k_q32t1_read 3:read_bandwidth_kb: 93309.00 🟢 ( previous job: 81357.00, improvement: 114.69%)
  • fedora-43-xfce_private_rnd4k_q1t1_read 3:read_bandwidth_kb: 7751.00 🔻 ( previous job: 8429.00, degradation: 91.96%)
  • fedora-43-xfce_private_rnd4k_q1t1_write 3:write_bandwidth_kb: 980.00 🟢 ( previous job: 567.00, improvement: 172.84%)
  • fedora-43-xfce_volatile_seq1m_q8t1_read 3:read_bandwidth_kb: 380056.00 🟢 ( previous job: 378957.00, improvement: 100.29%)
  • fedora-43-xfce_volatile_seq1m_q8t1_write 3:write_bandwidth_kb: 152420.00 🟢 ( previous job: 95037.00, improvement: 160.38%)
  • fedora-43-xfce_volatile_seq1m_q1t1_read 3:read_bandwidth_kb: 349408.00 🟢 ( previous job: 202428.00, improvement: 172.61%)
  • fedora-43-xfce_volatile_seq1m_q1t1_write 3:write_bandwidth_kb: 86543.00 🟢 ( previous job: 26016.00, improvement: 332.65%)
  • fedora-43-xfce_volatile_rnd4k_q32t1_read 3:read_bandwidth_kb: 77266.00 🔻 ( previous job: 79218.00, degradation: 97.54%)
  • fedora-43-xfce_volatile_rnd4k_q32t1_write 3:write_bandwidth_kb: 6277.00 🟢 ( previous job: 1720.00, improvement: 364.94%)
  • fedora-43-xfce_volatile_rnd4k_q1t1_read 3:read_bandwidth_kb: 8453.00 🔻 ( previous job: 8667.00, degradation: 97.53%)
  • fedora-43-xfce_volatile_rnd4k_q1t1_write 3:write_bandwidth_kb: 1027.00 🟢 ( previous job: 548.00, improvement: 187.41%)
  • debian-13-xfce_dom0-dispvm-api (mean:6.718): 80.61 🔻 ( previous job: 74.50, degradation: 108.21%)
  • debian-13-xfce_dom0-dispvm-gui-api (mean:8.104): 97.25 🔻 ( previous job: 93.50, degradation: 104.02%)
  • debian-13-xfce_dom0-dispvm-preload-2-api (mean:3.326): 39.91 🔻 ( previous job: 37.79, degradation: 105.63%)
  • debian-13-xfce_dom0-dispvm-preload-2-delay-0-api (mean:3.078): 36.93 🔻 ( previous job: 34.29, degradation: 107.70%)
  • debian-13-xfce_dom0-dispvm-preload-2-delay-minus-1d2-api (mean:3.393): 40.71 🔻 ( previous job: 38.52, degradation: 105.70%)
  • debian-13-xfce_dom0-dispvm-preload-4-api (mean:2.331): 27.97 🟢 ( previous job: 30.73, improvement: 90.99%)
  • debian-13-xfce_dom0-dispvm-preload-4-delay-0-api (mean:2.687): 32.24 🔻 ( previous job: 29.50, degradation: 109.29%)
  • debian-13-xfce_dom0-dispvm-preload-4-delay-minus-1d2-api (mean:2.557): 30.68 🔻 ( previous job: 28.74, degradation: 106.76%)
  • debian-13-xfce_dom0-dispvm-preload-2-gui-api (mean:4.842): 58.10 🔻 ( previous job: 53.65, degradation: 108.30%)
  • debian-13-xfce_dom0-dispvm-preload-4-gui-api (mean:3.686): 44.23 🟢 ( previous job: 44.40, improvement: 99.63%)
  • debian-13-xfce_dom0-dispvm-preload-6-gui-api (mean:3.41): 40.92 🟢 ( previous job: 41.47, improvement: 98.69%)
  • debian-13-xfce_dom0-vm-api (mean:0.03): 0.37 🟢 ( previous job: 0.62, improvement: 59.03%)
  • debian-13-xfce_dom0-vm-gui-api (mean:0.032): 0.39 🟢 ( previous job: 0.47, improvement: 82.80%)
  • fedora-43-xfce_dom0-dispvm-gui-api (mean:10.106): 121.27 🔻 ( previous job: 112.35, degradation: 107.94%)
  • fedora-43-xfce_dom0-dispvm-preload-2-gui-api (mean:5.323): 63.88 🟢 ( previous job: 63.91, improvement: 99.94%)
  • fedora-43-xfce_dom0-dispvm-preload-4-gui-api (mean:4.204): 50.44 🟢 ( previous job: 52.06, improvement: 96.90%)
  • fedora-43-xfce_dom0-dispvm-preload-6-gui-api (mean:3.502): 42.02 🟢 ( previous job: 46.35, improvement: 90.65%)
  • fedora-43-xfce_dom0-vm-api (mean:0.034): 0.40 🟢 ( previous job: 0.47, improvement: 86.67%)
  • fedora-43-xfce_dom0-vm-gui-api (mean:0.037): 0.44 🟢 ( previous job: 0.46, improvement: 96.70%)
  • whonix-workstation-18_dom0-dispvm-preload-2-delay-minus-1d2-api (mean:5.184): 62.20 🔻 ( previous job: 59.22, degradation: 105.05%)
  • whonix-workstation-18_dom0-dispvm-preload-4-api (mean:3.508): 42.10 🔻 ( previous job: 38.29, degradation: 109.96%)
  • whonix-workstation-18_dom0-dispvm-preload-4-delay-0-api (mean:3.576): 42.91 🟢 ( previous job: 43.23, improvement: 99.25%)
  • whonix-workstation-18_dom0-dispvm-preload-4-delay-minus-1d2-api (mean:3.595): 43.13 🔻 ( previous job: 42.67, degradation: 101.10%)
  • whonix-workstation-18_dom0-dispvm-preload-2-gui-api (mean:5.88): 70.56 🟢 ( previous job: 72.05, improvement: 97.94%)
  • whonix-workstation-18_dom0-dispvm-preload-6-gui-api (mean:4.239): 50.87 🔻 ( previous job: 50.72, degradation: 100.29%)
  • whonix-workstation-18_dom0-vm-api (mean:0.037): 0.44 🔻 ( previous job: 0.43, degradation: 101.62%)
  • whonix-workstation-18_dom0-vm-gui-api (mean:0.043): 0.51 🟢 ( previous job: 0.56, improvement: 92.09%)

@andersonarc

Copy link
Copy Markdown
Contributor Author

Tagging @marmarek

@andersonarc

Copy link
Copy Markdown
Contributor Author

New, related finding: The init script inside the VM initramfs always executes wrong code due to this. The "Qubes: Doing R/W setup for TemplateVM..." part is always executed, even in AppVMs - from listing /dev/xvdc*, /dev/xvdc1 and /dev/xvdc3 exist, confirming this. The reason this does not fail is that there is another CoW layer done by create-snapshot.sh which handles CoW on the dom0 side. The side effect is that dom0 root-cow.img images are used for CoW, which is unnoticeable unless you're running Qubes as a live/volatile system, where this becomes a problem. After the fix, the correct path is chosen which sets up /dev/xvdc1 and /dev/xvdc2 for AppVMs and writes to the volatile device selected in qubes.xml. I believe this is more appropriate, since in the other scenario the root-cow.img is created in the pool which owns the TemplateVM and not the AppVM, so you end up with one pool essentially absorbing writes from another. While with this fix, the AppVM's pool remains self-contained with volatile writes going to the volatile image.

@marmarek

Copy link
Copy Markdown
Member

Nice catch, I'm surprised it didn't exploded in more ways before!

@marmarek marmarek merged commit 66ca68a into QubesOS:main Jun 18, 2026
4 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants