bootloader-update service failed after update to FCOS 43 based image

[marys1234]

Describe the bug

Hi
After upgrading FCOS based ucore-hci:stable (https://github.com/ublue-os/ucore) form version: 42.20251012.3.0 to 43.20251024.3.0, bootloader-update service is failing to start during boot:

`bootloader-update.service - Update bootloader on boot
     Loaded: loaded (/usr/lib/systemd/system/bootloader-update.service; enabled; preset: enabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: failed (Result: exit-code) since Tue 2025-11-25 15:05:25 CET; 23h ago
 Invocation: 286cac2dde87435bbc6e1aade5506f8b
       Docs: https://github.com/coreos/bootupd
   Main PID: 1721 (code=exited, status=1/FAILURE)
   Mem peak: 49.7M
        CPU: 228ms

Nov 25 15:05:24 ser systemd[1]: Starting bootloader-update.service - Update bootloader on boot...
Nov 25 15:05:25 ser bootupctl[1721]: Installing for i386-pc platform.
Nov 25 15:05:25 ser bootupctl[1721]: /usr/sbin/grub2-install: warning: ../grub-core/partmap/gpt.c:194:this GPT partition label contains no BIOS Boot Partition; embedding won't be possible.
Nov 25 15:05:25 ser bootupctl[1721]: /usr/sbin/grub2-install: error: embedding is not possible, but this is required for RAID and LVM install.
Nov 25 15:05:25 ser bootupctl[1721]: error: Failed to update BIOS: Failed to run "/usr/sbin/grub2-install" "--target" "i386-pc" "--boot-directory" "/boot" "--modules" "mdraid1x part_gpt" "/>
Nov 25 15:05:25 ser systemd[1]: bootloader-update.service: Main process exited, code=exited, status=1/FAILURE
Nov 25 15:05:25 ser systemd[1]: bootloader-update.service: Failed with result 'exit-code'.
Nov 25 15:05:25 ser systemd[1]: Failed to start bootloader-update.service - Update bootloader on boot.`

On my x86, UEFI based server, I have md raid partitions for "boot" and "root" on two GPT disks (nvme0n1 and nvme1n1) and have also three MBR based sdX devices attached and mounted:

`lsblk 
NAME        MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
sda           8:0    0   1.9T  0 disk  
└─sda1        8:1    0   1.9T  0 part  /var/Vol1b
sdb           8:16   0 931.5G  0 disk  
└─sdb1        8:17   0 931.5G  0 part  /var/Bkpd
sdc           8:32   0 931.5G  0 disk  
└─sdc1        8:33   0 931.5G  0 part  /var/Bkps
nvme1n1     259:0    0 931.5G  0 disk  
├─nvme1n1p1 259:2    0   127M  0 part  
├─nvme1n1p2 259:3    0   384M  0 part  
│ └─md126     9:126  0 383.9M  0 raid1 /boot
├─nvme1n1p3 259:4    0    64G  0 part  
│ └─md127     9:127  0  63.9G  0 raid1 /var/lib/containers/storage/overlay
│                                      /usr/bin/swtpm
│                                      /var
│                                      /sysroot/ostree/deploy/fedora-coreos/var
│                                      /sysroot
│                                      /etc
├─nvme1n1p4 259:5    0     5G  0 part  [SWAP]
└─nvme1n1p5 259:6    0   862G  0 part  /var/Vol1
nvme0n1     259:1    0 931.5G  0 disk  
├─nvme0n1p1 259:7    0   127M  0 part  
├─nvme0n1p2 259:8    0   384M  0 part  
│ └─md126     9:126  0 383.9M  0 raid1 /boot
├─nvme0n1p3 259:9    0    64G  0 part  
│ └─md127     9:127  0  63.9G  0 raid1 /var/lib/containers/storage/overlay
│                                      /usr/bin/swtpm
│                                      /var
│                                      /sysroot/ostree/deploy/fedora-coreos/var
│                                      /sysroot
│                                      /etc
├─nvme0n1p4 259:10   0     5G  0 part  [SWAP]
└─nvme0n1p5 259:11   0   862G  0 part  /var/Vol2`

Not sure if the problem origin is ucore or baseline FCOS, so the issue is raised also for ucore: 325
Nevertheless please check the problem, as it looks serious and may potentially prevent booting on one of next restarts.

Reproduction steps

Boot FCOS 43 on UEDI based system with md raid on boot partition,

Expected behavior

bootloader-update systemd service is starting correctly during boot and properly updating bootloader.

Actual behavior

Systemd service bootloader-update failed to start during boot.

System details

BareMetal UEFI based x64 server.
FCOS 43.20251024.3.0

Butane or Ignition config

Additional information

No response

[dustymabe]

@HuijingHei could you look at this?

[HuijingHei]

@HuijingHei could you look at this?

Sure, I will take this.

Thank you @marys1234 for the reporting and detail info! bootupd support to update bootloader on RAID1 since 0.2.28.

Additionally, I can reproduce locally when I upgrade from 42.20251012.3.0 to 43.20251110.3.1 with only EFI (without BIOS), looks the BIOS actually is not installed as there is no BIOS-BOOT partition, but the bootupd-status.json includes the BIOS part in my environment.

Could you help to share the output of # sudo bootupctl status? Thank you!

[dustymabe]

Maybe this problem is already fixed in newer bootupd by #1010 ?

[HuijingHei]

Maybe this problem is already fixed in newer bootupd by coreos/bootupd#1010 ?

In my testing, upgrade to 0.2.32 and hit the same error.

Not sure if this is because bootupd-status.json includes the BIOS part, and bootupd upgrade does not check it.
In my testing env, start FCOS VM with removed the bios part in the ignition config using mirror, and the bootupd-status.json does not change, so it includes both.

[root@tutorial core]# rpm -q bootupd
bootupd-0.2.32-1.fc44.x86_64
[root@tutorial core]# bootupctl update -vvvv
[TRACE bootupd] executing cli
Running as unit: bootupd.service
[TRACE bootupd] executing cli
[DEBUG bootupd::blockdev] Found parent devices: ["/dev/vda", "/dev/vdb"]
[TRACE bootupd::bootupd] Gathering status for installed component: BIOS
[TRACE bootupd::bootupd] Gathering status for installed component: EFI
[DEBUG bootupd::efi] Unmounting
[TRACE bootupd::bootupd] Remaining known components: 0
Installing for i386-pc platform.
/usr/sbin/grub2-install: warning: ../grub-core/partmap/gpt.c:194:this GPT partition label contains no BIOS Boot Partition; embedding won't be possible.
/usr/sbin/grub2-install: error: embedding is not possible, but this is required for RAID and LVM install.
error: Failed to update BIOS: Failed to run "/usr/sbin/grub2-install" "--target" "i386-pc" "--boot-directory" "/boot" "--modules" "mdraid1x part_gpt" "/dev/vda"

[root@tutorial core]# bootupctl status
Running as unit: bootupd.service
Component BIOS
  Installed: grub2-tools-1:2.12-32.fc42.x86_64
  Update: Available: grub2-tools-1:2.12-40.fc43.x86_64
Component EFI
  Installed: grub2-efi-x64-1:2.12-32.fc42.x86_64,shim-x64-15.8-3.x86_64
  Update: Available: grub2-efi-x64-1:2.12-40.fc43.x86_64,shim-x64-15.8-3.x86_64
No components are adoptable.
CoreOS aleph version: 42.20251012.3.0
Boot method: EFI

[marys1234]

Could you help to share the output of # sudo bootupctl status? Thank you!

Here you are:

core@ser:~$ sudo bootupctl status
Running as unit: bootupd.service
Component BIOS
  Installed: grub2-tools-1:2.12-28.fc42.x86_64
  Update: Available: grub2-tools-1:2.12-40.fc43.x86_64
Component EFI
  Installed: grub2-efi-x64-1:2.12-28.fc42.x86_64,shim-x64-15.8-3.x86_64
  Update: Available: grub2-efi-x64-1:2.12-40.fc43.x86_64,shim-x64-15.8-3.x86_64
No components are adoptable.
CoreOS aleph version: 42.20250410.3.2
Boot method: EFI

[HuijingHei]

Here you are:

core@ser:~$ sudo bootupctl status
Running as unit: bootupd.service
Component BIOS
  Installed: grub2-tools-1:2.12-28.fc42.x86_64
  Update: Available: grub2-tools-1:2.12-40.fc43.x86_64
Component EFI
  Installed: grub2-efi-x64-1:2.12-28.fc42.x86_64,shim-x64-15.8-3.x86_64
  Update: Available: grub2-efi-x64-1:2.12-40.fc43.x86_64,shim-x64-15.8-3.x86_64
No components are adoptable.
CoreOS aleph version: 42.20250410.3.2
Boot method: EFI

Thank you! This is bug in bootupd, should skip BIOS upgrade and remove the installed part bootupd-state.json in this case.

The workaround is remove "BIOS":{"meta":{"timestamp":"2025-06-17T16:35:00Z","version":"grub2-tools-1:2.12-32.fc42.x86_64","versions":[{"name":"grub2","rpm_evr":"1:2.12-32.fc42"}]},"filetree":null,"adopted-from":null}, part in bootupd-state.json, and restart bootloader-update.service.

sudo mount -o remount,rw /boot
sudo vi /boot/bootupd-state.json

[marys1234]

Confirming, that after applied above workaround, bootloader-update.service is successful:

sudo systemctl status bootloader-update.service 
● bootloader-update.service - Update bootloader on boot
     Loaded: loaded (/usr/lib/systemd/system/bootloader-update.service; enabled; preset: enabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: active (exited) since Wed 2025-12-03 15:10:42 CET; 31s ago
 Invocation: c45130c7ee66457785d40d443e932bcc
       Docs: https://github.com/coreos/bootupd
    Process: 1991235 ExecStart=/usr/bin/bootupctl update (code=exited, status=0/SUCCESS)
   Main PID: 1991235 (code=exited, status=0/SUCCESS)
   Mem peak: 24.1M
        CPU: 165ms

Dec 03 15:10:41 ser systemd[1]: Starting bootloader-update.service - Update bootloader on boot...
Dec 03 15:10:42 ser bootupctl[1991235]: Previous EFI: grub2-efi-x64-1:2.12-28.fc42.x86_64,shim-x64-15>
Dec 03 15:10:42 ser bootupctl[1991235]: Updated EFI: grub2-efi-x64-1:2.12-40.fc43.x86_64,shim-x64-15.>
Dec 03 15:10:42 ser systemd[1]: Finished bootloader-update.service - Update bootloader on boot.

$ sudo bootupctl status
Running as unit: bootupd.service
Component EFI
  Installed: grub2-efi-x64-1:2.12-40.fc43.x86_64,shim-x64-15.8-3.x86_64
  Update: At latest version
Detected: BIOS: 42.20250410.3.2
CoreOS aleph version: 42.20250410.3.2
Boot method: EFI

Thank You!

[dustymabe]

This is bug in bootupd, should skip BIOS upgrade and remove the installed part

Can you file that bug including a reproducer? I'm really interested to know the details. We'd ideally not have to ask users to edit bootupd-state.json when we develop a fix for the bug.

[HuijingHei]

This is bug in bootupd, should skip BIOS upgrade and remove the installed part

Can you file that bug including a reproducer? I'm really interested to know the details. We'd ideally not have to ask users to edit bootupd-state.json when we develop a fix for the bug.

Sure, create bug https://issues.redhat.com/browse/RHEL-133325 to track this.

Agree with you, see the output of lsblk, there is no bios-boot part, so ideally should not install BIOS, but it has, guess that old version of bootupd in some cases installed it?

nvme1n1     259:0    0 931.5G  0 disk  
├─nvme1n1p1 259:2    0   127M  0 part   ----> esp part
├─nvme1n1p2 259:3    0   384M  0 part  
│ └─md126     9:126  0 383.9M  0 raid1 /boot
├─nvme1n1p3 259:4    0    64G  0 part  
│ └─md127     9:127  0  63.9G  0 raid1 /var/lib/containers/storage/overlay
│                                      /usr/bin/swtpm
│                                      /var
│                                      /sysroot/ostree/deploy/fedora-coreos/var
│                                      /sysroot
│                                      /etc

[dustymabe]

Sure, create bug https://issues.redhat.com/browse/RHEL-133325 to track this.

A bug against the upstream repo would be useful too I think.

Agree with you, see the output of lsblk, there is no bios-boot part, so ideally should not install BIOS, but it has, guess that old version of bootupd in some cases installed it?

Since we know CoreOS aleph version: 42.20250410.3.2 we can test that theory. I'd really like to know here exactly what was a contributing factor.

[HuijingHei]

Install fedora-coreos-42.20250410.3.2-live-iso.x86_64.iso on multipath env (refer to doc), by default it creates bios part and so install it.

cosa run -c --qemu-iso data/fedora-coreos-42.20250410.3.2-live-iso.x86_64.iso --add-disk 10G:mpath

[core@cosa-devsh ~]$ sudo mpathconf --enable
[core@cosa-devsh ~]$ sudo systemctl start multipathd
[core@cosa-devsh ~]$ sudo coreos-installer install /dev/mapper/mpatha \
    --ignition /run/ignition.json \
    --append-karg rd.multipath=default \
    --append-karg root=/dev/disk/by-label/dm-mpath-root \
    --append-karg rw
Installing Fedora CoreOS 42.20250410.3.2 x86_64 (512-byte sectors)
> Read disk 2.6 GiB/2.6 GiB (100%)     
Writing Ignition config
Modifying kernel arguments
Install complete.
[core@cosa-devsh ~]$ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
loop0         7:0    0   3.9G  0 loop  /run/ephemeral
                                       /var
                                       /etc
loop1         7:1    0 679.5M  1 loop  /usr
                                       /boot
                                       /
                                       /sysroot
sda           8:0    0    10G  0 disk  
└─mpatha    252:0    0    10G  0 mpath 
  ├─mpatha1 252:1    0     1M  0 part  
  ├─mpatha2 252:2    0   127M  0 part  
  ├─mpatha3 252:3    0   384M  0 part  
  └─mpatha4 252:4    0   2.1G  0 part  
sdb           8:16   0    10G  0 disk  
└─mpatha    252:0    0    10G  0 mpath 
  ├─mpatha1 252:1    0     1M  0 part  
  ├─mpatha2 252:2    0   127M  0 part  
  ├─mpatha3 252:3    0   384M  0 part  
  └─mpatha4 252:4    0   2.1G  0 part  
sr0          11:0    1   800M  0 rom   /run/media/iso
vda         253:0    0    12G  0 disk  

@marys1234 have you updated the ignition config with custom partition that removed bios part when installing on x86_64 server?

[marys1234]

It was some time ago, so don't have all details on top of my head, but there where a lot of trails and corrections, so this is possible that I removed unneeded parts from ignition (as bios partition).