Fedora ostree installer image installs broken on UEFI with 0.2.36 due to #1126

[AdamWill]

When 0.2.36 landed in Fedora Rawhide recently, ostree install tests on UEFI started failing (BIOS installs are still OK). The install apparently succeeds, but the installed system boots to the grub emergency shell.

In the install logs, we see this:

Could not find /boot/efi/EFI when installing "EFI/fedora/grub.cfg"

which looks pretty significant. Indeed if I run an install and do not reboot at the end, but go to a console and check /mnt/sysroot/boot/efi/EFI/fedora (/mnt/sysroot is where the Fedora installer mounts the installed system root) it contains no grub.cfg, which is very likely the issue. (It also seems odd that the failure to install this file is just logged as an info, but the bootupctl command still exits 0?)

On a hunch, I tested a scratch build with #1126 reverted, and it passed tests. So it looks like that PR is the cause, @Johan-Liebert1 .

I'll send an official Fedora build with the PR reverted for now, but it'd be great to get a proper fix for this. CC @cgwalters . To reproduce, just try installing from e.g. https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20260821.n.0/compose/Silverblue/x86_64/iso/Fedora-Silverblue-Installer-Rawhide-20260821.n.0.x86_64.iso on a UEFI system or VM, and booting the installed system.

[Johan-Liebert1]

Hi, could you guide me on how the installation is done here? I tried with bootc install to-disk and though I do not get the error

Could not find /boot/efi/EFI when installing "EFI/fedora/grub.cfg"

I do get grub getting stuck in the emergency shell. In my case that was due to bootuuid.cfg not being included in the ESP due to the following

# /usr/lib/bootc/install/10-boot.toml

# We don't want bootupd to stamp the boot and UEFI partitions
# with the boot uuid because these will change at first boot.
# See https://github.com/coreos/fedora-coreos-config/pull/3898
[install.bootupd]
skip-boot-uuid = true

which ultimately causes bootc to call bootupd as

env -i PATH="/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin" "bootupctl" "backend" "install" "--with-static-configs" "-vvvv" "--filesystem" "/sysroot" "/"

We need --with-uuid to add bootuuid.cfg in the ESP so grub can find the boot partition.

Ref1: coreos/fedora-coreos-config#3898
Ref2: coreos/coreos-assembler@5f003fa

But, this feels like a separate issue than what you're facing. I'm assuming this ISO was created using bootc somehow, and inspecting the ISO I don't find bootuuid.cfg which points to the same issue.

I'll update here once I try out the ISO

[Johan-Liebert1]

Okay, the ISO installation is just stuck at Software Installation for around 20+ minutes

[Rolv-Apneseth]

Can confirm that installing with that ISO leaves us with no grub.cfg and an emergency shell.

@Johan-Liebert1 I had the same issue when trying to install in a VM - I forgot to enable UEFI, which apparently isn't the default.

---

From looking at the code I assume it's this change causing the failure:

- let dest_efidir = target_root
-     .open_dir_optional("boot/efi/EFI")
-     .context("Opening /boot/efi/EFI")?;
- if let Some(efidir) = dest_efidir {
+
+ if let Some(efidir) = mounted_esp {

I guess we should keep the fallback?

And I was trying to figure out how the mounted_esp ends up as None, and the only way I can think of is none of these downcasts match i.e. no Efi component:

let mut esp_dir = None;

#[cfg(efi_arch)]
{
    for c in &target_components {
        use crate::efi::Efi;

        if let Some(efi) = c.as_any().downcast_ref::<Efi>() {
            let mounted_esp_path = efi.get_esp_mountpoint();
            let mounted_esp_path = mounted_esp_path.as_ref().ok_or_else(|| {
                anyhow!("Mounted ESP not found to install static grub config")
            })?;

            esp_dir = Some(
                Dir::open_ambient_dir(mounted_esp_path, ambient_authority())
                    .context("Opening mounted ESP dir")?,
            );

            break;
        }
    }
}

Which could mean something wrong with --auto? Exact call anaconda makes is here.

This failing should probably be a hard error though.

[Johan-Liebert1]

I guess we should keep the fallback?

the reason for my change was to support composefs grub systems. Having the fallback complicates things with an incorrect UUID being written to grubcfg. I'll try to pinpoint the root cause. Thanks for sharing the anaconda snippet. The bootupctl call inside the python script looks correct to me.

@Rolv-Apneseth Could you point me to how to create an ISO using anaconda

[Rolv-Apneseth]

@Rolv-Apneseth Could you point me to how to create an ISO using anaconda

I'm afraid I have no idea how to do that, sorry.

I did do a chroot /mnt/sysroot bootupctl status after finishing the install and confirmed that the set boot method is EFI with the installed component being EFI too so my theory about --auto is invalid. Not really sure how we reach the state required for the error.

[AdamWill]

Okay, the ISO installation is just stuck at Software Installation for around 20+ minutes

So, it does genuinely take a long time anyway, and there's also a bug where the UI gets stuck there even though the install is actually done. You can switch to a console with ctrl-alt-f2 and check /tmp/packaging.log to see if the install has actually finished; if it shows the last two lines I mentioned in the bug report, it's done.

You can look at the other log files in /tmp for some details on the install process - bootloader install happens near the end, obviously - and you can look in /mnt/sysimage to see how the installed system root looks.

Building an ostree installer ISO is unfortunately somewhat complicated. You can see the source of the test for it at https://forge.fedoraproject.org/quality/os-autoinst-distri-fedora/src/branch/main/tests/_ostree_build.pm - that's already quite complicated and it also calls some stuff in https://forge.fedoraproject.org/quality/os-autoinst-distri-fedora/src/branch/main/lib/utils.pm and the https://forge.fedoraproject.org/quality/os-autoinst-distri-fedora/src/branch/main/data/ostree-parse-pungi-ib.py script. So...if you want a specific change tested, or some changes to print more debugging details or anything, maybe the easiest way is if you post it here and I can build the ISO for you?

[AdamWill]

Building a non-ostree installer ISO is rather simpler, but that path doesn't use bootupctl :/ I guess I could see if there's an easy way to change that, and if so, whether the resulting ISO hits the same bug.

[Johan-Liebert1]

I think I have a hunch on what the issue might be. The ISO installation succeeded, unfortunately it rebooted before I could grab the logs. In the final qcow image I have for my VM, I see no ESP

Disk /dev/nbd0: 20 GiB, 21474836480 bytes, 41943040 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 131072 bytes
Disklabel type: gpt
Disk identifier: 41A72B6D-77F8-43AA-A39B-B1D2DAE744A1

Device        Start      End  Sectors Size Type
/dev/nbd0p1    2048     4095     2048   1M BIOS boot
/dev/nbd0p2    4096  4198399  4194304   2G Linux filesystem
/dev/nbd0p3 4198400 41940991 37742592  18G Linux filesystem

NAME     PARTTYPE                             PARTUUID                             SIZE FSTYPE
nbd0                                                                                20G 
├─nbd0p1 21686148-6449-6e6f-744e-656564454649 1ec9d264-274f-4f7e-9d18-950f08ae6776   1M 
├─nbd0p2 0fc63daf-8483-4772-8e79-3d69d8477de4 4e9ffb33-3b69-4ef5-a55b-1ca480c9faa4   2G ext4
└─nbd0p3 0fc63daf-8483-4772-8e79-3d69d8477de4 9e37f731-5c19-41ba-9a83-89c4f7d9e0c8  18G btrfs

The second one is supposed to be the ESP maybe? 4e9ffb33-3b69-4ef5-a55b-1ca480c9faa4 being the partuuid for Generic Linux Data Partition according to https://uapi-group.org/specifications/specs/discoverable_partitions_specification/

So my hunch is bootupd didn't find any ESP (due to no partition with parttype c12a7328-f81f-11d2-ba4b-00a0c93ec93b) and no vfat partition.

Prior to #1126, things worked out because we simply used boot/efi/EFI to place the configs.

What really confuses me is the final qcow after installing having nothing grub related. Is that intentional? I just kept on clicking next on the ISO installer

[Johan-Liebert1]

What really confuses me is the final qcow after installing having nothing grub related

what I mean by this is there is no bootloader at all in the final VM. It gets stuck at "Booting from hard disk..."

[AdamWill]

Um. I don't know what's going on there, but that partition table indicates you got a BIOS install, not a UEFI install. anaconda will only create a BIOS boot partition if it's doing a BIOS install. The three partitions there are the ones anaconda typically creates for a BIOS install to a small disk: a BIOS boot partition, a /boot partition, and a root partition. Are you sure you used a UEFI VM?

[Johan-Liebert1]

Are you sure you used a UEFI VM?

Welp, sorry, I wasn't. I had forgotten to press "apply" to use UEFI.

I tried now with UEFI firmware and my installation is stuck the following for a while now

[Johan-Liebert1]

The installation failed with the same error and after adding a bunch of log statements, building bootupd for f46, using cloudflared to send it into the chrooted system and running bootupctl install, everything succeeds :/

I even tried removing everything from /boot then running the same bootupctl cmd again, and things seem to be in order.

[Johan-Liebert1]

But yeah, most definitely something is not correct when the command is called when the ISO's being installed. @AdamWill could you create an ISO with https://github.com/Johan-Liebert1/bootupd/tree/iso-debug build of bootupd?

[AdamWill]

Here: https://adamwill.fedorapeople.org/06880571-149504956-Silverblue-ostree-x86_64.iso