FS#77780 - [systemd] boot fails, cannot find PARTUUID (systemd-boot with BTRFS-Subvolumes)

Attached to Project: Arch Linux
Opened by Peter Weber (hoschi) - Wednesday, 08 March 2023, 14:10 GMT
Last edited by Toolybird (Toolybird) - Thursday, 09 March 2023, 19:59 GMT
Task Type Bug Report
Category Packages: Core
Status Closed
Assigned To Christian Hesse (eworm)
Architecture All
Severity Critical
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 1
Private No

Details

Description:
Hello!

Today I've upgraded from 253.1-1 to 253.1-2 which rendered my system unbootable. Upon boot systemd failed to mount the BTRFS subvolume of the root directory.
The PARTUUID is not the one usually found in /etc/fstab but with blkid (look for PARTUUID).

Please take into account that the system was recently installed with archinstall, using BTRFS-Subvolumes with systemd-boot.
With both I'm not familiar. Maybe I did something stupid? These were the default settings presented by archinstall.
Usually I'm installing manually with EXT4 and GRUB2.




Additional info:
* package version(s): systemd-253.1-2, systemd-libs-253.1-2, systemd-sysvcompat-253.1-2
* config and/or log files etc.:

[2023-03-08T13:41:11+0100] [PACMAN] Running 'pacman -Syyu'
[2023-03-08T13:41:11+0100] [PACMAN] synchronizing package lists
[2023-03-08T13:41:12+0100] [PACMAN] starting full system upgrade
[2023-03-08T13:41:43+0100] [ALPM] transaction started
[2023-03-08T13:41:43+0100] [ALPM] upgraded systemd-libs (253.1-1 -> 253.1-2)
[2023-03-08T13:41:44+0100] [ALPM] upgraded systemd (253.1-1 -> 253.1-2)
[2023-03-08T13:41:45+0100] [ALPM] upgraded libedit (20210910_3.1-2 -> 20221030_3.1-1)
[2023-03-08T13:41:45+0100] [ALPM] upgraded sdl2 (2.26.3-1 -> 2.26.4-1)
[2023-03-08T13:41:45+0100] [ALPM] upgraded gtk-update-icon-cache (1:4.8.3-3 -> 1:4.10.0-3)
[2023-03-08T13:41:45+0100] [ALPM] upgraded chromium (110.0.5481.177-1 -> 111.0.5563.64-1)
[2023-03-08T13:41:45+0100] [ALPM] upgraded libwpe (1.14.1-1 -> 1.14.1-2)
[2023-03-08T13:41:45+0100] [ALPM] upgraded gnome-photos (1:43.0-2 -> 1:44.0-1)
[2023-03-08T13:41:45+0100] [ALPM] upgraded gtk4 (1:4.8.3-3 -> 1:4.10.0-3)
[2023-03-08T13:41:45+0100] [ALPM] upgraded gtk4-demos (1:4.8.3-3 -> 1:4.10.0-3)
[2023-03-08T13:41:46+0100] [ALPM] upgraded gtk4-docs (1:4.8.3-3 -> 1:4.10.0-3)
[2023-03-08T13:41:46+0100] [ALPM] upgraded inetutils (2.3-1 -> 2.4-1)
[2023-03-08T13:41:46+0100] [ALPM] upgraded libbpf (1.0.1-1 -> 1.1.0-1)
[2023-03-08T13:41:46+0100] [ALPM] upgraded lmdb (0.9.30-2 -> 0.9.30-3)
[2023-03-08T13:41:46+0100] [ALPM] upgraded phodav (3.0-1 -> 3.0-2)
[2023-03-08T13:41:46+0100] [ALPM] upgraded python-urllib3 (1.26.12-1 -> 1.26.13-1)
[2023-03-08T13:41:46+0100] [ALPM] installed spice-protocol (0.14.4-1)
[2023-03-08T13:41:46+0100] [ALPM] upgraded spice-gtk (0.42-1 -> 0.42-3)
[2023-03-08T13:41:46+0100] [ALPM] upgraded systemd-sysvcompat (253.1-1 -> 253.1-2)

* link to upstream bug report, if any: none

Preconditions:
* root filesystem is btrfs with subvolumes
* bootloader is systemd-boot

The archinstall-script will give you this setup as default on an UEFI-System.

Steps to reproduce:
1. pacman -Syu
2. reboot

Workaround/Recovery:
The command arch-chroot didn't worked for me. And the usual chroot also not because the subvolumes of BTRFS makes things more complicated. I roughly followed[1] this instructions.
Basically look first at hour /etc/fstab and then:
# mount /dev/${REPLACEDEVICE} /mnt/ -t btrfs -o subvol=/@
# mount /dev/${REPLACEDEVICE} /mnt/home -t btrfs -o subvol=/@honme
# mount /dev/${REPLACEDEVICE} /mnt//var/cache/pacman/pkg -t btrfs -o subvol=/@pkg
# mount /dev/${REPLACEDEVICE} /mnt/var/log -t btrfs -o subvol=/@log
# mount /dev/${REPLACEDEVICE} /mnt/.snapshots -t btrfs -o subvol=/@.snapshots
# mount /dev/${OTHERDEVICE} /mnt/boot

# now usual stuff
# mount --bind /dev /mnt/dev
# mount -t proc /proc /mnt/proc
# mount -t sysfs /sys /mnt/sys
# mount -t tmpfs tmpfs /mnt/run
# chroot /mnt
# cd /var/cache/pacmang/pkg
# pacman -U ...
# exit
# umount # I skipped...
# reboot

Thank you

[1] https://fedoramagazine.org/os-chroot-101-covering-btrfs-subvolumes/
This task depends upon

Closed by  Toolybird (Toolybird)
Thursday, 09 March 2023, 19:59 GMT
Reason for closing:  None
Additional comments about closing:  Reporter says "mkinitcpio fails due to hardware problem, corrupted initramfs. nothing to do with systemd!"
Comment by Peter Weber (hoschi) - Wednesday, 08 March 2023, 14:16 GMT
I forget to mention that the rescue shell was not usable. It didn't accepted any input.
Comment by Christian Hesse (eworm) - Wednesday, 08 March 2023, 14:28 GMT
There is no change in systemd 253.1-2 that would explain this... Anybody has an idea?
Comment by Peter Weber (hoschi) - Wednesday, 08 March 2023, 14:36 GMT
I also wonder. Your recent changes to the packages don't look dangerous. Should I run `# bootctl update` after the update?
Comment by Peter Weber (hoschi) - Wednesday, 08 March 2023, 14:48 GMT
https://unix.stackexchange.com/q/738829

Not from me but timing matches.

Archinstall didn't setup an XBOOTLDR here:

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
zram0 254:0 0 4G 0 disk [SWAP]
nvme0n1 259:0 0 476.9G 0 disk
├─nvme0n1p1 259:1 0 511M 0 part /boot
└─nvme0n1p2 259:2 0 476.4G 0 part /var/log
/var/cache/pacman/pkg
/home
/.snapshots
/
Comment by Peter Weber (hoschi) - Wednesday, 08 March 2023, 15:24 GMT
This also looks similar but I'm using AMD not Intel:
https://bbs.archlinux.org/viewtopic.php?id=284062

// edit
Probably same user as on Stackexchange.
Comment by Christian Hesse (eworm) - Wednesday, 08 March 2023, 15:37 GMT Comment by Peter Weber (hoschi) - Wednesday, 08 March 2023, 15:39 GMT
I've seen that but the machine is using an AMD-Processor.
btw. same for the other affect user
Comment by Peter Weber (hoschi) - Thursday, 09 March 2023, 09:36 GMT
I found one glitch in the update after which the boot failed:

[2023-03-08T13:41:47+0100] [ALPM-SCRIPTLET] ==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'default'
[2023-03-08T13:41:47+0100] [ALPM-SCRIPTLET] -> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux.img
[2023-03-08T13:41:48+0100] [ALPM-SCRIPTLET] ==> Starting build: 6.2.2-arch1-1
[2023-03-08T13:41:48+0100] [ALPM-SCRIPTLET] -> Running build hook: [base]
[2023-03-08T13:41:48+0100] [ALPM-SCRIPTLET] -> Running build hook: [udev]
[2023-03-08T13:41:48+0100] [ALPM-SCRIPTLET] -> Running build hook: [autodetect]
[2023-03-08T13:41:48+0100] [ALPM-SCRIPTLET] sort: cannot read: '/sys/devices/pci0000:00/0000:00:08.1/0000:07:00.4/usb3/3-1/3-1.2/3-1.2.2/3-1.2.2:1.2/ep_04/uevent': No such file or directory
[2023-03-08T13:41:48+0100] [ALPM-SCRIPTLET] modprobe: ERROR: missing parameters. See -h.
[2023-03-08T13:41:48+0100] [ALPM-SCRIPTLET] -> Running build hook: [keyboard]
[2023-03-08T13:41:48+0100] [ALPM-SCRIPTLET] -> Running build hook: [keymap]
[2023-03-08T13:41:48+0100] [ALPM-SCRIPTLET] -> Running build hook: [modconf]
[2023-03-08T13:41:48+0100] [ALPM-SCRIPTLET] -> Running build hook: [block]
[2023-03-08T13:41:48+0100] [ALPM-SCRIPTLET] -> Running build hook: [filesystems]
[2023-03-08T13:41:48+0100] [ALPM-SCRIPTLET] -> Running build hook: [fsck]
[2023-03-08T13:41:49+0100] [ALPM-SCRIPTLET] ==> Generating module dependencies
[2023-03-08T13:41:49+0100] [ALPM-SCRIPTLET] ==> Creating zstd-compressed initcpio image: /boot/initramfs-linux.img
[2023-03-08T13:41:49+0100] [ALPM-SCRIPTLET] ==> Image generation successful
[2023-03-08T13:41:49+0100] [ALPM-SCRIPTLET] ==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'fallback'
[2023-03-08T13:41:49+0100] [ALPM-SCRIPTLET] -> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-fallback.img -S autodetect
[2023-03-08T13:41:49+0100] [ALPM-SCRIPTLET] ==> Starting build: 6.2.2-arch1-1
[2023-03-08T13:41:49+0100] [ALPM-SCRIPTLET] -> Running build hook: [base]
[2023-03-08T13:41:49+0100] [ALPM-SCRIPTLET] -> Running build hook: [udev]
[2023-03-08T13:41:49+0100] [ALPM-SCRIPTLET] -> Running build hook: [keyboard]
[2023-03-08T13:41:49+0100] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: xhci_pci
[2023-03-08T13:41:50+0100] [ALPM-SCRIPTLET] -> Running build hook: [keymap]
[2023-03-08T13:41:50+0100] [ALPM-SCRIPTLET] -> Running build hook: [modconf]
[2023-03-08T13:41:50+0100] [ALPM-SCRIPTLET] -> Running build hook: [block]
[2023-03-08T13:41:51+0100] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: aic94xx
[2023-03-08T13:41:51+0100] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: bfa
[2023-03-08T13:41:52+0100] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: qed
[2023-03-08T13:41:52+0100] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: qla1280
[2023-03-08T13:41:52+0100] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: qla2xxx
[2023-03-08T13:41:52+0100] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: wd719x
[2023-03-08T13:41:53+0100] [ALPM-SCRIPTLET] -> Running build hook: [filesystems]
[2023-03-08T13:41:54+0100] [ALPM-SCRIPTLET] -> Running build hook: [fsck]
[2023-03-08T13:41:55+0100] [ALPM-SCRIPTLET] ==> Generating module dependencies
[2023-03-08T13:41:55+0100] [ALPM-SCRIPTLET] ==> Creating zstd-compressed initcpio image: /boot/initramfs-linux-fallback.img
[2023-03-08T13:41:56+0100] [ALPM-SCRIPTLET] ==> Image generation successful

This looks like a faulty USB-Device. Can this caused some side-effect?
Comment by Peter Weber (hoschi) - Thursday, 09 March 2023, 09:37 GMT
Upgrade and:
-> Running build hook: [autodetect]
sort: stat failed: '/sys/devices/pci0000:00/0000:00:08.1/0000:07:00.4/usb4/4-1/4-1.2/4-1.2.4/4-1.2.4:1.1/ep_81/uevent': No such file or directory
modprobe: ERROR: missing parameters. See -h.

Same error. Different USB-Device?
Comment by Peter Weber (hoschi) - Thursday, 09 March 2023, 09:50 GMT
It was not systemd! Sorry Heftig!

It looks like mkinitcpio falls over some weird USB-Device which is provided through my docking-station.
To chroot into my broken system I had to attach a USB-Thumbdrive, therefore I undocked my laptop and the image creation succeeded.

Maybe mkiniticpio can be improved?
So it either fails 'hard' when an error of this kind occurs or handles such issues elegantly?
Comment by AlexDeLorenzo.dev (alexdelorenzo.dev) - Thursday, 09 March 2023, 11:03 GMT
I ran into this exact issue about a week ago.

My UUIDs for my disks/partitions did not match the same UUIDs I had in my fstab or systemd-boot entries. I could not boot and was punted to an initramfs shell with the same error as in the first image attached to the OP.

My fix was to inspect my disk and partitions' UUIDs, and then edit /etc/fstab and my systemd-boot entries with the correct UUID values.

Either my boot config was updated with incorrect UUIDs, or the disks/partitions were given new UUIDs for some reason.

I was using dracut and not mkinitcpio, but I believe I might have had it installed at the time.
Comment by Toolybird (Toolybird) - Thursday, 09 March 2023, 19:58 GMT
> Maybe mkiniticpio can be improved?

It does seem like an odd failure mode. If you can repro and have the time, mkinitcpio bugs can be reported here [1].

[1] https://gitlab.archlinux.org/archlinux/mkinitcpio/mkinitcpio

Loading...