Arch Linux

Please read this before reporting a bug:

Do NOT report bugs when a package is just outdated, or it is in the AUR. Use the 'flag out of date' link on the package page, or the Mailing List.

REPEAT: Do NOT report bugs for outdated packages!

FS#77569 - [dracut] boot failure with systemd >= 253

Attached to Project: Arch Linux
Opened by Toolybird (Toolybird) - Saturday, 18 February 2023, 23:21 GMT
Last edited by Toolybird (Toolybird) - Thursday, 13 July 2023, 00:35 GMT
Task Type Bug Report
Category Packages: Core
Status Closed
Assigned To Christian Hesse (eworm)
Giancarlo Razzolini (grazzolini)
Architecture All
Severity High
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 4
Private No


This is mainly a heads-up because I haven't figured it out yet. It's the first time in yonks I've had to recover an Arch system with an installation USB flash drive :(

- standard LVM setup (root volume -> no raid, thin, luks or anything like that, just basic)
- dracut
- system fails to boot when systemd-253 is present in the initrd

Removing "quiet" and adding "rd.debug" to kernel params shows it's udev related...seems to be hanging somewhere around ""

Downgrading systemd and regenerating initrd works. Upgrading systemd but leaving initrd containing old systemd also works. Bit of a head-scratcher so far. I'm curious if anyone else is affected.
This task depends upon

Closed by  Toolybird (Toolybird)
Thursday, 13 July 2023, 00:35 GMT
Reason for closing:  Fixed
Additional comments about closing:  dracut 059-2
Comment by Toolybird (Toolybird) - Sunday, 19 February 2023, 04:55 GMT Comment by loqs (loqs) - Sunday, 19 February 2023, 12:27 GMT
If the issue is still present under dracut 059 what if you change the paths that are not made read only [1]? I would suggest trying changing /dev/shm/ to /dev/ if that fails then try adding /etc/ /var/ /usr/.

Comment by Janis König (LeonardK) - Sunday, 19 February 2023, 17:33 GMT
@loqs: I compiled dracut from git to have 059-* instead of 056 but nothing changed.

FWIW, I also run LVM but I do use LUKS on the individual logical volumes, but since this breaks even before cryptsetup comes into play, I'm probably hitting the same bug.

I've uploaded the /run/initramfs/rdsosreport.txt and journalctl output for systemd-253 with rd.debug enabled. While the logs mention linux-hardened, the same behavior can be seen with linux plain as well.

Note that `dmsetup` claims no devices exist, and there are no `/dev/dm*` devices and `/dev/disk/by-uuid` doesn't contain any mapped disks. `lvm` does list the physical and logical volumes as well as the volume groups, but the volume group `vgmain` (in my case) doesn't exist on `/dev/vgmain/*`.

From the log it does seem like lvm_scan is never run, maybe because it doesn't exist at the right place/its hook is not registered or because udev doesn't settle properly? I have no idea about the actual underlying workings unfortunately. I ~~can provide~~ have uploaded a tarball of the contents of a faulty and a correctly generated initramfs ~~but not here~~ on the linked GitHub issue (too big for flyspray)
Comment by loqs (loqs) - Sunday, 19 February 2023, 18:11 GMT
@LeonardK what if you rebuild systemd 253 with the attached patch?
Comment by Toolybird (Toolybird) - Sunday, 19 February 2023, 21:27 GMT
Thanks for the test patch @loqs, but no difference here.
Comment by Janis König (LeonardK) - Sunday, 19 February 2023, 21:40 GMT
Same for me unfortunately :/
Comment by Janis König (LeonardK) - Sunday, 19 February 2023, 21:52 GMT
I also rebuilt adding the other directories you mentioned:

- STRV_MAKE("/sys", "/run", "/proc", "/dev/shm", "/tmp"));
+ STRV_MAKE("/sys", "/run", "/proc", "/dev", "/tmp", "/etc/", "/var", "/usr"));

but unfortunately no change.
Comment by loqs (loqs) - Sunday, 19 February 2023, 22:00 GMT
Can you determine what is left mounted read only?
Skip the whole remount and see if the issue is still present.
Comment by Toolybird (Toolybird) - Sunday, 19 February 2023, 22:35 GMT
@loqs, that patch works. But that is to be expected because it effectively reverts the intent of the original MR. Maybe we just need to fix dracut? Unfortunately, I don't get an emergency shell when it fails so I cannot see the state of things.
Comment by loqs (loqs) - Sunday, 19 February 2023, 22:45 GMT
Something other than /dev /etc /proc /run /sys /tmp /usr /var seems to need write access.
Could you somehow place strace at the top of the two generators and find what call is getting blocked?
I would suggest using kernel audit rules but that would involve getting auditd installed and configured in the initrd.
Try asking upstream dracut?
Comment by Toolybird (Toolybird) - Monday, 20 February 2023, 01:02 GMT Comment by Toolybird (Toolybird) - Monday, 20 February 2023, 04:00 GMT
Upstream systemd produced a patch [1]. Works for me.

Comment by Christian Hesse (eworm) - Friday, 03 March 2023, 14:45 GMT
Upstream systemd will not fix / work around this. Please wait for dracut to fix this.
Comment by Toolybird (Toolybird) - Friday, 03 March 2023, 17:22 GMT
So those of us affected are basically screwed? Fedora have applied the patch [1] as a temporary workaround until dracut is fixed. Any chance you could do the same?

Comment by Christian Hesse (eworm) - Friday, 03 March 2023, 17:47 GMT
Fedora uses dracut by default, we do not. Anyway... A commit exists for dracut - yet to be merged. But perhaps that should be evaluated and pushed to dracut package.
Comment by loqs (loqs) - Friday, 03 March 2023, 20:28 GMT
Seems there are four commits that need to be cherry-picked to 056, which is the last signed release. At least there are no merge commits and the patch applies and builds. That is as far as I tested.
Comment by Christian Hesse (eworm) - Saturday, 04 March 2023, 18:44 GMT
Anybody can test this? I will happily push a fixed dracut package after confirmation.
Comment by Toolybird (Toolybird) - Sunday, 05 March 2023, 04:33 GMT
@loqs, thanks for the patch, it works a treat.

@eworm, please push, thanks.
Comment by Toolybird (Toolybird) - Friday, 07 July 2023, 22:11 GMT
dracut 059-1 just broke this again.

@grazzolini, it appears you dropped the patch that fixed this. The patch is *not* included in 059 upstream.
Comment by Toolybird (Toolybird) - Saturday, 08 July 2023, 00:02 GMT
Our GitLab merge requests don't appear to be "live" yet. But I've tested this [1] and can confirm it solves the problem.