FS#67597 - [dracut] Linux 5.8.1 fails to decrypt LUKS drive with hostonly dracut image

Attached to Project: Arch Linux
Opened by STPR (stpr) - Sunday, 16 August 2020, 01:38 GMT
Last edited by Jelle van der Waa (jelly) - Thursday, 14 September 2023, 17:54 GMT
Task Type Bug Report
Category Packages: Extra
Status Closed
Assigned To Giancarlo Razzolini (grazzolini)
Architecture x86_64
Severity Medium
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 4
Private No

Details

Description:


Additional info:
* Linux Zen 5.7.12 -> Linux Zen 5.8.1, Linux 5.7.12 -> Linux 5.8.1, dracut 050-1
* lsinitrd for both non-fallback and fallback attached.
* unknown

Steps to reproduce:

- Upgrade kernel to 5.8.1.
- Reboot system and try to use the non-fallback initramfs (i.e. --hostonly).
- Prompts for LUKS password, but errors after entering the password, with indications that it was unable to decrypt the drives. Error messages on screen:
"encrypted_key failed to alloc_cipher (-2)."
"table: 254:0 crypt: unknown target type."
"Failed to start Cryptography Setup for system"
- Reboot and use fallback kernel (i.e. --no-hostonly).
- Works fine. Now downgrade kernel to 5.7.12 and rebuilt both initramfs images. Reboot.
- Non-fallback initramfs works fine.

Additional notes:

I can reproduce this for both zen and regular kernels. I took a quick look at the lsinitrd outputs and I did not see any obvious differences. I am lost on why it is not booting. I do not know what other logs to attach, so please let me know if you need anything else.
This task depends upon

Closed by  Jelle van der Waa (jelly)
Thursday, 14 September 2023, 17:54 GMT
Reason for closing:  Deferred
Additional comments about closing:  Old kernel, please retry with the latest
Comment by freswa (frederik) - Sunday, 16 August 2020, 11:39 GMT
Please also provide your dracut.conf
Thanks
Comment by STPR (stpr) - Sunday, 16 August 2020, 16:09 GMT
I do not use any custom dracut *.conf (i.e. no files in /etc/dracut.conf.d/).
Comment by Leopold Bloom (leopoldbloom) - Monday, 17 August 2020, 04:45 GMT
Wanted to comment that I am also getting the same issue with linux 5.8.1-arch1, except I am using mkinitcpio only. Fallback kernel image does not work either. linux-lts is fine.
Comment by Leopold Bloom (leopoldbloom) - Monday, 17 August 2020, 04:48 GMT
After some experimenting and digging, this issue may be a consequence of the dm-crypt kernel module being somehow missing or broken from the linux 5.8.1-arch1 package.
I tried adding the dm-crypt kernel module to the MODULES array in mkinitcpio, and regenerated the kernel images. On linux-lts, same (correct) decryption behavior occurred. Conversely, using linux 5.8.1-arch1, I get an additional systemd error saying that the systemd-modules-load.service unit failed upon boot, right before drive decryption.
Comment by Giancarlo Razzolini (grazzolini) - Monday, 17 August 2020, 10:52 GMT
@STPR,

This is looking more like a systemd issue than a dracut one.

@Leopold

Are you using a systemd based mkinitcpio initramfs or a busybox based? If you're using systemd based, and this is is indeed a systemd issue, then it would explain it affecting both dracut and (systemd base) mkinitcpio.

I have no issues decrypting luks disks on 5.8.1 with a busybox based mkinitcpio initramfs.
Comment by STPR (stpr) - Monday, 17 August 2020, 12:34 GMT
@grazzolini That is a good point - though I upgraded systemd way before I did the Linux packages, and I didn't see any issues. That is why I assumed it was a Linux packages problem. I am still not sure why the fallback works at all in my case, given that the other user wasn't able to work it out al all.

I also realise that there is another bug open that points to systemd having issues (https://bugs.archlinux.org/task/67517), perhaps that might explain it?
Comment by Giancarlo Razzolini (grazzolini) - Monday, 17 August 2020, 12:43 GMT
@STPR,

It could be that  FS#67517  has something to do with it. If you can get the journal from that unit that's failing, it would be helpful to see if it's the full path issue.
Comment by STPR (stpr) - Monday, 17 August 2020, 12:58 GMT
@grazzolini Could you please point out how would I go about doing that? The logs would presumably not be stored anywhere since the root drive wasn't decrypted, and I am unable to get access to the emergency shell to do anything useful. Maybe I am missing something?
Comment by Giancarlo Razzolini (grazzolini) - Monday, 17 August 2020, 13:52 GMT
Add rd.shell rd.debug to your command line, that should be enough. Also, check /run/initramfs/rdsosreport.txt if you do get dropped to the emergency shell.
Comment by STPR (stpr) - Monday, 17 August 2020, 16:12 GMT
@grazzolini Ok, I was able to get shell access by adding rd.shell and waiting for the timeout to drop me into the shell.

From there, I get that two units are failing, with the indication that the units are crypsetup units that handle the decryption of the two LUKS encrypted drives I have on my PC. I have attached the systemctl status log corresponding to the root drive here, but the other one is similar as well.

Additionally, I was able to grab the rdsosreport as well, and I have attached that here. Seems similar to journalctl logs, so I will skip attaching that here.
Comment by Giancarlo Razzolini (grazzolini) - Monday, 17 August 2020, 17:44 GMT
It's looking like your machine is not loading the cryptographic modules. Also, that warning there, I don't remember if it used to happen before or not.
Comment by Leopold Bloom (leopoldbloom) - Tuesday, 18 August 2020, 18:52 GMT
Just a quick update: I tried removing the systemd-related hooks in mkinitcpio, switching to [udev] and [encrypt] instead, and changing the kernel command line arguments accordingly, and am getting the exact same error on boot. Things are still fine with systemd hooks using linux-lts and systems 246.2-1.
Comment by STPR (stpr) - Tuesday, 18 August 2020, 21:03 GMT
@grazzolini An update on this: I just received an update for systemd (246.1-1 -> 246.2-1). After the upgrade, I rebuilt the initramfs images and now everything appears to work without any problems. So then this was a systemd problem.
Comment by STPR (stpr) - Tuesday, 18 August 2020, 21:34 GMT
@grazzolini An update on this: I just received an update for systemd (246.1-1 -> 246.2-1). After the upgrade, I rebuilt the initramfs images and now everything appears to work without any problems. So then this was a systemd problem.
Comment by George Rawlinson (rawlinsong) - Wednesday, 19 August 2020, 00:32 GMT
I've only got around to spending some time on this issue, as it occurs on two of my devices.

I have the following package(s) installed:

- dracut 050-1
- systemd 246.2-1
- linux 5.7.12arch1-1

And the issue has persisted ever since kernel 5.8.0.

Following is journalctl output from recovery shell:

journalctl:
systemd-cryptsetup[324]: Set cipher aes, mode xts-plain64, key size 512 bits for device /dev/disk/by-uuid/<UUID>
kernel: encrypted_key: failed to alloc_cipher (-2)
kernel: encrypted_key: failed to alloc_cipher (-2)
kernel: encrypted_key: failed to alloc_cipher (-2)
systemd-cryptsetup[324]: device-mapper: reload ioctl on failed: invalid argument
kernel: device-mapper: table 254:0: crypt: unknown target type
kernel: device-mapper: ioctl: error adding target to table
kernel: encrypted_key: failed to alloc_cipher (-2)
kernel: encrypted_key: failed to alloc_cipher (-2)
systemd-cryptsetup[324]: Failed to activate with specified passphrase: Invalid argument
Comment by George Rawlinson (rawlinsong) - Wednesday, 19 August 2020, 02:17 GMT
Removed `hostonly="yes"` from my dracut configuration and the initramfs didn't have any issues booting kernel 5.8.0 or 5.8.1.
Comment by Leopold Bloom (leopoldbloom) - Wednesday, 19 August 2020, 03:24 GMT
I recompiled linux 5.8.1-arch1 using the recipe files from `asp checkout linux`, installed it, and somehow it works with both mkinitcpio and dracut. Your guess is as good as mine.
Comment by Alexandre Bouvier (doskoi) - Wednesday, 19 August 2020, 04:47 GMT
Well, it appears to boot just fine if the hostonly initramfs was built while on an already running linux 5.8 kernel (this being booted from a no-hostonly initramfs built on linux<5.8).
Comment by LaserEyess (LaserEyess) - Wednesday, 19 August 2020, 23:09 GMT
This is affecting me as well, I have the same errors as rawlinsong about failed to alloc_cipher. I'm using dracut 050-1, linux 5.8.1, and systemd 246.2.

Downgrading systemd to 245.7 did not help. Only downgrading linux or switching to the LTS kernel helped. Does anyone know what kernel module provides these crypto functions? I'd like to check my initramfs for it?
Comment by LaserEyess (LaserEyess) - Friday, 21 August 2020, 12:35 GMT
I can get around this by doing the following:


1. Boot into a working kernel (if this kernel is 5.8.1 w/ fallback, skip to 3)
2. Create a bootloader entry for 5.8.1 with the fallback initramfs generated with `--no-hostonly`
3. Boot into the 5.8.1 fallback initramfs
4. Re-generate your initramfs with `--hostonly`
5. Reboot into the non-fallback initramfs
6. It Just Werks™

I guess this is an issue with how dracut is parsing dm_crypt modules while running from 5.7 and not finding them correctly for 5.8? I'm not sure.
Comment by Alexandre Bouvier (doskoi) - Friday, 21 August 2020, 13:05 GMT
The files found in a hostonly 5.8.1 initramfs built on a 5.8.1 system, but missing when built on a 5.4 system are:

/usr/lib/modules/5.8.1-arch1-1/kernel/crypto/blake2b_generic.ko.xz
/usr/lib/modules/5.8.1-arch1-1/kernel/crypto/cbc.ko.xz
/usr/lib/modules/5.8.1-arch1-1/kernel/crypto/ecc.ko.xz
/usr/lib/modules/5.8.1-arch1-1/kernel/crypto/ecdh_generic.ko.xz

There is also references to these modules in /usr/lib/modules/5.8.1-arch1-1/modules.* files, that are missing in the 5.4 built initramfs.
Comment by loqs (loqs) - Friday, 21 August 2020, 17:37 GMT Comment by Alexandre Bouvier (doskoi) - Friday, 21 August 2020, 18:03 GMT
@loqs the two initramfs were built with the latest dracut-git 050.r126.fe761330-1
Comment by STPR (stpr) - Saturday, 22 August 2020, 16:55 GMT
After playing around a bit more, @LaserEyess and @doskoi are right. It was not systemd upgrade that "solved" the problem for me (as I stated in my previous comment), it was the fact that I forced an initramfs build on a system booted with fallback initramfs that then allowed me to be able to boot from a non-fallback initramfs. So basically, after upgrading from 5.7.x to 5.8.x., the only way to get a non-fallback initramfs to work is (as suggested by @LaserEyess and @doskoi) to rebuild the initramfs on a booted 5.8.x kernel with fallback initramfs.
Comment by loqs (loqs) - Wednesday, 02 September 2020, 23:53 GMT
Is encrypted-keys.ko.xz present in an initrd with the issue? See  FS#67802  for reasoning.
Comment by lotan rm (lotanrm) - Thursday, 03 September 2020, 14:32 GMT
I had copied and analyzed my initramfs image when it didn't boot.
It definitely had encrypted-keys.ko.xz in it.

I've now tried the suggestions in this thread. I installed dracut-git 050.r145.a5372b8b-1.

After manually booting from a rescue shell as described in  FS#67802  and generating a new initramfs, I was able to boot without a problem. No fallback necessary.

Thanks for the suggestions, guys.
Comment by LaserEyess (LaserEyess) - Saturday, 21 November 2020, 20:09 GMT
I ran into this again upgrading to the 5.9.x. Same exact symptoms.

What happened exactly was for another, unrelated bug I needed to spend some time in LTS (5.4.x), and I had been still updating the linux package during that time. When I built my initramfs for 5.9.9, I was on 5.4.77. After updating to 5.9.9, while still booted into 5.4.77, I rebooted and was greeted with the same error. The same steps resolved it.

1. boot into 5.9.9 fallback-initramfs
2. force rebuilt of initramfs while booted into 5.9.9
3. reboot into 5.9.9 again, with a working initramfs

None of this is new information I guess, but I wanted to mention that this bug very much still exists on the latest dracut and latest kernel. For anyone who has to switch between stable and LTS and uses encrypted disks, I hope this helps.
Comment by Laszlo Gombos (laszlogombos) - Friday, 07 July 2023, 22:52 GMT Comment by Buggy McBugFace (bugbot) - Tuesday, 08 August 2023, 19:11 GMT
This is an automated comment as this bug is open for more then 2 years. Please reply if you still experience this bug otherwise this issue will be closed after 1 month.

Loading...