Arch Linux

Please read this before reporting a bug:

Do NOT report bugs when a package is just outdated, or it is in the AUR. Use the 'flag out of date' link on the package page, or the Mailing List.

REPEAT: Do NOT report bugs for outdated packages!

FS#70272 - [systemd] 248-1 breakes LUKS with keyfile

Attached to Project: Arch Linux
Opened by Nikita (Slenderchat) - Saturday, 03 April 2021, 00:36 GMT
Last edited by Christian Hesse (eworm) - Tuesday, 13 April 2021, 14:28 GMT
Task Type Bug Report
Category Packages: Core
Status Assigned
Assigned To Christian Hesse (eworm)
Architecture x86_64
Severity Critical
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 0%
Votes 14
Private No


Suddenly after systemd upgrade my server and laptop, which is LUKS encrypted and uses systemd-cryptsetup with keys in initramfs and root filesystem to auto-unlock, started to boot to emergency shell (photo of output will be attached). I started to investigate the problem, and find out that culprit was systemd-cryptsetup@root unit. Logs for that unit contained this error: "Failed to load key file "root.key": Argument list too long". After i extracted keys from initramfs file and chrooted to my broken systems, I decided to downgrade systemd to 247.4-2. After it was done and after initcpios was updated, both systems booted succesfully. This is the only change which was done and it helped, so the culprit is systemd 248-1. Seems to be, that FS#70264 is related and has the same culprit.

Steps to reproduce:
1. Setup system with LUKS full disk encryption which uses systemd-cryptsetup to auto-unlock partitions on boot with keyfiles in initcpios;
2. Make sure that you've replaced udev hook with systemd hook in mkinitcpio.conf, added sd-encrypt hook, and included /etc/cryptsetup-keys.d/root.key in FILES array;
3. Make sure, that you use systemd >= 248;
4. Try to boot.

P.S. Unfortunately I only captured output of server, which has "quiet" in kernel parameters. All other information was gained on laptop by removing "quiet" option, and by including /etc/shadow in initcpio's to get emergency shell access.
This task depends upon

Comment by xgjmibzr (xgjmibzr) - Saturday, 03 April 2021, 02:12 GMT
I had a similar bug, but on a secondary disk rather than root.
Error was "Apr 02 17:37:32 arch-server systemd-cryptsetup[247]: Failed to activate with key file '/etc/keys/[UUID].key': Invalid argument."

Downgraded to 247.4-2 and boots without error.
Comment by Robert (robson) - Saturday, 03 April 2021, 11:32 GMT
And did you install libfido2, which is an optional dependency for systemd 248?
Comment by Frederick Zhang (FrederickZh) - Saturday, 03 April 2021, 12:07 GMT
Similar issue here. While my main disk is opened via SSH hence unaffected, other additional ones all failed with

> Failed to activate with key file '/etc/crypttab.key': Invalid argument

I filed a ticket upstream:
Comment by Vince (Heronymous) - Saturday, 03 April 2021, 17:59 GMT
I wanted to throw my hat in and confirm that I also have the same issue with systemd 248-1. Downgrading to systemd 247.3-1 fixed my issue (and only this downgrade was needed).
Comment by Vince (Heronymous) - Saturday, 03 April 2021, 19:39 GMT
I wanted to throw my hat in and confirm that I also have the same issue with systemd 248-1. Downgrading to systemd 247.3-1 fixed my issue (and only this downgrade was needed).
Comment by Nikita (Slenderchat) - Saturday, 03 April 2021, 21:27 GMT
@Robert Read the description for libfido2, it’s needed only for usb devices.
Comment by Robert (robson) - Saturday, 03 April 2021, 21:40 GMT
@Nikita Well, take a look at this
Comment by xgjmibzr (xgjmibzr) - Saturday, 03 April 2021, 22:14 GMT
@Robert, take a look at this

> libfido2 provides library functionality and command-line tools to communicate with a FIDO device over USB, and to verify attestation and assertion signatures.
Comment by Kawai Tomato (kawaitomato) - Sunday, 04 April 2021, 05:17 GMT
I got a similar problem with my computer Hasee Z6-CT5NA. I use LVM on LUKS to encrypt my disk, and use systemd-boot as boot manager. After upgrade systemd from version 247 to version 248-3(I haven't upgraded for a few days), I couldn't boot my system. The screenshot:
Comment by MMH (mmh) - Monday, 05 April 2021, 22:32 GMT
I get "Failed to load key file 'disk.key': Argument list too long" like the author but I don't see any log messages related to "Invalid argument".
Are those two different issues?
Comment by Aaron Maslen (microsoftenator) - Tuesday, 06 April 2021, 04:00 GMT
@mhh: I'd guess that's a separate issue.

The bug appears to be in systemd-cryptsetup, specifically the "Invalid Argument" error message appears to originate in the code that reads the keyfile from the disk -

My guess would be that the bug is triggered by the crypt_activate_by_passphrase function, but I don't know the codebase well enough to attempt to debug further than this.

I can confirm that this issue also occurs for me with systemd version 248 and that downgrading to 247.4-2 works.
Comment by Kevin (kp7) - Wednesday, 07 April 2021, 09:08 GMT
I have encountered the same bug.
My system is configured to unlock the encrypted /home partition (which is on a secondary drive) automatically with a keyfile - /etc/cryptsetup-keys.d/home.key.
After upgrading to systemd-248, the system boots in emergency mode. However, I am able to correctly mount my home partition using both the said keyfile and the passphrase.
The issue is not present anymore after downgrading to systemd-247
Comment by Simon Paul (smnpl) - Sunday, 11 April 2021, 18:24 GMT
Same here as Kevin. A couple of devices that should get unlocked by a keyfile. Broke with 248 and works with 247.
Right now I downgraded, but that is very unfortunate for a systems core component.

Is this already filed as bug at the systemd github repo?
Comment by Hrvoje Hodak (tribly) - Tuesday, 13 April 2021, 14:33 GMT
Same here. I have 3 disks in /etc/crypttab and only one of them is getting decrypted. The other 2 are giving an "Invalid argument" error.

Downgrading to 247.4-2 solves that issue.
Comment by Random (random237849) - Friday, 16 April 2021, 20:42 GMT
Version 248-5 is still affected. Seems the fix for FS#70264 (see initial report) did not solve this issue.
Also no indication that there will be a solution from upstream soon.
Comment by Cysioland (Cysioland) - Sunday, 02 May 2021, 18:26 GMT
I have the same issue. libfido2 is present on my system, but this doesn't help.

Relevant journal attached.
Comment by Hrvoje Hodak (tribly) - Sunday, 13 June 2021, 09:09 GMT
Looks like they fixed and merged it upstream
Comment by Christian Hesse (eworm) - Sunday, 13 June 2021, 19:12 GMT
Watching this myself... Sadly this does not easily apply to v248...