FS#54825 - [systemd] System 234 fails to start cryptography service

Attached to Project: Arch Linux
Opened by Hugo Osvaldo Barrera (hobarrera) - Saturday, 15 July 2017, 14:04 GMT
Last edited by Christian Hesse (eworm) - Monday, 31 July 2017, 06:55 GMT
Task Type Bug Report
Category Packages: Core
Status Closed
Assigned To Dave Reisner (falconindy)
Christian Hesse (eworm)
Architecture x86_64
Severity Critical
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 8
Private No

Details

Description:

Booting into systemd-234 fails to start when using the sd-* hooks and FDE (LVM on top of LUKS).

Failed to start Cryptography setup for luks-83745..
(no password prompt, it just times out and dies).

Applicable versions:
Downgrading to the last 233.* version works again.

No log files, since this happens on the rd.
This task depends upon

Closed by  Christian Hesse (eworm)
Monday, 31 July 2017, 06:55 GMT
Reason for closing:  Fixed
Additional comments about closing:  systemd 234.11-2 in [testing]
Comment by William (wvds) - Saturday, 15 July 2017, 18:33 GMT
Have experienced the same issue with just ext4 on top of LUKS

mkinitpcio:
base systemd autodetect modconf block keyboard sd-vconsole sd-encrypt filesystems

minimal kernel options:
rw luks.name=b8c28c67-aeda-4255-b5e4-4c8265a1811a=root root=/dev/mapper/root

I receive a passphrase prompt on 233 but not on 234, in which systemd-cryptsetup@root.service fails instantly, dumping me to emergency mode (which strangely does not give me a shell).
Comment by Hugo Osvaldo Barrera (hobarrera) - Saturday, 15 July 2017, 21:12 GMT
I don't an emergency shell either, just a dead end. Regrettably, I'm extremely busy until Wednesday and cannot debug this further, but it's worth mentioning that this has also been discussed in arch-general today too.
Comment by Eyal Kalderon (ebkalderon) - Sunday, 16 July 2017, 03:18 GMT
Same issue occurred for me as well; took me completely by surprise. Frustratingly enough, I was also not presented with an emergency shell nor access the system journal. I'm glad I had an Arch Live USB drive on hand and I still had the older systemd-233.75-3 package in cache. For the record, I am also running ext4 with systemd-cryptsetup on top of LUKS.

Even after chrooting into the dead machine, I could not find any trace of what went wrong with systemd-cryptsetup. Hope someone else might have better luck debugging this.
Comment by Hugo Osvaldo Barrera (hobarrera) - Sunday, 16 July 2017, 05:09 GMT
There's no trace after chrooting because journald never gets to persist anything -- because your disk can't even get mounted! (I kinda looked through journalctl a while before I realized this).

TBH, I've absolute no idea how to debug this, since it's purely happening inside the rd environment.
Comment by loqs (loqs) - Sunday, 16 July 2017, 09:50 GMT
Could you try adding the timeout parameter to crypttab or luks.options try the values 30s 0s and 0 please.
Comment by William (wvds) - Sunday, 16 July 2017, 10:09 GMT
Prepended the timeout to the kernel options. Values of '30s' and '0s' work normally, '0' fails as before

Eg:
rw luks.options=timeout=30s luks.name=b8c28c67-aeda-4255-b5e4-4c8265a1811a=root root=/dev/mapper/root
Comment by loqs (loqs) - Sunday, 16 July 2017, 11:21 GMT
Perhaps ask upstream if the change to default behavior was intentional or accidental.
It is not mentioned in the NEWS file for 234 and the documentation matches 233's behavior not 234.
Comment by William (wvds) - Sunday, 16 July 2017, 12:03 GMT
Since there's nothing about it in the news, I've posted an issue upstream:

https://github.com/systemd/systemd/issues/6381
Comment by Hugo Osvaldo Barrera (hobarrera) - Sunday, 16 July 2017, 14:04 GMT
I'm not personally using crypttab, so the timeout used should be the default, not one specified by me. Is the default simply "not working"?
Comment by loqs (loqs) - Sunday, 16 July 2017, 16:50 GMT
From the reports the default behavior if kernel options is used instead of crypttab has changed from 233 to 234.
If crypttab is used without specifying the timeout option the behavior is the same between 233 and 234.
man 5 crypttab
timeout=
Specifies the timeout for querying for a password. If no unit is
specified, seconds is used. Supported units are s, ms, us, min, h,
d. A timeout of 0 waits indefinitely (which is the default).
This documentation appears accurate apart from the case of 234 using kernel options again going from reports.
Comment by loqs (loqs) - Monday, 17 July 2017, 23:27 GMT Comment by Bartłomiej Piotrowski (Barthalion) - Tuesday, 18 July 2017, 06:17 GMT
Pushed 234.0-3 backporting this change to [testing]. Leaving closing the issue to Christian.
Comment by Christian Hesse (eworm) - Tuesday, 18 July 2017, 06:45 GMT
Thanks Bartłomiej!
Comment by Luca Weiss (z3ntu) - Thursday, 27 July 2017, 16:21 GMT
I've just hit this issue with systemd 234.11-1 . Adding luks.options=timeout=30s to my cmdline made it possible to boot the system again.
Comment by Stefan Brand (seiichiro0185) - Thursday, 27 July 2017, 16:35 GMT
Same here, systemd 234.11-1 fails to boot from an encrypted root partition (meaning there is no password prompt) without luks.options=timeout=30s in the kernel commandline.
Comment by Bartłomiej Piotrowski (Barthalion) - Thursday, 27 July 2017, 17:28 GMT
I guess that you are using regular encrypt hook, non sd-encrypt?
Comment by Stefan Brand (seiichiro0185) - Thursday, 27 July 2017, 17:41 GMT
I'm using a systemd-based mkinitcpio with sd-encrypt.
Comment by Luca Weiss (z3ntu) - Thursday, 27 July 2017, 17:51 GMT
For me:
HOOKS="base systemd autodetect modconf block keyboard sd-encrypt sd-lvm2 filesystems fsck"
Comment by Christian Hesse (eworm) - Thursday, 27 July 2017, 17:57 GMT
What does /etc/crypttab.initramfs look like? What kernel parameters do you use?
Comment by Stefan Brand (seiichiro0185) - Thursday, 27 July 2017, 18:02 GMT
There is no /etc/crypttab.initramfs on my system (or should this file be in the initrd?)

Kernel Commandline looks like this:
rd.luks.uuid=c3078eb3-9f67-4ae0-ae2c-32bbcd9eedf7 rd.lvm.lv=xps13/arch.root rd.lvm.lv=xps13/arch.swap luks.options=timeout=30s resume=UUID=ce9a6eec-b6ef-48fc-a107-9ee030578c41 root=UUID=fcb4a880-b970-46e0-8e62-063e935eccc3 elevator=noop ipv6.disable=1 pcie_aspm=force rw quiet loglevel=3 systemd.show_status=false nvme_core.default_ps_max_latency_us=180000

[edit]To clarify: The luks.options=timeout=30s was added as a workaround to get a booting system again, it wasn't there in my original kernel commandline.[/edit]

And for the sake of completeness here are my initrd hooks:
HOOKS="systemd autodetect modconf block keyboard keymap sd-encrypt sd-lvm2 filesystems fsck"
Comment by Hugo Osvaldo Barrera (hobarrera) - Thursday, 27 July 2017, 18:33 GMT
It works for you because you're using `luks.options=timeout=30s`. The problem is that the default is broken when not specifying this.
Comment by loqs (loqs) - Thursday, 27 July 2017, 18:57 GMT
@hobarrera seiichiro0185 already noted that https://bugs.archlinux.org/task/54825#comment159710
Using /etc/crypttab.initramfs also works without any timeout specified
Edit:
In the upstream bug the screenshots in response to the request for information since the report was reopened are both from systems using dracut.
Could an affected arch user post similar screenshots if textual output is not available to the upstream report.
Comment by Xaver Nitsch (xaverlalo) - Thursday, 27 July 2017, 21:39 GMT
I have had the same issue. Chroot from Live-USB and downgrading solved the problem.
Comment by Alvin (Bekcpear) - Saturday, 29 July 2017, 05:36 GMT
I haven't met this bug, maybe the option, `luks.key=`, also works? Refer https://wiki.archlinux.org/index.php/Dm-crypt/System_configuration#luks.key . I use btrfs on top of LUKS, and here is the cmdline in file /etc/default/grub: `luks.uuid=d97261f8-38cc-42be-9cc6-745489831a3a luks.name=d97261f8-38cc-42be-9cc6-745489831a3a=dcroot luks.key=/.ck.bin acpi_osi="!Windows 2012"`
Comment by Hugo Osvaldo Barrera (hobarrera) - Saturday, 29 July 2017, 19:03 GMT
Alvin: You're using a key rather than a passphrase? Or both?
Comment by Alvin (Bekcpear) - Sunday, 30 July 2017, 00:38 GMT
@hobarrera Just a key. I enter the passphrase in the grub phase, and then unlock the entire disk by a key automaticly.
Comment by Hugo Osvaldo Barrera (hobarrera) - Sunday, 30 July 2017, 00:57 GMT
Well then, of course you haven't met this issues, because you CAN'T meet this issue. In your scenario, the prompt that's failing to show up is never shown up.
Comment by loqs (loqs) - Sunday, 30 July 2017, 08:27 GMT Comment by Andrew Soutar (andrewsoutar) - Sunday, 30 July 2017, 19:52 GMT
I'm the author of that particular patch. It's worked for me, but that doesn't mean very much... If anyone wants to test it, I've got a PKGBUILD up at https://github.com/andrewsoutar/pkgbuild-systemd-fix-6381 .
EDIT: Just got confirmation from another user on Github that this worked for him.
Comment by loqs (loqs) - Sunday, 30 July 2017, 20:22 GMT
eworm or Barthalion could post built packages with the patch applied on pkgbuild.com if they think the patch is promising.
Otherwise wait for upstream to review the patch.

Loading...