Arch Linux

Please read this before reporting a bug:
https://wiki.archlinux.org/index.php/Reporting_Bug_Guidelines

Do NOT report bugs when a package is just outdated, or it is in the AUR. Use the 'flag out of date' link on the package page, or the Mailing List.

REPEAT: Do NOT report bugs for outdated packages!
Tasklist

FS#45086 - [systemd] system hangs/pauses forever after entering luks passphrase

Attached to Project: Arch Linux
Opened by Hussam Al-Tayeb (hussam) - Monday, 25 May 2015, 23:28 GMT
Last edited by Evangelos Foutras (foutrelis) - Wednesday, 27 May 2015, 19:24 GMT
Task Type Bug Report
Category Packages: Testing
Status Closed
Assigned To Thomas Bächler (brain0)
Dave Reisner (falconindy)
Bartłomiej Piotrowski (Barthalion)
Architecture All
Severity Critical
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 10
Private No

Details

after upgrading to systemd 220, the system hangs/pauses forever after entering luks passphrase. It simply shows a blinking cursor instead of fscking the unlocked partition and stays there forever.
This only happens if the initramfs image is built under systemd 220.
I don't know how to check what is happening in the background so I ctrl-alt-del and rebooted to rescue cd and reinstalled systemd 219 and then regenerated the initramfs image. System booted correctly again.
Similar report here https://bbs.archlinux.org/viewtopic.php?id=197844
This task depends upon

Closed by  Evangelos Foutras (foutrelis)
Wednesday, 27 May 2015, 19:24 GMT
Reason for closing:  Fixed
Additional comments about closing:  systemd 220-2
Comment by Darek (blablo) - Tuesday, 26 May 2015, 12:01 GMT Comment by Evangelos Foutras (foutrelis) - Tuesday, 26 May 2015, 14:20 GMT Comment by Pascal Ernster (hardfalcon) - Tuesday, 26 May 2015, 17:19 GMT
Darek: I've just tried systemd-git 220.24.g185abfc (http://cgit.freedesktop.org/systemd/systemd/commit/?id=185abfc3d6b4e8f804a3f7216cd8b0459593af87), but this version gives exactly the same behaviour. Please note that you have to rebuild your initrd by using mkinitcpio after installing the new version, otherwise you'll end up with the old systemd version in the initrd and won't be able to reproduce the problem.

Btw, are we certain this is a bug in systemd and not one in cryptsetup? After all, cryptsetup seems to be the only thing triggering this bug so far.
Comment by Hussam Al-Tayeb (hussam) - Tuesday, 26 May 2015, 17:28 GMT
This is what I did:
1) install the official archlinux cryptsetup 1.6.6 and systemd 220 -> re-generated initramfs image -> saw the bug.
2) I rebooted to rescue cd, installed systemd 219, regenerated initramfs image, rebooted to system.
3) Then I updated cryptsetup 1.6.7 by editing the pkgbuild file, compiled systemd 220 against it, regenerated initramfs image, rebooted and saw bug again.
4) repeated step 2.

So, if it is a cryptsetup bug, it's not fixed in 1.6.7
Comment by Carsten Feuls (CarstenF) - Tuesday, 26 May 2015, 20:06 GMT
I can confirm that the system does not boot anymore. I get the following error before I can enter the password.
inotify_add_watch(7,/dev/dm-0,10) failed: Bad file descriptor
inotify_add_watch(7,/dev/dm-1,10) failed: Bad file descriptor
My system is up to date and I have updated the initrd.
Comment by Christian Hesse (eworm) - Tuesday, 26 May 2015, 21:11 GMT
Commit 185abfc3 [0] should fix the inotify_add_watch() error.
Pascal E. (hardfalcon) reported that it does not fix the LUKS issue.

[0] http://cgit.freedesktop.org/systemd/systemd/commit/?id=185abfc3
Comment by Benjamin Flesch (bf) - Wednesday, 27 May 2015, 11:20 GMT
I think I am also affected by this issue. How can I regenerate a new kernel image from within Mac or Ubuntu? And why on earth doesn't any of the systemd people test their own updates on an hdd-encrypted computer?
Comment by Benjamin Flesch (bf) - Wednesday, 27 May 2015, 16:07 GMT
So now the bug report I created on this issue was closed by Evangelos Foutras (foutrelis) as duplicate and I am still unable to boot my machine.

Can anyone of you please point me to a tutorial on the rescue cd where I can downgrade systemd and regenerate my initramfs image?
Comment by Hussam Al-Tayeb (hussam) - Wednesday, 27 May 2015, 16:23 GMT
Boot a archlinux installation CD then:
cryptsetup luksOpen /dev/sda3 root (replace sda3 with whatever your encrypted partition is)
mount /dev/mapper/root /mnt
arch-chroot /mnt
mount /boot
pacman -Sy core/systemd core/systemd-sysvcompat core/libsystemd
mkinitcpio -p linux
sync
umount /boot
exit
umount /mnt
reboot

Comment by Benjamin Flesch (bf) - Wednesday, 27 May 2015, 16:41 GMT
Thank you Hussam for pointing me into the right direction.
I had already tried the cryptsetup commands with the current arch live cd, but I get the following error when trying to use it: http://i.imgur.com/RBoQNOf.jpg
Comment by Benjamin Flesch (bf) - Wednesday, 27 May 2015, 16:44 GMT
Okay, apparently the arch iso cannot be burnt using a windows machine, I'll try again with another computer and report back. Sorry for spamming, but I did not expect for systemd to interfere with my in such a way.. :-(
Comment by Hussam Al-Tayeb (hussam) - Wednesday, 27 May 2015, 16:50 GMT
your screenshot shows errors reading your cd/dvd. you may want to verify it was correctly burned.
If you have a usb disk, try dd instead:
https://wiki.archlinux.org/index.php/USB_flash_installation_media#Using_dd
Comment by Pascal Ernster (hardfalcon) - Wednesday, 27 May 2015, 17:30 GMT
Benjamin: Just off the top of my head:

1. Boot the machine using a (correctly burned) Archlinux CD. Login as root, set the correct keymap using the "loadkeys" command and configure your network connection if you don't have DHCP in your network.

2. Decrypt and mount the encrypted rootfs on your harddisk:
# cryptsetup luksOpen /dev/sda3 luks-sda3
# mount /dev/mapper/luks-sda3 /mnt

3. chroot into the installation on your harddisk:
# arch-chroot /mnt /bin/bash

4. mount the /boot partition:
# mount /boot

5. Reinstall the systemd packages from core instead of testing:
# pacman -Sy --asdeps core/libsystemd core/systemd core/systemd-sysvcompat

6. Rebuild the initrd for your installed kernel(s):
# mkinitcpio -p linux
# mkinitcpio -p linux-lts
# mkinitcpio -p linux-grsec

7. Exit the chroot and reboot:
# exit
# reboot
Comment by Evangelos Foutras (foutrelis) - Wednesday, 27 May 2015, 17:34 GMT
'pacman -S core/systemd core/libsystemd core/systemd-sysvcompat' should be enough for downgrading the systemd packages.

Please do NOT specify --asdeps as that will alter the install reason of explicitly installed packages (systemd and systemd-sysvcompat).
Comment by Evangelos Foutras (foutrelis) - Wednesday, 27 May 2015, 18:07 GMT
This issue should be fixed in systemd 220-2; a couple of confirmations would be nice.
Comment by Benjamin Flesch (bf) - Wednesday, 27 May 2015, 18:18 GMT
Okay, again I want to thank you all for your kind help. I've gotten it to work now after some time of tinkering. Plus, I've found out the hard way that I have testing enabled. I must have done that a couple of years ago as I first installed arch and compiled several "exotic" libraries.

Pascal and Hussam, your writeups really helped. Unfortunately the situation on my Thinkpad W530 was a bit complicated:
- The first recovery CD (Arch 2015/05) was burnt using the Win7 burning tool, resulting in this error with the "hybrid image" according to people on IRC: http://i.imgur.com/RBoQNOf.jpg
- The second recovery CD was burnt using an OSS tool on Windows called Infrarecorder, and the cd gave the same error when booting
- The third recovery attempt was using a bootable USB stick created on a Mac, which booted...
- .. until it hung at "triggering uevents" ( http://i.imgur.com/JfGcmt6.jpg ); an old and nearly forgotten issue that pops up from time to time with my laptop.
- The acpi / noapic kernel parameters allowed me to finally boot the rescue cd, but the root shell was unusable because my Thinkpad's keyboard did not work (???)
- When plugging in an USB keyboard, the recovery shell from the arch live usb stick was finally usable and I could downgrade systemd

I know it this is a string of unlucky issues and my fixing attempts were at first a bit.. clumsy, but I just wanted to inform you about the reason for all my requests and the annoyance I caused you.

Now everything works like a charm again, and I made sure to remove the testing repos from my pacman.conf.

Thanks again and keep up the good work.
Comment by William Shuman (wshuman3) - Wednesday, 27 May 2015, 18:19 GMT
systemd 220-2 fixes the issue for me
Comment by Carsten Feuls (CarstenF) - Wednesday, 27 May 2015, 19:21 GMT
Yes systemd 220-2 fixes the issue for me.

Loading...