FS#52061 - [systemd] 232 breaks boot by failing to start udev

Attached to Project: Arch Linux
Opened by Nicolas F. (fratti) - Tuesday, 06 December 2016, 19:24 GMT
Last edited by Dave Reisner (falconindy) - Saturday, 17 December 2016, 17:41 GMT
Task Type Bug Report
Category Packages: Core
Status Closed
Assigned To Dave Reisner (falconindy)
Architecture x86_64
Severity High
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

Description:
After upgrading to systemd 232-4, my system no longer booted. systemd complained multiple times that it couldn't start udev, then after 90 seconds of trying to get the disks by uuid it drops me into a non-functional emergency shell, as my keyboard is not working, possibly due to udev not running.
Downgrading to systemd 231-4 fixes the issue.

The systemd journal contains no logged activity of this incident.

Setting the boot options "systemd.log_level=debug systemd.log_target=console" results in no useful information, all it says is that the udev process died.

I'd paste the logs here but it's kind of hard to get volatile logs out of a broken system, and since I currently have no phone camera to take a snapshot of my screen, my descriptions of the logs will have to suffice.

Additional info:
My mkinitcpio.conf HOOKS are as follows:
HOOKS="base systemd autodetect modconf block filesystems keyboard fsck"

Steps to reproduce:
1. Install systemd 232-4 on an ASUS UX32LN
2. Reboot the system
This task depends upon

Closed by  Dave Reisner (falconindy)
Saturday, 17 December 2016, 17:41 GMT
Reason for closing:  Not a bug
Additional comments about closing:  Not a systemd bug, caused by zlib-asm
Comment by Bartłomiej Piotrowski (Barthalion) - Tuesday, 06 December 2016, 19:25 GMT
Did you rebuild the initramfs?
Comment by Nicolas F. (fratti) - Tuesday, 06 December 2016, 19:31 GMT
Yes, multiple times. I first thought this was a kernel issue, so I downgraded the kernel and rebuilt the initramfs, but the error still occurred. Even if I rebuilt the initramfs in a booted live system with my system chrooted, it didn't solve anything.
Comment by Nicolas F. (fratti) - Tuesday, 06 December 2016, 19:38 GMT
I'm open for any suggested boot options to get as many useful logs out of this as possible.
Comment by Christian Rebischke (Shibumi) - Wednesday, 07 December 2016, 18:39 GMT
Did you tried to downgrade systemd to version 231? Does the boot succeed with v231?
Comment by Daniel M. Capella (polyzen) - Wednesday, 07 December 2016, 18:41 GMT
> Downgrading to systemd 231-4 fixes the issue.

Yes, they did.
Comment by Nicolas F. (fratti) - Tuesday, 13 December 2016, 21:08 GMT
Still broken with systemd 232-6

Again, I'd be glad if anyone could suggest some options for me so that I can generate useful debug output.
Comment by Dave Reisner (falconindy) - Tuesday, 13 December 2016, 21:15 GMT
What is your kernel commandline? It's highly suspicious that log_target=console with log_level=debug doesn't give you anything useful.
Comment by Dave Reisner (falconindy) - Tuesday, 13 December 2016, 21:18 GMT
Also, presumably, you might be able to extract and run systemd-udevd from a v232 package from a working system... Probably need to stop the udev-related sockets/services before you can do that, though.
Comment by Nicolas F. (fratti) - Wednesday, 14 December 2016, 18:05 GMT
My command line (without the systemd specific options) is
✓ fratti@archbook ~ $ cat /proc/cmdline
initrd=\intel-ucode.img initrd=\initramfs-linux.img elevator=noop quiet video.use_native_backlight=1 acpi_osi= resume=UUID=e3a957e4-6a55-4490-b9cc-3fe217b86e89 root=UUID=b19b2880-f4c5-481a-aabd-e84deaea2713 rw
Comment by Dave Reisner (falconindy) - Wednesday, 14 December 2016, 18:09 GMT
"quiet" is probably why you don't see anything useful.
Comment by Nicolas F. (fratti) - Wednesday, 14 December 2016, 18:42 GMT
Good catch, but the output is exactly the same if I remove quiet.

Here's a screenshot of the output: https://0x0.st/9Kz.jpg
Comment by Dave Reisner (falconindy) - Wednesday, 14 December 2016, 18:45 GMT
Well, it says that udevd is failing with an exit code of 127, which is useful information that you neglected to include. Typically this means "not found", but it also might mean that the linker or some dependency isn't found. Unclear. Could you try the experiment I suggested of starting the 232 udevd from a running system?
Comment by Nicolas F. (fratti) - Wednesday, 14 December 2016, 18:47 GMT
>Could you try the experiment I suggested of starting the 232 udevd from a running system?

If you hold my hand enough to tell me what exactly I need to do to get udevd extracted and started on a running system, I can try.
Comment by Nicolas F. (fratti) - Wednesday, 14 December 2016, 18:54 GMT
✗ fratti@archbook systemd-232-6-x86_64.pkg $ sudo usr/lib/systemd/systemd-udevd
[sudo] password for fratti:
specified group 'plugdev' unknown

and then it seems to run normally.

Though, I have not done whatever "Probably need to stop the udev-related sockets/services before you can do that, though." encompasses
Comment by Nicolas F. (fratti) - Wednesday, 14 December 2016, 19:11 GMT
stopping systemd-udevd.service before running that binary does not change the console output. It seems to run fine.

I've also tried to move the keyboard hook in front of the systemd hook and running mkinitcpio, but the emergency shell systemd gives me is still non-functional.
Comment by Nicolas F. (fratti) - Saturday, 17 December 2016, 17:32 GMT
Adding systemd.journald.forward_to_console=1 to the kernel options got me the following interesting error:

[89]: /usr/lib/systemd/systemd-udevd: error while loading shared libraries: libz.so.1: cannot enable executable stack as shared object requires: Operation not permitted

I've had zlib-asm installed, trying again now with distro zlib.
Comment by Nicolas F. (fratti) - Saturday, 17 December 2016, 17:33 GMT
Was indeed caused by zlib-asm, works with regular zlib.

Loading...