FS#67517 - [mkinitcpio] [systemd] Systemd 246 is just broken

Attached to Project: Arch Linux
Opened by Hanabishi (Hanabishi) - Friday, 07 August 2020, 09:24 GMT
Last edited by Giancarlo Razzolini (grazzolini) - Monday, 14 December 2020, 12:08 GMT
Task Type Bug Report
Category Packages: Core
Status Closed
Assigned To Dave Reisner (falconindy)
Christian Hesse (eworm)
Giancarlo Razzolini (grazzolini)
Architecture x86_64
Severity Critical
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 3
Private No

Details

Latest systemd update (246-1) is just broken. It's even can't install properly. Post-transaction hooks failed, it was not able to restart some services, the system started to lag and randomly not responding.
After reboot tty can't start around a minute! Login prompt just doesn't appear. Seems like the tty service is not responding during this time. After it timed out and restarted, this message appears in the journal:
`systemd-logind[537]: Failed to start autovt@tty2.service: No buffer space available`

Also there is such message on boot:
`systemd[1]: initrd-switch-root.service: Current command vanished from the unit file, execution of the command list won't be resumed.`
Found that https://github.com/systemd/systemd/issues/15699 but described solution doesn't help.

And systemd drains 100% CPU constantly:
https://i.ibb.co/s26yZSr/image.png

And systemctl is basically dead:
https://i.ibb.co/t4htRFH/image.png
This task depends upon

Closed by  Giancarlo Razzolini (grazzolini)
Monday, 14 December 2020, 12:08 GMT
Reason for closing:  Fixed
Comment by Hanabishi (Hanabishi) - Friday, 07 August 2020, 10:14 GMT
Source of the problem is a mkinitcpio systemd hook:
HOOKS=(base systemd block keyboard fsck filesystems)

Changing systemd to udev fixes the problem:
HOOKS=(base udev block keyboard fsck filesystems)

So this can be used as a workaround.
Comment by Martin Sandsmark (sandsmark) - Friday, 07 August 2020, 13:34 GMT
For those that don't want to wade through the systemd bug report, the issue is mkinitcpio symlinking /usr/local/bin to /usr/bin in the initrd. and because systemd switched to non-absolute paths (https://github.com/systemd/systemd/commit/cdc6804b6046282a42117678751c126a022cd24a) the paths to executables don't match when switching from the initrd (i. e. systemd has resolved a lot of stuff to /usr/local/bin/foo, when foo is in /usr/bin/ in the real rootfs. fwiw the bug you linked to is a duplicate of https://github.com/systemd/systemd/issues/16076

Which solution didn't work for you?
Comment by Hanabishi (Hanabishi) - Friday, 07 August 2020, 14:07 GMT
I've said about https://github.com/systemd/systemd/issues/15699#issuecomment-645843333 - manual editing of /usr/lib/systemd/system/initrd-switch-root.service and providing full path to systemctl doesn't worked for me.
But after I've been noticed about hooks, systemd to udev change worked.
Comment by luca cunegondi (cune) - Saturday, 08 August 2020, 19:44 GMT
netctl profile fails to start with the following error:
* netctl@home.service - A basic static ethernet connection
Loaded: bad-setting (Reason: Unit netctl@home.service has a bad unit file setting.)
Active: inactive (dead)
Aug 08 21:25:23 ... systemd[1]: /etc/systemd/system/netctl@home.service:1: Assignment outside of section. Ignoring.

Downgraded to 245
Comment by loqs (loqs) - Saturday, 08 August 2020, 20:32 GMT
@cune see  FS#67530 
Comment by Hanabishi (Hanabishi) - Sunday, 09 August 2020, 11:04 GMT
I've figured out that `dracut` is also affected by this bug: https://github.com/systemd/systemd/issues/16076#issuecomment-671036968
Comment by Zach Smith (zaxmyth) - Saturday, 15 August 2020, 15:02 GMT
I'm not sure the best way to submit patches so please direct me to the right docs if this is the wrong way. I've attached a patch that essentially reverts the problematic cdc6804b6046282a42117678751c126a022cd24a commit in systemd upstream and adds absolute paths back to the included unit files. This isn't a great long-term fix and it would be ideal to revert the changes in systemd but it's unclear if that'll happen anytime soon so this may help Arch users using systemd with mkinitcpio for the time being.

NOTE: I can't delete attached files on this comment - Only systemd-256-67517-fix.patch is relevant
Comment by Gaurav Yadav (gaurav_ballia) - Tuesday, 18 August 2020, 10:42 GMT
@Hanabishi my `mkinitcpio.conf` never had any `systemd` hook. What should I do as a workaround? And @cune I downgraded `systemd`, `systemd-libs` and `systemd-sysvcompat` to `245.7-1` but it didn't help. Its still hogging my CPU. In my case, `systemd-udevd` is eating my CPU and there's a catch, its all fine after a suspend.
Comment by Hanabishi (Hanabishi) - Tuesday, 18 August 2020, 10:52 GMT
I don't know really, this is confirmed for the systemd hook. Are you sure this is the same issue?
Comment by Giancarlo Razzolini (grazzolini) - Tuesday, 18 August 2020, 11:43 GMT
Doesn't look like the same issue, I think this is unrelated.
Comment by Gaurav Yadav (gaurav_ballia) - Tuesday, 18 August 2020, 12:58 GMT
I guess its the same issue because it all started with the same version as the bug specified. My system was fine with `245.7-1`. I don't know why it shows the same behavior now even with downgraded package.
Comment by Giancarlo Razzolini (grazzolini) - Tuesday, 18 August 2020, 13:48 GMT
Just because it started (or you noticed it) with this version, doesn't mean it's the same issue. If you downgraded just systemd and it is still happening, one more indication your issue is unrelated, at least to this one.
Comment by luca cunegondi (cune) - Tuesday, 18 August 2020, 19:56 GMT
@Gaurav Yadav: at the end my problem was related to netctl. Never had a systemd hook in mkinitcpio.conf, like yours. Did you upgrade just systemd or other packages at the same time?
Comment by teamalpha5441 (teamalpha5441) - Wednesday, 19 August 2020, 22:06 GMT
@luca cunegondi (cune) How did you fix netctl? I have netctl fail the same way.
Comment by luca cunegondi (cune) - Thursday, 20 August 2020, 19:23 GMT
by re-enabling profiles as per  FS#67530  (see above comment by logs)
Comment by Gaurav Yadav (gaurav_ballia) - Friday, 21 August 2020, 06:07 GMT
@luca cunegondi I performed a full upgrade
Comment by Gaurav Yadav (gaurav_ballia) - Friday, 28 August 2020, 13:36 GMT
It was due to the linux kernel 5.8.x. I opened a new task: https://bugs.archlinux.org/task/67744
Comment by loqs (loqs) - Monday, 28 September 2020, 14:10 GMT Comment by Sven-Hendrik Haase (Svenstaro) - Saturday, 12 December 2020, 23:10 GMT
Can't this be closed?
Comment by Hanabishi (Hanabishi) - Saturday, 12 December 2020, 23:16 GMT
Definitely should be closed.

Loading...