FS#66908 - [systemd] Ordering cycle reported, prevents non-root login

Attached to Project: Arch Linux
Opened by Bastian Beranek (totsilence) - Friday, 05 June 2020, 08:14 GMT
Last edited by Dave Reisner (falconindy) - Thursday, 11 June 2020, 11:46 GMT
Task Type Bug Report
Category Packages: Core
Status Closed
Assigned To Christian Hesse (eworm)
Architecture All
Severity Low
Priority High
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 1
Private No

Details

With systemd 245.6-2 I get error messages about units which are deleted in order to break ordering cycles in the journal (see below). This does not happen with systemd 245.5-2. I also cannot login anymore with non-root users because of pam_nologin:

Jun 05 09:35:49 bastian-desktop sddm[601]: Authentication error: "\"System is booting up. Unprivileged users are not permitted to log in yet. Please come back later. For technical details, see pam_nologin(8).\""

I believe these two issues to be related (maybe through sytemd-tmpfiles-setup?).

Jun 05 09:35:40 bastian-desktop systemd[1]: sysinit.target: Job systemd-tmpfiles-setup.service/start deleted to break ordering cycle starting with sysinit.target/start
Jun 05 09:35:40 bastian-desktop systemd[1]: sysinit.target: Job systemd-update-done.service/start deleted to break ordering cycle starting with sysinit.target/start
Jun 05 09:35:40 bastian-desktop systemd[1]: sysinit.target: Job systemd-journal-catalog-update.service/start deleted to break ordering cycle starting with sysinit.target/start
Jun 05 09:35:40 bastian-desktop systemd[1]: sysinit.target: Job local-fs.target/start deleted to break ordering cycle starting with sysinit.target/start
Jun 05 09:35:41 bastian-desktop systemd[1]: sysinit.target: Job systemd-tmpfiles-setup.service/start deleted to break ordering cycle starting with sysinit.target/start
Jun 05 09:35:41 bastian-desktop systemd[1]: systemd-update-done.service: Job systemd-journal-catalog-update.service/start deleted to break ordering cycle starting with systemd-update-done.service/start
Jun 05 09:35:41 bastian-desktop systemd[1]: sysinit.target: Job local-fs.target/start deleted to break ordering cycle starting with sysinit.target/start
Jun 05 09:35:41 bastian-desktop systemd[1]: local-fs.target: Job systemd-tmpfiles-setup.service/start deleted to break ordering cycle starting with local-fs.target/start
Jun 05 09:35:41 bastian-desktop systemd[1]: local-fs.target: Job systemd-update-done.service/start deleted to break ordering cycle starting with local-fs.target/start
Jun 05 09:35:41 bastian-desktop systemd[1]: local-fs.target: Job systemd-journal-catalog-update.service/start deleted to break ordering cycle starting with local-fs.target/start
Jun 05 09:35:43 bastian-desktop systemd[1]: systemd-tmpfiles-setup.service: Job local-fs.target/start deleted to break ordering cycle starting with systemd-tmpfiles-setup.service/start
Jun 05 09:35:43 bastian-desktop systemd[1]: sysinit.target: Job systemd-tmpfiles-setup.service/start deleted to break ordering cycle starting with sysinit.target/start
Jun 05 09:35:43 bastian-desktop systemd[1]: sysinit.target: Job local-fs.target/start deleted to break ordering cycle starting with sysinit.target/start
Jun 05 09:35:45 bastian-desktop systemd[1]: sysinit.target: Job systemd-update-done.service/start deleted to break ordering cycle starting with sysinit.target/start
Jun 05 09:35:45 bastian-desktop systemd[1]: sysinit.target: Job systemd-journal-catalog-update.service/start deleted to break ordering cycle starting with sysinit.target/start
Jun 05 09:35:45 bastian-desktop systemd[1]: sysinit.target: Job local-fs.target/start deleted to break ordering cycle starting with sysinit.target/start
This task depends upon

Closed by  Dave Reisner (falconindy)
Thursday, 11 June 2020, 11:46 GMT
Reason for closing:  Not a bug
Additional comments about closing:  configuration issue
Comment by Tuomas Jaako (tomaso) - Friday, 05 June 2020, 10:06 GMT
I'm also getting ordering cycle messages when booting. It doesn't always prevent non-root login in my case. The source of the messages for me seem to be nfs mounts in fstab. Commenting them out the system boots normally.

For me it is enough to downgrade to systemd 245.6-1 to resolve the problem.
Comment by Bastian Beranek (totsilence) - Friday, 05 June 2020, 11:26 GMT
Yes, downgrade to 245.6-1 is enough. I don't have nfs mounts in fstab, but I do have custom systemd unit files which setup automount for a samba share and another fuse mount. "data-nas.automount" is enabled in systemd, the "data-nas.mount" isn't. When disabling data-nas.automount the problem goes away - so it's definitely related.

They look like:

$ cat /etc/systemd/system/data-nas.automount
[Unit]
Description=Automount NAS Share
After=network-online.target
Requires=network-online.target

[Automount]
Where=/data/nas

[Install]
WantedBy=multi-user.target

$ cat /etc/systemd/system/data-nas.mount
[Unit]
Description=NAS Share

[Mount]
What=//fritz.nas/fritz.nas
Where=/data/nas
Options=credentials=/etc/samba/creds/fritz.nas,noperm,noserverino,vers=3.0,iocharset=utf8,rw
Type=cifs
TimeoutSec=30

[Install]
WantedBy=multi-user.target

Comment by Tuomas Jaako (tomaso) - Friday, 05 June 2020, 13:02 GMT
I have a few nfs mounts like:

server.local:/mount/point /mount/point nfs noauto,noatime,x-systemd.automount,x-systemd.device-timeout=10,x-systemd.mount-timeout=10,timeo=10,x-systemd.idle-timeout=2min,x-systemd.requires=network-online.target,users,rsize=65536,wsize=65536,proto=tcp 0 0
Comment by Andreas Radke (AndyRTR) - Saturday, 06 June 2020, 07:58 GMT
@eworm: I'm also affected by this issue. Can you at least tell us if this caused by one of the two backport you've added or by the install scriptlet reordering.
Should we file an upstream bug? Why have you added the backports at all?
Comment by loqs (loqs) - Saturday, 06 June 2020, 10:09 GMT
The two backports added in 245.6-3 are for [1]. The initial report by totsilence listed systemd 245.6-2 which did not contain those backports.
systemd 245.6-2 was the first actual release using 245.6, 245.6-1 only updated the pkgver. 245.6 added 124 commits.
Edit:
If everyone affected is using automount, 245.6 included [2] and [3]

[1] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-13776
[2] https://github.com/systemd/systemd-stable/commit/e1c091b6d4c55de5c5356e8ca5564dba6769b49f
[3] https://github.com/systemd/systemd-stable/commit/f1fb1971767d7b8a1fd220ecd074bbceaf34d556
Comment by Andreas Radke (AndyRTR) - Saturday, 06 June 2020, 13:22 GMT
It turns out for me we have two new issues:

1) login failure with 245.6-3
2) ordering cycle on mnt-nfs.automount


1) I could fix my login issue reverting https://github.com/systemd/systemd-stable/commit/7bc4bcea15aa837100284f7379f066f72b34b5b0
Everyone affected please rebuild systemd with pkgrel=3.1 and put "7bc4bcea15aa837100284f7379f066f72b34b5b0" into the revert array of the PKGBUILD and report back please.

2) I cannot revert these single two commits listed above. There a many more mount related commits so git revert fails. Probably that whole set of patches needs to be reverted.

Comment by Andreas Radke (AndyRTR) - Saturday, 06 June 2020, 13:45 GMT
1) Reverting that single commit doesn't seem enough. Randomly the boot still fails with ordering cycle issues.

I wonder why they introduced such a major thing into the stable branch.
Comment by loqs (loqs) - Saturday, 06 June 2020, 13:57 GMT
If you go back to 245.5 _tag='a520e63382396661d79f630b2babe717a85b1209' leaving the backports and other PKGBUILD changes does that resolve both your issues?
Edit:
I mean leaving the PKGBUILD as was in 245.6.3, 7bc4bcea15aa837100284f7379f066f72b34b5b0 would need to be removed the reverts array.
Comment by Andreas Radke (AndyRTR) - Saturday, 06 June 2020, 15:30 GMT
With tag 245.5 (a520e63382396661d79f630b2babe717a85b1209) login and nfs.mount both seem to fine even with the latest backports in 245.6-3. So I guess I will have to bisect what commit to stable in 245.6 start the ordering cycle issue.
Comment by Thomas Blechinger (Gecko253) - Saturday, 06 June 2020, 16:02 GMT
I could solve the ordering cycle by removing x-systemd.requires=network-online.target from the fstab line. Without that it works fine as before.
Comment by Andreas Radke (AndyRTR) - Saturday, 06 June 2020, 19:39 GMT
Found it. Please try 245.6-4 reverting one commit.
Reported it upstream: https://github.com/systemd/systemd-stable/issues/69
Comment by Bastian Beranek (totsilence) - Saturday, 06 June 2020, 20:01 GMT
Seems good! Thanks.
Comment by Bastian Beranek (totsilence) - Sunday, 07 June 2020, 10:11 GMT
BTW: I think systemd uses github.com/systemd/systemd for bug reports, not systemd-stable?
Comment by Tuomas Jaako (tomaso) - Sunday, 07 June 2020, 12:20 GMT
Solves my problems as well. Thanks!
Comment by Christian Hesse (eworm) - Wednesday, 10 June 2020, 18:45 GMT
The revert has been dropped for next package. Dave's commit message:

> This was never the right thing to do. Cyclic dependencies existed because users
> decided to hand roll their own automount units with incorrect dependencies
> rather than relying on the fstab generator. Upstream changes simply exposed
> these always-wrong units as being wrong (albeit not in the best of ways).
> Upstream will *not* be reverting anything in the short or long term here,
> so neither should Arch. What upstream *is* doing is clarifying the documentation
> about automounts[0] to try to discouarge this in the future.
>
> I'm removing this revert to make sure that the next systemd build does not
> include this.
>
> [0] https://github.com/systemd/systemd/pull/16114

https://git.archlinux.org/svntogit/packages.git/commit/trunk?h=packages/systemd&id=6d04fea696912e467d295e02ab30163d15024f31

Loading...