FS#38210 - systemd: infinite timeout loop waiting for tmpfs.device (unbootable system)

Attached to Project: Arch Linux
Opened by Xyne (Xyne) - Friday, 20 December 2013, 23:56 GMT
Last edited by Thomas Bächler (brain0) - Sunday, 22 December 2013, 08:41 GMT
Task Type Bug Report
Category Packages: Core
Status Closed
Assigned To Thomas Bächler (brain0)
Dave Reisner (falconindy)
Architecture x86_64
Severity High
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 1
Private No

Details

This morning I upgraded the following:
boost-libs (1.55.0-3 -> 1.55.0-4)
boost (1.55.0-3 -> 1.55.0-4)
cfitsio (3.350-2 -> 3.360-1)
systemd (208-2 -> 208-3)
mkinitcpio (0.15.0-1 -> 16-1)
pcre (8.33-2 -> 8.34-1)
raptor (2.0.10-3 -> 2.0.12-1)
systemd-sysvcompat (208-2 -> 208-3)

Later in the day when I rebooted, the boot process got stuck in an infinite timeout loop after mounting disks. The error message said something about a timeout while waiting for "tmpfs.device" and a broken systemd dependency. The error message was followed by 3 warning messages informing me that I would be dropped into a recovery shell along with the commands to try to continue, etc. I don't have the exact wording

Despite the warnings, no shell appeared. The error message followed by the warning message reappeared in a loop with a multiminute interval.

Downgrading mkinitcpio, systemd and systemd-sysvcompat (and rebuilding the kernel image) from an archiso usb stick solved the problem. I suspect that systemd was enough but I haven't had time to test it as I need to work on this computer right now.

Aside from the default configuration with /tmp on tmpfs, I have an entry in /etc/fstab for a group-shared tmpfs partition on /home:

tmpfs /home/foo/bar/share tmpfs nodev,nosuid,gid=... 0 2

I do not know if this is related, but it is the only other system-specific tmpfs configuration that I can think of. I also have per-user tmpfs partitions mounted with pam_mount for users in a given group, but that should not be related as this error occurs during boot before ttys are even activated.

The system is x86_64.

So far I haven't found any related info through online searches. Let me know what I can test.

Thanks in advance.


Regards,
Xyne
This task depends upon

Closed by  Thomas Bächler (brain0)
Sunday, 22 December 2013, 08:41 GMT
Reason for closing:  Fixed
Additional comments about closing:  systemd 208-5

Please open a separate report for the failure to enter emergency mode.
Comment by Dave Reisner (falconindy) - Saturday, 21 December 2013, 00:28 GMT
Sounds similar to https://bbs.archlinux.org/viewtopic.php?id=174492. You're asking for fsck on tmpfs (which isn't possible).
Comment by Xyne (Xyne) - Saturday, 21 December 2013, 00:57 GMT
That was indeed the cause. Thanks Dave! :D

Incidentally, the exact error message (minus typos from manual copying) was

[ TIME ] Timed out waiting for device tmpfs.device
[DEPEND] Dependency failed for File System Check on /tmpfs.
[DEPEND] Dependency failed for /home/foo/bar/share.
[DEPEND] Dependency failed for Local File Systems.
[Welcome to the emergency mode! After loggin in, type "journalctl -xb" to view
system logs, "systemctl reboot" to reboot, "systemctl default" to try again
to boot into default mode.
[Welcome to the emergency mode! After loggin in, type "journalctl -xb" to view
system logs, "systemctl reboot" to reboot, "systemctl default" to try again
to boot into default mode.
...

The last 3 lines before the ellipsis repeat indefinitely so there seems to be a bug somewhere.
Comment by Dave Reisner (falconindy) - Saturday, 21 December 2013, 01:04 GMT
Hrm. Are you somehow missing sulogin? That looks like emergency.service respawning.
Comment by Xyne (Xyne) - Saturday, 21 December 2013, 01:43 GMT
/usr/bin/sulogin (owned by util-linux 2.24-2) exists on both systems (laptop & desktop, identical behavior as they both have the same entry in fstab). Running it gives the expected user prompt ("Give root password for...").

Can you reproduce this by adding a similar entry in fstab?
Comment by Dave Reisner (falconindy) - Saturday, 21 December 2013, 02:24 GMT
Well, I can reproduce the .device timeout (which is probably introduced by[1]), but I get a sulogin prompt as expected.

[ TIME ] Timed out waiting for device tmpfs.device.
[DEPEND] Dependency failed for File System Check on /tmpfs.
[DEPEND] Dependency failed for /tmp.
[DEPEND] Dependency failed for Local File Systems.
Welcome to emergency mode! After logging in, type "journalctl -xb" to view
system logs, "systemctl reboot" to reboot, "systemctl default" to try again
to boot into default mode.
Give root password for maintenance
(or press Control-D to continue):
┌┤butter->root master * ~
└➤

[1] http://cgit.freedesktop.org/systemd/systemd/commit/?id=4c8bda2442bfc6d84a5deb241dc29efcb81bf3af
Comment by Thomas Bächler (brain0) - Saturday, 21 December 2013, 09:13 GMT
First of all, this is a configuration error. Second, I found out why this didn't happen before, will try to fix it.

In the meantime, fix your configuration while replacing the 2 with a 0:
tmpfs /home/foo/bar/share tmpfs nodev,nosuid,gid=... 0 0
Comment by Thomas Bächler (brain0) - Saturday, 21 December 2013, 10:24 GMT
Fix: http://lists.freedesktop.org/archives/systemd-devel/2013-December/015777.html

I'll backport this to our systemd package later. It needs an incorrect fstab file to notice this regression, so nobody has caught this yet.
Comment by Xyne (Xyne) - Saturday, 21 December 2013, 16:53 GMT
I already admitted that it was a configuration error (and fixed it) a few posts up. ;)

I still don't understand why I wasn't dropped to the recovery shell, but that seems to be a separate issue.

Loading...