FS#40706 - [systemd] cannot boot with 213-5

Attached to Project: Arch Linux
Opened by Gergő Pintér (pintergreg) - Thursday, 05 June 2014, 10:25 GMT
Last edited by Dave Reisner (falconindy) - Monday, 09 June 2014, 14:19 GMT
Task Type Support Request
Category Packages: Core
Status Closed
Assigned To Dave Reisner (falconindy)
Architecture x86_64
Severity Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 9
Private No

Details

Description:
After upgrading my sytem it did not boot.
I got a message during boot: A start job is running for dev-disk-by\* for all my disk devices for 5 times and waited 1:30 sec then got a recorvery console.
I tried to downgrade all main components that was upgraded, kernel (3.14.5), filesystem(2014.5-02), systemd (213-5)
The solution was to downgraded systemd, systemd-sysvcompat and libsystemd to 212-3.

Additional info:

* 64 bit system
* affected package version: systemd, libsystemd, systemd-sysvcompat 213-5
This task depends upon

Closed by  Dave Reisner (falconindy)
Monday, 09 June 2014, 14:19 GMT
Reason for closing:  Fixed
Additional comments about closing:  systemd 213-7
Comment by Dave Reisner (falconindy) - Thursday, 05 June 2014, 12:23 GMT
You'll have to dig in further to see why your disks aren't being found. Does udev know about them? Are the by-* symlinks created? What kernel version? What's in /etc/fstab?

Really smells like something went wrong with your upgrade. Strongly suggest you try it again and provide some actual debug info if you can still repeat this.
Comment by Jiri Prochazka (ojirio) - Thursday, 05 June 2014, 16:30 GMT
I had the same problem, downgrade to systemd 212-3 allowed me to boot again. The problem was that /boot wasn't being mounted.

/dev/disks/by-uuid/ contained the disks with uuids (not sure if symlinks or files?).

Kernel version 3.14.5-1-ARCH #1 SMP PREEMPT (32 bit)

fstab contains just normal:
UUID=f8ed... /boot ext4 defaults 0 1
UUID=6c99... swap swap defaults 0 0
UUID=974f... / ext4 defaults 0 1
+ my manual iso9660 CD image stuff
Comment by Théo Bueno (Munrek) - Thursday, 05 June 2014, 16:38 GMT
Same problem here since the last update.

Systemd start with fsck on my root partition ( quite long, a minute or so ), then it fails to mount my swap and my read-only windows partition.
After disabling the windows partition in fstab, systemd agrees to continue booting instead of recovery mode.

The weird thing here is that when I am logged on the system, I can mount the win partition.

My fstab : http://pastebin.archlinux.fr/515712
There is no valuable information on journalctl : http://pastebin.archlinux.fr/515713
Comment by Jiri Prochazka (ojirio) - Thursday, 05 June 2014, 16:44 GMT
Hopefully the relevant systemd log lines:

Jun 05 18:11:08 hakunin systemd[1]: Mounted FUSE Control File System.
Jun 05 18:11:08 hakunin systemd[1]: Mounting Configuration File System...
Jun 05 18:11:08 hakunin systemd[1]: Mounted Configuration File System.
Jun 05 18:11:08 hakunin systemd[1]: Started Setup Virtual Console.
Jun 05 18:11:10 hakunin systemd[1]: Starting system-systemd\x2drfkill.slice.
Jun 05 18:11:10 hakunin systemd[1]: Created slice system-systemd\x2drfkill.slice.
Jun 05 18:11:10 hakunin systemd[1]: Starting Sound Card.
Jun 05 18:11:10 hakunin systemd[1]: Reached target Sound Card.
Jun 05 18:11:10 hakunin kernel: ppdev: user-space parallel port driver
Jun 05 18:11:10 hakunin kernel: kvm: disabled by bios
Jun 05 18:11:10 hakunin kernel: kvm: disabled by bios
Jun 05 18:11:10 hakunin systemd[1]: Starting system-systemd\x2dbacklight.slice.
Jun 05 18:11:10 hakunin systemd[1]: Created slice system-systemd\x2dbacklight.slice.
Jun 05 18:11:11 hakunin systemd-sysctl[228]: Overwriting earlier assignment of kernel/sysrq in file '/etc/sysctl.d/99-sysctl.conf'.
Jun 05 18:11:11 hakunin systemd-sysctl[229]: Overwriting earlier assignment of kernel/sysrq in file '/etc/sysctl.d/99-sysctl.conf'.
Jun 05 18:11:11 hakunin systemd[1]: Starting Bluetooth.
Jun 05 18:11:11 hakunin systemd[1]: Reached target Bluetooth.
Jun 05 18:12:04 hakunin systemd-fsck[116]: /dev/sda3: 2056815/60760064 files (0.2% non-contiguous), 219302283/243038208 blocks
Jun 05 18:12:04 hakunin systemd[1]: Started File System Check on Root Device.
Jun 05 18:12:04 hakunin systemd[1]: Starting Remount Root and Kernel File Systems...
Jun 05 18:12:04 hakunin systemd[1]: Started Remount Root and Kernel File Systems.
Jun 05 18:12:04 hakunin systemd[1]: Starting Load/Save RF Kill Switch Status of rfkill2...
Jun 05 18:12:04 hakunin kernel: EXT4-fs (sda3): re-mounted. Opts: (null)
Jun 05 18:12:04 hakunin systemd[1]: Starting Load/Save Screen Backlight Brightness of backlight:acpi_video0...
Jun 05 18:12:04 hakunin systemd[1]: Starting Load/Save RF Kill Switch Status of rfkill1...
Jun 05 18:12:04 hakunin systemd[1]: Starting Load/Save Screen Backlight Brightness of backlight:intel_backlight...
Jun 05 18:12:04 hakunin systemd[1]: Starting Load/Save RF Kill Switch Status of rfkill0...
Jun 05 18:12:04 hakunin systemd[1]: Starting Local File Systems (Pre).
Jun 05 18:12:04 hakunin systemd[1]: Reached target Local File Systems (Pre).
Jun 05 18:12:04 hakunin systemd[1]: Mounting /mnt/f2...
Jun 05 18:11:11 hakunin systemd[1]: Starting Bluetooth.
Jun 05 18:11:11 hakunin systemd[1]: Reached target Bluetooth.
Jun 05 18:12:04 hakunin systemd-fsck[116]: /dev/sda3: 2056815/60760064 files (0.2% non-contiguous), 219302283/243038208 blocks
Jun 05 18:12:04 hakunin systemd[1]: Started File System Check on Root Device.
Jun 05 18:12:04 hakunin systemd[1]: Starting Remount Root and Kernel File Systems...
Jun 05 18:12:04 hakunin systemd[1]: Started Remount Root and Kernel File Systems.
Jun 05 18:12:04 hakunin systemd[1]: Starting Load/Save RF Kill Switch Status of rfkill2...
Jun 05 18:12:04 hakunin kernel: EXT4-fs (sda3): re-mounted. Opts: (null)
Jun 05 18:12:04 hakunin systemd[1]: Starting Load/Save Screen Backlight Brightness of backlight:acpi_video0...
Jun 05 18:12:04 hakunin systemd[1]: Starting Load/Save RF Kill Switch Status of rfkill1...
Jun 05 18:12:04 hakunin systemd[1]: Starting Load/Save Screen Backlight Brightness of backlight:intel_backlight...
Jun 05 18:12:04 hakunin systemd[1]: Starting Load/Save RF Kill Switch Status of rfkill0...
Jun 05 18:12:04 hakunin systemd[1]: Starting Local File Systems (Pre).
Jun 05 18:12:04 hakunin systemd[1]: Reached target Local File Systems (Pre).
Jun 05 18:12:04 hakunin systemd[1]: Mounting /mnt/f2...
Jun 05 18:12:04 hakunin systemd[1]: Starting Load/Save Random Seed...
Jun 05 18:12:04 hakunin systemd[1]: Started Load/Save Screen Backlight Brightness of backlight:intel_backlight.
Jun 05 18:12:04 hakunin systemd[1]: Started Load/Save Screen Backlight Brightness of backlight:acpi_video0.
Jun 05 18:12:04 hakunin systemd[1]: Started Load/Save RF Kill Switch Status of rfkill2.
Jun 05 18:12:04 hakunin kernel: loop: module loaded
Jun 05 18:12:04 hakunin systemd[1]: Started Load/Save RF Kill Switch Status of rfkill0.
Jun 05 18:12:04 hakunin systemd[1]: Started Load/Save RF Kill Switch Status of rfkill1.
Jun 05 18:12:04 hakunin systemd[1]: Started Load/Save Random Seed.
Jun 05 18:12:04 hakunin systemd[1]: Mounted /mnt/f2.
Jun 05 18:12:35 hakunin systemd[1]: Job dev-disk-by\x2duuid-f8ed9f94\x2d7945\x2d4cdf\x2db744\x2d9c4176afc7ac.device/start timed out.
Jun 05 18:12:35 hakunin systemd[1]: Timed out waiting for device dev-disk-by\x2duuid-f8ed9f94\x2d7945\x2d4cdf\x2db744\x2d9c4176afc7ac.device.
Jun 05 18:12:35 hakunin systemd[1]: Dependency failed for /boot.
Jun 05 18:12:35 hakunin systemd[1]: Dependency failed for Local File Systems.
Jun 05 18:12:35 hakunin systemd[1]: Triggering OnFailure= dependencies of local-fs.target.
Jun 05 18:12:35 hakunin systemd[1]: Dependency failed for File System Check on /dev/disk/by-uuid/f8ed9f94-7945-4cdf-b744-9c4176afc7ac.
Jun 05 18:12:35 hakunin systemd[1]: Job dev-disk-by\x2duuid-6c993e40\x2ddc30\x2d46b3\x2dbd62\x2da4664e5b12d6.device/start timed out.
Jun 05 18:12:35 hakunin systemd[1]: Timed out waiting for device dev-disk-by\x2duuid-6c993e40\x2ddc30\x2d46b3\x2dbd62\x2da4664e5b12d6.device.
Jun 05 18:12:35 hakunin systemd[1]: Dependency failed for /dev/disk/by-uuid/6c993e40-dc30-46b3-bd62-a4664e5b12d6.
Jun 05 18:12:35 hakunin systemd[1]: Dependency failed for Swap.
Jun 05 18:12:35 hakunin systemd[1]: Job dev-sda2.device/start timed out.
Jun 05 18:12:35 hakunin systemd[1]: Timed out waiting for device dev-sda2.device.
Jun 05 18:12:35 hakunin systemd[1]: Dependency failed for Swap Partition.
Comment by Daniel Hellsson (scienceboy) - Friday, 06 June 2014, 06:33 GMT
I have the same problem. My forum post: https://bbs.archlinux.org/viewtopic.php?id=182466
In the forum they thought something was wrong with my fstab because of the "Job dev-disk-by\x2duuid". But it looks likes encoding problems to me. I use swedish UTF8 locale (sv_SE.UTF8). Are the other people with this problem also using non-english locale?
Comment by Théo Bueno (Munrek) - Friday, 06 June 2014, 07:03 GMT
I am using both fr_FR.UTF-8 and en_US.UTF-8.
How did you get to that conclusion so I can verify it on my system ?
Comment by Jean-Marc Reiter (derBlaueReiter) - Friday, 06 June 2014, 08:24 GMT
the problem is not related with encoding but with time zone, see post https://bbs.archlinux.org/viewtopic.php?id=182445
unfortunately, setting time zone to UTC is quite annoying when double booting with windows
for now, the solution is either to downgrade systemd packages, either to disable fsck in fstab
Comment by Théo Bueno (Munrek) - Friday, 06 June 2014, 08:34 GMT
So this is definitely a systemd bug. I always had localtime and fsck never caused trouble until 213.

Edit : may be related with http://cgit.freedesktop.org/systemd/systemd/commit/?id=c343be283b7152554bac0c02493a4e1759c163f7
Comment by daimonion (demon) - Friday, 06 June 2014, 10:17 GMT
I've build systemd with the patch Théo provided (http://cgit.freedesktop.org/systemd/systemd/commit/?id=c343be283b7152554bac0c02493a4e1759c163f7) and the problem is solved.
Comment by Dave Reisner (falconindy) - Friday, 06 June 2014, 22:54 GMT
213-6 in [testing] has this backport.
Comment by Karsten (Smokie) - Saturday, 07 June 2014, 08:40 GMT
I had the same problem. Simply reverting RTC to UTC helped as workaround.
Comment by Ralf Mardorf (Ralf) - Saturday, 07 June 2014, 15:00 GMT
Here systemd sucks too.

My fstab and other related information:

https://mailman.archlinux.org/pipermail/arch-general/2014-June/036468.html
https://mailman.archlinux.org/pipermail/arch-general/2014-June/036469.html
https://mailman.archlinux.org/pipermail/arch-general/2014-June/036488.html
Comment by Dave Reisner (falconindy) - Saturday, 07 June 2014, 15:02 GMT
Again, systemd in [testing] should fix this. If someone who can replicate this problem wants to confirm that, it would actually be useful.
Comment by Ralf Mardorf (Ralf) - Saturday, 07 June 2014, 15:35 GMT
systemd, libsystemd and systemd-sysvcompat 213-6 from testing only solve the fstab issue, but they don't solve the superblock issue.
Comment by Dave Reisner (falconindy) - Saturday, 07 June 2014, 15:55 GMT
Your "superblock issue" is solved by not using localtime. Nothing to be fixed in systemd.
Comment by Ralf Mardorf (Ralf) - Saturday, 07 June 2014, 20:50 GMT
Likely a misunderstanding. I don't care for the superblock warning, but I don't want that fsck does check root at each startup.

https://mailman.archlinux.org/pipermail/arch-general/2014-June/036506.html
https://mailman.archlinux.org/pipermail/arch-general/2014-June/036507.html
Comment by Firestone (Firestone) - Monday, 09 June 2014, 14:13 GMT
Same problem here. Attached a log here: http://pastebin.com/PiEtB28q
Problem was (temporary) solved by disabling all fs checking in fstab (0 in last column) via boot disc instead of switching from localtime. Notice that finding the source of the issue was problematic at first, as each boot entered a loop whilst spouting "[ SKIP ] Ordering cycle found, skipping <a systemd service>".
Symptoms look similar to https://bugs.archlinux.org/task/38210, although that this is on a larger scale.
Comment by Dave Reisner (falconindy) - Monday, 09 June 2014, 14:19 GMT
> Jun 09 14:34:06 Prometheus systemd[1]: Breaking ordering cycle by deleting job loadkeys.service/start
loadkeys.service isn't anything that Arch ships -- you created this ordering problem, not systemd. Likely cause is that you're trying to order this before some very early boot target and not using DefaultDependencies=no.

Closing this, since no one really seems interested in the original bug any longer.

Loading...