FS#34435 - [systemd] Shutdown ocassionally hangs

Attached to Project: Arch Linux
Opened by Cody Maloney (maloney) - Saturday, 23 March 2013, 22:22 GMT
Last edited by Dave Reisner (falconindy) - Sunday, 26 May 2013, 17:01 GMT
Task Type Bug Report
Category Packages: Core
Status Closed
Assigned To Dave Reisner (falconindy)
Architecture All
Severity Medium
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

Description:

Ocassionally when shutting down/rebooting machines they'll hang / get stuck unmounting devices. We utilize kerberized NFS for home directories (mounted at /fserv/home). In pam config pam_systemd.so has "kill-session-processes=true" set.

Here is the output on the screen when it hangs (typed from a photo of the screen, apologies for typos):

[15648.255691] watchdog watchdog0: watchdog did not stop!
Sending SIGTERM to remaining processes...
Sending SIGKILL to remaining processess...
Hardware watchdog 'iTCO_wdt', version 0
Unmounting file systems.
Unmounting /sys/fs/fuse/connections.
Unmounting /sys/kernel/config.
Unmounting /dev/mqueue.
Unmounting /sys/kernel/debug.
Unmounting /dev/hugepages.
Unmounting /tmp.
Unmounting /tmp.
Unmounting /tmp.
Could not unmount /tmp: Device or resource busy
Not all file systems unmounted, 1 left.
Deactivating swaps.
All swaps deactivated.
Detaching loop devices.
All loop devices detached.
Detaching DM devices.
All DM devices detached.
Unmounting file systems.
Unmounting /tmp.
Unmountg /tmp.
Could not unmount /tmp: Device or resource busy
Not all file systems unmounted, 1 left.
Cannot finalize remaining file systems and devices, giving up.
[15704.956147] INFO: task gmain:7582 blocked for more than 120 seconds.
[15704.956195] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[15704.956431] INFO: task xfdesktop:7512 blocked for more than 120 seconds.
[15704.956476] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[15704.956623] INFO: task xfsettingsd:7535 blocked for more than 120 seconds.
[15704.956668] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[15704.956828] INFO: task panel-6-systray:7554 blocked for more than 120 seconds.
[15704.956875] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[15704.957019] INFO: task panel-2-actions:7557 blocked for more than 120 seconds.
[15704.957066] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

NOTE: This seems to cause some ext3 filesystem issues after the reboot.

Additional info:
* systemd 197-4
* linux 3.7.9-2
This task depends upon

Closed by  Dave Reisner (falconindy)
Sunday, 26 May 2013, 17:01 GMT
Reason for closing:  None
Additional comments about closing:  Not a systemd bug
Comment by Kevin (anonymous_user) - Sunday, 24 March 2013, 05:28 GMT
Try updating your machine first. The current systemd and linux packages (non-testing) are 198-1 and 3.8.4-1 respectively.
Comment by Cody Maloney (maloney) - Sunday, 24 March 2013, 22:40 GMT
Still occurs after upgrade, and has actually been going on for a while / several months at this point.
Comment by Daniel Wallace (gtmanfred) - Sunday, 24 March 2013, 23:26 GMT
you should make an effort to figure out which process is using /tmp and making it impossible to unmount
Comment by Cody Maloney (maloney) - Thursday, 28 March 2013, 03:57 GMT
Any suggestions for how to get at the information?

This is late user space and systemd has so far killed the process I try to launch/keep running to inspect what is running.

From the error messages from the watchdog it sounds like a number of xfce processes started by a user are still running, which shouldn't have happened (There is an NFS "feature"/bug where after a kerberos ticket expires, any system calls by the user's processes just hang indefinitely...).

In other cases it is other user processes (I don't have a full list). I sort of suspect that systemd might have cleaned up the kerberos ticket before the processes were all killed for the user, which leads to the processes hanging (Although they should still be SIGKILLable if I remember correctly).

Loading...