FS#26337 - [util-linux] Very long mount of disks

Attached to Project: Arch Linux
Opened by Pierre Franco (nob) - Saturday, 08 October 2011, 11:42 GMT
Last edited by Tom Gundersen (tomegun) - Wednesday, 19 October 2011, 16:45 GMT
Task Type Bug Report
Category Packages: Core
Status Closed
Assigned To Dave Reisner (falconindy)
Tom Gundersen (tomegun)
Architecture All
Severity Medium
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 3
Private No

Details

Description:

Mounting of disks is very slow: one minute for a mount command.
I had to downgrade to the 2.20-2 version of util-linux to correct it.
I did a strace log, 95% of time is spent in nanosleeps.

Additional info:
* package version: util-linux 2.20-3

Steps to reproduce:

Download or recompile util-linux 2.20-2(Arch Rollback Machine, ...) and install it.
This task depends upon

Closed by  Tom Gundersen (tomegun)
Wednesday, 19 October 2011, 16:45 GMT
Reason for closing:  Fixed
Additional comments about closing:  The real fix will be to replace /etc/mtab with a symlink, but we solve this as best we can for the time being by clearing all locks on boot.
Comment by Tom Gundersen (tomegun) - Monday, 10 October 2011, 10:30 GMT
Thanks for reporting.

Could you possibly give some more info about your system:
* /etc/fstab
* dmesg
* verbose output of the relevant mount command.

Which of your mount points are affected by this? Do you use lvm/mdadm or anything like that?
Comment by Dave Reisner (falconindy) - Monday, 10 October 2011, 12:00 GMT
I had a VM which could replicate this, and then I managed to unbreak it.

Please try upgrading back to 2.20-3 and replacing /etc/mtab with a symlink to /proc/self/mounts.
Comment by Pierre Franco (nob) - Monday, 10 October 2011, 17:20 GMT
#
# /etc/fstab: static file system information
#
# <file system> <dir> <type> <options> <dump> <pass>
proc /proc proc defaults 0 0

/dev/sda2 / ext4 defaults 0 1
/dev/sda1 /boot ext4 defaults 0 2
/dev/sda7 /home btrfs defaults 0 2
-------------------------------------------------------------------------------------
dmesg is the attachement
-------------------------------------------------------------------------------------
All the mount points are affected. I don't use lvm/mdadm.
-------------------------------------------------------------------------------------
Verbose output for an example of mount
[foo@foodesktop ~]$ sudo mount -v /dev/sda6 /media/disk
mount : vous n'avez pas indiqué le type de système de fichiers de /dev/sda6
Je vais essayer le type btrfs
/dev/sda6 on /media/disk type btrfs (rw)
(i'm french, the first sentence means that I didn't give the filesystem type, so he tries btrfs(it's an btrfs partition, but ext4 and ntfs are affected too).

nob.dir
   dmesg (59.7 KiB)
Comment by Alexandre Leuck (Weisse) - Friday, 14 October 2011, 12:40 GMT
I have the same problem, boot takes to long to mount the local partitions since yesterday.

Follows my fstab configuration
#
# /etc/fstab: static file system information
#
# <file system> <dir> <type> <options> <dump> <pass>
tmpfs /tmp tmpfs nodev,nosuid 0 0
/dev/sda5 swap swap defaults 0 0
/dev/sda6 / ext4 defaults,user_xattr 0 1
/dev/sda3 /media/sda3 ntfs-3g uid=white,gid=users 0 0

Follows attached my mtab
(application/x-gzip)    mtab.gz (0.2 KiB)
Comment by Dave Reisner (falconindy) - Friday, 14 October 2011, 13:34 GMT
Going to need more debugging than just config files to hunt this down...

Please install gdb and a util-linux debug package below...

http://pkgbuild.com/~dreisner/util-linux-2.20-3-i686.pkg.tar.xz
http://pkgbuild.com/~dreisner/util-linux-2.20-3-x86_64.pkg.tar.xz

Run 'gdb --args mount -t tmpfs tmpfs /boot' (or some other mount operation that hangs)

At the gdb prompt, type 'run'. When it hangs, hit ctrl-c and then type 'bt full'. Post the output here.
Comment by Pierre Franco (nob) - Friday, 14 October 2011, 15:35 GMT
Sadly(or happily), when i upgraded again to 2.20-3 to give you the output, it didn't hang more... I can't help you more, sorry.
Comment by Dave Reisner (falconindy) - Friday, 14 October 2011, 15:39 GMT
No, that's roughly my experience. This is probably some oddity in the way mtab is being written moving to mount being powered by libmount.
Comment by Tom Gundersen (tomegun) - Sunday, 16 October 2011, 19:19 GMT
If anyone are able to get a working and a broken mtab from the same system, that would be great...
Comment by Tom Gundersen (tomegun) - Tuesday, 18 October 2011, 10:30 GMT
Hi guys, could I ask for the following from everyone who can reproduce this:

Confirmation that /etc/mtab is not a symlink, and that you are using the stock Arch kernel (not self-compiled).

Does deleting /etc/mtab, symlinking it to /proc/self/mounts, and rebooting make things work?
Comment by Wenzhi Liang (lang2) - Tuesday, 18 October 2011, 15:18 GMT
gdb back trace here. strace output also attached.


(gdb) bt full
#0 0x00007f332d8f49f0 in sigprocmask () from /lib/libc.so.6
No symbol table info available.
#1 0x000000000040ce6e in block_signals (how=1) at sundries.c:97
sigs = {__val = {18446744067267099631, 18446744073709551615 <repeats 15 times>}}
#2 0x0000000000406129 in try_mount_one (spec0=0x25493b0 "/dev/sdb1", node0=0x2549350 "/media/wubai", types0=0x2549590 "auto", opts0=0x2549390 "defaults", freq=0, pass=0, ro=0)
at mount.c:1688
mo = 0x2552d20 "rw"
tp = 0x2549320 "ext3"
res = 0
status = 0
special = 0
mnt5_res = 0
mnt_err = 0
flags = 0
extra_opts = 0x0
mount_opts = 0x0
opts = 0x2549430 "defaults"
spec = 0x25493d0 "/dev/sdb1"
node = 0x25493f0 "/media/wubai"
types = 0x2549320 "ext3"
user = 0x0
loop = 0
loopdev = 0x0
loopfile = 0x25493d0 "/dev/sdb1"
statbuf = {st_dev = 139857784245104, st_ino = 139857794379776, st_nlink = 4294967295, st_mode = 772338181, st_uid = 32563, st_gid = 769971368, __pad0 = 32563,
st_rdev = 39097168, st_size = 2290400, st_blksize = 0, st_blocks = 0, st_atim = {tv_sec = 0, tv_nsec = 139857790025728}, st_mtim = {tv_sec = 139857792372916,
tv_nsec = 1}, st_ctim = {tv_sec = 0, tv_nsec = 6377848}, __unused = {139857784265024, 0, 39097168}}
opts1 = 0x2549430 "defaults"
spec1 = 0x25493d0 "/dev/sdb1"
node1 = 0x25493f0 "/media/wubai"
types1 = 0x2549410 "auto"
#3 0x00000000004070a2 in mount_one (spec=0x25493b0 "/dev/sdb1", node=0x2549350 "/media/wubai", types=0x2549590 "auto", fstabopts=0x2549970 "defaults", cmdlineopts=0x0, freq=0,
pass=0) at mount.c:2018
nspec = 0x25493b0 "/dev/sdb1"
opts = 0x2549390 "defaults"
#4 0x000000000040848f in main (argc=1, argv=0x7fff9012e360) at mount.c:2666
c = -1
result = 0
specseen = 0
options = 0x0
test_opts = 0x0
node = 0x7fff9012e370 "\273\375\022\220\377\177"
spec = 0x0
label = 0x0
uuid = 0x0
types = 0x0
p = 0x7fff9012fda6 "/mount"
mc = 0x2549990
fd = 7


Comment by Dave Reisner (falconindy) - Tuesday, 18 October 2011, 15:29 GMT
Awesome! So, mount hangs because /etc/mtab~ exists (a lock file) and isn't actually held by a running process. I can reproduce this. Additionally, it explains why linking /etc/mtab to /proc/self/mounts will "fix" this.
Comment by Tom Gundersen (tomegun) - Tuesday, 18 October 2011, 16:41 GMT
initscripts in [testing] works around this be deleting all stale locks on boot. Could someone verify that this solves it for them?
Comment by Wenzhi Liang (lang2) - Wednesday, 19 October 2011, 16:39 GMT
I removed /etc/mtab~ and /etc/mtab.fuselock and the problem went away. Hope that's what you did to work around it too.
Comment by Tom Gundersen (tomegun) - Wednesday, 19 October 2011, 16:44 GMT
@lang2: yeah, we now delete these files at boot if they exist, so rebooting will at least fix the problem. I'll close as fixed.

Loading...