FS#27695 - [initscripts] netfs: fix endless waits for stale timeouts

Attached to Project: Arch Linux
Opened by Mark (voidzero) - Saturday, 24 December 2011, 00:26 GMT
Last edited by Eric Belanger (Snowman) - Friday, 16 November 2012, 17:58 GMT
Task Type Feature Request
Category Packages: Core
Status Closed
Assigned To Tom Gundersen (tomegun)
Architecture All
Severity Medium
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

Hello,

If there is a stale nfs handle then it can take forever until the netfs script terminates. This can take up to an hour on hosts with a lot of mounts. A stale nfs handle can also accur because the nfs server is currently unreachable.

I propose the following patch, which makes netfs always exit after 60 seconds unless there is a succesful unmount. (Or, actually, unless the mount processes ended).
This task depends upon

Closed by  Eric Belanger (Snowman)
Friday, 16 November 2012, 17:58 GMT
Reason for closing:  Won't fix
Comment by Mark (voidzero) - Saturday, 24 December 2011, 00:30 GMT
Of course I meant that it will exit sooner in case mount exited.
Comment by Dave Reisner (falconindy) - Monday, 26 December 2011, 20:18 GMT
I like the idea, but I have several issues with the implementation:

- Not a fan of waiting on the singular subshell. The first may hang for 60 seconds and the second might never execute (or could execute outside of our control).
- Your while loop depends on 'j', but as far as I can tell, it's never declared.
- I don't like the dot dot dot crap. If anything, state that you're waiting up to 60s for mounts to clear.
- At the end of 60s, umount could still be running. Why not send it a SIGTERM?
- I'd prefer that we do something more serious than just fail after waiting up to the timeout. repeating umount with -l and -f seems reasonable.

Attaching a diff (untested) that implements all of this. I have no idea if umount will actually respond to the SIGTERM (e.g. in the case of it blocking on a syscall).

Editorial remark: why don't netfs based umount helpers have some sort of timeout? Or do they?
Comment by Mark (voidzero) - Saturday, 28 April 2012, 10:34 GMT
If the mount is stale then umount -l and umount -f do not work, they also wait indefinitely. You could say that the 'dot dot' is crap but that's trivial. There are other scripts doing the same thing.

Hmm, j is indeed never declared, I must have screwed it up.

Sending umount a SIGTERM does nothing with umount, it'll ignore the signal. So that's why the script is just exiting while the umount is still running in the background. The bug has barely been addressed since december so in the mean time I'll just use my own script. It's called netmount and you can find it on the AUR.
Comment by Mark (voidzero) - Saturday, 28 April 2012, 10:35 GMT Comment by Tom Gundersen (tomegun) - Tuesday, 08 May 2012, 12:51 GMT
Could you confirm if this is still a problem with the most recent initscripts?
Comment by Mark (voidzero) - Sunday, 03 June 2012, 08:42 GMT
Sorry, your previous comment slipped from radar. Yes, this is still an issue.
Comment by Mark (voidzero) - Saturday, 23 June 2012, 17:29 GMT
Right. So today and over the last week I spent many many hours into finding a workaround that doesn't lock up the system. Basically I found that the umount.nfs4 binary is broken. Little recap: If an NFS server isn't around anymore and a client system is already shutting down, the final call (by netfs or rc.shutdown) to umount is passed on to umount.nfs4.

My netmount script (see above link) has a 30 second timeout after which it quits regardless of the mount status. But this does not prevent the system from becoming locked... because then it locks when the umount_all routine is started from rc.shutdown which calls umount.nfs4 again.

So then I started to get really annoyed and tried different things resulting in this version:
https://github.com/slacks42/arch-pkgbuilds/blob/netmount2/netmount/netmount.rc

At first I thought I had it solved, but then I saw that the sed line (52) transformed /etc/mtab from a symlink into a regular file. At least it removed the nfs4 lines from that (and from /run/mount/utab), so that the umount.nfs4 binary would not even be called. The final TERM signal sent by rc.shutdown took a little bit longer but the system would at least reboot.

Some suggested the attempt to assign to the client the IP address of the NFS server. But this wouldn't help my cause. In fact, even without any networking devices including lo, the umount.nfs4 process would get stuck. It is a broken program and does not even do anything different when you use the parameters -f, -l and/or -v.

I hope someone feels obliged to pitch in here lest the same future awaits for this bug report.
Comment by Dave Reisner (falconindy) - Saturday, 23 June 2012, 17:34 GMT
Why would you try to edit /etc/mtab (hint: you cant. its read only)? If the problem is in the nfs4 umount helper, then maybe we shouldn't be calling the umount helper. Does adding -i to umount help here?
Comment by Tom Gundersen (tomegun) - Saturday, 23 June 2012, 17:54 GMT
I almost agree with Dave: If the problem is in the nfs4 omount helper, then maybe we should fix that :-) Has a bug been filed to the nfs people?
Comment by Mark (voidzero) - Saturday, 23 June 2012, 18:50 GMT
Yes, I agree with Dave too. Was I not clear enough? If so then I apologise. I'll admit I was a bit in a hurry when I wrote my previous comment.

So just to be clear about my perspective, the mtab edit was not even supposed to occur in the way I described, it was just something I tried along the way on a testing system. And then I found out sed replaced the symlink with a file, but, before I noticed that I was surprised that sed didn't spit errors. No, editing mtab is certainly not something I would suggest. I was just explaining how I got to where I am now. :)

The crucial point I tried to make is that the nfs4 umount binary is indeed the problem. Certainly not anything from the Archlinux side. When I did a google search for "linux-nfs nfs4 stale" I was quite surprised to see so many results. The issue has never properly been adressed, or at least that's how it seems to me.

I just tried umount -i but that also did nothing.

I am not sure if skipping the entire nfs4 umount can cause problems with busy shares on servers that do work. At the least it would leave stale mount information on the server. The command 'showmount <servername>' shows a list of clients of which the server believes mounts are active. This list can be outdated - I just tried it on my server and I see many client addresses that haven't been active for months. And it seems that IPv6 client addresses aren't listed by it at all.

I am not an nfs4 expert, I wonder if anyone is, there is a lot of conflicting information out there. What a headache. Take for example the following link and the first reply to the OP:
http://comments.gmane.org/gmane.linux.nfs/46354

Is one of you guys interested to open an upstream bug report?
Comment by Mark (voidzero) - Saturday, 23 June 2012, 18:51 GMT
Also -- not doing 'umount' could cause an underlying filesystem to not be unmounted properly.
Comment by Tom Gundersen (tomegun) - Saturday, 23 June 2012, 18:53 GMT
I'm sadly no nfs expert either. I do think such a person exists though, probably on the nfs ml ;-)

@Mark: If you would be willing to take this upstream (as you seem to have a good grasp of the problem, and can actually reproduce it) that would be awesome.
Comment by Dave Reisner (falconindy) - Saturday, 23 June 2012, 19:02 GMT
We already pass -f and -l to the umount invocation in netfs. My advice was meant to imply that you should _also_ pass the -i flag.
Comment by Mark (voidzero) - Saturday, 23 June 2012, 19:23 GMT
Dave - ah, 'umount -i' so that umount.nfs4 won't get called. I mistook your message to mean 'umount.nfs4 -i', my bad.

Well I haven't made up my mind about it just yet. First I'll follow your suggestion Tom and file a bug upstream. But not until after this weekend.

I'll keep you guys posted in here.
Comment by Tom Gundersen (tomegun) - Saturday, 23 June 2012, 19:33 GMT
@Mark: thanks! Once you do, please link to it in here.
Comment by Mark (voidzero) - Saturday, 23 June 2012, 19:39 GMT
Ack, will do.
Comment by Mark (voidzero) - Wednesday, 01 August 2012, 20:05 GMT
I'm still on this. I have recompiled nfs-utils with a couple of different options and fiddled with CFLAGS and so. I did that all at one go. I have installed that version on client and server and... when the server is unreachable the client still unmounts!

I will filter out which option causes the endless timeout and report that here, tomorrow probably.
Comment by Tom Gundersen (tomegun) - Sunday, 04 November 2012, 16:21 GMT
If anyone wants this fixed, post a patch to the projects ml. I'm not going to be working on this.

Loading...