FS#27695 - [initscripts] netfs: fix endless waits for stale timeouts
Attached to Project:
Arch Linux
Opened by Mark (voidzero) - Saturday, 24 December 2011, 00:26 GMT
Last edited by Eric Belanger (Snowman) - Friday, 16 November 2012, 17:58 GMT
Opened by Mark (voidzero) - Saturday, 24 December 2011, 00:26 GMT
Last edited by Eric Belanger (Snowman) - Friday, 16 November 2012, 17:58 GMT
|
Details
Hello,
If there is a stale nfs handle then it can take forever until the netfs script terminates. This can take up to an hour on hosts with a lot of mounts. A stale nfs handle can also accur because the nfs server is currently unreachable. I propose the following patch, which makes netfs always exit after 60 seconds unless there is a succesful unmount. (Or, actually, unless the mount processes ended). |
This task depends upon
- Not a fan of waiting on the singular subshell. The first may hang for 60 seconds and the second might never execute (or could execute outside of our control).
- Your while loop depends on 'j', but as far as I can tell, it's never declared.
- I don't like the dot dot dot crap. If anything, state that you're waiting up to 60s for mounts to clear.
- At the end of 60s, umount could still be running. Why not send it a SIGTERM?
- I'd prefer that we do something more serious than just fail after waiting up to the timeout. repeating umount with -l and -f seems reasonable.
Attaching a diff (untested) that implements all of this. I have no idea if umount will actually respond to the SIGTERM (e.g. in the case of it blocking on a syscall).
Editorial remark: why don't netfs based umount helpers have some sort of timeout? Or do they?
Hmm, j is indeed never declared, I must have screwed it up.
Sending umount a SIGTERM does nothing with umount, it'll ignore the signal. So that's why the script is just exiting while the umount is still running in the background. The bug has barely been addressed since december so in the mean time I'll just use my own script. It's called netmount and you can find it on the AUR.
My netmount script (see above link) has a 30 second timeout after which it quits regardless of the mount status. But this does not prevent the system from becoming locked... because then it locks when the umount_all routine is started from rc.shutdown which calls umount.nfs4 again.
So then I started to get really annoyed and tried different things resulting in this version:
https://github.com/slacks42/arch-pkgbuilds/blob/netmount2/netmount/netmount.rc
At first I thought I had it solved, but then I saw that the sed line (52) transformed /etc/mtab from a symlink into a regular file. At least it removed the nfs4 lines from that (and from /run/mount/utab), so that the umount.nfs4 binary would not even be called. The final TERM signal sent by rc.shutdown took a little bit longer but the system would at least reboot.
Some suggested the attempt to assign to the client the IP address of the NFS server. But this wouldn't help my cause. In fact, even without any networking devices including lo, the umount.nfs4 process would get stuck. It is a broken program and does not even do anything different when you use the parameters -f, -l and/or -v.
I hope someone feels obliged to pitch in here lest the same future awaits for this bug report.
So just to be clear about my perspective, the mtab edit was not even supposed to occur in the way I described, it was just something I tried along the way on a testing system. And then I found out sed replaced the symlink with a file, but, before I noticed that I was surprised that sed didn't spit errors. No, editing mtab is certainly not something I would suggest. I was just explaining how I got to where I am now. :)
The crucial point I tried to make is that the nfs4 umount binary is indeed the problem. Certainly not anything from the Archlinux side. When I did a google search for "linux-nfs nfs4 stale" I was quite surprised to see so many results. The issue has never properly been adressed, or at least that's how it seems to me.
I just tried umount -i but that also did nothing.
I am not sure if skipping the entire nfs4 umount can cause problems with busy shares on servers that do work. At the least it would leave stale mount information on the server. The command 'showmount <servername>' shows a list of clients of which the server believes mounts are active. This list can be outdated - I just tried it on my server and I see many client addresses that haven't been active for months. And it seems that IPv6 client addresses aren't listed by it at all.
I am not an nfs4 expert, I wonder if anyone is, there is a lot of conflicting information out there. What a headache. Take for example the following link and the first reply to the OP:
http://comments.gmane.org/gmane.linux.nfs/46354
Is one of you guys interested to open an upstream bug report?
@Mark: If you would be willing to take this upstream (as you seem to have a good grasp of the problem, and can actually reproduce it) that would be awesome.
Well I haven't made up my mind about it just yet. First I'll follow your suggestion Tom and file a bug upstream. But not until after this weekend.
I'll keep you guys posted in here.
I will filter out which option causes the endless timeout and report that here, tomorrow probably.