FS#25149 - [initscripts] /dev/sda3, sometimes sda4, busy during shutdown

Attached to Project: Arch Linux
Opened by RMS (TechnicalRah) - Sunday, 17 July 2011, 18:22 GMT
Last edited by Tom Gundersen (tomegun) - Sunday, 06 November 2011, 01:14 GMT
Task Type Bug Report
Category Packages: Core
Status Closed
Assigned To Tom Gundersen (tomegun)
Architecture x86_64
Severity Very Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 1
Private No

Details

Description:

Following an update to the initscripts package, every time my system shuts down, now, the shutdown script reports that /dev/sda3 is busy and will be remounted read-only. This happens directly after killing all processes and, momentarily after receiving the message about the partition being busy the system powers off, so I cannot bring up a shell to check what's happening.

Here's some info:

* I have confirmed this to happen on 3 clean installations, as of first reboot after booting from HDD.
* Issue has been confirmed with EFI and MBR partitions.
* I have 8 partitions on my HDD, in order: /, /usr, /usr/local, /opt, /var, /tmp, swap, /home.
* Issue has also happened with the above partition scheme plus an additional /boot partition. With the added /boot partition, /dev/sda4 becomes affected.
* I have checked all system logs under /var but find no errors relating to halt.
* Partitions are listed in order as above in fstab, so I am certain that /usr isn't trying to unmount before /usr/local.
* There are no mounting errors on boot.
* Issue happens whether I'm rebooting or halting.
* I have run a read-write sector scan of my HDD with Spinrite and all sectors are clean.
* I have not confirmed whether this happens on single partition installations.

Additional info:
* package version(s): 2011.06.4-1

Steps to reproduce:

* Install Arch Linux.
* Reboot/halt system following initial HDD boot, and any subsequent boot.
This task depends upon

Closed by  Tom Gundersen (tomegun)
Sunday, 06 November 2011, 01:14 GMT
Reason for closing:  Works for me
Additional comments about closing:  If this can be reproduced without a separate /usr and with initscripts from testing, please open a new report.
Comment by Jelle van der Waa (jelly) - Monday, 18 July 2011, 12:19 GMT
sda3 would be /usr/local right?
Comment by Tom Gundersen (tomegun) - Monday, 18 July 2011, 12:32 GMT
What dir's are mounted on sda3 and sda4? Could you include your fstab.

Could you possibly try if this is also a problem if /usr and /usr/local are on the root partition? (Having /usr on a separate partition is known to cause problems).
Comment by RMS (TechnicalRah) - Monday, 18 July 2011, 17:37 GMT
jelly: That's right, yep!

tomegun: I have attached my fstab for you. Sure, I can try a single partition set-up, no problem; I'll keep you posted.
   fstab (1.4 KiB)
Comment by Tom Gundersen (tomegun) - Monday, 18 July 2011, 18:02 GMT
@RMS: Thanks!
Comment by RMS (TechnicalRah) - Monday, 18 July 2011, 19:11 GMT
Ok, so I have performed a 2 partition (root, swap) installation on a secondary HDD using the same base package selection and installation options as my primary install; both partitions unmount cleanly.
I also tried changing the verbosity level in rc.conf to 10, but that didn't yield anything on shutdown.

I don't think it's the partition scheme that's causing the problem as I have always had this scheme on any distro that I've used, including Arch; I've been using Arch daily for approx 4 years now. That said, I'm still not sure where the problem lies precisely as the rc.shutdown script is only running the standard umount command.

The mind boggles...
Comment by Tom Gundersen (tomegun) - Monday, 18 July 2011, 21:09 GMT
@RMS: thanks for the feedback. So some process is accessing /usr after everything should have been killed. Do you get any output if you add a call to lsof just before attempting to unmount?
Comment by RMS (TechnicalRah) - Monday, 18 July 2011, 21:50 GMT
tomegun: I did as you suggested and I've dumped the output to a text file and the offending process appears to be "/usr/lib/locale/locale-archive". Now, I have set my locale to demonise in rc.conf, so maybe that's it. I have attached the lsof output for you to see, too.
   lsof (24.9 KiB)
Comment by RMS (TechnicalRah) - Monday, 18 July 2011, 21:56 GMT
I can confirm that setting DAEMON_LOCALE to "no" prevents this umount error appearing when one's /usr is separately partitioned. Bug squashed.
Comment by Tom Gundersen (tomegun) - Monday, 18 July 2011, 22:50 GMT
@RMS: Brilliant! That (DAEMON_LOCALE) explains why the bug suddenly appeared. It would be still nice to fix this properly, but I'll have to think more about that.

(By the way: the meaning of DAEMON_LOCALE="yes" is that your LOCALE variable is passed to the daemons. If you set it to "no", we unset LOCALE before starting daemons.)
Comment by RMS (TechnicalRah) - Tuesday, 19 July 2011, 17:39 GMT
tomegun: Do you know why it's not stopped when the processes are killed? How does the process survive? Also, thank you for your kind help and input with this bug. :)
Comment by Tom Gundersen (tomegun) - Tuesday, 19 July 2011, 22:32 GMT
Something in rc.shutdown must be accessing /usr/lib/locale/locale-archive. From what I have read this could be anything that uses glibc.

If anyone can figure out exactly what happens I'd be happy to hear it. My suspicion is that this is something we cannot fix, and that we should officially stop pretending that we support a separate /usr, but I'd like to know for sure.
Comment by Dave Reisner (falconindy) - Wednesday, 20 July 2011, 00:21 GMT
fuser(1) will tell you who is using a file.
Comment by Tom Gundersen (tomegun) - Wednesday, 20 July 2011, 12:53 GMT
I just did a short test:

# strace -e trace=file mount -a
...
open("/usr/lib/locale/locale-archive", O_RDONLY) = 3


So it looks like even "mount" accesses locale-archive. This could be worked around by unsetting the locale in rc.shutdown, but knowing the number of other problems caused by using a separate /usr, I don't think this is worth it.

If someone is motivated to do this, I guess one option could be to support mounting and unmounting /usr in the mkinitramfs.
Comment by Tom Gundersen (tomegun) - Monday, 25 July 2011, 14:41 GMT
I have made patches for mkinitcpio and initscripts that allow us to completely unmount the rootfs and all its submounts. It might take some time for this to be released (it first has to be accepted by Thomas into mkinitcpio and that has to be released, and then an initscripts release has to follow), in the meantime I can only urge people to migrate away from having /usr on a separate partition. Until mkinitcpio gets support for mounting /usr (which noone has offered to code) this will continue to be unstable.

Loading...