Arch Linux

Please read this before reporting a bug:

Do NOT report bugs when a package is just outdated, or it is in the AUR. Use the 'flag out of date' link on the package page, or the Mailing List.

REPEAT: Do NOT report bugs for outdated packages!

FS#28163 - mounted/available /usr filesystem now a required boot dependency

Attached to Project: Arch Linux
Opened by Anonymous Submitter - Monday, 30 January 2012, 02:20 GMT
Last edited by Allan McRae (Allan) - Monday, 30 January 2012, 03:55 GMT
Task Type Bug Report
Category General
Status Closed
Assigned To Dave Reisner (falconindy)
Tom Gundersen (tomegun)
Architecture All
Severity Medium
Priority Normal
Reported Version 4.0.1
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No


Summary and Info:

summary -

My system has /usr as a stand alone file system, which Arch Linux is mounting at the same time as e.g. /home. A recent pacman -Syu upgrade is causing my system to fail to boot properly - it can't fsck the already mounted /usr file system, causing it to fall into single user repair mode.

info -

Since learning and starting to use unix in the early 1990s, I've followed the convention that different parts of the file system are spread across different disks or partitions. The layout I've used is as follows -

/dev/md1 on /
/dev/md6 on /usr
/dev/md3 on /root
/dev/md4 on /tmp
/dev/md5 on /var
/dev/md7 on /opt
/dev/md8 on /home
/dev/md9 on /home2

My ArchLinux system(s) have been configured this way since I first installed it in around 2008 or perhaps earlier.

The have been a number of reasons to do this, which I think are still valid -

- the root file system is small, and is rarely written to, minimising the possibility that a system crash will corrupt it
- the other non-root file systems were where non-system essential software was to be kept. Specifically, /usr was where the applications or non-system essential services software were to be installed (e.g. the print spooler).
- other than for the root file system, when a file system becomes full, it only impacts the use of that file system, and will not prevent the system booting. In particular, /home*, /var and /tmp are file systems which can be prone to filling up, so isolating them from the root file system and /usr is a useful way to protect the system.
- having the different file systems on different devices means that an error on the device only impacts that file system. For example, as shown above, I have all these file systems on RAID1 devices, all residing on the same pair of 250GB disks. An error on /dev/md8 or /dev/md9, the largest of the RAID1 devices, will only impact the /home file systems. If I'd made the pair of 250GB disks a single RAID1 file system, then an error would impact the whole system.

I'd still like to continue to have my system set up this way, and I'd prefer not to have to spend time rebuild it to suit the recent changes to the way Arch Linux boots, which now seem to assume that /usr is part of the root file system.

I think there could be two possible solutions -

- add mounting /usr to the list of file systems that are "early" mounted i.e. when root is first mounted, also mount /usr if it is on a separate file system

- identify the binaries, libraries etc. that are currently being run from /usr during boot before "mount -a" (i.e. mounting of /etc/fstab file systems), and move them into /bin, /sbin, /lib etc., so that /usr doesn't need to be available before "mount -a"

I've worked around this in the interim by mounting /usr just after

findmnt / --options ro &>/dev/null ||
status "Mounting Root Read-Only" mount -o remount,ro /

in /etc/rc.sysinit. Of course, if this file is upgraded, my system will fail to boot again.


Steps to Reproduce:

Install Archlinux with /usr on a different device to /.
This task depends upon

Closed by  Allan McRae (Allan)
Monday, 30 January 2012, 03:55 GMT
Reason for closing:  Duplicate
Additional comments about closing:   FS#28026   FS#28058 
Comment by Dan McGee (toofishes) - Monday, 30 January 2012, 02:30 GMT
This has nothing to do with Pacman, changing projects. It has also already been covered in numerous postings around the Arch websites and mailing lists why this has happened, as well as a working implementation of mounting /usr from the initrd.
Comment by Dave Reisner (falconindy) - Monday, 30 January 2012, 02:44 GMT
too long; didn't read.

Your setup, which uses a separate /usr partition, is in fact, broken. Using the default initscripts (or even systemd), you were told that this is an unsupported setup on every bootup.
More reading:

As Dan mentioned, I've already gone out of my way to placate folks who think that this is still a valid setup by allowing /usr to be mounted from early userspace.
I posted about this on arch-dev-public and arch-general:
I added a section to the wiki:

You're not the first to report this either:  FS#28026 ,  FS#28058 

Please consider fixing your partitioning and merging some of these partitions (including /usr) back into the root FS. Until then, mkinitcpio will allow you to continue with this setup.
Comment by Anonymous Submitter - Monday, 30 January 2012, 03:36 GMT

"Your setup, which uses a separate /usr partition, is in fact, broken". So have you been using unix for 20 years like I have, and know what conventions have been followed for that long? If you can't be bothered reading my report, then you're choosing to remain ignorant about why I and others have done and do what we do with filesystems.

The though of move of /usr to the root file system is only a recent opinion, and certainly not one that is both universal and widely adopted. It's easy to find a webpage on the Internet that agrees with your opinion and everything different "broken". In fact, your system is broken because you've merged /usr with root. Here's a webpage that agrees with me - - have a look at the reasons to split up the file systems across multiple devices.

Your attitude is one that makes people give up on distros. I've recommended Arch Linux to a lot of people of the years I've been using it, and contributed a number of packages to AUR, including a few that have been adopted by [community], such as the radvd and ndisc6 packages. I won't bother recommending it or spending my own time making it better if people such as yourself are starting to be unhelpful and unfriendly towards people who're doing things different, with valid reasons, to what you believe is "the one single way".

If I have to re-install my system, then I may as well review whether I continue to use Arch Linux or not, and whether I abandon the dozen AUR packages I maintain.
Comment by Anonymous Submitter - Monday, 30 January 2012, 03:40 GMT

"working implementation of mounting /usr from the initrd."

it's not working then, because my system broke.

The hooks I have enabled in /etc/mkinitcpio.conf are -

HOOKS="base udev autodetect sata mdadm filesystems usbinput"
Comment by Dave Reisner (falconindy) - Monday, 30 January 2012, 03:42 GMT
Since you won't read even the wiki page, I'll quote it here for you:

/usr as a separate partition

If you keep /usr as a separate partition, you must adhere to the following requirements:
- Add the shutdown hook. At shutdown, initscripts will pivot to a saved copy of the initramfs and allow for /usr (and root) to be properly unmounted from the VFS.
- Add the fsck hook. While recommended for everyone, this is a hard requirement if you want your /usr partition to be fsck'ed at bootup. Without this hook, rc.sysinit will attempt to fsck /usr while still mounted and will fail.

Even without these hooks in place, /usr will be automatically found and mounted by the initramfs, but a setup not using these hooks is not supported.
Comment by Anonymous Submitter - Monday, 30 January 2012, 03:46 GMT
Oh, and as for a warning at boot about /usr on a separate file system not being a supported, good luck seeing it. It's not stored in the kernel buffer, so can't be viewed via dmesg; booting to a console login clears the screen such that the boot messages can't be viewed via the console scroll back buffer (I used to disable that clearing by editing /etc/issue, but now that doesn't work, and I haven't got around to finding somewhere else to fix it). Booting into an XDM suffers from the same problem. IOW, boot messages are mostly useless because they're displayed too fast and you can't go back to look at them, and they're not stored in any of the /var/log files either. For a successful boot, the messages show nothing more than the system is doing something.
Comment by Anonymous Submitter - Monday, 30 January 2012, 03:52 GMT

"Even without these hooks in place, /usr will be automatically found and mounted by the initramfs," That's not the case any more - it fails because the later fsck fails.

The other question is where I'm supposed to have found out about this change of functionality. The place that I'm aware of that notifications about significant changes that could break people's systems is the main Arch Linux web page. There's nothing that mentions of this change going back to 2008-05-28 - was this change to separate /usr support made before then?

Comment by Dave Reisner (falconindy) - Monday, 30 January 2012, 03:52 GMT
console messages are stored in /var/log/boot.