Release Engineering

This project is intented for all release related issues (isos, installer, etc), under the umbrella of the ArchLinux Release Engineers
Tasklist

FS#26344 - mkinitcpio segfaults , an error message "/proc must be mounted" is produced

Attached to Project: Release Engineering
Opened by dv (dv) - Saturday, 08 October 2011, 19:52 GMT
Last edited by Dave Reisner (falconindy) - Thursday, 13 October 2011, 00:39 GMT
Task Type Bug Report
Category AIF
Status Closed
Assigned To Dieter Plaetinck (Dieter_be)
Dave Reisner (falconindy)
Tom Gundersen (tomegun)
Architecture All
Severity Critical
Priority Normal
Reported Version 2011.08.19
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 8
Private No

Details

While trying to install Arch using a 2011.08.19 netinstall image, this error was encountered.

Log output:

installing linux...
>>> Updating module dependencies. Please wait ..
>>> Generating initial ramdisk, using mkinitcpio. Please wait...
/sbin/mkinitcpio: line 160: 2558 Segementation fault mountpoint -q /p
==> ERROR: /proc must be mounted!

The bug seems to originate from the mkinitcpio and util-linux packages.
This task depends upon

Closed by  Dave Reisner (falconindy)
Thursday, 13 October 2011, 00:39 GMT
Reason for closing:  Fixed
Additional comments about closing:  mkinitcpio 0.7.4
Comment by dv (dv) - Saturday, 08 October 2011, 20:03 GMT
As a workaround, continue the installation after the error occurs (AIF warns, but say yes), finish the installation,
then chroot into the installed system:

chroot /mnt/ /bin/bash/

then, manually mount sys, dev, and proc:

mount -t proc proc /proc
mount -t sysfs sys /sys
mount -t devtmpfs udev /dev

then call mkinitcpio:

mkinitcpio -p linux

Everything should be OK afterwards.
Comment by Dave Reisner (falconindy) - Saturday, 08 October 2011, 20:04 GMT
Or better yet, just run mkinitcpio with a basedir:

mkinitcpio -b /mnt -p linux
Comment by Dave Reisner (falconindy) - Saturday, 08 October 2011, 20:25 GMT
To expand on what's happening here:

- This was brought out by a change I made to mkinitcpio [1].
- mountpoint is what's segfaulting (not bash). It's not entirely clear to me why it occurs, but the scenario is that /etc/mtab doesn't exist inside the chroot (but /proc is mounted). I can't reproduce this outside of AIF. This is fixed in util-linux-git [1] but does not solve the problem as mountpoint still fails for a reason that I can't quite grasp.

Possible fixes:
1) Revert the switch from sysvinit's mountpoint to util-linux's. u-l's mountpoint tool makes the grave mistake of requiring /proc/self/mountinfo to be available and fails unconditionally when /proc isn't mounted. sysvinit's implementation relies on dead simple stat(2) calls and compares device and inode numbers. It will never fail.
2) Fix AIF to populate /etc/mtab in the chroot by creating a symlink to /proc/self/mounts. I'm not convinced this will work.
3) Fix AIF to call mkinitcpio outside of the chroot as I've noted above.

I'm in favor of #1 and sending patches to u-l providing the fallback behavior of sysvinit's mountpoint tool when /proc isn't mounted. We're up against a similar problem in initscripts when a user doesn't have an initramfs. /proc is the first thing we check to ensure that its mounted, but mountpoint will fail for the wrong reason. Even though the correct thing happens as a result of that, it's fairly hackish and we shouldn't be relying on it.

[1] http://projects.archlinux.org/mkinitcpio.git/commit/?id=39d7c29
[2] https://github.com/karelzak/util-linux/commit/96bc858
Comment by Tom Gundersen (tomegun) - Sunday, 09 October 2011, 01:12 GMT
@Dave: thanks for figuring it out.

I'd be in favour of pinging Karel to hear his take on the problem before we revert (in the unlikely event this is expected behaviour and we should really be doing XYZ instead; or in case he will cough up a quick patch), I would be doing this myself but I'm sort of incommunicado at the moment.

Also, I might need some help with rebuilding packages when we revert or patch as I don't have a linux box available... (that is, I could build and patch, but I would not be able to test).
Comment by Dave Reisner (falconindy) - Sunday, 09 October 2011, 01:54 GMT
In conjunction with another patch to libmount that fixes a null deref I have a pair of patches lined up for submission that assuming are accepted we should backport to solve this issue. I've beeen drinking all inght so expectbthem to be submitted tomorrow.
Comment by Tom Gundersen (tomegun) - Sunday, 09 October 2011, 02:59 GMT
Awesome!
Comment by Joe (n3hima) - Sunday, 09 October 2011, 13:09 GMT
I'm getting this behaviour from within the installed system so I don't think it can be anything to do with the installer.
Steps to reproduce:
1. Install Arch from the 2010.05 image (the latest one I had already downloaded), following AIF steps as normal.
2. Boot into installed system and configure network.
3. Run a full system upgrade with 'pacman -Syyu'
4. mkinitcpio fails with the above error.

As a workaround, I've commented out lines 160 and 161 (the two lines which call mountpoint) in /sbin/mkinitcpio and made sure manually that /dev and /proc are mounted. This is really really bad but it works.
Comment by Dave Reisner (falconindy) - Sunday, 09 October 2011, 13:50 GMT
Ok, with a clear head I went back and figured out the core of the problem here:

mount -t proc proc /proc # good
mount -t proc none /proc # bad

I wish I was joking. The patches I sent to util-linux are still valid, but they solve an apparently unrelated problem. I'll need to dig into libmount and figure out where this goes sour.

@Joe: Thanks for reporting that this happens outside of the installer.
Comment by Dave Reisner (falconindy) - Sunday, 09 October 2011, 15:22 GMT
It actually looks like, remarkably, the 2nd patch here solves everything (I'm really not sure how/why). Regardless, I went ahead and added another feature missing from the sysvinit impl for the greater good:

http://marc.info/?l=util-linux-ng&m=131813110308558&w=2
http://marc.info/?l=util-linux-ng&m=131813110308560&w=2
http://marc.info/?l=util-linux-ng&m=131817311701163&w=2
http://marc.info/?l=util-linux-ng&m=131817311701173&w=2

@Tom: Assuming these are accepted, we should merge the first two as well as the other patch I submitted recently (that was pulled) to fix the segfault in mountpoint:

http://marc.info/?l=util-linux-ng&m=131732272005806&w=2
Comment by melodos (melodos) - Sunday, 09 October 2011, 16:58 GMT
I followed everything the first two solutions and nothing works.. Is there a fix for this issue?
Comment by Joe (n3hima) - Sunday, 09 October 2011, 17:06 GMT
@melodos:
Can you be a little more specific?
- What is it exactly you are trying to do?
- What happens when you try to do it? Post logs or error messages if possible.
Comment by melodos (melodos) - Sunday, 09 October 2011, 17:15 GMT
Fresh install using a x64 iso "archlinux-2011.08.19-core-x86_64.iso". After pacman -Syu The the mkinitcpio command fails and I'm stuck with no initrd.img. Do I have to use the archlinux live cd and then chroot and do all the steps mentioned in the first answer of this thread?
Comment by Dave Reisner (falconindy) - Sunday, 09 October 2011, 17:39 GMT
@melodos: The problem is just as simple as I've stated it. /proc must be mounted with the source name 'proc' and not 'none'.

findmnt -nuro SOURCE /proc

If that returns 'none' or nothing at all, then yes... I would expect that mkinitcpio will fail. There's no shortage of workarounds in this thread but /proc being mounted "properly" is the crux of the matter.
Comment by Joe (n3hima) - Sunday, 09 October 2011, 18:02 GMT
@melodos
Do what Dave said so as to make sure that this is the bug you're running into and not some other bug with the same consequences.

Then, as a workaround comment out lines 160 and 161 of /sbin/mkinitcpio:
lines 159-161 should now read:
# insist that /proc and /dev be mounted (important for chroots)
#mountpoint -q /proc || die "/proc must be mounted!"
#mountpoint -q /dev || die "/dev must be mounted!"

Then, run mkinitcpio -p linux as root, which will generate a working initrd.img.

Once you've rebooted, make sure to uncomment lines 160 and 161 of /sbin/mkinitcpio as this does not affect the most recent kernel.
Comment by Tom Gundersen (tomegun) - Sunday, 09 October 2011, 21:38 GMT
@Dave: Thanks for your work on this, feel free to apply any accepted patches and push to [testing] when the time comes. Otherwise, I'll do it next week when I have my linux box back (I assume people will not be too happy with packages that are "Built and tested on my MacBook").
Comment by Dave Reisner (falconindy) - Monday, 10 October 2011, 00:16 GMT
I've chosen the path of least resistance and fixed this in mkinitcpio:

http://projects.archlinux.org/mkinitcpio.git/commit/?id=1d86ae6e

mkinitcpio 0.7.4 is in testing which has this commit. This'll stay open for a few days to see what becomes of my util-linux patches.
Comment by Francis (hoistyler) - Thursday, 13 October 2011, 00:08 GMT
I had similar issue when I was upgrading from archlinux-2011.08.19-core-x86_64.iso media install and performing pacman -Syu.
Now, I might be wrong (sorry I am still a newbie) but I believe at this point this process is not done via chroot because you are already logged in your system?
mkinitcpio still fails with /proc not mounted error message, and I have applied the mkinitcpio patch from
http://projects.archlinux.org/mkinitcpio.git/commit/?id=1d86ae6e
manually, and
mkinitcpio -p linux
was successful.

Issue here is, I can't boot. I get Kernel Panic after selecting it from grub. Both fallback and normal.
The error message is:
[2.686223] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[2.686309] Pid: 1, comm: swapper Not tainted 3.0-ARCH #1
...

Could someone please explain what I have done wrong?
Comment by Dave Reisner (falconindy) - Thursday, 13 October 2011, 00:11 GMT
> Now, I might be wrong (sorry I am still a newbie) but I believe at this point this process is not done via chroot because you are already logged in your system?

If you had read the rest of this bug report you'd find in the 2nd post that I suggested running this with a basedir flag. You generated an initramfs for your live environment, not your actual install.
Comment by Francis (hoistyler) - Thursday, 13 October 2011, 00:21 GMT
I am confused now. I am not on my live environment but I have installed ARCH from CD image and booted into my ARCH, then ran pacman -Syu.
Do you still have to follow the chroot method? I thought it was only if you're still in your installation boot(live environment)
Comment by Dave Reisner (falconindy) - Thursday, 13 October 2011, 00:38 GMT
This isn't the place to learn how to use mkinitcpio, nor is your situation relevant to this bug report.

Loading...