FS#45737 - [arch-install-scripts] pacstrap fails to set up API FS in chroot, blocks install from another distro

Attached to Project: Arch Linux
Opened by Bastien Traverse (Neitsab) - Tuesday, 21 July 2015, 22:05 GMT
Last edited by Dave Reisner (falconindy) - Sunday, 12 June 2016, 14:45 GMT
Task Type Bug Report
Category Arch Projects
Status Closed
Assigned To Dave Reisner (falconindy)
Architecture All
Severity Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 1
Private No

Details

I'm installing Arch on a remote server from within a modified Debian 7 host my provider uses as netboot environment, following the instructions on the wiki[1].

When running pacstrap within the chroot (an Arch bootstrap env), it fails with the error "mount: mount point mnt/dev/pts does not exist", whether it exists or not (see logs below).

I tried to bind mount the API filesystems previously mounted on the Arch bootstrap env on "$newroot/{dev,proc,run,sys}", both inside the chroot and from the host, but then pacstrap fails on "mount: proc is already mounted or /mnt/proc busy" (it does this for every API FS depending on which ones are mounted).

I found that commenting out line 104 of pacstrap [2] would make it proceed correctly, with only systemd complaining about missing /dev/urandom (which it needs to generate a machine-id as part of systemd-machine-id-setup [3]), so bind mounting only /dev seems sufficient.

To sum up, pacstrap fails to proceed within a chroot whether "$newroot/dev/pts" exists or not, which I believe is a bug.

[1] https://wiki.archlinux.org/index.php/Install_from_existing_Linux
[2] https://projects.archlinux.org/arch-install-scripts.git/tree/pacstrap.in#n104
[3] http://www.freedesktop.org/software/systemd/man/systemd-machine-id-setup.html

Additional info:
* package version(s): arch-install-scripts 15-1
* config and/or log files etc.

From within the chroot:
# mount
none on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
tmpfs on /dev type tmpfs (rw,relatime,size=10240k,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=402084k,mode=755)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /run/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=804160k)
/dev/mapper/lvm-root on /mnt type ext4 (rw,noatime,data=ordered)
/dev/sda1 on /mnt/boot type ext4 (rw,noatime,data=ordered)
/dev/mapper/lvm-home on /mnt/home type ext4 (rw,noatime,data=ordered)
/proc on /proc type proc (rw,relatime)
tmpfs on /mnt/dev type tmpfs (rw,relatime,size=10240k,mode=755)
**devpts on /mnt/dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)**

[root@rescue /]# pacstrap -i mnt base base-devel
==> Creating install root at mnt
mount: mount point mnt/dev/pts does not exist
==> ERROR: failed to setup chroot mnt
# ls -l /dev/pts
total 0
crw------- 1 root tty 136, 0 Jul 21 21:21 0
c--------- 1 root root 5, 2 Jul 21 02:12 ptmx

Steps to reproduce:

- on a Debian 7 system, follow the steps outlined at https://wiki.archlinux.org/index.php/Install_from_existing_Linux#Method_A:_Using_the_bootstrap_image_.28recommended.29
- when ready to install (pacstrap assumptions[4] are all cleared), run pacstrap and see whether it fails
- then try to manually --rbind mount /dev (already a bind mount from the host system) on /mnt/dev and verify with ls
- retry: still error message
- comment out line 104 (chroot_setup) of pacstrap and execute it again: it works.

[4] https://projects.archlinux.org/arch-install-scripts.git/tree/pacstrap.in#n4
This task depends upon

Closed by  Dave Reisner (falconindy)
Sunday, 12 June 2016, 14:45 GMT
Reason for closing:  Won't fix
Additional comments about closing:  arch-install-scripts requires util-linux>=2.23 (released over 3 years ago)
Comment by Bastien Traverse (Neitsab) - Tuesday, 21 July 2015, 22:21 GMT
Ok, I should have been more precise about "editing pacstrap": I just noticed I linked to the source file (pacstrap.in) and not the actual script (/usr/bin/pacstrap), which seems to do much more stuff dealing with chroot and bind mounts than what I thought.

So to rephrase it: I've applied the steps present in the source file **manually**, copying keyring and mirrorlist after doing pacman -r "$newroot". I didn't comment out the line in the script, I thought it could work but only doing so in the source /might/ work.

It seems nested chrooting isn't a thing... Which may cause pacstrap error.
Comment by Bastien Traverse (Neitsab) - Tuesday, 21 July 2015, 22:28 GMT
All right, and while scanning through the logs I just noticed the initramfs wasn't generated because /proc was missing:

>>> Updating module dependencies. Please wait ...
>>> Generating initial ramdisk, using mkinitcpio. Please wait...
==> ERROR: /proc must be mounted!
error: command failed to execute correctly

After manually bind mounting all the API fs and reinstalling the linux package, mkinitcpio was successfull. However pacstrap still ends on the same error: "mount: proc is already mounted or /mnt/proc busy".
Comment by Bastien Traverse (Neitsab) - Tuesday, 21 July 2015, 23:26 GMT
I get the same errors when I try to chroot into /mnt with arch-chroot:

# arch-chroot /mnt /bin/bash
mount: proc is already mounted or /mnt/proc busy
==> ERROR: failed to setup chroot /mnt
...
# umount -R mnt/proc
# arch-chroot /mnt /bin/bash
mount: sys is already mounted or /mnt/sys busy
==> ERROR: failed to setup chroot /mnt
# umount -R mnt/proc
# arch-chroot /mnt /bin/bash
mount: sys is already mounted or /mnt/sys busy
==> ERROR: failed to setup chroot /mnt
# umount -R mnt/sys
# arch-chroot /mnt /bin/bash
mount: mount point /mnt/dev/pts does not exist
==> ERROR: failed to setup chroot /mnt
# ls -l mnt/dev | grep pts
drwxr-xr-x 2 root root 4096 Jul 21 19:55 pts
# umount -R mnt/dev
# arch-chroot /mnt /bin/bash
mount: mount point /mnt/dev/pts does not exist
==> ERROR: failed to setup chroot /mnt
# ls -l mnt/dev | grep pts
drwxr-xr-x 2 root root 4096 Jul 21 19:55 pts

However there is no problem to do so with classic chroot. So I tried to umount all the API FS and then use arch-chroot, to no avail.
Comment by Dave Reisner (falconindy) - Friday, 24 July 2015, 13:12 GMT
> When running pacstrap within the chroot
This isn't how you use pacstrap. The point is that you don't need to do *any* chroot setup at all, and you certainly do not run it from *inside* the chroot.

Most of the errors here seem to be self-inflicted. Can you report what happens when you *actually* follow the guidance from the wiki, essentially:

1) mount your block devices on /mnt (and none of the kernel mounts)
2) run: pacstrap /mnt
Comment by Bastien Traverse (Neitsab) - Friday, 24 July 2015, 15:08 GMT
> The point is that you don't need to do *any* chroot setup at all, and you certainly do not run it from *inside* the chroot.

Then why does the "Install from existing Linux" article contains sections such as "Creating the chroot" (https://wiki.archlinux.org/index.php/Install_from_existing_Linux#Creating_the_chroot) or "Using the chroot environment" (https://wiki.archlinux.org/index.php/Install_from_existing_Linux#Using_the_chroot_environment)? Is it utterly outdated or bootstrapped installation unsupported?

If you mean I'm supposed to do something like "cd root.x86_64/ && ./usr/bin/pacstrap mnt/" (because pacstrap isn't in the host PATH), all right but that's not how the wiki reads.

The main point is, I believe I've spotted an installation procedure bug in the way pacstrap operates, since on systems where arch-chroot (which pacstrap leverages?) doesn't work because "bash 4 or later [isn't] installed, and unshare [doesn't] support the --fork and --pid options" (https://wiki.archlinux.org/index.php/Install_from_existing_Linux#Method_A:_Using_the_bootstrap_image_.28recommended.29), pacstrap will fail whether inside chroot or not.

I have drawn too much attention to the "nested chroot" thing; rather the problem is: pacstrap relies too much on arch-chroot (I guess), which doesn't work in a few environments (here Debian 7 oldstable, possibly quite modified). Latest trial log:

root@rescue:/# mount
none on / type tmpfs (rw,relatime)
none on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
178.33.XX.XX:/home/pub/rescue.v7 on /nfs type nfs (ro,relatime,vers=3,rsize=8192,wsize=8192,namlen=255,hard,nolock,proto=udp,port=2049,timeo=11,retrans=3,sec=sys,local_lock=all,addr=178.33.XXX.XX)
178.33.XXX.XX:/home/pub/pro-power on /power type nfs (ro,relatime,vers=3,rsize=8192,wsize=8192,namlen=255,hard,nolock,proto=udp,port=2049,timeo=11,retrans=3,sec=sys,local_lock=all,addr=178.33.XXX.XX)
178.33.XXX.XX:/home/pub/commonnfs on /common type nfs (ro,relatime,vers=3,rsize=8192,wsize=8192,namlen=255,hard,nolock,proto=udp,port=2049,timeo=11,retrans=3,sec=sys,local_lock=all,addr=178.33.XXX.XX)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev type tmpfs (rw,relatime,size=10240k,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=402084k,mode=755)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /run/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=804160k)
rpc_pipefs on /nfs/var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
/dev/mapper/lvm-root on /tmp/root.x86_64/mnt type ext4 (rw,noatime,data=ordered)
/dev/sda1 on /tmp/root.x86_64/mnt/boot type ext4 (rw,noatime,data=ordered)
/dev/mapper/lvm-home on /tmp/root.x86_64/mnt/home type ext4 (rw,noatime,data=ordered)

root@rescue:/# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 931.5G 0 disk
|-sda1 8:1 0 128M 0 part /tmp/root.x86_64/mnt/boot
`-sda2 8:2 0 931.4G 0 part
`-crypt (dm-0) 251:0 0 931.4G 0 crypt
|-lvm-root (dm-1) 251:1 0 20G 0 lvm /tmp/root.x86_64/mnt
|-lvm-home (dm-2) 251:2 0 900G 0 lvm /tmp/root.x86_64/mnt/home
`-lvm-swap (dm-3) 251:3 0 1G 0 lvm

root@rescue:/# cd /tmp/root.x86_64
root@rescue:/tmp/root.x86_64# ./bin/arch-chroot mnt/
mount: mount point mnt//dev/pts does not exist
==> ERROR: failed to setup chroot mnt/
root@rescue:/tmp/root.x86_64# ./bin/arch-chroot ./mnt
mount: mount point ./mnt/dev/pts does not exist
==> ERROR: failed to setup chroot ./mnt
root@rescue:/tmp/root.x86_64# ./bin/pacstrap -i mnt
==> Creating install root at mnt
mount: mount point mnt/dev/pts does not exist
==> ERROR: failed to setup chroot mnt
Comment by Dave Reisner (falconindy) - Friday, 24 July 2015, 16:40 GMT
Okay, let's try and focus on what you point out as problems here. In summary:

> pacstrap fails to proceed within a chroot whether "$newroot/dev/pts" exists or not, which I believe is a bug.
Using pacstrap from inside a chroot isn't expected. It's meant for bootstrapping Arch containers from an Arch host, or a new Arch install from an Arch live CD. If you could explain more about *why* you need to do this, maybe we can talk about whether or not it's the right approach or if it's something I even want to support.

> The main point is, I believe I've spotted an installation procedure bug in the way pacstrap operates
Nothing to be done about that here. I didn't write the wiki page you keep referring to, nor can I condone it. Please take your concerns to the wiki page's discussion.

> since on systems where arch-chroot (which pacstrap leverages?) doesn't work because "bash 4 or later [isn't] installed, and unshare [doesn't] support the --fork and --pid options"
Yes, Debian userspace is old. bash4 is over half a decade old. Admittedly, the util-linux incompatability is newer, but it's not something I can't "just drop" without reintroducing bugs in Arch. Suggestions welcome.

> rather the problem is: pacstrap relies too much on arch-chroot (I guess)
I don't know what this means. The only relationship between the two scripts is that they reuse the same code to mount kernel filesystems in preparation for doing "real work". There is no "prerequisite" for calling arch-chroot prior to pacstrap. pacstrap and arch-chroot simply mount kernel filesystems, which are basic operations.
Comment by Bastien Traverse (Neitsab) - Friday, 24 July 2015, 21:37 GMT
> Using pacstrap from inside a chroot isn't expected.
> I didn't write the wiki page you keep referring to, nor can I condone it.

All right, but you'll excuse my confusion then because here is the content of the README contained in the *official* archlinux-bootstrap image [1]:

> To install Arch Linux from another distribution, a root server rescue system or
> a live system, follow the instructions at
> https://wiki.archlinux.org/index.php/Install_from_Existing_Linux. A short version
> is provided below:
>
> * Make sure your system is running Linux 2.6.32 or later.
> * Extract this tarball. If you have enough memory, you can extract it into a
> tmpfs file system. In the example below, we'll assume that you extracted it in
> /tmp.
> * Enter the chroot:
> + If you have bash 4 or later installed, run
> # /tmp/root.x86_64/bin/arch-chroot /tmp/root.x86_64/
> + If you don't have bash 4, run the following commands:
> # cp /etc/resolv.conf /tmp/root.x86_64/etc
> # mount --rbind /proc /tmp/root.x86_64/proc
> # mount --rbind /sys /tmp/root.x86_64/sys
> # mount --rbind /dev /tmp/root.x86_64/dev
> (if /run exists on your system:
> # mount --rbind /run /tmp/root.x86_64/run
> )
> # chroot /tmp/root.x86_64/
> * In the chroot
> + Initialize the pacman keyring:
> # pacman-key --init
> # pacman-key --populate archlinux
> + Mount the destination partition under /mnt
> * Follow the Arch Linux installation guide
> https://wiki.archlinux.org/index.php/Installation_Guide#Install_the_base_system
> starting at the "Install the base system" step.

Which leaves us straight to running pacstrap in a chroot.
[1] Downloaded from https://mirrors.kernel.org/archlinux/iso/2015.07.01/archlinux-bootstrap-2015.07.01-x86_64.tar.gz

> If you could explain more about *why* you need to do this, maybe we can talk about whether or not it's the right approach or if it's something I even want to support.
Exactly what's written in this README: to install Arch from another distribution, a root server rescue system. In more details:

I started to rent a server. I want to install Arch on it, and the first way I thought to do so (it seemed the cleanest one to me) was to use what they call a "netboot rescue environment" to bootstrap Arch installation and do everything from there: partitioning, formatting etc.

So I used the web interface to boot the server on this rescue system, a slightly modified version of Debian 7, ssh-ed into it and started to follow the instructions on that wiki page (which I understand is by no means your responsibility btw, but it seems like it was condoned by at least some Arch devs in the past).

Following those instructions, I downloaded and unpacked the official Arch bootstrap image from a mirror, bind mounted kernel filesystems and chrooted into it to start the installation process.

Then I carried on with the installation until I hit this issue with pacstrap, which I decided to report because I thought it was a not-so-corner-case situation, installing from another GNU/Linux distro.

> The only relationship between the two scripts is that they reuse the same code to mount kernel filesystems in preparation for doing "real work".
After some more testing, this issue appears to stem from the chroot_setup() code in the common file (https://projects.archlinux.org/arch-install-scripts.git/tree/common#n76), which explains why pacstrap and arch-chroot are both affected and return the same error when run ("mount: mount point /dev/pts does not exist").

The chroot variable has actually no incidence here since I can reproduce the error without using one, running pacstrap and arch-chroot directly from the unpacked bootstrap image.

Something in those two lines:

chroot_add_mount udev "$1/dev" -t devtmpfs -o mode=0755,nosuid &&
chroot_add_mount devpts "$1/dev/pts" -t devpts -o mode=0620,gid=5,nosuid,noexec

doesn't play well with the host system it seems, as `/dev/pts` is always the first error invoked. Relevant lines from the rescue system `mount` output:

none on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev type tmpfs (rw,relatime,size=10240k,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=402084k,mode=755)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /run/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=804160k)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)

The thing is, there is apparently something in the chroot_setup() code that differs from the standard manual chroot procedure (like the one outlined at the bottom of https://wiki.archlinux.org/index.php/Install_from_existing_Linux#Method_A:_Using_the_bootstrap_image_.28recommended.29), because I can manually bind-mount kernel FS on the target "$newroot" and then use `pacman -r` option and have everything install properly. I can't use arch-chroot but I can bind mount+chroot. What gives?

> Debian userspace is old. bash4 is over half a decade old. Admittedly, the util-linux incompatability is newer, but it's not something I can't "just drop" without reintroducing bugs in Arch. Suggestions welcome.
The venerable age of Debian 7 indeed probably explains why the shared mounting logic fails, and I understand the benefit of relying on newer features (this is one of the main reasons I use Arch for).

My suggestion is then to include a logic in the chroot_setup() code to fallback on the previously mentionned "regular chroot" procedure in case it cannot proceed normally, so to become more robust in case the operation is done from an old platform (Debian, CentOS...).

BTW, here are the versions of bash and util-linux on this system:

# bash --version
GNU bash, version 4.2.37(1)-release (i486-pc-linux-gnu)
# apt-cache show util-linux
Version: 2.20.1-5.3
Comment by Dave Reisner (falconindy) - Tuesday, 28 July 2015, 14:50 GMT
> Which leaves us straight to running pacstrap in a chroot.
There's a section called "In the chroot", followed by a brand new section pointing you to the basic installation guide. I see why you might be confused by this because there's no explicit mention of "exit the chroot", but I have to point out that you're also the first person to bring this up, hence my skepticism.

> doesn't work because "bash 4 or later [isn't] installed,
> GNU bash, version 4.2.37(1)-release (i486-pc-linux-gnu)

These two things are quite at odds. So is it only the lack of a more recent util-linux?

> My suggestion is then to include a logic in the chroot_setup() code to fallback on the previously mentionned "regular chroot" procedure in case it cannot proceed normally,
No, this just reintroduces bugs. A replacement for the current usage of unshare needs to kill all processes which still remain in the chroot after leaving -- this was the point of PID namespacing. There's a *lot* of problems with re-implementing this in pretty much every other way.

Perhaps if the unshare on the host isn't up to the job, arch-chroot can try to use the unshare from the chroot itself. I don't know what side effects might come with this, though.
Comment by Dave Reisner (falconindy) - Sunday, 12 June 2016, 14:44 GMT
Swinging back here to mark this as wontfix. The necessary version of util-linux is now over 3 years old (tagged April 25th, 2013). If your distribution doesn't offer this, then it's time to upgrade your distro.

Loading...