FS#46137 - [arch-install-scripts] pacstrap breaks nfs-mounted pkg cache and sync folders, in non-mountpoint dir

Attached to Project: Arch Linux
Opened by James Harvey (jamespharvey20) - Saturday, 29 August 2015, 03:18 GMT
Last edited by Dave Reisner (falconindy) - Wednesday, 05 October 2016, 11:57 GMT
Task Type Bug Report
Category Packages: Extra
Status Closed
Assigned To Dave Reisner (falconindy)
Architecture All
Severity Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

FIRST the bug report I almost submitted for pacman, before figuring out it is a pacstrap bug

Nfs-server machine already has /var/cache/pacman/pkg and /var/lib/pacman/sync. I want to nfs mount these to other systems running the live ISO, so up-to-date packages already in my internal network aren't downloaded from the public mirrors. (I will also be nfs mounting my local repository which has built AUR packages, which is what drove me to this point, but that's left out of this bug report as not needed.)

Saw quite a few people saying they've done this, but it wasn't working for me.

Turns out this works:
=====
(these commands on the live Arch ISO, no partitions for reduced test-case, assuming 192.168.1.2 has the shares)
mkfs.btrfs -f /dev/vda
mount /dev/vda /mnt
mkdir -p /mnt/var/cache/pacman/pkg
mount 192.168.1.2:/var/cache/pacman/pkg /mnt/var/cache/pacman/pkg
mkdir -p /mnt/var/lib/pacman/sync
mount 192.168.1.2:/var/lib/pacman/sync /mnt/var/lib/pacman/sync
pacstrap -d /mnt man-pages
umount /mnt/var/cache/pacman/pkg
umount /mnt/var/lib/pacman/sync
umount /mnt
=====

But, this does not work:
=====
mkfs.btrfs -f /dev/vda
mount /dev/vda /mnt
mkdir /mnt/root
mkdir -p /mnt/root/var/cache/pacman/pkg
mount 192.168.1.2:/var/cache/pacman/pkg /mnt/root/var/cache/pacman/pkg
mkdir -p /mnt/root/var/lib/pacman/sync
mount 192.168.1.2:/var/lib/pacman/sync /mnt/root/var/lib/pacman/sync
pacstrap -d /mnt/root man-pages
umount /mnt/root/var/cache/pacman/pkg
umount /mnt/root/var/lib/pacman/sync
rm -r /mnt/root
umount /mnt
=====

The ONLY difference between these lists of commands is the first one uses /mnt as the new root, and the second one uses /mnt/root, where /mnt is the mountpoint. Im my actual case, /mnt is a btrfs volume and /mnt/root is a btrfs subvolume that will be mounted as root in the system being installed. But, I removed btrfs from the test scenario because I found it acts the same way whether it's a btrfs subvolume directory or just a normal directory.

I'm just installing "man-pages" with pacstrap in these test cases, because that's a base package that has no dependencies. (Quicker testing.)

By this "works", for the first list, I mean that pacman says core/extra/community is up to date (assuming that they are - i.e. that the nfs-server has refreshed them recently) and installs man-pages without downloading it.

By this "does not work", for the second list, yes, it technically installs. But, my point is that pacman downloads the package databases from scratch, and downloads man-pages from one of the official mirrors, when it should be doing neither.


If I install strace in the live Arch ISO environment (so not using /mnt or pacstrap), and edit /usr/bin/pacstrap:361 from "pacman -r..." to "strace pacman -r..." and re-run the second list, above, and add to pacstrap "> pacs 2>&1", I see in the pacs output file unexpected ENOENT's, including:

=====
access("/mnt/root/var/lib/pacman/sync/core.db", R_OK) = -1 ENOENT (No such file or directory)
stat("/mnt/root/var/cache/pacman/pkg/man-pages-4.02-1-any.pkg.tar.xz", 0x7ffd71225070) = -1 ENOENT (No such file or directory)
=====

But these files very much exist: (before I perform the umount's in the lists above)
=====
# ll /mnt/root/var/cache/pacman/pkg/man-pages*
-rw-r--r-- 1 root root 5669120 Aug 11 16:42 /mnt/root/var/cache/pacman/pkg/man-pages-4.02-1-any.pkg.tar.xz
# ll /mnt/root/var/lib/pacman/sync/core.db
-rw-r--r-- 1 root root 124041 Aug 28 18:33 /mnt/root/var/lib/pacman/sync/core.db
=====



NOW to show this is a pacstrap bug, not pacman...

Re-ran these two lists, except ran pacstrap through "bash -x " to get bash debug output, redirecting each list to a separate output file. Edited the second list's file, and replaced "/mnt/root" with "/mnt" to eliminate the expected differences, and see what the unexpected differences were, if any. I then saw these entries, which are listed below in their original form (not my search-replace form):

=====
chroot_add_mount /mnt/root /mnt/root --bind
mount /mnt/root /mnt/root --bind
=====

If in the archiso console I have the nfs directories mounted, then this shows these commands are the issue because:

=====
# ll /mnt/root/var/cache/pacman/pkg | wc -l
243
# mount /mnt/root /mnt/root --bind
# ll /mnt/root/var/cache/pacman/pkg | wc -l
1
=====

The command "mount /mnt/root /mnt/root --bind" hides the nfs mounts.

So, I think the bug is in pacstrap's chroot_maybe_add_mount(). Unless this is a bug in mount, not sure what the expected behavior of this command would be... But even still, pacstrap could guard against it.
This task depends upon

Closed by  Dave Reisner (falconindy)
Wednesday, 05 October 2016, 11:57 GMT
Reason for closing:  No response
Comment by James Harvey (jamespharvey20) - Saturday, 29 August 2015, 04:40 GMT
Being unsure about the expected behavior of mount in this scenario, checked with util-linux which said this is acting as intended. See http://article.gmane.org/gmane.linux.utilities.util-linux-ng/11609
Comment by Dave Reisner (falconindy) - Saturday, 29 August 2015, 13:38 GMT
Have you seen pacstrap's -c option?
Comment by James Harvey (jamespharvey20) - Saturday, 29 August 2015, 21:44 GMT
Without pacstrap's -c, pacman's args gets '--cachedir="$newroot/var/cache/pacman/pkg"' With -c, pacman defaults to /var/cache/pacman/pkg

Using -c still leaves the db path as /mnt/root/var/lib/pacman, and the lock file in that directory as db.lck. And the visibility of these nfs-mounts is hidden to pacman.

pacman has a -b (--dbpath) option which isn't directly supported by pacstrap. I can put "--dbpath /var/lib/pacman" after the packages list argument on pacstrap, and it forwards those options along to pacman, and doing this successfully installs with the nfs mounts under / rather than /mnt/root.

Another workaround (which I can use since I'm using btrfs, not just directories like my example shows) is to mount btrfs subvolid=0 to /mnt, and btrfs subvol=root to /mnt_root, and nfs-mount underneath /mnt_root, and use this directory with pacstrap. Since the nfs mounts are directly under the new root instead of having an intermediate subdirectory, the nfs mounts aren't hidden by pacstrap.

Even though there are workarounds to get past this, I think either the "bug" (as I see it) should be fixed, or this behavior should be documented if no code change will happen, explaining one or both of the work-arounds needed.

Ultimately, if I can nfs mount without workarounds to a directory right under my new root, it's quite unexpected (to me anyway) that I can't nfs mount in a subdirectory of my new root.
Comment by James Harvey (jamespharvey20) - Sunday, 30 August 2015, 04:14 GMT
Just realized, I think (haven't tried it since I'm using the btrfs mount subvol=root workaround) using --dbpath isn't an option, because that would also change the location of the local db directory (such as /var/lib/pacman/local) being used, so the local database would be put in the arch iso filesystem rather than the new root filesystem, and be gone upon shutdown/reboot. My guess is that missing the /local/ entries would be very problematic, although I don't know for sure how that directory is used.
Comment by Dave Reisner (falconindy) - Sunday, 12 June 2016, 14:58 GMT
Can you just mount the NFS share "locally" and pass --cachedir to pacstrap to point to it? The other alternative would be to plumb in some sort of --bind option to the chroot setup logic, but that would still require passing the path (so not really any better than using --cachedir). I'm not keen on trying to guess about the contents of the chroot.

Loading...