FS#78702 - [util-linux][linux] overlayfs mount broken

Attached to Project: Arch Linux
Opened by Andrei Pozolotin (random-archer) - Sunday, 04 June 2023, 17:01 GMT
Last edited by Christian Hesse (eworm) - Sunday, 11 June 2023, 19:55 GMT
Task Type Bug Report
Category Packages: Core
Status Closed
Assigned To Christian Hesse (eworm)
Architecture All
Severity Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

PROBLEM:
kernel switch from 6.3.2-arch1-1 into 6.3.5-arch1-1
breaks setup of existing overlayfs mounts

WORKAROUND:
rollback to
Server=https://archive.archlinux.org/repos/2023/05/15/$repo/os/$arch
that is, reverting back to linux-6.3.2 restores normal working behavior


REPRODUCE:

1) use existing mount config, such as

/usr/bin/mount -t overlay \
-o lowerdir=/var/lib/nspawn/extract/image/data/serv/2023-06-01.tar.gz/:/var/lib/nspawn/extract/image/data/extn-tsdb/2023-06-01.tar.gz/:/var/lib/nspawn/extract/image/data/extn-repl/2023-06-01.tar.gz/:/var/lib/nspawn/extract/image/data/extn-uuid/2023-06-01.tar.gz/:/var/lib/nspawn/extract/image/data/extn-host/2023-06-01.tar.gz/:/var/lib/nspawn/extract/image/data/base/2023-06-01.tar.gz/:/var/lib/nspawn/extract/image/arch/base/2023-06-01.tar.gz/:/var/lib/nspawn/extract/archive.archlinux.org/iso/2023.06.01/archlinux-bootstrap-2023.06.01-x86_64.tar.gz/root.x86_64:/var/lib/nspawn/runtime/db-main-n0/zero,upperdir=/var/lib/nspawn/runtime/db-main-n0/root,workdir=/var/lib/nspawn/runtime/db-main-n0/work \
overlay /var/lib/machines/db-main-n0

2) issue above mount command

3) observe non-informative error message from mount command

mount: /var/lib/machines/db-main-n0: wrong fs type, bad option, bad superblock on overlay, missing codepage or helper program, or other error.
dmesg(1) may have more information after failed mount system call.

4) observe lack of any extra information in dmesg or journal


SUGGESTION:

the bug seems to come from kernel overlay module / mount fs-api truncation
of lowerdir/upperdir/workdir mount option, which supposed to support up to 4K long option strings

https://unix.stackexchange.com/questions/563871/is-there-any-way-around-the-length-limitation-in-mount-options
This task depends upon

Closed by  Christian Hesse (eworm)
Sunday, 11 June 2023, 19:55 GMT
Reason for closing:  Fixed
Additional comments about closing:  util-linux 2.39-7
Comment by loqs (loqs) - Sunday, 04 June 2023, 17:38 GMT
Please update the kernel to 6.3.5-arch1-1 again along with any packages that provide kernel modules and test if the issue is introduced by only updating the kernel.
If it is not I would suspect the issue is in util-linux [1][2] rather than the kernel.

[1] https://github.com/util-linux/util-linux/issues/2287
[2] https://github.com/util-linux/util-linux/issues/1992
Comment by Andrei Pozolotin (random-archer) - Sunday, 04 June 2023, 20:25 GMT
1) I confirm that in actual use case reported above I switch whole system by means of
/etc/pacman.d/mirrors, using 2 reference dates: 2023/05/15 vs 2023/06/01
Server=https://archive.archlinux.org/repos/2023/05/15/$repo/os/$arch
Server=https://archive.archlinux.org/repos/2023/06/01/$repo/os/$arch

2) it seems there are no recent changes to the kernel overlayfs module (last 3 month):
https://github.com/torvalds/linux/tree/master/fs/overlayfs

3) thank you for the links to the util-linux issues:
they both do look very similar to use case reported above
Comment by loqs (loqs) - Sunday, 04 June 2023, 22:16 GMT
You could also test a fully updated system with util-linux built with the configure option --disable-libmount-mountfd-support which I think will switch mount back to the older API.
Edit:
util-linux built with --disable-libmount-mountfd-support
https://drive.google.com/file/d/18FH9bYHL1-u_hIcRrW2GYTwWvS4DTOmq/view?usp=share_link util-linux-2.39-4.2-x86_64.pkg.tar.zst
https://drive.google.com/file/d/16fXXiY5Rssg5kk8OGFxEVmTYSrfvB54r/view?usp=share_link util-linux-libs-2.39-4.2-x86_64.pkg.tar.zst
Comment by Andrei Pozolotin (random-archer) - Monday, 05 June 2023, 13:21 GMT
1) thank you for providing custom built util-linux packages

2) I confirm that system roll to:
Server=https://archive.archlinux.org/repos/2023/06/01/$repo/os/$arch
plus using the above packages:
util-linux built with --disable-libmount-mountfd-support
resolves current issue, thank you.

3) please note: there was further conversation in:
https://github.com/util-linux/util-linux/issues/2287#issuecomment-1575702372
and another proposed solution:
https://github.com/util-linux/util-linux/pull/2297
Comment by loqs (loqs) - Monday, 05 June 2023, 14:23 GMT Comment by Andrei Pozolotin (random-archer) - Monday, 05 June 2023, 20:23 GMT
1) thanks again for custom built util-linux packages v-2.39-4.3

2) I confirm that:

a) a system roll to:
Server=https://archive.archlinux.org/repos/2023/06/01/$repo/os/$arch

b) plus using the above custom packages v-2.39-4.3:

c) with injection of environment variable LIBMOUNT_FORCE_MOUNT2
https://github.com/util-linux/util-linux/pull/2297
to affect only mounts in systemd subsystem via systemd-system.conf:
https://www.freedesktop.org/software/systemd/man/systemd-system.conf.html
---
[Manager]
DefaultEnvironment="LIBMOUNT_FORCE_MOUNT2=yes"
---

also resolves current issue, thank you.

3) perhaps this approach (LIBMOUNT_FORCE_MOUNT2) is more flexible
than the previous one (--disable-libmount-mountfd-support)

Comment by Andrei Pozolotin (random-archer) - Thursday, 08 June 2023, 17:42 GMT
note:

1) there were further changes in util-linux:
libmount: introduce LIBMOUNT_FORCE_MOUNT2={always,never,auto}
https://github.com/util-linux/util-linux/pull/2297

2) there is an upcoming change in overlayfs kernel module:
ovl: port to new mount api
https://github.com/brauner/linux/commit/213f92f573accb6638bf108cad3e620073fdcb99
Comment by Andrei Pozolotin (random-archer) - Friday, 09 June 2023, 11:05 GMT
this is now "official", lets test it:
https://github.com/util-linux/util-linux/pull/2297
libmount: introduce LIBMOUNT_FORCE_MOUNT2={always,never,auto}
Comment by loqs (loqs) - Friday, 09 June 2023, 11:51 GMT Comment by Christian Hesse (eworm) - Friday, 09 June 2023, 12:43 GMT
Cherry-picked 39032646fe0f572949024fa23a5b322c5dd9a5bf into util-linux 2.39-7... Anybody can confirm it is fixed?
Comment by Andrei Pozolotin (random-archer) - Sunday, 11 June 2023, 14:50 GMT
thank you for update of util-linux to 2.39-6

I confirm that this also resolves the issue:

a) system roll to:
Server=https://archive.archlinux.org/repos/2023/06/01/$repo/os/$arch

b) update to the above 2.39-6:
util-linux-2.39-6.1 and util-linux-libs-2.39-6.1

c) inject environment variable
LIBMOUNT_FORCE_MOUNT2=always


Comment by Andrei Pozolotin (random-archer) - Sunday, 11 June 2023, 17:06 GMT
I also confirm that as-of-now-current v 2.39-8 resolves this issue as well:

a) system roll to:
Server=https://archive.archlinux.org/repos/2023/06/01/$repo/os/$arch

b) update to the current v 2.39-8:
https://archlinux.org/packages/core-testing/x86_64/util-linux/download/
https://archlinux.org/packages/core-testing/x86_64/util-linux-libs/download/

c) inject environment variable
LIBMOUNT_FORCE_MOUNT2=always

Loading...