FS#78702 - [util-linux][linux] overlayfs mount broken
Attached to Project:
Arch Linux
Opened by Andrei Pozolotin (random-archer) - Sunday, 04 June 2023, 17:01 GMT
Last edited by Christian Hesse (eworm) - Sunday, 11 June 2023, 19:55 GMT
Opened by Andrei Pozolotin (random-archer) - Sunday, 04 June 2023, 17:01 GMT
Last edited by Christian Hesse (eworm) - Sunday, 11 June 2023, 19:55 GMT
|
Details
PROBLEM:
kernel switch from 6.3.2-arch1-1 into 6.3.5-arch1-1 breaks setup of existing overlayfs mounts WORKAROUND: rollback to Server=https://archive.archlinux.org/repos/2023/05/15/$repo/os/$arch that is, reverting back to linux-6.3.2 restores normal working behavior REPRODUCE: 1) use existing mount config, such as /usr/bin/mount -t overlay \ -o lowerdir=/var/lib/nspawn/extract/image/data/serv/2023-06-01.tar.gz/:/var/lib/nspawn/extract/image/data/extn-tsdb/2023-06-01.tar.gz/:/var/lib/nspawn/extract/image/data/extn-repl/2023-06-01.tar.gz/:/var/lib/nspawn/extract/image/data/extn-uuid/2023-06-01.tar.gz/:/var/lib/nspawn/extract/image/data/extn-host/2023-06-01.tar.gz/:/var/lib/nspawn/extract/image/data/base/2023-06-01.tar.gz/:/var/lib/nspawn/extract/image/arch/base/2023-06-01.tar.gz/:/var/lib/nspawn/extract/archive.archlinux.org/iso/2023.06.01/archlinux-bootstrap-2023.06.01-x86_64.tar.gz/root.x86_64:/var/lib/nspawn/runtime/db-main-n0/zero,upperdir=/var/lib/nspawn/runtime/db-main-n0/root,workdir=/var/lib/nspawn/runtime/db-main-n0/work \ overlay /var/lib/machines/db-main-n0 2) issue above mount command 3) observe non-informative error message from mount command mount: /var/lib/machines/db-main-n0: wrong fs type, bad option, bad superblock on overlay, missing codepage or helper program, or other error. dmesg(1) may have more information after failed mount system call. 4) observe lack of any extra information in dmesg or journal SUGGESTION: the bug seems to come from kernel overlay module / mount fs-api truncation of lowerdir/upperdir/workdir mount option, which supposed to support up to 4K long option strings https://unix.stackexchange.com/questions/563871/is-there-any-way-around-the-length-limitation-in-mount-options |
This task depends upon
Closed by Christian Hesse (eworm)
Sunday, 11 June 2023, 19:55 GMT
Reason for closing: Fixed
Additional comments about closing: util-linux 2.39-7
Sunday, 11 June 2023, 19:55 GMT
Reason for closing: Fixed
Additional comments about closing: util-linux 2.39-7
If it is not I would suspect the issue is in util-linux [1][2] rather than the kernel.
[1] https://github.com/util-linux/util-linux/issues/2287
[2] https://github.com/util-linux/util-linux/issues/1992
/etc/pacman.d/mirrors, using 2 reference dates: 2023/05/15 vs 2023/06/01
Server=https://archive.archlinux.org/repos/2023/05/15/$repo/os/$arch
Server=https://archive.archlinux.org/repos/2023/06/01/$repo/os/$arch
2) it seems there are no recent changes to the kernel overlayfs module (last 3 month):
https://github.com/torvalds/linux/tree/master/fs/overlayfs
3) thank you for the links to the util-linux issues:
they both do look very similar to use case reported above
Edit:
util-linux built with --disable-libmount-mountfd-support
https://drive.google.com/file/d/18FH9bYHL1-u_hIcRrW2GYTwWvS4DTOmq/view?usp=share_link util-linux-2.39-4.2-x86_64.pkg.tar.zst
https://drive.google.com/file/d/16fXXiY5Rssg5kk8OGFxEVmTYSrfvB54r/view?usp=share_link util-linux-libs-2.39-4.2-x86_64.pkg.tar.zst
2) I confirm that system roll to:
Server=https://archive.archlinux.org/repos/2023/06/01/$repo/os/$arch
plus using the above packages:
util-linux built with --disable-libmount-mountfd-support
resolves current issue, thank you.
3) please note: there was further conversation in:
https://github.com/util-linux/util-linux/issues/2287#issuecomment-1575702372
and another proposed solution:
https://github.com/util-linux/util-linux/pull/2297
https://drive.google.com/file/d/1NcAToo9F0Hp5RR4xIYXEboSXiYXyL72u/view?usp=share_link util-linux-2.39-4.3-x86_64.pkg.tar.zst
https://drive.google.com/file/d/19ojAAPKUcjZPCcPfCNzQ3Fr5dsxp5ytI/view?usp=share_link util-linux-libs-2.39-4.3-x86_64.pkg.tar.zst
2) I confirm that:
a) a system roll to:
Server=https://archive.archlinux.org/repos/2023/06/01/$repo/os/$arch
b) plus using the above custom packages v-2.39-4.3:
c) with injection of environment variable LIBMOUNT_FORCE_MOUNT2
https://github.com/util-linux/util-linux/pull/2297
to affect only mounts in systemd subsystem via systemd-system.conf:
https://www.freedesktop.org/software/systemd/man/systemd-system.conf.html
---
[Manager]
DefaultEnvironment="LIBMOUNT_FORCE_MOUNT2=yes"
---
also resolves current issue, thank you.
3) perhaps this approach (LIBMOUNT_FORCE_MOUNT2) is more flexible
than the previous one (--disable-libmount-mountfd-support)
1) there were further changes in util-linux:
libmount: introduce LIBMOUNT_FORCE_MOUNT2={always,never,auto}
https://github.com/util-linux/util-linux/pull/2297
2) there is an upcoming change in overlayfs kernel module:
ovl: port to new mount api
https://github.com/brauner/linux/commit/213f92f573accb6638bf108cad3e620073fdcb99
https://github.com/util-linux/util-linux/pull/2297
libmount: introduce LIBMOUNT_FORCE_MOUNT2={always,never,auto}
Edit:
Updated for 2.39-6
https://drive.google.com/file/d/1r21BsHFWagbl6I06Lc3jRlcH-8eSJ5aO/view?usp=sharing util-linux-2.39-6.1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/11ZfGPTTJcsnd99uRXw9qdESas-FPuTVs/view?usp=sharing util-linux-libs-2.39-6.1-x86_64.pkg.tar.zst
I confirm that this also resolves the issue:
a) system roll to:
Server=https://archive.archlinux.org/repos/2023/06/01/$repo/os/$arch
b) update to the above 2.39-6:
util-linux-2.39-6.1 and util-linux-libs-2.39-6.1
c) inject environment variable
LIBMOUNT_FORCE_MOUNT2=always
a) system roll to:
Server=https://archive.archlinux.org/repos/2023/06/01/$repo/os/$arch
b) update to the current v 2.39-8:
https://archlinux.org/packages/core-testing/x86_64/util-linux/download/
https://archlinux.org/packages/core-testing/x86_64/util-linux-libs/download/
c) inject environment variable
LIBMOUNT_FORCE_MOUNT2=always