FS#79654 - [btrfs-progs] 6.5-1 mkfs.btrfs doesn't seem to be compatible with PR_SET_MDWE
Attached to Project:
Arch Linux
Opened by Frantisek Sumsal (mrc0mmand) - Monday, 11 September 2023, 15:06 GMT
Last edited by Toolybird (Toolybird) - Thursday, 14 September 2023, 22:20 GMT
Opened by Frantisek Sumsal (mrc0mmand) - Monday, 11 September 2023, 15:06 GMT
Last edited by Toolybird (Toolybird) - Thursday, 14 September 2023, 22:20 GMT
|
Details
Description:
One of our upstream systemd jobs started reporting an error in systemd-homed, more specifically when running mkfs.btrfs from homed's context. The systemd-homed service uses MemoryDenyWriteExecute=yes, which with recent kernels started using PR_SET_MDWE prctl() instead of seccomp[0], which mkfs.btrfs doesn't seem to be compatible with: # pacman -Q linux btrfs-progs systemd linux 6.5.2.arch1-1 btrfs-progs 6.5-1 systemd 254.3-1 # strace -p1 -s500 -y -f -e prctl -o strace.log & [4] 173831 # systemd-run -t -p MemoryDenyWriteExecute=yes mkfs.btrfs Running as unit: run-u49.service Press ^] three times within 1s to disconnect TTY. strace: Process 173835 attached /usr/bin/mkfs.btrfs: error while loading shared libraries: cannot make segment writable for relocation: Permission denied # grep MDWE strace.log 173835 prctl(PR_SET_MDWE, PR_MDWE_REFUSE_EXEC_GAIN, 0, 0, 0) = 0 Working example with an older kernel: # pacman -Q linux btrfs-progs systemd linux 6.4.3.arch1-1 btrfs-progs 6.3.2-1 systemd 253.6-2 # strace -p1 -s500 -y -f -e prctl -o strace.log & [1] 1139 # systemd-run -t -p MemoryDenyWriteExecute=yes /bin/mkfs.btrfs Running as unit: run-u24.service Press ^] three times within 1s to disconnect TTY. strace: Process 1143 attached btrfs-progs v6.3.2 See https://btrfs.readthedocs.io for more information. usage: mkfs.btrfs [options] <dev> [<dev...>] ... # grep MDWE strace.log <no output> [0] https://github.com/systemd/systemd/commit/7a114ed4b39e9670f6a511f3eecb6fd58274d27b |
This task depends upon
Closed by Toolybird (Toolybird)
Thursday, 14 September 2023, 22:20 GMT
Reason for closing: Fixed
Additional comments about closing: btrfs-progs 6.5.1-1
Thursday, 14 September 2023, 22:20 GMT
Reason for closing: Fixed
Additional comments about closing: btrfs-progs 6.5.1-1
# systemd-run -t -p MemoryDenyWriteExecute=yes btrfs --help
Running as unit: run-u21.service
Press ^] three times within 1s to disconnect TTY.
/usr/bin/btrfs: error while loading shared libraries: cannot make segment writable for relocation: Permission denied
# systemd-run -t -p MemoryDenyWriteExecute=yes btrfsck
Running as unit: run-u22.service
Press ^] three times within 1s to disconnect TTY.
/usr/bin/btrfsck: error while loading shared libraries: cannot make segment writable for relocation: Permission denied
And on a somewhat unrelated note - while testing this I noticed that after installing btrfs-progs, btrfs-convert doesn't seem to work out-of-the box:
# pacman -Syu
:: Synchronizing package databases...
core is up to date
extra is up to date
:: Starting full system upgrade...
there is nothing to do
# btrfs-convert --help
btrfs-convert: error while loading shared libraries: libreiserfscore.so.0: cannot open shared object file: No such file or directory
https://github.com/systemd/systemd/issues/29160
I do not believe that diagnosis is correct:
Cause linker text relocations to be an error with:
LDFLAGS+=' -Wl,-z,text'
This gives:
/usr/bin/ld: crypto/crc32c-pcl-intel-asm_64.o: warning: relocation in read-only section `.rodata'
/usr/bin/ld: crypto/crc32c-pcl-intel-asm_64.o: warning: relocation in read-only section `.rodata'
Which is caused by a commit in btrfs-progs 6.5 [1].
[1] https://github.com/kdave/btrfs-progs/commit/992be8b50a65974e52a484911e72bc5a5e1b412c
To further confirm this, running readelf -d on the btrfs binaries shows that on Arch the TEXTREL flag is present, whereas on Fedora Rawhide it's not:
# readelf -d btrfs | grep TEXTREL
0x0000000000000016 (TEXTREL) 0x0
0x000000000000001e (FLAGS) TEXTREL
[0] https://github.com/kdave/btrfs-progs
https://github.com/kdave/btrfs-progs/commit/83ac6e0a72c4ba6ce44e6b86f0cc32050af6924d.patch
https://github.com/kdave/btrfs-progs/commit/992be8b50a65974e52a484911e72bc5a5e1b412c.patch
@loqs yup, reverting the two commits seems to fix the issue:
# git revert --no-commit 83ac6e0a72c4ba6ce44e6b86f0cc32050af6924d 992be8b50a65974e52a484911e72bc5a5e1b412c
Auto-merging crypto/hash-vectest.c
Auto-merging Makefile
Auto-merging crypto/hash-vectest.c
# make -j
...
# systemd-run -t -p MemoryDenyWriteExecute=yes $PWD/btrfs
Running as unit: run-u50.service
Press ^] three times within 1s to disconnect TTY.
usage: btrfs [global] <group> [<group>...] <command> [<args>]
Global options:
--format <format> if supported, print subcommand output in that format (text, json)
...
# readelf -d btrfs | grep TEXTREL; echo $?
1
As for the flags, I don't think any "explicit" flags are at fault here, since even with
# CFLAGS= LDFLAGS= ./configure
# CFLAGS= LDFLAGS= make -j
# systemd-run -t -p MemoryDenyWriteExecute=yes $PWD/btrfs
everything works as expected on Fedora (and fails on Arch). I also, out of curiosity, checked gcc + ld.gold and clang + lld combinations, both with the same results.
And, also, I noticed one more difference between the Arch and Fedora binaries:
[root@fedora btrfs-progs]# readelf -h btrfs | grep Type
Type: EXEC (Executable file)
[root@archlinux btrfs-progs]# readelf -h btrfs | grep Type
Type: DYN (Position-Independent Executable file)
but that's just me grasping at straws, as this is getting into a mostly uncharted territory for me.
[0] https://github.com/systemd/systemd/issues/29160
```
sudo homectl create prueba
Please enter new password for user prueba: ****
Please enter new password for user prueba (repeat): ****
Operation on home prueba failed: Failed to execute operation: Protocol error
```
Produces the following output
```
Sep 11 16:29:30 archlinux systemd-homed[612]: prueba: changing state absent → creating
Sep 11 16:29:30 archlinux systemd-homework[13236]: Using automatic default storage of 'luks'.
Sep 11 16:29:30 archlinux systemd-homework[13236]: Sizing home to 83% of available disk space, which is 346.0G.
Sep 11 16:29:30 archlinux systemd-homework[13236]: Allocating image file completed.
Sep 11 16:29:30 archlinux systemd-homework[13236]: Writing of partition table completed.
Sep 11 16:29:30 archlinux systemd-homework[13236]: Setting up loopback device /dev/loop1 completed.
Sep 11 16:29:31 archlinux systemd-homework[13236]: LUKS formatting completed.
Sep 11 16:29:33 archlinux systemd-homework[13236]: Writing password to LUKS keyslot 0 completed.
Sep 11 16:29:33 archlinux systemd-homework[13236]: LUKS activation by volume key succeeded.
Sep 11 16:29:33 archlinux systemd-homework[13236]: Writing user record as LUKS token completed.
Sep 11 16:29:33 archlinux systemd-homework[13236]: Setting up LUKS device /dev/mapper/home-prueba completed.
Sep 11 16:29:33 archlinux systemd-homed[13246]: mkfs.btrfs: error while loading shared libraries: cannot make segment writable for relocation: Permission denied
Sep 11 16:29:33 archlinux systemd-homework[13236]: (mkfs) failed with exit status 127.
Sep 11 16:29:33 archlinux systemd-homed[612]: block device /sys/devices/virtual/block/dm-1 has been removed.
Sep 11 16:29:33 archlinux systemd-homed[612]: block device /sys/devices/virtual/block/dm-1 has been removed.
Sep 11 16:29:33 archlinux systemd-homed[612]: block device /sys/devices/virtual/block/loop1 has been removed.
Sep 11 16:29:33 archlinux systemd-homed[612]: Operation on prueba failed: Protocol error
```
I also had some problems trying to mount a systemd-homed user home with luks and btrfs, but I dont have any logs of that. It just said the device was not a valid LUKS device. I was able to recover my data though, the luks header was intact.
Could @freswa be added to the issue as one of the maintainers of binutils and gcc. Possibly also @anthraxx as the devtools and pacman maintainer which set the build environment.
$ namcap /var/cache/pacman/pkg/btrfs-progs-6.5-1-x86_64.pkg.tar.zst | grep "text rel"
btrfs-progs W: ELF file ('usr/bin/btrfs') has text relocations.
btrfs-progs W: ELF file ('usr/bin/btrfs-convert') has text relocations.
btrfs-progs W: ELF file ('usr/bin/btrfs-find-root') has text relocations.
btrfs-progs W: ELF file ('usr/bin/btrfs-image') has text relocations.
btrfs-progs W: ELF file ('usr/bin/btrfs-map-logical') has text relocations.
btrfs-progs W: ELF file ('usr/bin/btrfs-select-super') has text relocations.
btrfs-progs W: ELF file ('usr/bin/btrfstune') has text relocations.
btrfs-progs W: ELF file ('usr/bin/mkfs.btrfs') has text relocations.
link time warnings are also evident [1]:
/usr/bin/ld: warning: creating DT_TEXTREL in a PIE
@mrc0mmand, when building on Fedora are you creating PIEs? From memory, Fedora toolchain does it differently to Arch (our GCC is built with --enable-default-pie whereas Fedora fiddle with GCC specs files via redhat-rpm-config)
[1] https://reproducible.archlinux.org/api/v0/builds/502580/log
Actually, that's already answered above (i.e. no)
> [root@fedora btrfs-progs]# readelf -h btrfs | grep Type
> Type: EXEC (Executable file)
And also fixes the bug! @loqs, could you kindly submit your patch upstream for their consideration? I won't involve the others just yet because (as per above) it appears the toolchain thing might be a red herring.
Note that Fedora haven't moved to btrfs-progs 6.5 yet, which might explain why upstream haven't picked up yet on this textrel stuff.
@Toolybird I think Fedora does not build its gcc with --enable-default-pie as shown in mrc0mmand's outputs. crc32c-pcl-intel-asm_64.S was copied to btrfs-progs from the kernel which builds with -no-pie. Adding -no-pie to both CFLAGS and LDFLAGS stops the DT_TEXTREL however namcap then complains:
btrfs-progs W: ELF file ('usr/bin/btrfs') lacks PIE.
btrfs-progs W: ELF file ('usr/bin/btrfs-convert') lacks PIE.
btrfs-progs W: ELF file ('usr/bin/btrfs-find-root') lacks PIE.
btrfs-progs W: ELF file ('usr/bin/btrfs-image') lacks PIE.
btrfs-progs W: ELF file ('usr/bin/btrfs-map-logical') lacks PIE.
btrfs-progs W: ELF file ('usr/bin/btrfs-select-super') lacks PIE.
btrfs-progs W: ELF file ('usr/bin/btrfstune') lacks PIE.
btrfs-progs W: ELF file ('usr/bin/mkfs.btrfs') lacks PIE.
I will raise the issue with upstream btrfs-progs. I would like @jcasmar to test my proposed fix first.
Edit:
@jcasmar can you please try the attached btrfs-progs to verify.
[1] https://github.com/kdave/btrfs-progs/commit/839b2d587da9d691c2f2e7cdb5301eed23781d1d
Doh. Sorry @loqs! I should have checked.