FS#79654 - [btrfs-progs] 6.5-1 mkfs.btrfs doesn't seem to be compatible with PR_SET_MDWE

Attached to Project: Arch Linux
Opened by Frantisek Sumsal (mrc0mmand) - Monday, 11 September 2023, 15:06 GMT
Last edited by Toolybird (Toolybird) - Thursday, 14 September 2023, 22:20 GMT
Task Type Bug Report
Category Packages: Core
Status Closed
Assigned To Tobias Powalowski (tpowa)
Sébastien Luttringer (seblu)
Architecture All
Severity Medium
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

Description:
One of our upstream systemd jobs started reporting an error in systemd-homed, more specifically when running mkfs.btrfs from homed's context. The systemd-homed service uses MemoryDenyWriteExecute=yes, which with recent kernels started using PR_SET_MDWE prctl() instead of seccomp[0], which mkfs.btrfs doesn't seem to be compatible with:

# pacman -Q linux btrfs-progs systemd
linux 6.5.2.arch1-1
btrfs-progs 6.5-1
systemd 254.3-1
# strace -p1 -s500 -y -f -e prctl -o strace.log &
[4] 173831
# systemd-run -t -p MemoryDenyWriteExecute=yes mkfs.btrfs
Running as unit: run-u49.service
Press ^] three times within 1s to disconnect TTY.
strace: Process 173835 attached
/usr/bin/mkfs.btrfs: error while loading shared libraries: cannot make segment writable for relocation: Permission denied
# grep MDWE strace.log
173835 prctl(PR_SET_MDWE, PR_MDWE_REFUSE_EXEC_GAIN, 0, 0, 0) = 0

Working example with an older kernel:
# pacman -Q linux btrfs-progs systemd
linux 6.4.3.arch1-1
btrfs-progs 6.3.2-1
systemd 253.6-2
# strace -p1 -s500 -y -f -e prctl -o strace.log &
[1] 1139
# systemd-run -t -p MemoryDenyWriteExecute=yes /bin/mkfs.btrfs
Running as unit: run-u24.service
Press ^] three times within 1s to disconnect TTY.
strace: Process 1143 attached
btrfs-progs v6.3.2
See https://btrfs.readthedocs.io for more information.

usage: mkfs.btrfs [options] <dev> [<dev...>]
...
# grep MDWE strace.log
<no output>

[0] https://github.com/systemd/systemd/commit/7a114ed4b39e9670f6a511f3eecb6fd58274d27b
This task depends upon

Closed by  Toolybird (Toolybird)
Thursday, 14 September 2023, 22:20 GMT
Reason for closing:  Fixed
Additional comments about closing:  btrfs-progs 6.5.1-1
Comment by Frantisek Sumsal (mrc0mmand) - Monday, 11 September 2023, 17:50 GMT
Similar stuff happens with other binaries from btrfs-progs:

# systemd-run -t -p MemoryDenyWriteExecute=yes btrfs --help
Running as unit: run-u21.service
Press ^] three times within 1s to disconnect TTY.
/usr/bin/btrfs: error while loading shared libraries: cannot make segment writable for relocation: Permission denied

# systemd-run -t -p MemoryDenyWriteExecute=yes btrfsck
Running as unit: run-u22.service
Press ^] three times within 1s to disconnect TTY.
/usr/bin/btrfsck: error while loading shared libraries: cannot make segment writable for relocation: Permission denied

And on a somewhat unrelated note - while testing this I noticed that after installing btrfs-progs, btrfs-convert doesn't seem to work out-of-the box:
# pacman -Syu
:: Synchronizing package databases...
core is up to date
extra is up to date
:: Starting full system upgrade...
there is nothing to do
# btrfs-convert --help
btrfs-convert: error while loading shared libraries: libreiserfscore.so.0: cannot open shared object file: No such file or directory
Comment by Tobias Powalowski (tpowa) - Monday, 11 September 2023, 17:52 GMT
reiserfsprogs is optdepend
Comment by Tobias Powalowski (tpowa) - Monday, 11 September 2023, 17:53 GMT
I read the bug on systemd and I don't understand how to fix it. btrfs-progs has no build flag that takes care of this.
https://github.com/systemd/systemd/issues/29160
Comment by loqs (loqs) - Monday, 11 September 2023, 18:40 GMT
I believe pottering meant the issue is Arch's default LDFLAGS or toolchain built in default linker options.
I do not believe that diagnosis is correct:
Cause linker text relocations to be an error with:
LDFLAGS+=' -Wl,-z,text'
This gives:
/usr/bin/ld: crypto/crc32c-pcl-intel-asm_64.o: warning: relocation in read-only section `.rodata'
/usr/bin/ld: crypto/crc32c-pcl-intel-asm_64.o: warning: relocation in read-only section `.rodata'
Which is caused by a commit in btrfs-progs 6.5 [1].

[1] https://github.com/kdave/btrfs-progs/commit/992be8b50a65974e52a484911e72bc5a5e1b412c
Comment by Frantisek Sumsal (mrc0mmand) - Monday, 11 September 2023, 19:09 GMT
I'm a bit puzzled here. I was curious why this happens only on Arch and not Fedora (Rawhide), and even after building the latest btrfs-progs from [0] the results are the same - i.e. on Fedora no warnings during compilation are emitted and the binary works with MDWE, on Arch the aforementioned error occurs.

To further confirm this, running readelf -d on the btrfs binaries shows that on Arch the TEXTREL flag is present, whereas on Fedora Rawhide it's not:

# readelf -d btrfs | grep TEXTREL
0x0000000000000016 (TEXTREL) 0x0
0x000000000000001e (FLAGS) TEXTREL

[0] https://github.com/kdave/btrfs-progs
Comment by Juan Jose (jjcasmar) - Monday, 11 September 2023, 19:14 GMT
I am the author of the original bug report in systemd github. Can this bug report also explain why I am unable to log into an existing luks2 btrfs home?
Comment by loqs (loqs) - Monday, 11 September 2023, 20:44 GMT
I believe reverting [1][2] stops the issue. @mrc0mmand can you confirm this? Can you share what flags Fedora is using? Unsetting all Arch's build flags has no effect. Could be a built in default or something is missing.

https://github.com/kdave/btrfs-progs/commit/83ac6e0a72c4ba6ce44e6b86f0cc32050af6924d.patch
https://github.com/kdave/btrfs-progs/commit/992be8b50a65974e52a484911e72bc5a5e1b412c.patch
Comment by Frantisek Sumsal (mrc0mmand) - Monday, 11 September 2023, 21:13 GMT
@jjcasmar please paste the logs & other necessary information in the original bug report [0], so we can debug it there further without polluting this issue

@loqs yup, reverting the two commits seems to fix the issue:

# git revert --no-commit 83ac6e0a72c4ba6ce44e6b86f0cc32050af6924d 992be8b50a65974e52a484911e72bc5a5e1b412c
Auto-merging crypto/hash-vectest.c
Auto-merging Makefile
Auto-merging crypto/hash-vectest.c
# make -j
...
# systemd-run -t -p MemoryDenyWriteExecute=yes $PWD/btrfs
Running as unit: run-u50.service
Press ^] three times within 1s to disconnect TTY.
usage: btrfs [global] <group> [<group>...] <command> [<args>]

Global options:
--format <format> if supported, print subcommand output in that format (text, json)
...
# readelf -d btrfs | grep TEXTREL; echo $?
1

As for the flags, I don't think any "explicit" flags are at fault here, since even with

# CFLAGS= LDFLAGS= ./configure
# CFLAGS= LDFLAGS= make -j
# systemd-run -t -p MemoryDenyWriteExecute=yes $PWD/btrfs

everything works as expected on Fedora (and fails on Arch). I also, out of curiosity, checked gcc + ld.gold and clang + lld combinations, both with the same results.

And, also, I noticed one more difference between the Arch and Fedora binaries:

[root@fedora btrfs-progs]# readelf -h btrfs | grep Type
Type: EXEC (Executable file)

[root@archlinux btrfs-progs]# readelf -h btrfs | grep Type
Type: DYN (Position-Independent Executable file)

but that's just me grasping at straws, as this is getting into a mostly uncharted territory for me.

[0] https://github.com/systemd/systemd/issues/29160
Comment by Juan Jose (jjcasmar) - Monday, 11 September 2023, 22:25 GMT
Not really much logs:


```
sudo homectl create prueba
Please enter new password for user prueba: ****
Please enter new password for user prueba (repeat): ****
Operation on home prueba failed: Failed to execute operation: Protocol error
```

Produces the following output

```
Sep 11 16:29:30 archlinux systemd-homed[612]: prueba: changing state absent → creating
Sep 11 16:29:30 archlinux systemd-homework[13236]: Using automatic default storage of 'luks'.
Sep 11 16:29:30 archlinux systemd-homework[13236]: Sizing home to 83% of available disk space, which is 346.0G.
Sep 11 16:29:30 archlinux systemd-homework[13236]: Allocating image file completed.
Sep 11 16:29:30 archlinux systemd-homework[13236]: Writing of partition table completed.
Sep 11 16:29:30 archlinux systemd-homework[13236]: Setting up loopback device /dev/loop1 completed.
Sep 11 16:29:31 archlinux systemd-homework[13236]: LUKS formatting completed.
Sep 11 16:29:33 archlinux systemd-homework[13236]: Writing password to LUKS keyslot 0 completed.
Sep 11 16:29:33 archlinux systemd-homework[13236]: LUKS activation by volume key succeeded.
Sep 11 16:29:33 archlinux systemd-homework[13236]: Writing user record as LUKS token completed.
Sep 11 16:29:33 archlinux systemd-homework[13236]: Setting up LUKS device /dev/mapper/home-prueba completed.
Sep 11 16:29:33 archlinux systemd-homed[13246]: mkfs.btrfs: error while loading shared libraries: cannot make segment writable for relocation: Permission denied
Sep 11 16:29:33 archlinux systemd-homework[13236]: (mkfs) failed with exit status 127.
Sep 11 16:29:33 archlinux systemd-homed[612]: block device /sys/devices/virtual/block/dm-1 has been removed.
Sep 11 16:29:33 archlinux systemd-homed[612]: block device /sys/devices/virtual/block/dm-1 has been removed.
Sep 11 16:29:33 archlinux systemd-homed[612]: block device /sys/devices/virtual/block/loop1 has been removed.
Sep 11 16:29:33 archlinux systemd-homed[612]: Operation on prueba failed: Protocol error
```

I also had some problems trying to mount a systemd-homed user home with luks and btrfs, but I dont have any logs of that. It just said the device was not a valid LUKS device. I was able to recover my data though, the luks header was intact.

Comment by loqs (loqs) - Tuesday, 12 September 2023, 02:25 GMT
Moving the symbols from the rodata section to the data section stops the relocations. Does not explain what is causing them on Arch while not on Fedora.
Could @freswa be added to the issue as one of the maintainers of binutils and gcc. Possibly also @anthraxx as the devtools and pacman maintainer which set the build environment.
Comment by Toolybird (Toolybird) - Tuesday, 12 September 2023, 02:30 GMT
Just to confirm the bleeding obvious. namcap issues these new warnings on 6.5 (not shown on 6.3.3):

$ namcap /var/cache/pacman/pkg/btrfs-progs-6.5-1-x86_64.pkg.tar.zst | grep "text rel"
btrfs-progs W: ELF file ('usr/bin/btrfs') has text relocations.
btrfs-progs W: ELF file ('usr/bin/btrfs-convert') has text relocations.
btrfs-progs W: ELF file ('usr/bin/btrfs-find-root') has text relocations.
btrfs-progs W: ELF file ('usr/bin/btrfs-image') has text relocations.
btrfs-progs W: ELF file ('usr/bin/btrfs-map-logical') has text relocations.
btrfs-progs W: ELF file ('usr/bin/btrfs-select-super') has text relocations.
btrfs-progs W: ELF file ('usr/bin/btrfstune') has text relocations.
btrfs-progs W: ELF file ('usr/bin/mkfs.btrfs') has text relocations.

link time warnings are also evident [1]:

/usr/bin/ld: warning: creating DT_TEXTREL in a PIE

@mrc0mmand, when building on Fedora are you creating PIEs? From memory, Fedora toolchain does it differently to Arch (our GCC is built with --enable-default-pie whereas Fedora fiddle with GCC specs files via redhat-rpm-config)

[1] https://reproducible.archlinux.org/api/v0/builds/502580/log
Comment by Toolybird (Toolybird) - Tuesday, 12 September 2023, 02:47 GMT
> @mrc0mmand, when building on Fedora are you creating PIEs?

Actually, that's already answered above (i.e. no)

> [root@fedora btrfs-progs]# readelf -h btrfs | grep Type
> Type: EXEC (Executable file)
Comment by Toolybird (Toolybird) - Tuesday, 12 September 2023, 03:29 GMT
> Moving the symbols from the rodata section to the data section stops the relocations

And also fixes the bug! @loqs, could you kindly submit your patch upstream for their consideration? I won't involve the others just yet because (as per above) it appears the toolchain thing might be a red herring.

Note that Fedora haven't moved to btrfs-progs 6.5 yet, which might explain why upstream haven't picked up yet on this textrel stuff.
Comment by loqs (loqs) - Tuesday, 12 September 2023, 09:37 GMT
@jcasmar can you please try the attached btrfs-progs to see if it has resolved the issue.

@Toolybird I think Fedora does not build its gcc with --enable-default-pie as shown in mrc0mmand's outputs. crc32c-pcl-intel-asm_64.S was copied to btrfs-progs from the kernel which builds with -no-pie. Adding -no-pie to both CFLAGS and LDFLAGS stops the DT_TEXTREL however namcap then complains:
btrfs-progs W: ELF file ('usr/bin/btrfs') lacks PIE.
btrfs-progs W: ELF file ('usr/bin/btrfs-convert') lacks PIE.
btrfs-progs W: ELF file ('usr/bin/btrfs-find-root') lacks PIE.
btrfs-progs W: ELF file ('usr/bin/btrfs-image') lacks PIE.
btrfs-progs W: ELF file ('usr/bin/btrfs-map-logical') lacks PIE.
btrfs-progs W: ELF file ('usr/bin/btrfs-select-super') lacks PIE.
btrfs-progs W: ELF file ('usr/bin/btrfstune') lacks PIE.
btrfs-progs W: ELF file ('usr/bin/mkfs.btrfs') lacks PIE.
I will raise the issue with upstream btrfs-progs. I would like @jcasmar to test my proposed fix first.
Comment by Juan Jose (jjcasmar) - Tuesday, 12 September 2023, 10:05 GMT
Works fine
Comment by loqs (loqs) - Tuesday, 12 September 2023, 10:34 GMT Comment by loqs (loqs) - Tuesday, 12 September 2023, 19:30 GMT
Was already fixed by [1].
Edit:
@jcasmar can you please try the attached btrfs-progs to verify.

[1] https://github.com/kdave/btrfs-progs/commit/839b2d587da9d691c2f2e7cdb5301eed23781d1d
Comment by Toolybird (Toolybird) - Tuesday, 12 September 2023, 20:16 GMT
> Was already fixed

Doh. Sorry @loqs! I should have checked.
Comment by Juan Jose (jjcasmar) - Wednesday, 13 September 2023, 08:45 GMT
6.5-1.2 works fine
Comment by Tobias Powalowski (tpowa) - Wednesday, 13 September 2023, 14:05 GMT
Thanks guys for investigating. 6.5-2 should work now.
Comment by Juan Jose (jjcasmar) - Wednesday, 13 September 2023, 14:06 GMT
Thanks. Its not yet in core repo, no?
Comment by Tobias Powalowski (tpowa) - Wednesday, 13 September 2023, 14:08 GMT
It has to go through testing repository first.
Comment by Tobias Powalowski (tpowa) - Thursday, 14 September 2023, 07:39 GMT
6.5.1 is in [testing] now.

Loading...