FS#73708 - [glibc] Disable CET support in glibc until kernel interface is finalized

Attached to Project: Arch Linux
Opened by loqs (loqs) - Thursday, 10 February 2022, 04:09 GMT
Last edited by Buggy McBugFace (bugbot) - Saturday, 25 November 2023, 20:26 GMT
Task Type Feature Request
Category Packages: Testing
Status Closed
Assigned To Giancarlo Razzolini (grazzolini)
freswa (frederik)
Architecture All
Severity Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 1
Private No


CET comprises of Shadow Stack and Indirect Branch Tracking. Shadow stack is a secondary stack allocated from memory and cannot be directly modified by applications. When executing a CALL instruction, the processor pushes the return address to both the normal stack and the shadow stack. Upon function return, the processor pops the shadow stack copy and compares it to the normal stack copy. If the two differ, the processor raises a control-protection fault.
Indirect branch tracking verifies indirect CALL/JMP targets are intended as marked by the compiler with 'ENDBR' opcodes. Indirect branch tracking relies on shadow stack.
There have been 30 versions of the patchset to add shadow stack support for userspace to the kernel followed by a reset to version one for a reboot of the series.
The indirect branch tracking support for userspace patchset which was split out from the shadow stack patchset also had 30 versions. It has not yet been rebooted.
Glibc relies on kernel support to implement shadow stack and indirect branch tracking in userspace.

CET as shipped in glibc 2.35 assumes the kernel interface as proposed in versions of 1-11 of the kernel patchset shadow stack for userspace.
CET support is shipped disabled by upstream and was not expected to be enabled by distributions until after support was in the mainline kernel [1].
The proposed interface included ARCH_X86_CET_MMAP_SHSTK to allocate a new shadow stack in userspace [2]. This was rejected as being a security hole.
As glibc implemented support before the kernel it could not check kernel headers for support so absence of support does not stop glibc being built with support for the original interface. If glibc detects a kernel supports shadow stack it will uses ARCH_X86_CET_MMAP_SHSTK as part of its implementation of makecontext the kernel will return ENOSYS which glibc does not check and the result is a segfault. Users of makecontext include openssl 3.0. Proposed fixes for glibc are available [3]. They have not been merged as the current kernel patchset [4] has not been merged.
A separate issue with shadow stack is it requires all dynamic libraries to support it, which is automatic with Arch's current build flags except if the library uses assembler which requires the assembly to be annotated to note its support [5]. Packages lacking such annotation hence disabling shadow stack support for any binary linked to them include pcre, gmp, zstd and openssl. The last two will gain support from upstream updates.

Proposed Solution:
Disable CET in glibc. Prevents the risk of segfaults due to interfaces mismatches. Will not reduce security due to missing kernel support and incomplete userspace support.

Alternative Solutions Considered:
Patch kernels packaged by Arch as well as glib to support the latest version of the proposal. Increased maintenance burden and goes against policy of not accepting none upstreamed patches.

Additional info:
* glibc 2.35-1
[1] https://lore.kernel.org/all/a7e59ae16e0e05579b087caf4045e42b174e2167.camel%40intel.com/
[2] https://lore.kernel.org/lkml/20200825002540.3351-26-yu-cheng.yu%40intel.com/
[3] https://gitlab.com/x86-glibc/glibc/-/commits/users/hjl/cet/2.35
[4] https://lore.kernel.org/all/20220130211838.8382-1-rick.p.edgecombe%40intel.com/
[5] This can be detected at build time by adding -z cet-report=error to LDFLAGS

Steps to reproduce:
Run the openssl 3 or glibc 2.35 testsuite with glibc 2.35 on a CPU that supports shadow stack using a kernel with the current shadow stack for userspace patchset applied.
This task depends upon

Closed by  Buggy McBugFace (bugbot)
Saturday, 25 November 2023, 20:26 GMT
Reason for closing:  Moved
Additional comments about closing:  https://gitlab.archlinux.org/archlinux/p ackaging/packages/glibc/issues/2
Comment by Toolybird (Toolybird) - Thursday, 10 February 2022, 07:39 GMT
Interesting stuff @loqs! Thanks for raising it. I don't claim to understand any of the above but just wanted to make 2 points:

1. Fedora 36 is due to be released in April and they still have `--enable-cet' set in their f36 glibc spec file. We should keep an eye on it..

2. This blurb was in your link [4]:

"Many distros (Fedora and others) have binaries already marked with
shadow stack, waiting for kernel support. Unfortunately their glibc
binaries expect the original arch_prctl() interface for allocating
shadow stacks, as those changes were pushed ahead of kernel support.
The net result of it all is, when updating to a kernel with shadow
stack these binaries would suddenly get shadow stack enabled and expect
the arch_prctl() interface to be there. And so calls to makecontext()
will fail, resulting in visible breakages. This series deals with this
problem as described below in "Updates"."

I think it confirms what you are saying, but it also suggests the problem will be fixed by the kernel devs before it becomes a problem for mainstream kernels. Am I on the right track?

You also mention "using a kernel with the current shadow stack for userspace patchset applied". Do any Arch kernels have this?
Comment by loqs (loqs) - Thursday, 10 February 2022, 08:12 GMT
None of Arch's kernels applies any version of the shadow stack for userspace patchset. Fedora's kernels do not either [1].

Will apply the patchset from [2] see if I can still trigger a segfault.
No segfaults running the openssl 3.0.1 test suite, I used that as it is quicker to build and run than glibc.
That just leaves it will not work with whatever the kernel developers decide upon but it will not cause unexpected behavior.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1802686
[2] https://github.com/hjl-tools/linux/commits/hjl/cet/linux-5.16.8
Comment by freswa (frederik) - Thursday, 10 February 2022, 17:17 GMT
Have you checked if 32 bit binaries are still an issue as described in  FS#63015 ?
Comment by loqs (loqs) - Thursday, 10 February 2022, 19:31 GMT
@freswa good thought. No output is produce for a hello world `gcc -v -m32 test.c` with CET disabled for both 32bit and 64bit.

As the current kernel patchset combined with glibc 2.35 with CET enabled results in CET being none functional rather than producing segfaults leaving CET enabled in glibc is harmless.
I think [1] is the patch for glibc that the new kernel interface requires to enable CET for a binary. The current glibc never calls arch_prctl(ARCH_X86_FEATURE_1_ENABLE).
Not suggesting adding the patchset, just documenting what I think would need to happen.

There seems to be more potential risk than benefit in changing the option then changing it back later.

[1] https://gitlab.com/x86-glibc/glibc/-/commit/ffcf6732559eb6fa29e53f14d320079269f8afbc

Comment by freswa (frederik) - Friday, 11 February 2022, 16:13 GMT
I've thought about this a lot. I'd like to keep our glibc in this regard rather near to the other distros.
After all this makes sure that any future solution for Debian and Fedora will also work for us. Going a different way is more risky imo.

I'll keep this open though, to keep the issue in mind.

Thank you guys helping us :)
Comment by loqs (loqs) - Thursday, 12 October 2023, 17:43 GMT
The kernel support for shadow stack in userspace is queued for 6.6 [1]. I expect glibc support will then be updated to be compatible. The current update series can be found at [2].
Please consider changing --enable-cet to --enable-cet=permissive once that support lands until after all libraries Arch builds have shadow stack support or disable shadow stack support in the binaries that use those libraries to avoid the binary refusing to run due to missing library support. Examples of packages missing library support include ffmpeg gmp gtk4 libjpeg-turbo libunwind nspr nvidia-utils. Example of the issue this causes the udisks service fell into a reboot loop due to missing support in nspr required by the crypto module and gmp required by the mdraid module.

[1] https://github.com/torvalds/linux/commit/df57721f9a63e8a1fb9b9b2e70de4aa4c7e0cd2e
[2] https://gitlab.com/x86-glibc/glibc/-/commits/users/hjl/cet/v8/master/
I used the following to find libraries missing shadow stack support on my system
$ find /usr/lib -name '*.so.*' -exec bash -c 'readelf -n {} | grep -q SHSTK || echo {}' \;