Arch Linux

Reopening.  FS#45295  was closed with "Not a glibc issue - file bugs against the segfaulting packages."

Right, which is not what this is. This is a complete duplicate unless I'm reading it completely wrong?

 FS#45295  is against glibc. This is against nvidia-libgl.

Try updating the microcode if you haven't already:

https://wiki.archlinux.org/index.php/Microcode

Since there are no microcode updates available for this processor, I assume it has a non-buggy implementation.

I don't think there is any way I can do anything here?

I fear nVidia will just point back to glibc, and Allan is adamant about keeping lock elision enabled.

I'm pulling in Allan and see.

This is a nvidia issue. They are unlocking something that does not exist.

Is this still an issue in the current version?

you mean the current glibc version? (I think there was no nvidia update)
yes it is, I had to recompile it again

Hi, I can confirm the bug. Same behaviour on my system (just other programms crashing: vlc, sddm, kde-lockscreen). As soon as I recompile glibc with lock elison disabled or link libEGL.so.1 to the mesa lib instead of the nvidia lib manually (ln -s /usr/lib/mesa/libEGL.so.1 /usr/lib) everything appears to work correctly.
Also in my case there are no microcode updates as well (skylake cpu).

Can confirm. I'm also on skylake and running into the same problems, as soon as I install nvidia-libgl gnome-shell doesn't launch and vim (from the gvim package) segfaults on quit.
Installing a custom version of glibc with lock elision disabled makes the problems go away.

Even with lock elision disabled, running helgrind with say vim (from gvim) shows that libEGL is unlocking a lock that wasn't locked: http://pastie.org/pastes/10443914/text

I'm sure helgrind will show the same issue on systems that don't have TSX enabled.

Minimal test case.

test.c:
int main() { return 0; }

compile with: gcc test.c -o test -lEGL

test with helgrind: valgrind --tool=helgrind ./test

I have a glibc built without lock elision (literally everything was crashing on my Skylake box, so I killed the lock elision feature). I see a different crash-on-exit with anything using libEGL.so:

==3673== Invalid read of size 8
==3673== at 0xAFA9BE1: __eglTeardownVendor (in /usr/lib/nvidia/libEGL.so.355.11)
==3673== by 0x400F884: _dl_fini (in /usr/lib/ld-2.22.so)
==3673== by 0x63F8F87: __run_exit_handlers (in /usr/lib/libc-2.22.so)
==3673== by 0x63F8FD4: exit (in /usr/lib/libc-2.22.so)
==3673== by 0x63E3616: (below main) (in /usr/lib/libc-2.22.so)
==3673== Address 0x8 is not stack'd, malloc'd or (recently) free'd
==3673==
==3673==
==3673== Process terminating with default action of signal 11 (SIGSEGV)
==3673== Access not within mapped region at address 0x8
==3673== at 0xAFA9BE1: __eglTeardownVendor (in /usr/lib/nvidia/libEGL.so.355.11)
==3673== by 0x400F884: _dl_fini (in /usr/lib/ld-2.22.so)
==3673== by 0x63F8F87: __run_exit_handlers (in /usr/lib/libc-2.22.so)
==3673== by 0x63F8FD4: exit (in /usr/lib/libc-2.22.so)
==3673== by 0x63E3616: (below main) (in /usr/lib/libc-2.22.so)
==3673== If you believe this happened as a result of a stack
==3673== overflow in your program's main thread (unlikely but
==3673== possible), you can try to increase the size of the
==3673== main thread stack using the --main-stacksize= flag.
==3673== The main thread stack size used in this run was 8388608.

"exo-open" with no arguments is a nice example of this behavior.

The __eglTeardownVendor crash doesn't happen if I downgrade the NVIDIA packages from 355.11 to 352.41.

Still get lock elision crashes when that's built in.

Attaching phire's helgrind output in case it vanishes from pastie.

Maybe it will help?
driver 358.09 (beta) https://devtalk.nvidia.com/default/topic/884727

With a Skylake CPU (sig=0x506e3, pf=0x2, revision=0x39) this still happens using the 358.09 beta driver if lock elision in glibc is enabled. (It doesn't with nvidia < 352 regardless of lock elision.)

Does NVIDIA already know about this? They should really fix it, having to compile glibc every new release could get pretty annoying...

I've reported the bug using the NVIDIA partners site. That should get it the appropriate engineering attention.

Confirmed this issue still exists with the same skylake as Gerhard (sig=0x506e3, pf=0x2, revision=0x39) and the latest NVIDIA driver (358.09). Also confirmed that disabling lock elision fixes this issue.

I can confirm this issue still exists on an Intel Core i5-6500 Skylake system (sig=0x506e3, pf=0x2, revision=0x49) and the NVIDIA driver 352.55. Again, using glibc without --enable-lock-elision fixes it.

Still exists with nvidia driver 358.16-1.

You guys with reproducible problems, please post a bug report on the nvidia dev forums and link it here. Nvidia appears to be unaware of it.

As I said before, I reported it through the NVIDIA partners site (bug #1701106 if you're able to request access somehow). They're aware of it.

Alright, so basically there's nothing to do here for us since glibc is going to be kept the way it is in Arch? Is Aaron Plattner aware of this? He's an Arch user himself IIRC and works at nvidia.

Until it's fixed by NVIDIA, there is an awful patch that could be applied to glibc (I don't like the patch *at all*, but it has made my systems usable in the meantime):

http://git.uplinklabs.net/snoonan/projects/archlinux/ec2/ec2-packages.git/tree/glibc/glibc-2.22-lock-elision-crash-nvidia.patch

I hadn't heard Aaron Plattner's name until now. I haven't received many updates on the NVIDIA bug report other than simple status changes (no comments yet). I believe it's currently marked "in progress" (was "pending review" a couple weeks ago).

@Svenstaro
Aaron Plattner said: NOTE: I'm on paternity leave until early 2016.

That glibc patch is awful and will not be applied. Someone could just create a "glibc-for-stupid-nvidia" package that disables lock elision and provides/replaces glibc.

Good news. NVIDIA closed the bug today and has marked it as "fixed". An upcoming driver release should have the fix (352.67 or later, 361.10 or later).

Any idea on the release of 352.67 or later?

I do have a similar issue with Intel(R) Xeon(R) CPU E3-1535M v5 (sig=0x506e3, pf=0x20, revision=0x49), nvidia driver 358.16, lots of stuff segfaulting all over the place.

I've solved it by patching the glibc with the tsx blacklist code from debian (http://sources.debian.net/data/main/g/glibc/2.22-0experimental1/debian/patches/amd64/local-blacklist-for-Intel-TSX.diff), just added model 94 with stepping <= 3 aswell.

Maybe this would be an approach to solve this issue? I've tried to use the latest microcode from intel, as intel-ucode is outdated, but even with that it doesn't work properly.

Of course this could also be a flawed CPU, but I don't really have a way to figure that one out.

Driver 361.16 (beta) https://devtalk.nvidia.com/default/topic/908423

Great news! I'll wait for the corresponding Arch Package, and test it with the original glibc libraries - should be working without core dumps, according to the last bullet point mentioned in Aaron Plattner's dev note.

Will the corresponding arch package wait until the BETA is complete, or will there be a BETA package made available?

@Tarr3128

That's not a viable approach, as it would require backlisting every single CPU with TSX support, including future CPUs.
It's exactly the same as compiling glibc with lock elison disabled.

The problem is not a CPU issue, the TSX feature is finally working correctly, that's why there is no microcode
update for the newer CPUs. It's an issue with the Nvidia package, which is acting outside the spec. Only an update
to the Nvidia package will fix it. Everything else (including microcode updates which disable TSX) is a workaround.

I see, thank you @phire. The reason I posted that solution is that there is a difference in between disabling lock elision and the blacklisting patch, I still had segfaults, even though not as many, might be some code issue with glibc. Once the driver is available I'll do some testing.

I have a workaround for this issue with a 5 byte binary patch:

In libEGL_nvidia.so.0 (md5sum 36a6edacefcd2893b3bc5c6c282943b6), replace bytes "e8 64 d3 f8 ff" @ offset 0x95987 with "90 90 90 90 90".

It replaces an unnecessary call to pthread_mutex_unlock just before pthread_mutex_destroy with a sequence of NOPs. This problem is silently ignored in glibc built without lock elision or on CPUs with no TSX.

Please test the new nvidia beta driver. You can use the AUR package.

I have not tested the beta driver but I did extract the package and analyze libEGL_nvidia.so disassembly with objdump and the offending call to pthread_mutex_unlock is gone.

In other words: this issue SHOULD be fixed in the beta.

I stuck the new driver into [testing]. Please test.

There is a thread over at nvidia (https://devtalk.nvidia.com/default/topic/908506/several-essential-kde-applications-sddm-krunner-plasmashell-segfault-on-startup-with-361-16/) that suggests that the new 361.16 driver has a rather serious bug.

The original crash (in __lll_unlock_elision) seemts to be fixed in 361-16 from testing. (Gnome works, I didn't try KDE.)

Today Nvidia released another beta driver 361.18
From the description I see that the "phtread_mutex_unlock" issue is mentioned again, same as for 361.16:
"Fixed a bug in the EGL driver where a mutex was unlocked more than once. This triggers undefined behavior, and in particular, if lock elision is enabled in glibc, may result in a segmentation fault."
Apart from that, I don't find any new fixes, esp. nothing what might be different as compared to 361.16
Does it make sense to test 361.18, too, with respect to the "plasmashell segfault" issues?

According to this: https://devtalk.nvidia.com/default/topic/908506/several-essential-kde-applications-sddm-krunner-plasmashell-segfault-on-startup-with-361-16/#4776281
the "plasmashell segfault issues" should be fixed in 361.18.
I'll give it a try now...

Edit: Can confirm both the sddm/plasmashell segfault and the original "glibc lock elision segfaults" are gone with 361.18.