FS#59550 - [GLIBC]glibc 2.28 cause core dump on electron based apps

Attached to Project: Arch Linux
Opened by Pryka (Pryka) - Tuesday, 07 August 2018, 08:47 GMT
Last edited by Bartłomiej Piotrowski (Barthalion) - Friday, 10 August 2018, 16:54 GMT
Task Type Bug Report
Category Packages: Core
Status Closed
Assigned To Bartłomiej Piotrowski (Barthalion)
Architecture x86_64
Severity Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 22
Private No

Details

Description:

It seams that update to glibc 2.28 cause some apps to core dump.

So far I have found that Singal(AUR - from source), Mailspring(AUR-bin), Skype(AUR-bin), Won't start with similar error:

Mailspring GDB:
Program received signal SIGSEGV, Segmentation fault.
0x0000000000a4fa30 in ?? ()
(gdb) bt full
#0 0x0000000000a4fa30 in ()
#1 0x00007ffff7d832b2 in node::http2::Http2Session::Callbacks::Callbacks(bool) () at /usr/share/mailspring/libnode.so
#2 0x00007ffff7d83375 in () at /usr/share/mailspring/libnode.so
#3 0x00007ffff7fe36da in call_init.part () at /lib64/ld-linux-x86-64.so.2
#4 0x00007ffff7fe37da in _dl_init () at /lib64/ld-linux-x86-64.so.2
#5 0x00007ffff7fd503a in _dl_start_user () at /lib64/ld-linux-x86-64.so.2
#6 0x0000000000000001 in ()
#7 0x00007fffffffea7d in ()
#8 0x0000000000000000 in ()


Signal:
[pryka@Iluvatar ~]$ LC_ALL=C signal-desktop
NODE_ENV production
NODE_CONFIG_DIR /usr/lib/signal/resources/app.asar/config
NODE_CONFIG {}
ALLOW_CONFIG_MUTATIONS undefined
HOSTNAME undefined
NODE_APP_INSTANCE undefined
SUPPRESS_NO_CONFIG_WARNING undefined
Set Windows Application User Model ID (AUMID) { appUserModelId: 'org.whispersystems.signal-desktop' }
userData: /home/pryka/.config/Signal
making app single instance
{"name":"log","hostname":"Iluvatar","pid":2868,"level":30,"msg":"app ready","time":"2018-08-07T08:26:51.377Z","v":0}
{"name":"log","hostname":"Iluvatar","pid":2868,"level":30,"msg":"Ensure attachments directory exists","time":"2018-08-07T08:26:51.396Z","v":0}
{"name":"log","hostname":"Iluvatar","pid":2868,"level":30,"msg":"updateSchema: Current schema version: 0; Most recent schema version: 1; SQLite version: 3.20.1; SQLCipher version: 3.4.2;","time":"2018-08-07T08:26:51.414Z","v":0}
{"name":"log","hostname":"Iluvatar","pid":2868,"level":30,"msg":"updateToSchemaVersion1: starting...","time":"2018-08-07T08:26:51.417Z","v":0}
/usr/bin/signal-desktop: line 3: 2868 Segmentation fault (core dumped) electron /usr/lib/signal/resources/app.asar $@

From journal:
#2 0x00005583eb8abc1b n/a (electron)
#3 0x00005583eb886307 n/a (electron)
#4 0x00005583eb88695e n/a (electron)
#5 0x00005583eb84cb73 n/a (electron)
#6 0x00007f7861e27a8d start_thread (libpthread.so.0)
#7 0x00007f785818a823 __clone (libc.so.6)

Stack trace of thread 3047:
#0 0x00007f7861e2daec pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
#1 0x00007f786312b607 uv_cond_wait (libnode.so)
#2 0x00007f786311ea62 n/a (libnode.so)
#3 0x00007f7861e27a8d start_thread (libpthread.so.0)
#4 0x00007f785818a823 __clone (libc.so.6)

Stack trace of thread 3042:
#0 0x00007f7861e2daec pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
#1 0x00005583eb9b3688 n/a (electron)
#2 0x00005583eb86820e n/a (electron)
#3 0x00005583eb84cb73 n/a (electron)
#4 0x00007f7861e27a8d start_thread (libpthread.so.0)
#5 0x00007f785818a823 __clone (libc.so.6)

Stack trace of thread 3048:
#0 0x00007f7861e2daec pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
#1 0x00007f786312b607 uv_cond_wait (libnode.so)
#2 0x00007f786311ea62 n/a (libnode.so)
#3 0x00007f7861e27a8d start_thread (libpthread.so.0)
#4 0x00007f785818a823 __clone (libc.so.6)

Stack trace of thread 3043:
#0 0x00007f7861e2daec pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
#1 0x00005583eb8abd5f n/a (electron)
#2 0x00005583eb8abc1b n/a (electron)
#3 0x00005583eb84314e n/a (electron)
#4 0x00005583eb85110c n/a (electron)
#5 0x00005583eb889a02 n/a (electron)
#6 0x00005583eb84cb73 n/a (electron)
#7 0x00007f7861e27a8d start_thread (libpthread.so.0)
#8 0x00007f785818a823 __clone (libc.so.6)

Stack trace of thread 3044:
#0 0x00007f785817f991 __poll (libc.so.6)
#1 0x00007f785a05c673 n/a (libpulse.so.0)
#2 0x00007f785a04d990 pa_mainloop_poll (libpulse.so.0)
#3 0x00007f785a04dfe0 pa_mainloop_iterate (libpulse.so.0)
#4 0x00007f785a04e091 pa_mainloop_run (libpulse.so.0)
#5 0x00007f785a05c5ae n/a (libpulse.so.0)
#6 0x00007f784d1009fc n/a (libpulsecommon-12.2.so)
#7 0x00007f7861e27a8d start_thread (libpthread.so.0)
#8 0x00007f785818a823 __clone (libc.so.6)


Skype(from journal):
sie 07 10:31:08 Iluvatar systemd-coredump[3085]: Process 3082 (skypeforlinux) of user 1000 dumped core.

Stack trace of thread 3082:
#0 0x0000000000d20020 n/a (skypeforlinux)



Additional info:
Related topics:
https://github.com/signalapp/Signal-Desktop/issues/2600
https://github.com/signalapp/Signal-Desktop/issues/2610

Didn't test this on my own but found on the Internet that whatsapp and slack are also affected by this. So it's look like mostly electron based apps have difficult time.

Reverting back to older glibc is fixing all the issues. Not really sure if this bug should be on Arch tracker... feel free to close it.
This task depends upon

Closed by  Bartłomiej Piotrowski (Barthalion)
Friday, 10 August 2018, 16:54 GMT
Reason for closing:  Fixed
Additional comments about closing:  glibc 2.28-2
Comment by Matthias Lisin (matthias.lisin) - Tuesday, 07 August 2018, 11:33 GMT
Looks like it applies for Electon apps with ELF executables.
Atom, for example, is not affected by this.

Also affected, for the sake of completeness:
https://aur.archlinux.org/packages/rocketchat-desktop/
https://aur.archlinux.org/packages/rocketchat-client-bin/
Comment by Peter (protake) - Tuesday, 07 August 2018, 12:43 GMT
Camunda Modeler is also affected.
Comment by Eli Schwartz (eschwartz) - Tuesday, 07 August 2018, 14:41 GMT
I'm... not even sure what to do about this. glibc is supposed to be ABI compatible with stuff built against older electron, so if it isn't, then this is a pretty bad upstream bug I would guess. But the fact that only electron is broken -- and only prebuilt electron, not the system electron which Arch provides [community] packages for -- makes me nervous.

It almost sounds like electron is doing something terribly wrong.

@tensor5, maybe since you deal with electron you might know if this is fundamentally an electron issue?

EDIT: would anyone be willing to try bisecting glibc to figure out at what point electron started breaking?
Comment by Maxime Wack (SataMaxx) - Tuesday, 07 August 2018, 17:01 GMT
It doesn't seem to be restricted to electron apps: some R packages are also affected. (although shiny makes heavy use of node.js, it might be linked)
Simply loading the shiny package causes a core dump with message:

terminate called after throwing an instance of 'std::runtime_error'
what(): Mutex creation failed
[1] 2599 abort (core dumped) R

Reverting glibc fixes the problem.
Comment by John Smith (macrophage) - Tuesday, 07 August 2018, 17:44 GMT
Also having the same issue. How did you revert back to older glibc? I'm still trying to get used to arch/pacman. Thanks!
Comment by Nick Stefanov (mozo) - Tuesday, 07 August 2018, 17:53 GMT
Same here - Skype doesn't launch anymore.
Comment by panman (panman) - Tuesday, 07 August 2018, 20:50 GMT
After the update to glibc 2.28 and gcc 8.2.0 the following R packages do not work/cannot be installed anymore:
later
promises
httpuv
shiny

Probably many other are affected, just in my case these are what I am using at the moment. I see nobody is assigned to this issue.
Comment by loqs (loqs) - Tuesday, 07 August 2018, 22:54 GMT
@SataMaxx and @panman can you provide a backtrace from a coredump from one of the R applications?
Comment by panman (panman) - Tuesday, 07 August 2018, 23:28 GMT
@loqs: Here is the installation error of package later:


** testing if installed package can be loaded
sh: line 1: 13980 Aborted (core dumped) '/usr/lib64/R/bin/R' --no-save --slave 2>&1 < '/tmp/Rtmp9qGF9i/file366f4967975f'
terminate called after throwing an instance of 'std::runtime_error'
what(): Mutex creation failed
ERROR: loading failed
* removing '/home/panman/R/x86_64-pc-linux-gnu-library/3.5/later'
* restoring previous '/home/panman/R/x86_64-pc-linux-gnu-library/3.5/later'

The downloaded source packages are in
'/tmp/Rtmppyhlu1/downloaded_packages'
Warning message:
In install.packages("later") :
installation of package 'later' had non-zero exit status



And here is the error when I try to load it:

terminate called after throwing an instance of 'std::runtime_error'
what(): Mutex creation failed
Aborted (core dumped)




By the way, I have made these libraries install and run by downgrading the following Arch packages to their previous versions:

gcc-8.1.1+20180531-1-x86_64.pkg.tar.xz
gcc-fortran-8.1.1+20180531-1-x86_64.pkg.tar.xz
gcc-libs-8.1.1+20180531-1-x86_64.pkg.tar.xz
glibc-2.27-3-x86_64.pkg.tar.xz
lib32-glibc-2.27-3-x86_64.pkg.tar.xz
Comment by 0f4784cf (0f4784cf) - Wednesday, 08 August 2018, 10:04 GMT
Hello,

I confirm what @SataMaxx and @panman have reported. This bug prevents me from working on my Shiny project.

Launching the command `install.packages('shiny')` in R produces an error and coredump.

@loqs: My logfile and coredump are available here: https://send.firefox.com/download/d9e78a1144/#rJ665uGdP-TSa8MbW9e-zA

(Link valid for 20 downloads or 24 hours)
Comment by Nicola Squartini (tensor5) - Wednesday, 08 August 2018, 10:59 GMT
Apps using [community] electron are not affected. I even tried running skype using system installed electron (cd /usr/share/skypeforlinux/resources && electron --app=app.asar) and it works fine. Maybe upstream they need to rebuild.
Comment by Eli Schwartz (eschwartz) - Wednesday, 08 August 2018, 15:06 GMT
Upstream bugreport for electron: https://github.com/electron/electron/issues/13972

I discussed this on #glibc on Freenode, as mentioned on the electron issue -- it would seem to be a glibc issue, dunno why only prebuilt electron triggers it though. :D
Comment by Eli Schwartz (eschwartz) - Wednesday, 08 August 2018, 15:10 GMT
OTOH this is a good excuse to encourage more people to depend on the system electron package...
Comment by M. Edward (Ed) Borasky (znmeb) - Wednesday, 08 August 2018, 18:22 GMT
Add R to the list. :-( Symptom is crashes in package installs with mysterious messages about mutexes.
Comment by loqs (loqs) - Wednesday, 08 August 2018, 18:40 GMT
@znmeb you have tested glibc 2.28 with e7feec374c635b6a29d65c39ae5e1855528fed39 reverted and that resolved the issue for R as well?
Comment by M. Edward (Ed) Borasky (znmeb) - Wednesday, 08 August 2018, 18:51 GMT
Not yet - I downgraded to glibc 2.27 so I could get my R packages to build. How do I get the reverted glibc to test?
Comment by loqs (loqs) - Wednesday, 08 August 2018, 19:20 GMT
@eschwartz could you post a 2.28 with e7feec374c635b6a29d65c39ae5e1855528fed39 reverted to pkgbuild.com for the R users to test?
Comment by Eli Schwartz (eschwartz) - Wednesday, 08 August 2018, 20:10 GMT
Well, it seems like it's a bug in lld, manifesting via electron's vendored llvm version, see https://github.com/electron/electron/issues/13972 and https://github.com/electron/electron/pull/13988

Devendoring vindicated. ;)

So the question is, how are these R packages being built?
Comment by Patrick Schratz (pat-s) - Wednesday, 08 August 2018, 20:37 GMT
Just wanna push on the severity of this issue. My e-mail client is broken (electron based), my work language (R) and more minor stuff.
Reverting to glib 2.27 triggers more libraries to be reverted (including icu) and it seems to be never ending so I stopped reverting.
My whole system is pretty much unusable right now, so hopefully a fix can be provided soon.

Thanks for all your work!
Comment by loqs (loqs) - Wednesday, 08 August 2018, 20:51 GMT
The R issue upstream is being dicussed https://github.com/r-lib/later/issues/63 and has been noted as similar to https://github.com/r-lib/later/issues/45
but the r package as built by arch does not have clang / llvm listed in .BUILDINFO
I was suggesting building with e7feec374c635b6a29d65c39ae5e1855528fed39 reverted to see if it the same change that is the cause or if 2.28 has two distinct bugs.
Edit:
Better wordingg would be "if glibc has triggered two distinct issues".
Comment by loqs (loqs) - Wednesday, 08 August 2018, 20:55 GMT Comment by Eli Schwartz (eschwartz) - Wednesday, 08 August 2018, 21:29 GMT
https://wiki.archlinux.org/index.php/Unofficial_user_repositories#eschwartz

See the "glibc-fix-r-and-electron-git" package signed by my TU signing key. You can directly pacman -U https://pkgbuild.com/~eschwartz/repo/x86_64/glibc-fix-r-and-electron-git-2.28.r24.g92a4cba760-1-x86_64.pkg.tar.xz and it will be verified before installing.
Comment by M. Edward (Ed) Borasky (znmeb) - Wednesday, 08 August 2018, 22:16 GMT
OK - ready for testing. Here's the "before"

```
> install.packages("learnr")
Installing package into ‘/home/znmeb/R/x86_64-pc-linux-gnu-library/3.5’
(as ‘lib’ is unspecified)
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 286k 100 286k 0 0 1858k 0 --:--:-- --:--:-- --:--:-- 1858k
* installing *source* package ‘learnr’ ...
** package ‘learnr’ successfully unpacked and MD5 sums checked
** R
** inst
** byte-compile and prepare package for lazy loading
terminate called after throwing an instance of 'std::runtime_error'
what(): Mutex creation failed
/usr/lib64/R/bin/INSTALL: line 34: 29151 Done echo 'tools:::.install_packages()'
29152 Aborted (core dumped) | R_DEFAULT_PACKAGES= LC_COLLATE=C "${R_HOME}/bin/R" $myArgs --slave --args ${args}

The downloaded source packages are in
‘/tmp/RtmpeQGwZe/downloaded_packages’
Warning message:
In install.packages("learnr") :
installation of package ‘learnr’ had non-zero exit status
>
```
Comment by M. Edward (Ed) Borasky (znmeb) - Wednesday, 08 August 2018, 22:20 GMT
Nope - still crashes the same way. R just got updated, by the way. Should I open a new bug for R? This may be another failure mode.
Comment by M. Edward (Ed) Borasky (znmeb) - Wednesday, 08 August 2018, 22:24 GMT
Wait - it gets worse - when I build R from source it crashes the same way! I was counting on that to find this. :-(
Comment by Eli Schwartz (eschwartz) - Wednesday, 08 August 2018, 22:34 GMT
Yes, it would seem that electron and R are separate bugs. Please open a separate report for it.
Comment by M. Edward (Ed) Borasky (znmeb) - Wednesday, 08 August 2018, 22:48 GMT
Meanwhile if you downgrade glibc and lib32-glibc, the GNOME desktop no longer comes up. Sigh. :-(

For those of you for whom everything except R is working, I highly recommend running R / RStudio server in Docker via the "rocker/rstudio" image.
Comment by Allan McRae (Allan) - Thursday, 09 August 2018, 05:23 GMT
The R issue is completely separate to the electron one. Stop commenting on it here.


While the electron issues not a glibc bug as such, we need to revert the identified patch until electron upstream releases rebuilt binaries with their lld fix and all users of those libraries are updated. Not ideal, but this is a workaround we can provide.
Comment by David McKay (rawkode) - Friday, 10 August 2018, 09:35 GMT
Yeah, downgrading `glibc` isn't an option; everything else breaks. Hopefully this can be reverted soon :(
Comment by Adam (blackjok3r) - Friday, 10 August 2018, 13:22 GMT
I installed the glibc package above and its fixed my issue with this. Thanks.

Loading...