FS#41505 - [gdb] hangs when debugging multithreaded application.

Attached to Project: Arch Linux
Opened by Milian Wolff (milianw) - Friday, 08 August 2014, 13:49 GMT
Last edited by Anatol Pomozov (anatolik) - Sunday, 31 August 2014, 21:17 GMT
Task Type Bug Report
Category Packages: Extra
Status Closed
Assigned To Anatol Pomozov (anatolik)
Architecture x86_64
Severity High
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 11
Private No

Details

Description:

GDB hangs when debugging any multithreaded application. It was reproduced on at least two different ArchLinux machines with current updates installed.
Here's a backtrace of GDB when it happens (missing debug symbols, but maybe better than nothing):

(gdb) thread apply all bt

Thread 4 (Thread 0x7ff44423d700 (LWP 481)):
#0 0x00007ff446193b2f in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1 0x00007ff445376457 in ?? () from /usr/lib/libgc.so.1
#2 0x00007ff44536c62f in ?? () from /usr/lib/libgc.so.1
#3 0x00007ff445374757 in ?? () from /usr/lib/libgc.so.1
#4 0x00007ff44618f124 in start_thread () from /usr/lib/libpthread.so.0
#5 0x00007ff4456ad4bd in clone () from /usr/lib/libc.so.6

Thread 3 (Thread 0x7ff443a3c700 (LWP 482)):
#0 0x00007ff446193b2f in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1 0x00007ff445376457 in ?? () from /usr/lib/libgc.so.1
#2 0x00007ff44536c62f in ?? () from /usr/lib/libgc.so.1
#3 0x00007ff445374757 in ?? () from /usr/lib/libgc.so.1
#4 0x00007ff44618f124 in start_thread () from /usr/lib/libpthread.so.0
#5 0x00007ff4456ad4bd in clone () from /usr/lib/libc.so.6

Thread 2 (Thread 0x7ff44323b700 (LWP 483)):
#0 0x00007ff446193b2f in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1 0x00007ff445376457 in ?? () from /usr/lib/libgc.so.1
#2 0x00007ff44536c62f in ?? () from /usr/lib/libgc.so.1
#3 0x00007ff445374757 in ?? () from /usr/lib/libgc.so.1
#4 0x00007ff44618f124 in start_thread () from /usr/lib/libpthread.so.0
#5 0x00007ff4456ad4bd in clone () from /usr/lib/libc.so.6

Thread 1 (Thread 0x7ff4474d4740 (LWP 479)):
#0 0x00007ff4455f8157 in sigsuspend () from /usr/lib/libc.so.6
#1 0x000000000049eae8 in ?? ()
#2 0x000000000049edae in ?? ()
#3 0x00000000004a060c in ?? ()
#4 0x000000000049f689 in iterate_over_lwps ()
#5 0x00000000004a1cd4 in ?? ()
#6 0x00000000004a8276 in ?? ()
#7 0x00000000005d394d in ?? ()
#8 0x00000000005dae63 in target_wait ()
#9 0x00000000005a70e9 in fetch_inferior_event ()
#10 0x00000000005bc77a in inferior_event_handler ()
#11 0x00000000005ba921 in ?? ()
#12 0x00000000005bacba in gdb_do_one_event ()
---Type <return> to continue, or q <return> to quit---
#13 0x00000000005baf2e in start_event_loop ()
#14 0x00000000005b4783 in ?? ()
#15 0x00000000005b1b2a in catch_errors ()
#16 0x00000000005b5766 in ?? ()
#17 0x00000000005b1b2a in catch_errors ()
#18 0x00000000005b5b0b in gdb_main ()
#19 0x000000000045adf5 in main ()

Additional info:
* package version(s)

$ uname -a
Linux milian-kdab2 3.15.8-1-ARCH #1 SMP PREEMPT Fri Aug 1 08:51:42 CEST 2014 x86_64 GNU/Linux

$ gcc --version
gcc (GCC) 4.9.1
Copyright (C) 2014 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

$ gdb --version
GNU gdb (GDB) 7.8
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".

* config and/or log files etc.


Steps to reproduce:

$ cat test.cpp
#include <thread>
#include <iostream>

using namespace std;

void doStuff()
{
cout << "hello world!\n";
}

int main()
{
thread t1(doStuff);
thread t2(doStuff);
thread t3(doStuff);
thread t4(doStuff);

t1.join();
t2.join();
t3.join();
t4.join();
return 0;
}

$ g++ -std=c++11 -lpthread test.cpp -o test
$ gdb ./test
GNU gdb (GDB) 7.8
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./test...(no debugging symbols found)...done.
(gdb) run
Starting program: /tmp/test
warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
[New Thread 0x7ffff6fe6700 (LWP 398)]
hello world!
[New Thread 0x7ffff67e5700 (LWP 399)]
[Thread 0x7ffff6fe6700 (LWP 398) exited]
hello world!

## it is stuck!
## I can't even interrupt it with CTRL+C:
^C
^C
^C
^C

^C^C^C^C^C^C

## output from a different terminal's ps aux | grep test
## milian 389 0.1 0.3 111340 28656 pts/5 Sl 15:43 0:00 gdb ./test
## milian 394 0.0 0.0 39352 1588 pts/5 tl+ 15:44 0:00 /tmp/test

## now I'll do a killall -s SIGCONT gdb from a different terminal

Program received signal SIGINT, Interrupt.
[Switching to Thread 0x7ffff67e5700 (LWP 399)]
0x00007ffff7bc4020 in __nptl_death_event () from /usr/lib/libpthread.so.0
(gdb)
(gdb) ## now I'll do a killall -s SIGCONT gdb from a different terminal
(gdb) ## huh? where am I now?
(gdb) bt
#0 0x00007ffff7bc4020 in __nptl_death_event () from /usr/lib/libpthread.so.0
#1 0x00007ffff7bc52c7 in start_thread () from /usr/lib/libpthread.so.0
#2 0x00007ffff70d04bd in clone () from /usr/lib/libc.so.6
(gdb) thread apply all bt
[New Thread 0x7ffff5fe4700 (LWP 400)]
(gdb) cont
Continuing.
[Thread 0x7ffff67e5700 (LWP 399) exited]
hello world!
## stuck again! sending SIGCONT one more time
[New Thread 0x7ffff57e3700 (LWP 451)]
hello world!
[Thread 0x7ffff5fe4700 (LWP 400) exited]
[Thread 0x7ffff57e3700 (LWP 451) exited]
[Inferior 1 (process 394) exited normally]
(gdb) ## stuck again! sending SIGCONT one more time
(gdb) bt
No stack.
This task depends upon

Closed by  Anatol Pomozov (anatolik)
Sunday, 31 August 2014, 21:17 GMT
Reason for closing:  Fixed
Additional comments about closing:  Fixed in 7.8-2
Comment by sergio (sergio) - Friday, 08 August 2014, 13:52 GMT
Same result with gdb from master
Comment by Dave Reisner (falconindy) - Friday, 08 August 2014, 14:04 GMT
Not necessarily a bug in gdb. Using pthread_create instead of the c++ <thread> library (which eventually just uses pthread_create), but still compiling with the same flags does *not* replicate this error...

Compiling with clang++ also avoids the hang...
Comment by Milian Wolff (milianw) - Friday, 08 August 2014, 14:15 GMT
It also happens with QThread-based applications in GCC for me (which is how I originally found this). Afaik, QThread (like std::thread) is just a wrapper around pthread_create, no?
Comment by sergio (sergio) - Saturday, 09 August 2014, 13:36 GMT Comment by Alex (nylocx) - Tuesday, 26 August 2014, 12:01 GMT
Just read the report and using this PKGBUILD file to build gdb without guile
http://dpaste.com/18RJWBQ
Comment by sergio (sergio) - Tuesday, 26 August 2014, 12:12 GMT
Are you saying it works fine without guile ?
Comment by Alex (nylocx) - Wednesday, 27 August 2014, 04:36 GMT
Yeah sorry, I really shouldn't stop writing mid sentence. Yes for me compiling gdb without guile solved the issue. I now can debug my boost multithread and qt multithread stuff again.
Comment by Johan R (cleanrock) - Wednesday, 27 August 2014, 05:44 GMT
Success here also without guile.
How about a 7.8-2 without guile ?
Comment by Anatol Pomozov (anatolik) - Wednesday, 27 August 2014, 06:22 GMT
I confirm that disabling guile makes the problem go away.

> How about a 7.8-2 without guile ?
Sounds good to me. I doubt that anyone actively uses guile in gdb. Just pushed new version to [testing].
Comment by Jaap Crezee (jaapcrezee) - Wednesday, 27 August 2014, 18:36 GMT
I just tried 7.8-2 from testing but it does not help me. As soon as I am stepping over system (system(3), stdlib.h) gdb hangs.
After some time:
NON-CRITICAL TIMEOUT

Debug, interrupt (qtcreator):
CALL: INTERRUPT INFERIOR
Attempting to interrupt.
Stop requested...
TRYING TO INTERRUPT INFERIOR
Interrupted 10232

But then everything hangs.
Gdb is no longer eating away 100% on one core though....
Comment by sergio (sergio) - Wednesday, 27 August 2014, 18:39 GMT
That's maybe a different bug, can you reproduce it with the attached test-case, or other threading scenarios ?
Comment by Jaap Crezee (jaapcrezee) - Wednesday, 27 August 2014, 18:41 GMT
Uhoh, where do I find this attachment?
Comment by sergio (sergio) - Wednesday, 27 August 2014, 18:43 GMT
It's in the first comment
Comment by Jaap Crezee (jaapcrezee) - Wednesday, 27 August 2014, 18:59 GMT
Seems to work:

[jaap@jaap /tmp ]$ g++ -std=c++11 -lpthread test.cpp -ggdb -o test
[jaap@jaap /tmp ]$ gdb ./test
GNU gdb (GDB) 7.8
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./test...done.
(gdb) r
Starting program: /tmp/test
Got object file from memory but can't read symbols: File truncated.
warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
[New Thread 0x7ffff6fe6700 (LWP 10538)]
hello world!
[New Thread 0x7ffff67e5700 (LWP 10539)]
hello world!
[New Thread 0x7ffff5fe4700 (LWP 10540)]
hello world!
[Thread 0x7ffff67e5700 (LWP 10539) exited]
[Thread 0x7ffff5fe4700 (LWP 10540) exited]
[New Thread 0x7ffff57e3700 (LWP 10541)]
hello world!
[Thread 0x7ffff57e3700 (LWP 10541) exited]
[Thread 0x7ffff6fe6700 (LWP 10538) exited]
[Inferior 1 (process 10534) exited normally]
(gdb) q
[jaap@jaap /tmp ]$


Maybe it is some other bug; I was triggered by system -> using fork -> using pthread => threading

Loading...