Arch Linux

I have a i686 cpu ( pentium 4 2.4 Ghz ) and I don't have the bug with svn,

maybe the problem occurs only on atom processor

Ok, in case that helps, it is an Intel "D510" cpu, two cores with HyperThreading.
1.66 GHz. CPU family 6, model 28.

How can I find out which dynamic shared library ld-linux open_path is being asked to open?
Or probably 'dlopen'.

I have reproduced the problem on an ARM v5 architecture
in addition to x86. What all the failing systems have in common is
that they are headless machines with no X11 or graphical desktop packages
installed. The machine where it works (x86_64) happens to have a full XFCE
desktop.

My suspicion is that this is a packaging problem, and svn is counting on
being able to dynamically load something that is not there in the minimal
server systems, and is not listed as a dependency.

If I download the svn sources and compile it myself, setting ./configure
to omit all dependencies except 'serf', it works.

I've tried it with a clean i686 chroot on Intel Atom machine, but cannot reproduce it.

Maybe you can try to build it yourself with devtools (sudo extra-i686-build) and try again, since the problem doesn't seem to be dependency.

For me it segfaults when I don't have kdelibs installed. kdelibs is only in the makedeps.

Cannot reproduce here. I've installed svn in a clean chroot, and then removed kdelibs. But it still runs.

A little experiment on an ARM v7 machine, the the symptoms are the same on i686 though with
different exact module names: Logged in as root, the command "svn -h" works
as expected. But logged in as a regular user that same command results in a segmentation
fault. Even running under "sudo" is enough to make it work. The backtrace this time
is:
````
#0 0x76fde92c in open_path () from /lib/ld-linux-armhf.so.3
#1 0x76fe0a78 in _dl_map_object () from /lib/ld-linux-armhf.so.3
#2 0x76fe5314 in openaux () from /lib/ld-linux-armhf.so.3
#3 0x76fe77ac in _dl_catch_error () from /lib/ld-linux-armhf.so.3
#4 0x76fe5508 in _dl_map_object_deps () from /lib/ld-linux-armhf.so.3
#5 0x76febcb0 in dl_open_worker () from /lib/ld-linux-armhf.so.3
#6 0x76fe77ac in _dl_catch_error () from /lib/ld-linux-armhf.so.3
#7 0x76feb5ac in _dl_open () from /lib/ld-linux-armhf.so.3
````

kdelibs is installed.

The stack says that it tries to load "platform specific auth client provider", cannot find the file (or maybe it is corrupted) and it fails.

Could you please run with strace: "strace -f svn ...."

Somewhat similar bug http://stackoverflow.com/questions/20577638/library-path-order-for-alternate-glibc-dynamic-linker-ld-so shows SIGSEGV when system has several incompatible libc versions. Having strace log would be helpful to understand what exactly you are loading and what ld-linux.so does not like.

I did as you suggested and used strace. It turns out the missing package is libgnome-keyring.
This needs to be added as a dependency on subversion. In case there is more to glean, I have
attached the log from strace.

At least, that worked on the ARMv7 machine. Over on the i686 machine I installed libgnome-keyring
as well, but that did not satisfy it - now it is looking for libQtCore.so.4. I have attached that
log as well. The ARMv7 machine already had a graphical environment installed so it already had
libQtCore; the i686 machine is a simple headless server and did not.

One has to wonder why svn, a simple command line utility, has to drag in the entire Qt4 library
(nearly 100 MB) of graphical display functionality, all the way down to xorg.

checking the python script according to the guidelines on main page gives subversion as erraneous in pacman.txt. SVN::_Client
SVN::_Repos
SVN::_Wc
SVN::_Fs
SVN::_Ra
SVN::_Delta
SVN::_Core

If this is not relevant plz delete this msg. I know you all find me a hopeless noob :-)

I don't think it relevant, since I still cannot reproduce any of the segfaults :(

In order to recreate the segfaults, you have to NOT have one of the essential libraries
installed. Such as libQtCore and libgnome-keyring. These packages are NOT marked as required
by svn, so if you install svn on a machine that does not already have them (such as a
server machine with no graphical packages) they do not get pulled in, and the segfault will
happen.

Yes, I've tried with clean chroots as well as a headless server, but still cannot reproduce this.

I also have the same problem. I can still use it last month, but somehow (don't know which updated package), now I have segmentation fault on all svn commands.Program received signal SIGSEGV, Segmentation fault.

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7de1171 in open_path () from /lib64/ld-linux-x86-64.so.2
(gdb) bt
#0 0x00007ffff7de1171 in open_path () from /lib64/ld-linux-x86-64.so.2
#1 0x00007ffff7de36b7 in _dl_map_object () from /lib64/ld-linux-x86-64.so.2
#2 0x00007ffff7de7b82 in openaux () from /lib64/ld-linux-x86-64.so.2
#3 0x00007ffff7dea0c4 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#4 0x00007ffff7de7de0 in _dl_map_object_deps ()
from /lib64/ld-linux-x86-64.so.2
#5 0x00007ffff7dee4ea in dl_open_worker () from /lib64/ld-linux-x86-64.so.2
#6 0x00007ffff7dea0c4 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#7 0x00007ffff7dede63 in _dl_open () from /lib64/ld-linux-x86-64.so.2
#8 0x00007ffff3b9c03b in ?? () from /usr/lib/libdl.so.2
#9 0x00007ffff7dea0c4 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#10 0x00007ffff3b9c619 in ?? () from /usr/lib/libdl.so.2
#11 0x00007ffff3b9c0e1 in dlopen () from /usr/lib/libdl.so.2
#12 0x00007ffff6df3f4d in apr_dso_load () from /usr/lib/libapr-1.so.0
#13 0x00007ffff703eff3 in svn_dso_load () from /usr/lib/libsvn_subr-1.so.0
#14 0x00007ffff702d991 in svn_auth_get_platform_specific_provider ()
from /usr/lib/libsvn_subr-1.so.0
#15 0x00007ffff702dca4 in svn_auth_get_platform_specific_client_providers ()
from /usr/lib/libsvn_subr-1.so.0
#16 0x00007ffff7033688 in svn_cmdline_create_auth_baton ()
from /usr/lib/libsvn_subr-1.so.0
#17 0x000000000041b8d7 in ?? ()
#18 0x0000000000406c27 in ?? ()
---Type <return> to continue, or q <return> to quit---
#19 0x00007ffff683e040 in __libc_start_main () from /usr/lib/libc.so.6
#20 0x0000000000406c63 in ?? ()
(gdb) bt
#0 0x00007ffff7de1171 in open_path () from /lib64/ld-linux-x86-64.so.2
#1 0x00007ffff7de36b7 in _dl_map_object () from /lib64/ld-linux-x86-64.so.2
#2 0x00007ffff7de7b82 in openaux () from /lib64/ld-linux-x86-64.so.2
#3 0x00007ffff7dea0c4 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#4 0x00007ffff7de7de0 in _dl_map_object_deps ()
from /lib64/ld-linux-x86-64.so.2
#5 0x00007ffff7dee4ea in dl_open_worker () from /lib64/ld-linux-x86-64.so.2
#6 0x00007ffff7dea0c4 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#7 0x00007ffff7dede63 in _dl_open () from /lib64/ld-linux-x86-64.so.2
#8 0x00007ffff3b9c03b in ?? () from /usr/lib/libdl.so.2
#9 0x00007ffff7dea0c4 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#10 0x00007ffff3b9c619 in ?? () from /usr/lib/libdl.so.2
#11 0x00007ffff3b9c0e1 in dlopen () from /usr/lib/libdl.so.2
#12 0x00007ffff6df3f4d in apr_dso_load () from /usr/lib/libapr-1.so.0
#13 0x00007ffff703eff3 in svn_dso_load () from /usr/lib/libsvn_subr-1.so.0
#14 0x00007ffff702d991 in svn_auth_get_platform_specific_provider ()
from /usr/lib/libsvn_subr-1.so.0
#15 0x00007ffff702dca4 in svn_auth_get_platform_specific_client_providers ()
from /usr/lib/libsvn_subr-1.so.0
#16 0x00007ffff7033688 in svn_cmdline_create_auth_baton ()
from /usr/lib/libsvn_subr-1.so.0
#17 0x000000000041b8d7 in ?? ()
#18 0x0000000000406c27 in ?? ()
#19 0x00007ffff683e040 in __libc_start_main () from /usr/lib/libc.so.6
#20 0x0000000000406c63 in ?? ()

Recently apr was updated maybe subversion just need to be rebuilt?

You can try to rebuild subversion in a clean chroot (using extra-x86_64-build from devtools) and see if it fixes the problem. I still cannot reproduce the segfault myself :/

Most likely you have modified ~/.subversion/config and set 'password-stores=gnome-keyring'. And you don't have gnome-keyring package installed. Am I correct?

@anatolik
I just tried to configure it this way on my headless box, and I don't have any graphics components installed (including gnome-keyring, of course), but still I cannot reproduce the crash.

To reproduce the issue you need to enable following options in ~/.subversion/config

[auth]
password-stores = gnome-keyring
store-passwords = yes

then you need to uninstall 'libgnome-keyring' package. It is the one containing /usr/lib/libgnome-keyring.so.0. Package 'libgnome-keyring' is optional for svn.

Well, I have gnome-keyring installed, and I don't have password-stores=gnome-keyring in my configuration.

Hong, provide output of 'strace -f svn YOURCOMMAND' (see discussion above).

Well, I did have these settings (double checked), and I don't have libgnome-keyring installed. I've tried to clone a new repository and a few other operations, but still didn't get any segfault.

I spotted the problem: I have a non-existent directory in $LD_LIBRARY_PATH. After create the directory, problem solved. However, I don't understand why only svn crashes in this case. The output of strace is attached.

Interesting... I tried to repro the issue locally but it works fine... I looked at your strace output but I don't see any obvious reasons for this SIGSEGV.

> After create the directory, problem solved.
If this true then it sounds like glibc bug in shared libraries loader.

There is other way to debug the issue. The crash happens in open_path() function that comes from glibc http://osxr.org/glibc/source/elf/dl-load.c#1931
To get more information about stacktrace (e.g. function parameters, source line number) please recompile glibc with debug compile options. Get glibc pkgbuild files from ABS and add (!strip debug) to options= array. Then rerun the crash case and provide its stacktrace with additional debug information.

Seems there is no much difference in the log.. Do you mean to write `options=('!strip debug')` or `options=('!strip' 'debug')`?

You should use `options=('!strip' 'debug')`, and the difference will be in stack trace (bt) instead of the output of strace.

OK, here's the bt

Program received signal SIGSEGV, Segmentation fault.
open_path (name=0x7ffff02bdef0 "libkdecore.so.5", namelen=16, mode=-2147483648,
sps=0x7ffff7ffcdc0 <env_path_list>, realname=0x7fffffffd0c8, fbp=0x7fffffffd0d8, loader=0x673ac0,
whatcode=2, found_other_class=0x7fffffffd0c7) at dl-load.c:1903
1903 sps->dirs = (void *) -1;
(gdb) bt
#0 open_path (name=0x7ffff02bdef0 "libkdecore.so.5", namelen=16, mode=-2147483648,
sps=0x7ffff7ffcdc0 <env_path_list>, realname=0x7fffffffd0c8, fbp=0x7fffffffd0d8, loader=0x673ac0,
whatcode=2, found_other_class=0x7fffffffd0c7) at dl-load.c:1903
#1 0x00007ffff7de3597 in _dl_map_object (loader=0x673ac0, name=0x7ffff02bdef0 "libkdecore.so.5",
type=0, trace_mode=0, mode=-136442972, nsid=0) at dl-load.c:2041
#2 0x00007ffff7de7a72 in openaux (a=a@entry=0x7fffffffd8a8) at dl-deps.c:63
#3 0x00007ffff7de9fc4 in _dl_catch_error (objname=objname@entry=0x7fffffffd8a0,
errstring=errstring@entry=0x7fffffffd898, mallocedp=mallocedp@entry=0x7fffffffd897,
operate=operate@entry=0x7ffff7de7a40 <openaux at dl-deps.c:60>, args=args@entry=0x7fffffffd8a8)
at dl-error.c:187
#4 0x00007ffff7de7cd0 in _dl_map_object_deps (map=map@entry=0x673ac0, preloads=preloads@entry=0x0,
npreloads=npreloads@entry=0, trace_mode=trace_mode@entry=0, open_mode=open_mode@entry=-2147483648)
at dl-deps.c:254
#5 0x00007ffff7dee3fa in dl_open_worker (a=a@entry=0x7fffffffdb18) at dl-open.c:261
#6 0x00007ffff7de9fc4 in _dl_catch_error (objname=objname@entry=0x7fffffffdb08,
errstring=errstring@entry=0x7fffffffdb10, mallocedp=mallocedp@entry=0x7fffffffdb07,
operate=operate@entry=0x7ffff7dee2f0 <dl_open_worker at dl-open.c:186>,
args=args@entry=0x7fffffffdb18) at dl-error.c:187
#7 0x00007ffff7dedd73 in _dl_open (file=0x64eb98 "libsvn_auth_kwallet-1.so.0", mode=-2147483390,
caller_dlopen=0x7ffff6df3f4d <apr_dso_load+29>, nsid=-2, argc=<optimized out>,
argv=<optimized out>, env=0x7fffffffe400) at dl-open.c:650
#8 0x00007ffff3b9d03b in ?? () from /usr/lib/libdl.so.2
#9 0x00007ffff7de9fc4 in _dl_catch_error (objname=0x664a20, errstring=0x664a28, mallocedp=0x664a18,
operate=0x7ffff3b9cfe0, args=0x7fffffffdd30) at dl-error.c:187
#10 0x00007ffff3b9d619 in ?? () from /usr/lib/libdl.so.2
#11 0x00007ffff3b9d0e1 in dlopen () from /usr/lib/libdl.so.2
#12 0x00007ffff6df3f4d in apr_dso_load () from /usr/lib/libapr-1.so.0
#13 0x00007ffff703eff3 in svn_dso_load () from /usr/lib/libsvn_subr-1.so.0
#14 0x00007ffff702d991 in svn_auth_get_platform_specific_provider () from /usr/lib/libsvn_subr-1.so.0
#15 0x00007ffff702dca4 in svn_auth_get_platform_specific_client_providers ()
from /usr/lib/libsvn_subr-1.so.0
#16 0x00007ffff7033688 in svn_cmdline_create_auth_baton () from /usr/lib/libsvn_subr-1.so.0
#17 0x000000000041b8d7 in ?? ()
#18 0x0000000000406c27 in ?? ()
#19 0x00007ffff683e040 in __libc_start_main (main=0x406be0, argc=2, argv=0x7fffffffe3e8,
init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe3d8)
at libc-start.c:289
#20 0x0000000000406c63 in ?? ()

Thanks for the last backtrace. Now the bug becomes more searchable. I found two similar issues:

https://sourceware.org/bugzilla/show_bug.cgi?id=10411
https://sourceware.org/bugzilla/show_bug.cgi?id=15378

In short - the code tries to write into RO memory. Apply the patch from the last bug and see if it helps.

Yes, it is resolved by applying this patch. Seems the upstream is not going to include this patch though...

Upstream does not apply because there is not enough feedback provided and there is not enough pressure to fix the bug. Everyone who has this issue please speak up in https://sourceware.org/bugzilla/show_bug.cgi?id=15378 and ask them to fix this bug.

Thanks - I'll handle this upstream.

Hong Xu: Did you have only non-existent paths in your $LD_LIBRARY_PATH, or were some of them OK? I'm trying to determine whether upstream  bug 10411  was valid or only 15378.

Allan: I have only one path in $LD_LIBRARY_PATH, which is a non-existent path.

This is upstream bug https://sourceware.org/bugzilla/show_bug.cgi?id=15378 . I am pushing the patch upstream, but this is really a configuration issue.

I pushed the fix upstream. glibc-2.21 is about to be released, so I will not rebuild glibc until then.