Arch Linux

Sounds to me like something you need to take upstream

Thanks, will do.

This is a readline 6.3 API change. The python package should be patched/recompiled. I think that all readline dependencies should be checked to make sure they do not use old API.

Here is related message in readlines maillist http://lists.gnu.org/archive/html/bug-readline/2014-03/msg00002.html

Here is related bug in Ruby https://bugs.ruby-lang.org/issues/9578 with patch (already applied to Arch package).

Just to clarify - I am not sure it is the same we saw with ruby, but it looks very similar.

Try one of the things:
- revert readline to previous version
- recompile python (and patch it if needed)
and see if the error still happens.

Good call! It seems to be readline 6.3, since downgrading to 6.2 solves the problem for me.
Contrary to my initial report, the bug occurs with ipython2 1.1 as well.

Steps to reproduce:
- start ipython session
- enter a sequence of characters which is not in your history and hit upcursor

How should I proceed?

python2/3 should be patched and recompiled for new readline. Reassign it to python maintainers.

I also believe that any binary that depends on /usr/lib/libreadline.so.6 should be checked as well.

And this is a bug (+patch) at the python website http://bugs.python.org/issue20374

API changes should not require a recompile. The ABI should be stable without an soname bump... So there is perhaps something else going on here.

I suspect that 6.3 brings ABI changes as well. The easiest thing is to recompile python against 6.3 and check if it still crashes.

A rebuild of python 3.3.4 against readline 6.3 still crashes here.

I'm looking around in upstream issue tracker, and doesn't seem to find a same problem.

And first lines of stack trace here is:

#0 0x00007ffff1335849 in _rl_dispatch_callback () from /usr/lib/libreadline.so.6
#1 0x00007ffff134bca0 in rl_callback_read_char () from /usr/lib/libreadline.so.6
#2 0x00007ffff156aacb in ?? () from /usr/lib/python3.3/lib-dynload/readline.cpython-33m.so
#3 0x00007ffff79c2e4f in PyOS_Readline (sys_stdin=0x7ffff77484e0 <_IO_2_1_stdin_>, sys_stdout=0x7ffff77482a0 <_IO_2_1_stdout_>,
prompt=0x7ffff350fdc0 "\n\001\033[0;32m\002In [\001\033[1;32m\002\061\001\033[0;32m\002]: \001\033[0m\002") at Parser/myreadline.c:214
#4 0x00007ffff7a6c6f6 in builtin_input (self=<optimized out>, args=<optimized out>) at Python/bltinmodule.c:1734
#5 0x00007ffff7a7794c in call_function (oparg=<optimized out>, pp_stack=0x7fffffffca50) at Python/ceval.c:4069
#6 PyEval_EvalFrameEx (f=f@entry=0xafc6c0, throwflag=throwflag@entry=0) at Python/ceval.c:2679

Ok, if it repro even after recompile then it sounds like a bug in readline. The best of all to contact readline maillist https://lists.gnu.org/mailman/listinfo/bug-readline

I can repro it (with current python3). A bit more debug information:

$ systemd-coredumpctl gdb

(gdb) bt
#0 0x00007fb714b0e849 in _rl_dispatch_callback () from /usr/lib/libreadline.so.6
#1 0x00007fb714b24ca0 in rl_callback_read_char () from /usr/lib/libreadline.so.6
#2 0x00007fb714d43acb in ?? () from /usr/lib/python3.3/lib-dynload/readline.cpython-33m.so
#3 0x00007fb71b0bae4f in PyOS_Readline (sys_stdin=0x7fb71ae404e0 <_IO_2_1_stdin_>, sys_stdout=0x7fb71ae402a0 <_IO_2_1_stdout_>,
prompt=0x7fb7169f0960 "\n\001\033[0;32m\002In [\001\033[1;32m\002\061\001\033[0;32m\002]: \001\033[0m\002") at Parser/myreadline.c:214
#4 0x00007fb71b1646f6 in builtin_input (self=<optimized out>, args=<optimized out>) at Python/bltinmodule.c:1734
#5 0x00007fb71b16f94c in call_function (oparg=<optimized out>, pp_stack=0x7fff753b13a0) at Python/ceval.c:4069
#6 PyEval_EvalFrameEx (f=f@entry=0x2923ed0, throwflag=throwflag@entry=0) at Python/ceval.c:2679
#7 0x00007fb71b1703b1 in PyEval_EvalCodeEx (_co=_co@entry=0x7fb7170b0e40, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=argcount@entry=2, kws=0x293a630, kwcount=0,
defs=0x7fb716881f68, defcount=1, kwdefs=0x0, closure=0x0) at Python/ceval.c:3439
#8 0x00007fb71b16f3b9 in fast_function (nk=<optimized out>, na=2, n=<optimized out>, pp_stack=0x7fff753b15c0, func=<optimized out>) at Python/ceval.c:4167
#9 call_function (oparg=<optimized out>, pp_stack=0x7fff753b15c0) at Python/ceval.c:4090
#10 PyEval_EvalFrameEx (f=f@entry=0x293a470, throwflag=throwflag@entry=0) at Python/ceval.c:2679
#11 0x00007fb71b1703b1 in PyEval_EvalCodeEx (_co=_co@entry=0x7fb7170b0db0, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=argcount@entry=1, kws=0x28c84d8, kwcount=1,
defs=0x7fb716881f28, defcount=1, kwdefs=0x0, closure=0x0) at Python/ceval.c:3439
#12 0x00007fb71b16f3b9 in fast_function (nk=<optimized out>, na=1, n=<optimized out>, pp_stack=0x7fff753b17e0, func=<optimized out>) at Python/ceval.c:4167
#13 call_function (oparg=<optimized out>, pp_stack=0x7fff753b17e0) at Python/ceval.c:4090
#14 PyEval_EvalFrameEx (f=f@entry=0x28c8340, throwflag=throwflag@entry=0) at Python/ceval.c:2679
#15 0x00007fb71b1703b1 in PyEval_EvalCodeEx (_co=_co@entry=0x7fb7170b0c90, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=argcount@entry=1, kws=0x28bc9e8, kwcount=0,
defs=0x7fb716881ee8, defcount=1, kwdefs=0x0, closure=0x0) at Python/ceval.c:3439
#16 0x00007fb71b16f3b9 in fast_function (nk=<optimized out>, na=1, n=<optimized out>, pp_stack=0x7fff753b1a00, func=<optimized out>) at Python/ceval.c:4167
#17 call_function (oparg=<optimized out>, pp_stack=0x7fff753b1a00) at Python/ceval.c:4090
#18 PyEval_EvalFrameEx (f=f@entry=0x28bc860, throwflag=throwflag@entry=0) at Python/ceval.c:2679
#19 0x00007fb71b16f6af in fast_function (nk=<optimized out>, na=1, n=1, pp_stack=0x7fff753b1b60, func=<optimized out>) at Python/ceval.c:4157
#20 call_function (oparg=<optimized out>, pp_stack=0x7fff753b1b60) at Python/ceval.c:4090
#21 PyEval_EvalFrameEx (f=f@entry=0x25a6b80, throwflag=throwflag@entry=0) at Python/ceval.c:2679
#22 0x00007fb71b1703b1 in PyEval_EvalCodeEx (_co=0x7fb71856e5d0, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0x7fb71a01ba68, argcount=1, kws=kws@entry=0x7fb71a00d968,
kwcount=kwcount@entry=1, defs=defs@entry=0x7fb716ff6a28, defcount=defcount@entry=1, kwdefs=0x0, closure=0x0) at Python/ceval.c:3439
#23 0x00007fb71b0e7353 in function_call (func=0x7fb717b818c0, arg=0x7fb71a01ba50, kw=0x7fb71854ec68) at Objects/funcobject.c:633
#24 0x00007fb71b0c129c in PyObject_Call (func=func@entry=0x7fb717b818c0, arg=arg@entry=0x7fb71a01ba50, kw=kw@entry=0x7fb71854ec68) at Objects/abstract.c:2035
#25 0x00007fb71b16b1bc in ext_do_call (nk=<optimized out>, na=<optimized out>, flags=<optimized out>, pp_stack=0x7fff753b1e78, func=0x7fb717b818c0) at Python/ceval.c:4384
#26 PyEval_EvalFrameEx (f=f@entry=0x2559e60, throwflag=throwflag@entry=0) at Python/ceval.c:2720
#27 0x00007fb71b1703b1 in PyEval_EvalCodeEx (_co=_co@entry=0x7fb719fcc390, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=argcount@entry=0, kws=0x24e15a8, kwcount=0,
defs=0x7fb71855d4a8, defcount=1, kwdefs=0x0, closure=0x0) at Python/ceval.c:3439
#28 0x00007fb71b16f3b9 in fast_function (nk=<optimized out>, na=0, n=<optimized out>, pp_stack=0x7fff753b2090, func=<optimized out>) at Python/ceval.c:4167
#29 call_function (oparg=<optimized out>, pp_stack=0x7fff753b2090) at Python/ceval.c:4090
#30 PyEval_EvalFrameEx (f=f@entry=0x24e1420, throwflag=throwflag@entry=0) at Python/ceval.c:2679
#31 0x00007fb71b1703b1 in PyEval_EvalCodeEx (_co=_co@entry=0x7fb71a373930, globals=globals@entry=0x7fb71a3167a0, locals=locals@entry=0x7fb71a3167a0, args=args@entry=0x0, argcount=argcount@entry=0,
kws=kws@entry=0x0, kwcount=kwcount@entry=0, defs=defs@entry=0x0, defcount=defcount@entry=0, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0) at Python/ceval.c:3439
#32 0x00007fb71b17047b in PyEval_EvalCode (co=co@entry=0x7fb71a373930, globals=globals@entry=0x7fb71a3167a0, locals=locals@entry=0x7fb71a3167a0) at Python/ceval.c:771
#33 0x00007fb71b189bd4 in run_mod (mod=<optimized out>, filename=filename@entry=0x7fb71a2d7168 "/usr/bin/ipython", globals=globals@entry=0x7fb71a3167a0, locals=locals@entry=0x7fb71a3167a0,
flags=flags@entry=0x7fff753b2350, arena=arena@entry=0x24e0ee0) at Python/pythonrun.c:1996
#34 0x00007fb71b18b9a8 in PyRun_FileExFlags (fp=fp@entry=0x24e0ca0, filename=filename@entry=0x7fb71a2d7168 "/usr/bin/ipython", start=start@entry=257, globals=globals@entry=0x7fb71a3167a0,
locals=locals@entry=0x7fb71a3167a0, closeit=closeit@entry=1, flags=flags@entry=0x7fff753b2350) at Python/pythonrun.c:1952
#35 0x00007fb71b18c6d1 in PyRun_SimpleFileExFlags (fp=fp@entry=0x24e0ca0, filename=<optimized out>, closeit=closeit@entry=1, flags=flags@entry=0x7fff753b2350) at Python/pythonrun.c:1452
#36 0x00007fb71b18d4e3 in PyRun_AnyFileExFlags (fp=fp@entry=0x24e0ca0, filename=<optimized out>, closeit=closeit@entry=1, flags=flags@entry=0x7fff753b2350) at Python/pythonrun.c:1174
#37 0x00007fb71b1a1138 in run_file (p_cf=0x7fff753b2350, filename=0x246eff0 L"/usr/bin/ipython", fp=0x24e0ca0) at Modules/main.c:307
#38 Py_Main (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:744
#39 0x0000000000400b29 in main ()


Dump of assembler code for function _rl_dispatch_callback:
0x00007fb714b0e840 <+0>: push %rbp
0x00007fb714b0e841 <+1>: mov %rdi,%rbp
0x00007fb714b0e844 <+4>: push %rbx
0x00007fb714b0e845 <+5>: sub $0x8,%rsp
=> 0x00007fb714b0e849 <+9>: testb $0x1,(%rdi)
0x00007fb714b0e84c <+12>: je 0x7fb714b0e8c0 <_rl_dispatch_callback+128>
0x00007fb714b0e84e <+14>: mov 0x30(%rdi),%ebx
0x00007fb714b0e851 <+17>: cmp $0xfffffffd,%ebx
0x00007fb714b0e854 <+20>: je 0x7fb714b0e86c <_rl_dispatch_callback+44>
0x00007fb714b0e856 <+22>: mov 0x0(%rbp),%ecx
0x00007fb714b0e859 <+25>: mov 0x20(%rbp),%edx
0x00007fb714b0e85c <+28>: mov %ebx,%edi
0x00007fb714b0e85e <+30>: mov 0x18(%rbp),%rsi
0x00007fb714b0e862 <+34>: and $0x2,%ecx
0x00007fb714b0e865 <+37>: callq 0x7fb714b0e6b0 <_rl_subseq_result>
0x00007fb714b0e86a <+42>: mov %eax,%ebx
0x00007fb714b0e86c <+44>: mov 0x22a2e5(%rip),%rax # 0x7fb714d38b58
0x00007fb714b0e873 <+51>: mov (%rax),%edx
0x00007fb714b0e875 <+53>: test %edx,%edx
0x00007fb714b0e877 <+55>: jne 0x7fb714b0e8e8 <_rl_dispatch_callback+168>
0x00007fb714b0e879 <+57>: test %ebx,%ebx
0x00007fb714b0e87b <+59>: je 0x7fb714b0e8f3 <_rl_dispatch_callback+179>
0x00007fb714b0e87d <+61>: cmp $0xfffffffd,%ebx
0x00007fb714b0e880 <+64>: je 0x7fb714b0e910 <_rl_dispatch_callback+208>



(gdb) info register
rax 0x2c0006 2883590
rbx 0x7fb714d3f630 140424305178160
rcx 0x0 0
rdx 0x0 0
rsi 0x7fff753b0e78 140735160192632
rdi 0x0 0
rbp 0x0 0x0
rsp 0x7fff753b10b0 0x7fff753b10b0
r8 0x7fff753b0de0 140735160192480
r9 0x28c27b0 42739632
r10 0x8 8
r11 0x202 514
r12 0x7fb714d3f028 140424305176616
r13 0x0 0
r14 0x1 1
r15 0x7fb71b6d2690 140424415880848
rip 0x7fb714b0e849 0x7fb714b0e849 <_rl_dispatch_callback+9>
eflags 0x10202 [ IF RF ]
cs 0x33 51
ss 0x2b 43
ds 0x0 0
es 0x0 0
fs 0x0 0
gs 0x0 0




So rl_callback_read_char() passes NULL pointer to _rl_dispatch_callback() and it is something the callback does not expect.

@Anatol: Can you post on the readline list given you have all the information?

Ok, will do

http://lists.gnu.org/archive/html/bug-readline/2014-03/msg00010.html

A bug that sounds similar although has different crash stack http://bugs.python.org/issue20631

I just tried to apply the patch in the issue20631, but readline seems to be completely broken after applying the patch.

building 'readline' extension
gcc -pthread -fPIC -fno-strict-aliasing -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector --param=ssp-buffer-size=4 -DNDEBUG -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector --param=ssp-buffer-size=4 -I. -IInclude -I./Include -I/usr/local/include -I/home/felix/projects/arch/packages/python2/trunk/src/Python-2.7.6/Include -I/home/felix/projects/arch/packages/python2/trunk/src/Python-2.7.6 -c /home/felix/projects/arch/packages/python2/trunk/src/Python-2.7.6/Modules/readline.c -o build/temp.linux-x86_64-2.7/home/felix/projects/arch/packages/python2/trunk/src/Python-2.7.6/Modules/readline.o
/home/felix/projects/arch/packages/python2/trunk/src/Python-2.7.6/Modules/readline.c: In function ‘setup_readline’:
/home/felix/projects/arch/packages/python2/trunk/src/Python-2.7.6/Modules/readline.c:914:24: error: ‘Function’ undeclared (first use in this function)
rl_startup_hook = (Function *)on_startup_hook;
^
/home/felix/projects/arch/packages/python2/trunk/src/Python-2.7.6/Modules/readline.c:914:24: note: each undeclared identifier is reported only once for each function it appears in
/home/felix/projects/arch/packages/python2/trunk/src/Python-2.7.6/Modules/readline.c:914:34: error: expected expression before ‘)’ token
rl_startup_hook = (Function *)on_startup_hook;
^
/home/felix/projects/arch/packages/python2/trunk/src/Python-2.7.6/Modules/readline.c:916:36: error: expected expression before ‘)’ token
rl_pre_input_hook = (Function *)on_pre_input_hook;
^
/home/felix/projects/arch/packages/python2/trunk/src/Python-2.7.6/Modules/readline.c:919:41: error: ‘CPPFunction’ undeclared (first use in this function)
rl_attempted_completion_function = (CPPFunction *)flex_complete;
^
/home/felix/projects/arch/packages/python2/trunk/src/Python-2.7.6/Modules/readline.c:919:54: error: expected expression before ‘)’ token
rl_attempted_completion_function = (CPPFunction *)flex_complete;
^

To compile against latest readline you need several patches from upstream. See http://bugs.python.org/issue20374

Oops sorry, forgot about this...

Just applied the patches (incl. issue20374 & issue20631) and tried again, but I'm still getting the same crash in ipython2.

Potential patch: http://lists.gnu.org/archive/html/bug-readline/2014-03/msg00031.html

The patch fixed the "press DEL twice then crash" problem here, but not for the ipython2 "some random chars then up up" crash.

Just pushed readline 6.3-2 to [testing] with the above patch applied. Better than nothing.

http://lists.gnu.org/archive/html/bug-readline/2014-03/msg00034.html

The new patch fixed both cases here.