FS#77868 - When installing a custom-built Linux kernel depmod (kmod) crashes with a segmentation fault.

Attached to Project: Arch Linux
Opened by Martin Clauss (martinclauss) - Wednesday, 15 March 2023, 21:41 GMT
Last edited by Toolybird (Toolybird) - Monday, 17 April 2023, 06:25 GMT
Task Type Bug Report
Category Packages: Core
Status Closed
Assigned To No-one
Architecture x86_64
Severity Medium
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

Description:

When installing a custom-built Linux kernel depmod (kmod) crashes with a segmentation fault.

Additional info:

pacman -Qi kmod
Name : kmod
Version : 30-3
Description : Linux kernel module management tools and library
Architecture : x86_64
URL : https://git.kernel.org/pub/scm/utils/kernel/kmod/kmod.git
Licenses : GPL2
Groups : None
Provides : module-init-tools=3.16 libkmod.so=2-64
Depends On : glibc zlib openssl xz zstd
Optional Deps : None
Required By : linux linux-dbg mkinitcpio pciutils systemd
Optional For : None
Conflicts With : module-init-tools
Replaces : module-init-tools
Installed Size : 291.06 KiB
Packager : Evangelos Foutras <foutrelis@archlinux.org>
Build Date : Tue 01 Nov 2022 12:46:41 PM UTC
Install Date : Wed 15 Mar 2023 03:20:43 PM UTC
Install Reason : Installed as a dependency for another package
Install Script : No
Validated By : Signature

Steps to reproduce:

sudo pacman -U linux-dbg-6.2.6.arch1-1-x86_64.pkg.tar.zst

the crash happens during the execution of /usr/lib/initcpio/functions more conrecte:

depmod -b "$BUILDROOT" "$KERNELVERSION"

near the end of the shell script

I gathered the following crash information with gdb:

gdb -ex "set disassembly-flavor intel" -ex "run" -ex "bt full" -ex "info registers" -ex 'x/50i $pc' -ex 'x/50gx $sp' -args depmod -b "$BUILDROOT" "$KERNELVERSION"

Program received signal SIGSEGV, Segmentation fault.
index_insert (node=0x21, node@entry=0x555555fe4d60,
key=key@entry=0x7fffffffa9b0 "symbol:\360\003", value=<optimized out>,
priority=<optimized out>) at tools/depmod.c:276
276 for (j = 0; node->prefix[j]; j++) {
#0 index_insert (node=0x21, node@entry=0x555555fe4d60,
key=key@entry=0x7fffffffa9b0 "symbol:\360\003", value=<optimized out>,
priority=<optimized out>) at tools/depmod.c:276
j = 0
child = <optimized out>
i = 8
ch = <optimized out>
#1 0x000055555555b9a4 in output_symbols_bin (depmod=0x7fffffffaf60,
out=0x55555558a4f0) at tools/depmod.c:2352
duplicate = <optimized out>
sym = 0x555555f869e0
len = 2
idx = 0x555555fe4d60
alias = "symbol:\360\003\000\274\001\000\000\313F\034\234\245n\265\000\021I\243\235\020\006\000\177\000\000\002\000\000\000\000\000\000\000J\262\000 \000\000\000\000\020\307\aVUU\000\000\002\000\000\000\000\000\000\000\002\000\000\000\000\000\000\0001V\214\367\377\177\000\000\360\244XUUU\000\000\360\244XUUU\000\000\004\000\000\000\000\000\000\0001V\214\367\377\177\000\000\004\000\000\000\000\000\000\000xN\214\367\377\177\000\000\034\265\aVUU\000\000:\000\000\000\000\000\000\000 ԡ\367\377\177\000\000\000\000\000\000\000\000\000\000\001\000\000\000\000\000\000\0001V\214\367\377\177\000\000\360\244XUUU\000\000\360\244XUUU\000\000\004\000\000\000\000\000\000\0001"...
salias = {bytes = 0x7fffffffa9b0 "symbol:\360\003", size = 1024,
need_free = false}
baselen = 7
iter = {hash = 0x5555555824d0, bucket = 217, entry = 1}
v = 0x555555f869e0
ret = 0
#2 0x00005555555678e2 in depmod_output (out=0x0, depmod=0x7fffffffaf60)
at tools/depmod.c:2622
fp = 0x55555558a4f0
tmp = "modules.symbols.bin.1031958.275132.1678913714", '\000' <repeats 209 times>
r = <optimized out>
ferr = <optimized out>
dname = 0x7fffffffb1d8 "/tmp/mkinitcpio.EPpxx8/root/lib/modules/6.2.6-arch1-1-dbg"
dfd = <optimized out>
err = 0
tv = {tv_sec = 1678913714, tv_usec = 275132}
itr = <optimized out>
depfiles = <optimized out>
dname = <optimized out>
dfd = <optimized out>
err = <optimized out>
tv = <optimized out>
fp = <optimized out>
tmp = <optimized out>
r = <optimized out>
ferr = <optimized out>
flags = <optimized out>
mode = <optimized out>
fd = <optimized out>
#3 do_depmod (argc=<optimized out>, argv=<optimized out>)
at tools/depmod.c:3112
out = 0x0
err = 0
all = <optimized out>
maybe_all = <optimized out>
n_config_paths = <optimized out>
root = 0x55555557d2a0 "/tmp/mkinitcpio.EPpxx8/root"
config_paths = 0x0
system_map = <optimized out>
module_symvers = <optimized out>
null_kmod_config = 0x0
un = {sysname = '\000' <repeats 64 times>,
nodename = '\000' <repeats 64 times>,
release = '\000' <repeats 64 times>,
version = '\000' <repeats 64 times>,
machine = '\000' <repeats 64 times>,
domainname = '\000' <repeats 64 times>}
ctx = 0x0
cfg = {kversion = 0x7fffffffeddb "6.2.6-arch1-1-dbg",
dirname = "/tmp/mkinitcpio.EPpxx8/root/lib/modules/6.2.6-arch1-1-dbg", '\000' <repeats 4038 times>, dirnamelen = 57, sym_prefix = 0 '\000',
check_symvers = 0 '\000', print_unknown = 0 '\000',
warn_dups = 0 '\000', overrides = 0x0, searches = 0x55555558b820,
externals = 0x0, excludes = 0x0}
depmod = {cfg = 0x7fffffffb1d0, ctx = 0x55555557d2d0, modules = {
array = 0x5555555cfac0, count = 624, total = 640, step = 128},
modules_by_uncrelpath = 0x55555557e490,
modules_by_name = 0x5555555804b0, symbols = 0x5555555824d0}
#4 0x00007ffff786b790 in __libc_start_call_main (
main=main@entry=0x5555555580f0 <main>, argc=argc@entry=4,
argv=argv@entry=0x7fffffffeb68)
at ../sysdeps/nptl/libc_start_call_main.h:58
self = <optimized out>
result = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140737488350056,
9154261971346381771, 0, 140737488350096, 93824992389880,
140737354125312, -9154261970923299893, -9154243749688648757},
mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x7fffffffeb68,
0x4}, data = {prev = 0x0, cleanup = 0x0, canceltype = -5272}}}
not_first_call = <optimized out>
#5 0x00007ffff786b84a in __libc_start_main_impl (main=0x5555555580f0 <main>,
argc=4, argv=0x7fffffffeb68, init=<optimized out>, fini=<optimized out>,
rtld_fini=<optimized out>, stack_end=0x7fffffffeb58)
at ../csu/libc-start.c:360
No locals.
#6 0x0000555555558305 in _start () at ../sysdeps/x86_64/start.S:115
No locals.
rax 0xfffffff0 4294967280
rbx 0xfffffffffffffff0 -16
rcx 0x21 33
rdx 0x8 8
rsi 0xfffffff0 4294967280
rdi 0x555555fe4d60 93825003310432
rbp 0x6 0x6
rsp 0x7fffffffa8e0 0x7fffffffa8e0
r8 0x1 1
r9 0x20 32
r10 0x7fffffffa9b0 140737488333232
r11 0x0 0
r12 0x0 0
r13 0x555555fe5180 93825003311488
r14 0x21 33
r15 0x7 7
rip 0x555555559380 0x555555559380 <index_insert+32>
eflags 0x10202 [ IF RF ]
cs 0x33 51
ss 0x2b 43
ds 0x0 0
es 0x0 0
fs 0x0 0
gs 0x0 0
=> 0x555555559380 <index_insert+32>: mov r13,QWORD PTR [r14]
0x555555559383 <index_insert+35>: movsxd r8,edx
0x555555559386 <index_insert+38>: xor ebp,ebp
0x555555559388 <index_insert+40>: lea rbx,[r10+r8*1]
0x55555555938c <index_insert+44>: mov r15,r8
0x55555555938f <index_insert+47>: movsx r12,BYTE PTR [r13+0x0]
0x555555559394 <index_insert+52>: test r12b,r12b
0x555555559397 <index_insert+55>:
jne 0x5555555593b7 <index_insert+87>
0x555555559399 <index_insert+57>:
jmp 0x55555555945c <index_insert+252>
0x55555555939e <index_insert+62>: xchg ax,ax
0x5555555593a0 <index_insert+64>: add rbp,0x1
0x5555555593a4 <index_insert+68>: add rbx,0x1
0x5555555593a8 <index_insert+72>: movsx r12,BYTE PTR [r13+rbp*1+0x0]
0x5555555593ae <index_insert+78>: test r12b,r12b
0x5555555593b1 <index_insert+81>:
je 0x555555559480 <index_insert+288>
0x5555555593b7 <index_insert+87>: mov eax,ebp
0x5555555593b9 <index_insert+89>: cmp BYTE PTR [rbx],r12b
0x5555555593bc <index_insert+92>:
je 0x5555555593a0 <index_insert+64>
0x5555555593be <index_insert+94>: add edx,ebp
0x5555555593c0 <index_insert+96>: mov QWORD PTR [rsp+0x18],r10
0x5555555593c5 <index_insert+101>: lea r15,[rbp+r8*1+0x0]
0x5555555593ca <index_insert+106>: mov esi,0x1
0x5555555593cf <index_insert+111>: mov DWORD PTR [rsp+0x28],edx
0x5555555593d3 <index_insert+115>: mov edi,0x418
0x5555555593d8 <index_insert+120>:
call QWORD PTR [rip+0x228a2] # 0x55555557bc80
0x5555555593de <index_insert+126>: mov r11d,0x83
0x5555555593e4 <index_insert+132>: mov rsi,r14
0x5555555593e7 <index_insert+135>: mov rdi,rax
0x5555555593ea <index_insert+138>: mov rcx,r11
0x5555555593ed <index_insert+141>: mov QWORD PTR [rsp+0x10],rax
0x5555555593f2 <index_insert+146>:
rep movs QWORD PTR es:[rdi],QWORD PTR ds:[rsi]
0x5555555593f5 <index_insert+149>: lea rdi,[r13+rbp*1+0x1]
0x5555555593fa <index_insert+154>: mov QWORD PTR [rsp+0x8],rcx
0x5555555593ff <index_insert+159>:
call QWORD PTR [rip+0x22b53] # 0x55555557bf58
0x555555559405 <index_insert+165>: lea rdi,[r14+0x8]
0x555555559409 <index_insert+169>: mov rcx,r14
0x55555555940c <index_insert+172>: mov edx,r12d
0x55555555940f <index_insert+175>: mov r9,QWORD PTR [rsp+0x10]
0x555555559414 <index_insert+180>: and rdi,0xfffffffffffffff8
0x555555559418 <index_insert+184>: sub rcx,rdi
0x55555555941b <index_insert+187>: mov QWORD PTR [r9],rax
0x55555555941e <index_insert+190>: add ecx,0x418
0x555555559424 <index_insert+196>: mov QWORD PTR [r14+0x410],0x0
0x55555555942f <index_insert+207>: mov rax,QWORD PTR [rsp+0x8]
0x555555559434 <index_insert+212>: shr ecx,0x3
0x555555559437 <index_insert+215>: rep stos QWORD PTR es:[rdi],rax
0x55555555943a <index_insert+218>: movzx eax,r12b
0x55555555943e <index_insert+222>: mov BYTE PTR [r13+rbp*1+0x0],0x0
0x555555559444 <index_insert+228>: mov ah,dl
0x555555559446 <index_insert+230>: mov QWORD PTR [r14],r13
0x7fffffffa8e0: 0x0000555555fab1f8 0x00007fffffffa9b0
0x7fffffffa8f0: 0x000000000000017b 0x0000555555559337
0x7fffffffa900: 0x000055555558b708 0x0000026f55fb7ae0
0x7fffffffa910: 0x0000555555573cb2 0x0000555555fe4d60
0x7fffffffa920: 0x0000000000000002 0x00007fffffffa978
0x7fffffffa930: 0x0000555555f869f0 0x00007fffffffaf60
0x7fffffffa940: 0x0000555555f869e0 0x000055555555b9a4
0x7fffffffa950: 0x0000000000000004 0x00007fffffffa980
0x7fffffffa960: 0x00007fffffffa9b0 0x000055555558a4f0
0x7fffffffa970: 0x0000555555fa7bbc 0x0000555555f869e0
0x7fffffffa980: 0x00005555555824d0 0x00000001000000d9
0x7fffffffa990: 0x00007fffffffa9b0 0x0000000000000400
0x7fffffffa9a0: 0x000055555558a400 0x000055555558a4f0
0x7fffffffa9b0: 0xf03a6c6f626d7973 0x46cb000001bc0003
0x7fffffffa9c0: 0x491100b56ea59c1c 0x00007f0006109da3
0x7fffffffa9d0: 0x0000000000000002 0x000000002000b24a
0x7fffffffa9e0: 0x000055555607c710 0x0000000000000002
0x7fffffffa9f0: 0x0000000000000002 0x00007ffff78c5631
0x7fffffffaa00: 0x000055555558a4f0 0x000055555558a4f0
0x7fffffffaa10: 0x0000000000000004 0x00007ffff78c5631
0x7fffffffaa20: 0x0000000000000004 0x00007ffff78c4e78
0x7fffffffaa30: 0x000055555607b51c 0x000000000000003a
0x7fffffffaa40: 0x00007ffff7a1d420 0x0000000000000000
0x7fffffffaa50: 0x0000000000000001 0x00007ffff78c5631
0x7fffffffaa60: 0x000055555558a4f0 0x000055555558a4f0

in

mov r13,QWORD PTR [r14]

r14 will be dereferenced but is 0x21, so not a valid address.

The package is quite large

-rw-r--r-- 1 vagrant vagrant 1.6G Mar 15 20:27 linux-dbg-6.2.6.arch1-1-x86_64.pkg.tar.zst

but I could upload it somewhere if necessary.

The kernel was build like this: https://wiki.archlinux.org/title/Kernel/Arch_Build_System and the PKGBUILD file is attached.
It is modified quite a bit to build a kernel with debug information.
I marked all the lines that I modified with # MOD

Thanks!
   PKGBUILD (6.5 KiB)
This task depends upon

Closed by  Toolybird (Toolybird)
Monday, 17 April 2023, 06:25 GMT
Reason for closing:  Upstream
Additional comments about closing:  Hopefully upstream will address the GitHub issue soon. Nothing we can do until then.
Comment by Toolybird (Toolybird) - Wednesday, 15 March 2023, 23:10 GMT
kmod segfaulting is an upstream issue and should be reported there. The standard Arch method for getting backtraces is here [1].

Re: debug kernels -- have you looked at  FS#75652 ?

[1] https://wiki.archlinux.org/title/Debugging/Getting_traces#Debuginfod
Comment by Martin Clauss (martinclauss) - Thursday, 16 March 2023, 08:10 GMT
Thanks for the reply :) I managed to build the kernel with debug information before so that's not the problem. I think it happens when I also leave the debug information in the *modules*.

Debuginfod works... sometimes...

my running kernel compressed:

[vagrant@archlinux data]$ file vmlinuz-linux
vmlinuz-linux: Linux kernel x86 boot executable bzImage, version 6.2.6-arch1-1 (linux@archlinux) #1 SMP PREEMPT_DYNAMIC Mon, 13 Mar 2023 17:02:08 +0000, RO-rootFS, swap_dev 0XB, Normal VGA

uncompressed, stripped and w/o debug info (extracted with extract-vmlinux script):

[vagrant@archlinux data]$ file vmlinux
vmlinux: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=bdb4a56fad97b891ecbccb5d194884721c85b4d2, stripped

[vagrant@archlinux data]$ debuginfod-find debuginfo bdb4a56fad97b891ecbccb5d194884721c85b4d2
Server query failed: No such file or directory

other examples where debug information can be found or not... no idea why it's inconsistent like this:

[vagrant@archlinux data]$ debuginfod-find debuginfo $(pwd)/vmlinux
Server query failed: No such file or directory

[vagrant@archlinux data]$ debuginfod-find debuginfo /bin/ls
Server query failed: No such file or directory

[vagrant@archlinux data]$ debuginfod-find debuginfo /bin/ping
/home/vagrant/.debuginfod_client_cache/2a82604d45603ca094b5c3f60747dd06f5b04586/debuginfo

[vagrant@archlinux data]$ debuginfod-find debuginfo /bin/tr
Server query failed: No such file or directory

[vagrant@archlinux data]$ debuginfod-find debuginfo /bin/whoami
Server query failed: No such file or directory

[vagrant@archlinux data]$ echo $DEBUGINFOD_URLS
https://debuginfod.elfutils.org/

[vagrant@archlinux data]$ readelf -n /bin/ls | rg -i "build id"
GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring)
Build ID: 588ca812c340997ca8660ce0e15ee31a542568ad

The Build ID is there so it should not be a problem... also tried https://debuginfod.archlinux.org/ and https://debuginfod.elfutils.org/

I think I will report it upstream because it is a bug (segfault) either way :)

Thanks for your time!
Comment by Martin Clauss (martinclauss) - Thursday, 16 March 2023, 18:47 GMT

Loading...