FS#75652 - [linux] Kernel vmlinux files do not contain debug info

Attached to Project: Arch Linux
Opened by qookie (qookie) - Friday, 19 August 2022, 15:24 GMT
Last edited by Buggy McBugFace (bugbot) - Saturday, 25 November 2023, 20:13 GMT
Task Type Bug Report
Category Kernel
Status Closed
Assigned To Tobias Powalowski (tpowa)
Jan Alexander Steffens (heftig)
David Runge (dvzrv)
Architecture x86_64
Severity Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

Description:

The kernel vmlinux files (contained in the linux-headers packages) have their debug info stripped, which makes it impossible to debug kernel crashes using "crash".

This applies to at least linux-headers-5.19.2.arch1-1 and linux-zen-headers-5.19.2-zen1-1, and presumably to all kernels based on the same PKGBUILD.

Steps to reproduce:
1. Follow https://wiki.archlinux.org/title/Kdump to set up a dump kernel using a kernel from the repo (instead of a manually compiled one),
2. Trigger a crash,
3. Try to inspect the dump file with "crash":

$ crash /usr/lib/modules/5.19.2-zen1-1-zen/build/vmlinux crashdump-2022-08-19-16\:44\:58

crash 8.0.1
...
For help, type "help".
Type "apropos word" to search for commands related to "word"...

crash: /usr/lib/modules/5.19.2-zen1-1-zen/build/vmlinux: no debugging data available
This task depends upon

Closed by  Buggy McBugFace (bugbot)
Saturday, 25 November 2023, 20:13 GMT
Reason for closing:  Moved
Additional comments about closing:  https://gitlab.archlinux.org/archlinux/p ackaging/packages/linux/issues/2
Comment by Toolybird (Toolybird) - Saturday, 20 August 2022, 00:44 GMT
> have their debug info stripped

Are you sure? I see references to source .c files with "strings". vmlinux is an ELF file and can be inspected with readelf etc, though I'm guessing kernel debug is special. Have you looked at [1]? Maybe the problem is with "crash" utility?

[1] https://bbs.archlinux.org/viewtopic.php?id=264253
Comment by qookie (qookie) - Saturday, 20 August 2022, 11:56 GMT
> I see references to source .c files with "strings". vmlinux is an ELF file and can be inspected with readelf etc, though I'm guessing kernel debug is special.

The files indeed do have a symbol table with proper strings, and the string table also includes the file names for whatever reason (this is common to all executables).
The problem is that the DWARF info is stripped out, so there's no way for the debugger to map instructions to source code lines, know the stack frame or struct layout etc.

For vmlinux, "file" shows:
$ file /usr/lib/modules/5.19.2-arch1-1/build/vmlinux
/usr/lib/modules/5.19.2-arch1-1/build/vmlinux: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=1368c994e25e19983709ee8b14ef7d9de0c6a97a, not stripped

Note the omission of "with debug_info", which denotes the DWARF debug sections (.debug_aranges, .debug_info, etc).

It is also worth noting that the kernel PKGBUILD has the following lines:
echo "Stripping vmlinux..."
strip -v $STRIP_STATIC "$builddir/vmlinux"

And STRIP_STATIC is set in the makepkg.conf file (pacman/repos/core-x86_64/makepkg.conf):
#-- Options to be used when stripping binaries. See `man strip' for details.
STRIP_BINARIES="--strip-all"
#-- Options to be used when stripping shared libraries. See `man strip' for details.
STRIP_SHARED="--strip-unneeded"
#-- Options to be used when stripping static libraries. See `man strip' for details.
STRIP_STATIC="--strip-debug"
Comment by Toolybird (Toolybird) - Sunday, 21 August 2022, 05:24 GMT
> echo "Stripping vmlinux..."
> strip -v $STRIP_STATIC "$builddir/vmlinux"

Cripes! I missed that bit.. It was added back here [1] so it's clearly on purpose. I'm unsure how all of this is meant to work so will defer to the PM's.

[1] https://github.com/archlinux/svntogit-packages/commit/663b0866
Comment by Jan Alexander Steffens (heftig) - Sunday, 21 August 2022, 18:53 GMT
We only ship BTF debug info, not DWARF. The latter is too large.
Comment by qookie (qookie) - Sunday, 21 August 2022, 19:03 GMT
Seems like BTF is not good enough for GDB then. Perhaps this is a good candidate for a split debug package? Linux has native support for split debug via CONFIG_DEBUG_INFO_SPLIT.
Comment by Jan Alexander Steffens (heftig) - Sunday, 21 August 2022, 19:05 GMT
Our split debug packages don't support kernel modules, as far as I can tell. That would need fixing in makepkg first.

Loading...