Decode a Linux Kernel Call Trace (kernel panic or kernel oops)

Kernel Call Trace

A Linux kernel call trace, like any traceback, call stack, stacktrace or backtrace, lists a most-recent-first chain of function calls which led to a crash. For the linux kernel, a crash is a panic or oops.

Without debug symbols, a kernel stack trace may come only with a line-by-line list of function symbols, e.g.

[  844.569701][ T7073] Call Trace:
[  844.569734][ T7073]  <IRQ>
[  844.569805][ T7073]  dump_stack+0x188/0x20d
[  844.569823][ T7073]  ? uncore_pmu_event_add+0xa25/0x16d0

• Here, '?' means that the information about this stack entry is probably not reliable (ref).
• uncore_pmu_event_add is the name of the function symbol.
• +0xa25 represents the offset within the function.
• /0x16d0 is the size of the function.

This cannot be directly correlated to the lines of kernel code for debugging and further analysis. For that we need debug symbols, and a script to decode the call trace.

Kernel image and debug symbols

Debug symbols map memory addresses in the binary image to named variables and functions in the source code. Kernel images do not typically include debug symbols. Stripping debug symbols ensures a lightweight kernel image that can be booted quickly by the bootloader. However, the drawback is that there is limited debug information.

With debug symbols, the same kernel version is 744MB:

$ file /usr/lib/debug/boot/vmlinux-5.4.0-80-generic 
  /usr/lib/debug/boot/vmlinux-5.4.0-80-generic: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=eb7923de948c85242c36d3f0f9d0e1e0f6053a67, with debug_info, not stripped
$ ll -h /usr/lib/debug/boot/vmlinux-5.4.0-80-generic 
-rw-r--r-- 1 root root 744M Jul  9 23:49 /usr/lib/debug/boot/vmlinux-5.4.0-80-generic

With debug symbols stripped (the default), the uncompressed kernel is 43MB:

$ file /tmp/kernel-extract/vmlinux-5.4.0-80-generic 
/tmp/kernel-extract/vmlinux-5.4.0-80-generic: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=eb7923de948c85242c36d3f0f9d0e1e0f6053a67, stripped
$ ll -h /tmp/kernel-extract/vmlinux-5.4.0-80-generic 
-rw-rw-r-- 1 dough dough 43M Aug  9 10:20 /tmp/kernel-extract/vmlinux-5.4.0-80-generic

The decode_stacktrace.sh script

To decode a kernel stacktrace to the line of code in the kernel, we need 4 things:

1. The kernel stacktrace e.g. stacktrace.log.
2. The decode_stacktrace.sh script.
3. The kernel image (of the same version that produced the stack trace) with debug symbols.
4. The kernel source code tree for the same version that produced the stack trace.

scripts$ ./decode_stacktrace.sh <kernel-image-with-debug-symbols> <kernel-source-tree> < stacktrace.log 

The decode_stacktrace.sh script can be found in the kernel source tree, in the scripts directory. Alternatively, it may be found in the kernel headers directory e.g. /usr/src/linux-headers-5.4.0-80-generic/scripts/decode_stacktrace.sh.

To decode a stacktrace, execute decode_stacktrace.sh:

~/linux-kernel-dev/linux_stable/scripts$ ./decode_stacktrace.sh /usr/lib/debug/boot/vmlinux-5.4.0-80-generic /home/dough/linux-kernel-dev/linux_stable/ < ../../stacktrace.log 

Get a kernel image with debug symbols

To get a kernel image with debug symbols, you may need to add a debug symbols repository for the package manager of your linux distribution. On Ubuntu, you can follow the guidance in Debug Symbol Packages. To find and install a kernel image with debug symbols for the kernel currently running:

$ apt search linux-image-`uname -r`
Sorting... Done
Full Text Search... Done
linux-image-5.4.0-80-generic/focal-updates,focal-security,now 5.4.0-80.90 amd64 [installed,automatic]
  Signed kernel image generic

linux-image-5.4.0-80-generic-dbgsym/focal-updates,now 5.4.0-80.90 amd64 [installed]
  Signed kernel image generic

$ sudo apt install linux-image-`uname -r`-dbgsym

Once installed, the kernel image can be found in /usr/lib/debug/boot/ directory.