Troubleshoot a Linux Kernel Crash on CentOS / RHEL 6

Intallation

Red Hat

yum install --enablerepo=rhel-6-server-debug-rpms crash

Find out the kernel version you need to install for the crash dump

crash --osrelease 127.0.0.1-2017-09-13-02\:18\:39/vmcore

yum install kernel-debuginfo-<version> kernel-debuginfo-common-<version>

CentOS

yum install --enablerepo=base-debuginfo crash kernel-debuginfo kernel-debuginfo-common

Analize the dump

crash /usr/lib/debug/lib/modules/2.6.32-642.13.2.el6.x86_64/vmlinux /var/crash/127.0.0.1-2017-02-22-12\:56\:17/vmcore

If the vmcoare is incomplete

crash /usr/lib/debug/lib/modules/2.6.32-642.13.2.el6.x86_64/vmlinux /var/crash/127.0.0.1-2017-02-22-12\:56\:17/vmcore --minimal

Commands

  • Check the kernel version of the vmcore: crash --osrelease 127.0.0.1-2017-09-13-02\:18\:39/vmcore or strings vmcore | head
  • Find the reason for the panic: crash> sys | grep -e RELEASE -e PANIC
  • Dump the kernel message buffer: crash> log
  • Display process status information: crash> ps
  • Display blocked processes: crash> ps | grep UN
  • Display a kernel stack backtrace: crash> bt
  • Display kernel command line: crash> p saved_command_line
  • Display a sysctl parameter: crash> p panic_on_oops
  • Display load average: crash> sys |grep -e CPUS -e LOAD
  • Display memory usage: crash> kmem -i
  • Display a summary consisting of the number of tasks in a task state: ps -S
  • List Running/Runnable: ps |grep RU |wc -l

Links