How to configure kdump on CentOS / RHEL 6

Note: Prior to the 6.3GA release, crashkernel=auto will only reserve memory on systems with 4GB or more physical memory. If the system has less than 4GB of memory the memory must be reserved by explicitly requesting the reservation size, for example: crashkernel=128M. Since the 6.3GA release (kernel-2.6.32-279.el6), this limit has been lowered to 2GB

+-------------------------+  
| RAM       | crashkernel |  
| size      | memory      |  
|-----------+-------------+  
|  0 - 2G   | 128M        |   
| 2G - 6G   | 256M        |   
| 6G - 8G   | 512M        |  
|  > 8G     | 768M        |  
+-------------------------+  

Install the kexec-tools package

yum install kexec-tools

chkconfig kdump on

Adding boot parameters if memory < 2GB

grubby --update-kernel=ALL --args="crashkernel=128M"

reboot

Check the kdump service and if the crashkernel has been loaded

service kdump status

cat /proc/iomem | grep Crash\ kernel

Force a vmcore

echo 1 > /proc/sys/kernel/sysrq
echo c > /proc/sysrq-trigger

After reboot check /var/crash for the dump

ls -l /var/crash

Useful commands

  • Check how much memory was reserved for the kdump kernel: cat /proc/cmdline or cat /sys/kernel/kexec_crash_size

Controling when kdump is activated

Edit /etc/sysctl.conf and run sysctl -p

  • The system must panic if a kernel oops occurs (activated by default)
kernel.panic_on_oops=1
  • The system hang is detected through the Non-Maskable Interrupt (NMI) Watchdog mechanism (activated by default)
kernel.nmi_watchdog=1
  • A hardware NMI button is pressed

Fist disable NMI Watchdog by appending "nmi_watchdog=0" to the kernel line in /boot/grub/grub.conf file and "panic_on_io_nmi = 1" in /etc/sysctl.conf

kernel.unknown_nmi_panic=1
  • An "unrecovered" NMI has occurred
kernel.panic_on_unrecovered_nmi=1
  • Panic if OOM-Killer is initiated
kernel.panic=10
vm.panic_on_oom=1
  • Hung task
kernel.hung_task_panic=1

One time vmcore collection: echo '1' > /proc/sys/kernel/hung_task_panic

Links