Installing and configuring Crash utility for kernel dumps analysis on RHEL OR CentOS 7

Installing and configuring Crash utility for kernel dumps analysis on RHEL OR CentOS 7

What is Crash utility?

Crash is a linux utility which analyses the core dump file created by kdump. Crash utility depends on the kdump and kexec utilities to obtain its input file. A linux kernel, when boots with the crashkernel argument, it reserves some memory space to capture kernel dump in panic conditions. In case kernel panic's, the kexec utility triggers a warm reboot into a dump kernel, where the memory contents of the panicked kernel get's backed up. Once the core dump copied to the configured location then system does cold reboot and loads up standard default kernel.

Step 1. Install Crash utility and Kernel-debug package:

# yum install crash

OR You can download "Crash" source, compile it and install:

# wget http://people.redhat.com/anderson/crash-7.2.5.tar.gz

# tar -zxvf crash-7.2.5.tar.gz

# cd crash-7.2.5

# make && make install

You can get more information on Crash utility from it's whitepaper "http://people.redhat.com/anderson/crash_whitepaper/#BUILD_PROCEDURE"

Step 2. Once Crash is installed, install kernel-debuginfo package:

In my case i am running CentOS 7 so we will have to enable "CentOS-Debuginfo.repo" first,

# vi /etc/yum.repos.d/CentOS-Debuginfo.repo

[base-debuginfo]

name=CentOS-7 - Debuginfo

baseurl=http://debuginfo.centos.org/7/$basearch/

gpgcheck=1

gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-Debug-7

enabled=1

save and exit.

If you are running RHEL 7 follow instructions from below link,

https://access.redhat.com/solutions/9907

# yum install kernel-debuginfo*

Step 3. Enable cump of the vmcore file:

Set "crashkernel=auto" option in GRUB_CMDLINE_LINUX variable.

# vi /etc/default/grub

GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet"

Regenerate "grub.cfg" configuration file:

# grub2-mkconfig -o /boot/grub2/grub.cfg

Step 4. Set and verify default location to capture kernel crash core files:

Note: the core dump can be stored in a local filesystem or directly on a device, or sent via NFS or SSH. If unset, will use the default "/var/crash" directory.

# vi /etc/kdump.conf

path /var/crash

Now we are done with Crash configuration. Once above procedure is done reboot the server.

To test if this configuration is working let's initiate kernel panic :)

Before you inject kernel panic verify if: 

kdump is active and running

# systemctl is-active kdump

active

if not active use below command -

# systemctl enable kdump.service

# systemctl start kdump.service

Once kdump is running run below two commands to initiate kernel panic with sysrq, also make sure you have enough space in core directory "/var/crash" to store core dump:

# echo 1 > /proc/sys/kernel/sysrq

# echo c > /proc/sysrq-trigger

running last command will initiate kernel panic > store core file and the reboots system. So once host is up after reboot run crash utility to analyse vmcore:

# crash /usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux /var/crash/127.0.0.1-2019-04-06-21\:12\:55/vmcore

crash>