Rebooting a Centos server ignoring CIFS or other broken / hung mount

We had a problem on a server df would hang and we knew that a cifs / samba share died and recovered. Trying unmount / with any parameters -l -f failed.

Experience and google told us we need to reboot. This is a remote server and we know the shutdown hangs on going down so another way to restart was needed.

We needed magic !!

Wouldn’t it be nice if there was a way to ask the kernel to reboot without needing to access the failing drive? Well, there is a way, and it is remarkably simple.

The “magic SysRq key” provides a way to send commands directly to the kernel through the /proc filesystem. It is enabled via a kernel compile time option, CONFIG_MAGIC_SYSRQ, which seems to be standard on most distributions. First you must activate the magic SysRq option:

echo 1 > /proc/sys/kernel/sysrq

When you are ready to reboot the machine simply run the following:

echo b > /proc/sysrq-trigger

This does not attempt to unmount or sync filesystems, so it should only be used when absolutely necessary, but if your drive is already failing then that may not be a concern.

In addition to rebooting the system the sysrq trick can be used to dump memory information to the console, sync all filesystems, remount all filesystems in read-only mode, send SIGTERM or SIGKILL to all processes except init, or power off the machine entirely, among other things.

Also, instead of echoing into /proc/sys/kernel/sysrq each time you can activate the magic SysRq key at system boot time using sysctl, where supported:

echo “kernel.sysrq = 1” >> /etc/sysctl.conf

If you would like to learn more about magic SysRq you can read the sysrq.txt file in the kernel documentation.

nJoy 😉

Leave a Reply

Your email address will not be published. Required fields are marked *