29

Now I encounter a problem about Linux NMI Watchdog. I want to use Linux NMI watchdog to detect and recovery OS hang. So, I add "nmi_watchdog=1" into grub.cfg. And then check the /proc/interrupt, NMI were triggered per second. But after I load a module with deadlock (double-acquire spinlock), system were hanged totally, and nothing occurs (never panic!). It looks like that NMI watchdog did not work!

Then I read the Documentation/nmi_watchdog.txt, it says:

Be aware that when using local APIC, the frequency of NMI interrupts it generates, depends on the system load. The local APIC NMI watchdog, lacking a better source, uses the "cycles unhalted" event.

What's the "cycles unhalted" event?

It added:

but if your system locks up on anything but the "hlt" processor instruction, the watchdog will trigger very soon as the "cycles unhalted" event will happen every clock tick...If it locks up on "hlt", then you are out of luck -- the event will not happen at all and the watchdog won't trigger.

Seems like that watchdog won't trigger if processor executes "hlt" instruction, then I search "hlt" in "Intel 64 and IA-32 Architectures Software Developer's Manual, Volumn 2A", it describes it as follow:

Stops instruction execution and places the processor in a HALT state. An enabled interrupt (including NMI and SMI), a debug exception, the BINIT# signal, the INIT# signal, or the RESET# signal will resume execution.

Then I am lost...

My question is:

  • How does Linux NMI watchdog work?
  • Who trigger the NMI?

My OS is Ubuntu 10.04 LTS, Linux-2.6.32.21, CPU Pentium 4 Dual-core 3.20 GHz.

I didn't read the whole source code about NMI watchdog(no time), if I couldn't understand how NMI watchdog work, I want use performance monitoring counter interrupt and inter-processor interrupt (be provided by APIC) to send NMI instead of NMI watchdog.

Rachid K.
  • 2,562
  • 2
  • 3
  • 17
silverbullettt
  • 768
  • 1
  • 9
  • 12

3 Answers3

9

The answer depends on your hardware.

Non-maskable interrupts (NMI) can be triggered 2 ways: 1) when the kernel reaches a halting state that can't be interrupted by another method, and 2) by hardware -- using an NMI button.

On the front of some Dell servers, for example, you will see a small circle with a zig-zag line inside it. This is the NMI symbol. Nearby there is a hole. Insert a pin to trigger the interrupt. If your kernel is built to support it, this will dump a kernel panic trace to the console, then reboot the system.

This can happen very fast. So if you don't have a console attached to save the output to a file, it might look like only a reboot.

Courtney Schwartz
  • 131
  • 1
  • 3
3

As I know, nmi_watchdog would only triggered for non-interruptible hangs. I found an code example by google: http://oslearn.blogspot.in/2011/04/use-nmi-watchdog.html

If your deadlock is not non-interruptiable, you can try enable sysRq to trigger some trace (Alt-printscreen-t) or crash (Alt-printscreen-c) to get more information.

Johnlcf
  • 572
  • 6
  • 7
  • 1
    Thanks. I read the all sources about nmi_watchdog (in linux-2.6.32) already, and now I know how it work :) – silverbullettt Apr 16 '12 at 10:47
  • The code from the original link was [reposted on kernelnewbies](https://lists.kernelnewbies.org/pipermail/kernelnewbies/2011-June/002122.html). It only works on uniprocessor systems. A [reply](https://lists.kernelnewbies.org/pipermail/kernelnewbies/2011-June/002129.html) in that thread includes code that works on multicore and SMP systems. – bain Sep 06 '17 at 21:56
  • The blog behind the link is non public (anymore) "This blog is open to invited readers only" – Benjamin Peter Apr 25 '20 at 12:54
1

Non-Maskable Interrupt(NMI) is the highest priority interrupt that can not be masked by any software. Basically, this is the scenario when you press CTRL+ALT+DELETE on Windows computer.

NMI watchdog is available for i386 and amd64 architectures.In order to use the NMI watchdog, operating systems’ kernel must support APIC(Advanced Programmable Interrupt Controller) protocol.

NMI watchdog can be enabled by the kernel parameters;

kernel.nmi_watchdog=1 →(I/O APIC)

kernel.nmi_watchdog=2 → (Locall APIC)

When NMI is enabled , system will periodically generate a NMI call. Each NMI invokes a handler in Linux kernel and check the number of interrupts. If the NMIs’ handler detects the number of interrupts hasn’t changed for a certain period of time, it assumes that kernel is hung.Then it invokes a kernel panic enter image description here

Command shows the interrupts per CPUs.$ cat /proc/interrupts CPU0 CPU1 CPU2 CPU3
NMI: 24 18 21 18 Non-maskable interrupts

NMI watchdog can be useful to detect server hung and reduce down time. But highly recommended to analyze system performance before enable NMI watchdog. NMI generates occasionally high number of interrupts and reduce server performance.

You can follow these steps to enable and disable NMI watchdog. I always recommend to enable NMI watchdog after performance analysis. Because sometimes it can be more complicated to detect kernel hung in order to analyze server performance.

Enable NMI:

kernel.nmi_watchdog=1 →(I/O APIC)

kernel.nmi_watchdog=2 → (Locall APIC)

sysctl -w kernel.nmi_watchdog=1

Disable NMI:

kernel.nmi_watchdog=0

sysctl -w kernel.nmi_watchdog=0

You can only run this command which kernel version is 2.6.18–238 and later. Otherwise you have to edit grub.conf then reboot server to disable NMI watchdog.

#vim /boot/grub/grub.conf
kernel /vmlinuz-2.6.18–194.el5 ro root=/dev/VolGroup00/LogVol00   
nmi_watchdog=0

Output shows that NMI watchdog is disabled. Because there is no interrupt on CPUs.$ cat /proc/interrupts

   CPU0     CPU1    CPU2     CPU3             
   NMI:    0        0        0        0   Non-maskable interrupts

https://medium.com/@yildirimabdrhm/nmi-watchdog-on-linux-ae3b4c86e8d8

farhad kh
  • 13
  • 1
  • 4