Windows Server crash still responds to PING, but no CPU or RAM statistics available, and no alerts generated

We are experiencing a unique issue with several of our monitored Windows Servers and I am hoping you guys can provide some guidance.

There are occasions where our Windows Servers crashed and are sitting at the BSOD screen but the server still responds to pings (yes these are virtualized in VMware). So as far as SolarWinds NPM is concerned there is no gap in monitoring and the server is still in an "UP" state. Also, as far as the VMware host is concerned the VM is still in a running state. The only thing I've noticed when this happens is that NPM shows no gaps in network latency, but the CPU and RAM statistic gathering is blank.

I know the best work around is this --- use WMI and monitor a service. So if the server doesn't respond to WMI then we can alert against that. But I have been forbidden to use WMI due to the fact that it causes memory issues with these specific application servers. I cannot use WMI in this case.

Is there a way to monitor for gaps in CPU or RAM statistic gathering? Such as when there are gaps of time where there is no vital statistics gathered from a monitored node even though it is responding to PING?

- Joe

Find more posts tagged with

network_monitoring

crash

network_management

npm

server

Windows

Accepted answers

automag928

Take a look at this thread here and it should solve your problems - NPM Alert on Node UP (Ping) SNMP DOWN (unknown / unresponsive)

Bascially you are collecting cpu / ram via snmp - and the query goes out to see if you've gotten data - and this will alert you if ICMP is still responding, but snmp data collection is failed. Specifically look at this message by Leon Adato -

Re: NPM Alert on Node UP (Ping) SNMP DOWN (unknown / unresponsive)

I've implemented this last week and it indeed lets me know when snmp processes get hung and devices stop responding to snmp but still respond to ICMP.

All comments

automag928

Take a look at this thread here and it should solve your problems - NPM Alert on Node UP (Ping) SNMP DOWN (unknown / unresponsive)

Re: NPM Alert on Node UP (Ping) SNMP DOWN (unknown / unresponsive)

I've implemented this last week and it indeed lets me know when snmp processes get hung and devices stop responding to snmp but still respond to ICMP.

joedissmeyer

Thanks Automag. I will give this a try and let you know if this solves the issue.

- Joe

joedissmeyer

Thanks Jay! Your answer was the correct one.

Here is what I did:

Logged into the SW server and created a new Advanced alert. The "Trigger Condition" area is the important one. Here is what I have:

Type of Property to Monitor: Custom SQL

Set up your Trigger Query: Node

WHERE

(

(DATEDIFF(mi, Nodes.LastSystemUpTimePollUtc, getutcdate()) > 30) AND

(Nodes.Status = '1') AND

(Nodes.ObjectSubType = 'SNMP') AND

(Nodes.MachineType LIKE '%Server%') AND

(Nodes.Assigned_Group LIKE '%Web%')

)

Do not trigger until this condition exists for more than "0" seconds.

Screenshot:

I'll explain a few things in the query:

Nodes.ObjectSubType = 'SNMP' ----- We are using SNMP, ICMP, and WMI monitoring protocols. I want this alert to only include nodes polled using SNMP.
Nodes.MachineType LIKE '%Server%' ----- We do have several different types of nodes including Linux Servers, Windows Client workstations, and other systems. This limits the alert to anything with "server" in the machine type description. If you look closely, all Windows Server OSes have "Server" in the description. So this limitation works for my situation.
Nodes.Assigned_Group LIKE '%Web%' ------ The "Assigned_Group" is a custom property that is unique to my organization's Orion setup. I am limiting the alert to only a specific assigned group in our IT department.

You'll want to set the "Reset Condition" in this alert too. It will be the same as the trigger condition, except for the item highlighted in the screenshot below. Make sure to change the "greater than" in the trigger condition to "less than" in the reset condition. That is the only change you need to make.

I hope this helps others trying to figure out what is being described in the link in the answer to this question.

- Joe

automag928

Nice job on the detailed write up!! Glad I could help, I stumbled across that thread myself and its been perfect!

adatole

Nice work. What we found in our environment is that sometimes the LastSystemUptimePollUTC *WASN'T* getting updated even when SNMP was running. I know that sounds crazy, but we were seeing a lot of false alarms on this logic Which is why I went with checking the CPU statistics instead.

joedissmeyer

Leon Adato, yes I've seen things like that too. Checking the CPU would definitely work with your unique issue. Windows servers are definitely strange beasts sometimes.

- Joe

joedissmeyer

Thanks Jay! Yup, Leon's post is the one that helped me out too - http://thwack.solarwinds.com/message/231657#231657

- Joe

Jfrazier

You have discovered why ping is useful for "reachability" tests only but not availability tests...the server is reachable on the network but is not available.

dstip

I see SNMP is the fix here, but what if your polling your Windows servers via WMI?