Node Warning Status?

So the following is a quote from the NPM Administrators Guide...

A device may drop packets or fail to respond to a poll for many reasons. Should the device fail to respond, the device status is changed from Up to Warning. On the Node Warning Interval tab, you specify how long it will remain in the Warning status before it is marked as Down. During the interval specified, the service performs "fast polling" to continually check the node status.

In the events I see a Node Warning event type; however, I don't see any events that have ever indicated that a node has gone into a warning status. We have had nodes go down so I don't understand how those nodes were never flagged in a warning status prior to being flagged as down. If I am understanding the quoted documentation above correctly, all nodes that go down due to ICMP inaccessibility should be flagged in a warning state prior (120 Seconds by default) before they are flagged a down.

Please help me better understand, thanks!

Find more posts tagged with

Accepted answers

pacetti

Byron,

I've just confirmed with Development that, while there is a Warning status to indicate when a device is "fast-polled" to determine if the device really is down, there is no corresponding event. You can filter on Warning as a status, but there isn't a Warning event to see anywhere.

Thanks for helping us clarify this,

All comments

byrona

Any ideas on this?

pacetti

Byron,

Thanks for helping us clarify this,

byrona

So, while there isn't an event, it's still put into an actual Warning Status and if I filter on that with an alert I will get notified any time a node is being fast-polled? Is that correct?

pacetti

Yes, that is my understanding.

SimonEvans

Based on this conversation, it appears there is no way that a 'Node Warning' event can be logged. Yet here we are two years later and this is still one of the event types that is present in the 'Event Type' drop down box.

Is there any plans for NPM to add 'Node Warning' events to the Event Log when a Node goes into warming state? If not, is there a plan to remove the 'Node Warning' event from the 'Event Type' drop down box?

adatole

My understanding (in talking with several SW techs) is that the fast-poll works like this:

ping every xx seconds (120 by default)

if a ping fails:

mark node as warning and ping every 5 seconds

IF 10 pings in a row fail, mark the node down

IF you don't get 10 failed pings, stop "fast ping" after 1 minute

wait polling interval and try again

The same is true when a node comes back up (ping once every 120 seconds, if you get a good ping, go into fast ping mode, when you get 10 good pings in a row mark the node up)

With that said, the window for a node in warning status is typically very short - down is down, so less than a minute after the first failed ping, the node is down.

Since the typical interval to check for an alert is 1 minute (that's on the first page of the alert properties), you never catch it.

We did have devices in warning status for longer: they were on the other side of a mis-configured circuit where the xmit was 1Mbps but the rcv was only 512kbps. So we were dropping 50% of our pings all the time.

What I'm getting at is that you can probably set up a SQL-based alert to look for packet loss within a short timeframe, and have that alert check the db every 10 seconds (or whatever). But you would be creating a lot of "churn" on the system to catch what is usually an extremely transient state.

IM(ns)HO, YMMV, caveate redor, egos may be larger than they appear, etc etc.

SimonEvans

Yes, I understand that the period of time that a node may be in the warning state may be brief. It would be useful for NPM to be able to log when a node goes into a Warning state. At the moment if a node in NPM goes into warning there does not appear to be a record that it ever went into warning state. As there is an 'Event Type' called 'Node Warning' it stands to reason that under some circumstances events of this type should be able to be added to the event log. If NPM is not intending on be able to log 'Node Warning' events then it should not be listed as a possible event type.

We suspect there is some sort of issue in part of our network that is causing some packet loss. This results in nodes briefly going into a Warning state and then returning to an Up state. If NPM logged when a node went into a Warning state then we could potentially do some sort of historical examination to see whether the issue started happening after a specific date or if the number of warning has been slowly increasing over time. As this logging does not appear to happen, we are left guessing.

We could set up some SQL-based monitoring of the node status table to advise us when a node goes into warning, however yes, it would be resource intensive. We try not to make these sorts of changes outside the normal operation of the product as product upgrades can break this sort of monitoring.

patriot

Adatole's description of the way the Node Warning level and fast polling function is a little different than what I have always understood. From what I understand, if a node fails to respond to a poll, the Node Warning counter immediately starts and the node is fast polled every 10 seconds. If the node responds before the Node Warning interval expires, then the node is set to UP and normal polling resumes. If the Node Warning interval expires with the node still is not responding, it is set to DOWN, and normal polling resumes as well.

I would appreciate knowing which understanding is correct (or for that matter if neither is correct ). Would a knowledgeable SolarWinds engineer care to weigh in? Thanks.

patriot

No one? Both descriptions of the Node Warning Interval cannot be correct. Anyone at SolarWinds?

byrona

is a SolarWinds developer and he confirmed with development that it functions they way that he suggests in his response.