Trigger Node Down Alert Based on Quantity of Polling Cycles Instead of Time

dmvrtisjr

What I'm looking to do is create a single node down alert in NPM that would allow me to trigger the alert based on the node being down for 2 polling cycles instead of "X" amount of minutes.

Use case is: We have some nodes being polled every 2 minutes, some every 5 minutes, some every 10 minutes. It's completely possible that these times might change, or additional times may be added. We want the alert to trigger if any node has missed 2 polling cycles. So for nodes polled every 2 minutes the trigger would be at 4 minutes, nodes polled every 5 minutes would trigger at 10 minutes, and nodes polled every 10 minutes would trigger at 20 minutes.

There are about 10,000 nodes, so we don't want to break these into multiple node down alerts with different times, we simply want it to trigger after being marked down for 2 polling cycles. Does anybody know of a good clean way of accomplishing this?

I tried using the Skipped Polling Cycles option as a trigger but it did not trigger after being down for multiple polling cycles. Any additional help would be great! Thanks.

brscott

I think this is possible in Orion Platform 2019.2 or newer.  It's a bit different from what you are asking though.  I would prefer to also implement this functionality in the alert conditions as you put forth and I suggest that you go ahead and put in a feature request.

NOTE: I have *not* tested the below suggestion, but I believe it will work.

If you have not already, then upgrade to 2019.2HF1 or 2019.4.  Starting in the 2019 versions, there are additional fields on the packet loss thresholds.  For example, the older verisons had "Critical [Greater than] [80]%".  Now it has "Critical [Greater than] [80]% for [Single poll]", where the [words] are drop downs or edit boxes.  So in effect you can control how many polls until the node goes critical, then make the wait time on the alert really short (or not at all).

There is a key difference between handling this in the node threshold vs alert conditions.

Node Thresholds: The Packet Loss critical threshold is set to 4 polling cycles

1) 10:00 Node goes down

2) 10:01 Orion polls the node

3) 10:01 wait X polling cycles in the threshold

4) 10:21 Node status changes to "down"

5) 10:21 Alert is generated

Alert Condition: (this method does not exist currently) The alert is set to wait 4 polling cycles and packet loss critical threshold is set to (default)single poll

1) 10:00 Node goes down

2) 10:01 Orion polls the node

3) 10:01 Node status changes to "down"

4) 10:01 wait X polling cycles in the alert condition

5) 10:21 alert is generated.

The difference is #3 and #4 are swapped. 

Assuming a polling cycle of 5 minutes and we want 4 polls before the alert.  

In both situations the alert is generated after (roughly) 20 minutes.  In the node threshold method the node is green the entire 20 minutes in Orion even though it's down, and this can be confusing.  In the alert condition method, the node is red on the first pool after it went down (a max of 5 minutes), then it's red for 20 minutes before the alert.

So great thanks to the developers.  I love the new packet loss threshold settings, I am sure I will use the new options eventually, BUT I still want the "condition must exist for [XX] polls" in the alert conditions, the way @dmvrtisjr is asking. 

Now someone is probably going to tell me this is already in the 2019.4 version, I haven't had time to upgrade to yet....