Large outage alert?

rileymat

I'm looking to create an alert for large-outage scenarios.  Currently, I have the following setup:

But what I'm noticing....if 33 nodes are already down, and 2 new nodes go down at the same time, this Outage alert triggers.  When really, only 2 new nodes went down. I'm wanting to only capture large-scale outages, such as 35 simultaneous down nodes.

I'm thinking some sort of time window, like 35 nodes that have gone down within the past 5 minutes....or something similar.

Is this possible?  What am I missing?

rboyer

rileymat

The problem may be with the condition, since it hasn't changed for the 33 nodes, the 2 new triggered nodes would then set off the alert. The time frame would be awesome, but the normal advanced only gives us a 'more than' and not a 'between/within' figure for time, which would be required for what you're after.

I started to build out an alert based on node status change from up to down, with the parameters you have in your alert, but I've not had time to try it out. I believe this would be an event type, not a field, when building out the alert. I'm drawing a blank on how to manipulate it further to account for a new trigger and ignore the 33 previous changed, just like you've run into.

I personally use the group setup for mass outages and use the dependency structure to alert when the particular site/application goes out. This doesn't quite account for a numerical value like you're after, but it serves its purpose in my datacenter.

I'm curious if anyone else has input on this topic, as i'd be curious to know. I don't quite have a use case for it, but always good to share the knowledge.