Send notification if the same alert triggers\resolves {X} times in a period of time

LatteLarry

Does anyone know of a way I can send notification if an alert triggers and resolves itself multiple times in a period of time for the same object (node, interface, volume, etc)? For instance, if an interface is flapping intermittently and I have an alert triggering and resetting itself. Is there a way to send notification that the alert has occurred {x} amount of times in the past 24 hours? I would even be ok with the original alert triggering on the interface flapping containing a detail that the alert has occurred {x} amount of times in the past 24 hours.

Thanks for the help!

jrouviere

You can see how many times the alert triggered if you click on the alert details link that's included with the default e-mail template.

However, it was an interesting thought experiment so I dug some:

I didn't see a default flapping interface alert, but I had some ideas for creating one.

I started looking directly at the alerting at first, but then I saw the Interface Downtime History (InterfaceNetObjectDowntime) and realized that was likely the best fit, so this is what I've come up with:

It is a custom SWQL alert, so you'll need to select those options and the top line will be hard coded for you, but you can copy/paste from the WHERE to the end.

SELECT Interfaces.Uri, Interfaces.DisplayName FROM Orion.NPM.Interfaces AS Interfaces

where interfaces.statusdescription like '%down%' and

(interfaces.interfacedowntimehistory.datetimefrom > (ADDDAY(-1,GETUTCDATE())) and interfaces.interfacedowntimehistory.state > 1)

group by interfaces.uri, interfaces.displayname

having count(interfaces.uri) > 3

A couple of things:

For the first line, you could be more specific, but Orion was being touchy about my testing with Down vs Admin Down so I made it general, this is so the alert will only trigger the next time the interface is down and not trigger on an up interface.

Next line limits the scope to the last 24 hours and for an interface state that isn't Up. Again, I was having some issues finding specifics here. It should have been state = 4 for me from the data, but I just made it general to test the alert, you can query this table for your interface to see how it's reporting to catch your specific use case.

Finally I group them so that I can use having, so this is where you can set it needs to happen X times in 24 hours. X will be whatever you want, but mine could go 3+ as that's how many times I tested it.

I didn't really see anything else out of the box that struck me that it would help here, but I'd be curious if anyone comes up with anything else.