Create a new Alert if x Alerts are triggered in y minutes

rhether

I'd like to build an alert to detect a mass trigger of alerts.
Basically, something is happening in the datacenter.

Example trigger this 'Mass Alert', if we see 10 alerts created in the last 15 minutes.

Any help would be appreciated.
Most likely an SQL or SWQL alert needed for this.

Thanks,
Richard

Find more posts tagged with

npm

Alerts

Accepted answers

All comments

sguido

So this produces a report of alerts over the last 12 hours. You may be able to edit it to give you the alerts over the last 10 minutes and then just perform a count on the number of alerts returned.

SELECT ToLocal(Alert.TimeStamp) as Time, Nodes.Caption AS Name, SubString(Alert.Message, 1, 100) AS Alert_Description,Nodes.DetailsUrl AS DetailsUrl, Nodes.IP AS IP
FROM Orion.AlertHistory Alert
INNER JOIN Orion.Nodes AS Nodes ON Alert.AlertObjects.RelatedNodeId = Nodes.NodeID
WHERE (MinuteDiff(Alert.TimeStamp,ToUtc(GetDate())) < 720)
AND (Alert.Message NOT LIKE '%email%' AND Alert.Message NOT LIKE '%texttospeech%')
ORDER BY Alert.TimeStamp ASC

rhether

Yup, I was able to modify the alert to my needs.
Forgot about the time being in UTC (ToUtc)

Thank you

rhether

I have the SWQL code narrowed down to this:

SELECT COUNT(Nodes.Caption) As Node FROM Orion.AlertHistory Alert INNER JOIN Orion.Nodes AS Nodes ON Alert.AlertObjects.RelatedNodeId = Nodes.NodeID WHERE (MinuteDiff(Alert.TimeStamp,ToUtc(GetDate())) < 120) AND (Alert.Message LIKE '%triggered%')

My issue now is getting it into the SWQL Alert

The part in yellow is where I get thrown off, as I can't modify it.

monitoringlife

Not directly what your asking for but an alternative method. If you have a ticketing tool I would recommended settings up a storm protection feature on that end.

Depending on the type of conditions/alerts your talking about (I got triggered on 'happening in the datacenter'). You can setup a dependency to suppress the alerts (down stream suppression). If WAN link to DC goes down, only make one ticket for WAN link, don't make 5,000 ping alerts because we expect everything to be down when we cant route past you.