Suppressing Reset Alerts

I'm currently trying to tweak some of my alerts to make them less annoying afterhours and seem to have hit a wall. Basically, our alerts are setup to go out after a node is down for 5 minutes to an e-mail address 24/7 then send out the reset message when the node comes back up. Afterhours, I have it setup that a node has to be down for 30 minutes before sending out a page to the afterhours pager. These are the alert trigger actions that I have setup, the first line is for all hours, the third line is for afterhours weekedays, the fourth line is for weekends.

This works great on the alert end, the normal e-mail goes out as it should, then the tech gets paged to his phone 30 minutes later which repeats every 15 minutes until acknowledged.

The issue I'm running into is the reset trigger. There is no delay option for the reset triggers that I can find. It gives options to set the e-mail address, message, and time of day but no escalation option. What this does is sends out pages that a node has reset even though the tech never received a page that it was down.

I know its possible to setup seperate alerts for each time frame but that is what we are trying to get away with, plus these are much cleaner.

Any suggestions out there?

Find more posts tagged with

advanced_alerts

advanced_alert

Alert

alert_condition

Alerts

Accepted answers

All comments

julrich

I wrote this handy little SQL query that MAY help, or at least maybe get us moving in one direction.

You could use this: {SQL:Select CASE WHEN Dateadd(MINUTE,15,'${AlertTriggerTime}') > GETDATE() THEN 'put your whole alert email in here' END } to send a blank email if the alert clears before 15 minutes.

Or you could use this {SQL:Select CASE WHEN Dateadd(MINUTE,15,'${AlertTriggerTime}') < GETDATE() THEN 'put something that will KILL the email here' END } to kill (not send) the email if it clears before 15 minutes. Maybe someone from Solarwinds, or anyone else smarter than me can figure out how to kill or abort the email in the middle of it.

rspoelstra

This may seem like a dumb question but where do you use the query?

julrich

Put it in the body of the Reset Email.

The first one, put what ever you already have in the reset email inside the sql query where it says put your whole alert email in here

rspoelstra

I think that's moving in the right direction. What I'd like to do is what you've started on in the second query but wander what would be needed to make it not send out. Obviously there is a variable available that is used on the trigger alerts.

rspoelstra

Just out of curiosity, if I were to use this and use a different e-mail address to send to if it's been less than 15 minutes, would it still send to the main address setup in the condition?

In a better explanation, I have that particular reset trigger set to send a page out to our afterhours. If I put that query in my reset e-mail and the reset is more than 15 minutes after the AlertTriggerTime, would it still send the e-mail out to the pager?