Community
- Command Central
- MVP Program
- Monthly Mission
- Blogs
- Groups
- Events
- Media Vault
Products
- Observability
- Network Management
- Application Management
- IT Security
- IT Service Management
- System Management
- Database Management
Content Exchange
- SolarWinds Platform
- Server & Application Monitor
- Database Performance Analyzer
- Server Configuration Monitor
- Network Performance Monitor
- Network Configuration Manager
- SQL Sentry
- Web Help Desk
Free Tools & Trials
Store

Alert Escalation - Reset Action

I have an alert setup to send out an email notification for Escalation Level 1. After 1 hour if there have been no acknowledgement or status change it will send out a SMS.

I also have a reset action to send out an email after the status has been changed from Critical to UP. Is there a way configure the alert to send out a Reset SMS only after it reaches Escalation Level 2 ? I want it to send out a Reset email after the Escalation Level 1 clears, and a SMS after Escalation Level 2 clears.

Find more posts tagged with

Accepted answers

All comments

kylemock

I had the same problem. Here is what I did to get around the escalation Level reset action.

My alert was set to if an alert was triggered then it would send an email to the IT dept that there was a problem. If they didn't respond in 5 minutes then it was escalated to a text and voice alert, but still to the IT dept. 10 minutes after it was escalated as an email alert was sent to management. Then 10 minutes after that it was escalated again sending Email, text, and voice messages to management. Reset Action was just emails.

So what I did was copy the alerts that I created and made management there own alert. In the trigger conditions, I just checked the box that the condition has to exist for 15 minutes before it triggered. This made it so that Management would only get a notification if some was wrong for at least 15 minutes.

gwest1972

Am looking for similar, but wanting the "update" email to be a reply to the original, so that all are contained in a single "thread".

silverbacksays

Hmm. This could be a fiddly one. Currently, the different escalation TRIGGER actions do not have a separate RESET action section. The alert engine uses global resets, regardless.

You could do something with Custom Properties (my favourite piece of the Orion puzzle), and have an alert set to trigger whenever a node has a specific Custom Property value assigned to it, and the reset for _THAT_ alert performs the fancy escalation tier two actions you're looking for.

So:

When your alert escalates to level two, add a trigger action to change 'Alert_Escallatrion' node custom property set to yes.
Create a new alert, I'll call it 'Escalation_level_Two', which specifically looks for this value, and have it's trigger action send an event to the NPM event log, so you can audit and report on when this happens.
Set the reset actions on this new alert to reset when the trigger conditions are no longer active, but have the reset actions to carry out the actions you're looking to take, but only for tier 2 escallated alerts.
Then, in the reset actions of the FIRST alert, have it reset the 'Alert-Escalation' node custom property to 'No'.
So, when the first alert triggers and is reset, regardless of whether it's a tier 1 or 2 escalation, it'll RESET any of your 'Escalation_Level_Two' alerts triggered due to a breach of tier 1 in the original alert.

It's a little complicated/convoluted, but it'll give you a solution to taking certain actions only where an alert has been escallated to level two.

I use similar logic for what I call 'management alerts', where i use the power of Custom Properties to perform various actions, treating these Custom Properties as a binary switch.

itengineer

Hi @silverbacksays ,

Is suggested method above can be combined with the PagerDuty alerting as well?

silverbacksays

Sure @itengineer

You can add as many trigger actions as you need