Extend alerting to allow for criticality escalations

score 12
Voted on 12 times. You have not voted. Open for Voting

To have to create three different alerts to alert on essentially the same thing is a lot to maintain, and actually increases the query load to process alerts in a large system. By having a single alert covering three different levels (or more), will streamline things, make maintenance of the alerts easier, and clean up the tracking down of alerts overall.


Ability to drive criticality scaling and alert escalation priorities based on a single alert definition with a little more inteligence built in.


Example: free space on a node


Set the level of criticality:

if ( 10% < disk_free% < 15%) : set alert level = Warning

if ( 5% < disk_free% < 10%) : set alert level = Major

if ( 0% < disk_free% < 5% ) : set alert level = Critical

--> this would utilize the trigger conditions, but as you can see, it is a nested condition that needs to affect another aspect of the alert.


Then Set the alert frequency:

if ( alert level = Warning ) : email frequency = every 24 hours

if ( alert level = Major ) : email frequency = every 8 hours

if ( alert level = Critical ) : email frequency = every hour


You can already set the alert level in the message (such as in the subject), so this would further simplify the management of the alert as the message becomes custom to the level, timing and possibly even the target audience.


How would this be done?

Not knowing the underpinnings of the product, I'm assuming that the alert level logic could be in the trigger definition page. If this block matches that, then set this var to that, etc.

then, in the trigger action. This would require a rework of logic in the trigger screen, and an addition of logic in the trigger action screen. Similarly, you may need to tweak the reset action screen.


In the message, you can then take advantage (in the subject, for instance) of the alert level, and you can set the frequency based on the frequency setting for that level.


The result would be this: a single alert with reset condition that escalates both the alert level as well as the reminder frequency depending on the alert level; once the condition is reset, only one alert branch would be affected, instead of three (traditionally), and your maintenance of the alert(s) is simplified.

Similarly, if you go beyond critical, then fall back to major, the frequency changes to the major frequency till the alert moves below major or warning.


Vote history

SolarWinds uses cookies on its websites to make your online experience easier and better. By using our website, you consent to our use of cookies. For more information on cookies, see our cookie policy.