Alerting - Dropping From Critical Back To Warning ! Is there anyway we can avoid Warning Alert Action

Question

Hi !

We have disk space thresholds configured ! 80% to 89.99% for Warning - 90% to 100% for Critical. We get email in our ticketing system each time any threshold is met. (And act accordingly by calling customer to act etc).

Now last time when it was in critical band (around 96%), customer acted and reduced the space and the 'volume space used' came down to 84%.

Now theoretically warning alert is STILL in place !

My question is: is still solarwinds warning alert will be fired again ?

If Yes:

Is there any way we can avoid warning alert OR atleast the action of warning alert, so that it should not raise a new ticket for warning, when customer has brought it back from critical zone to warning zone.

Many Thanks

mesverrum · Accepted Answer

It depends on what your team actually does at these various levels of warning.  How do you notify on warning, how do you notify on critical, and what is the expected action of the person you notify?

In most cases I push for a model where we don't even alert on warning thresholds, I just send out a daily report of nodes that breached their warning since in 90% of cases people ignore warning level stuff anyway and don't actually take action until they get the second notification that the issue has gone critical.  I mostly try to train people to figure than any alert message they get from the monitoring tool requires immediate response, but the only way to get them into that frame of mine is to be really aggressive about eliminating your alert noise.  If you send a user 50-100+ emails a day they aren't reading most of them, if you only send a few then they will focus on those few better.

If the notification model is something like "warnings we send an email" and "criticals we send an email and make a phone call" then you can just set up your alert logic where there is an alert that triggers at anything above warning with the email notification tied to that, and the critical is a separate alert that just tells whoever to make the call.