I think I am missing a trick on alerting because I suck at it.

Question

On the final stages of rebuilding solarwinds, but I am a little annoyed.  I will explain what we have currently and what I did...Though I don't fully understand why we are receiving random phantom alerts when I have not set them up.  (I turned on three alerts - one for CPU, one for RAM and one for HDD.  Really simple stuff.  I set out a small template that would be for CPU, RAM and HDD.  But it came back every 30 ish minutes with a random email that I KNOW I did not setup.)  I have to also say, this is the one thing that I actually hate more than anything else: is our alerting.  And for the last few weeks it's driven me insane to no end.

So, the current setup is based as thus...

Lots of compartmental alerts (one for OOH, one for ESXL, one for x, y, z.)  It equates to about 100 + alerts.

I wanted to change this and have a few really large alerts that would cover all of this.

So I set out the following:

- One for high CPU (95%) and then I drilled down and did conditional statements for SQL, ESXL, ect, ect.  It's a lot larger, but I don't want that many alerts as I think it's stupid and I like things cut down.

So, usually we receive about 50 ish with the current setup.  When I turned my alerting on, I got about 200 +.  Which I kinda understand (I think it does the conditional statements a linear fashion maybe.  That's my only theory on it) but IDK.  In my brain I think that somewhere I can make it so much better, but no matter what I try I still cannot reduce the amount of alerts and make it more efficient.  It's driving me insane actually.

shaun_9999 · Answer

See I thought it was how I was doing my logic.  But I really hate 137 alerts sitting there being a f*****g annoyance.  I don't even know where this hate has come from with alerting, but it's all I can think about.  I know I can make it better, but I am unsure how at the moment.

I might think about it the weekend.  If you are about the weekend, could you provide a little more personal assistance?  As in I provide you with screenshots, ect???  Or maybe sometime during next week>?

zackm · Answer

It sounds like your conditional logic is breaking your alerts, but without actually seeing the trigger conditions I wouldn't know that for sure.

Generally, we try to leverage custom properties and baselines as much as possible in our alerting schema. To give you perspective, I have worked with >500 clients in the past 3.5 years and can usually get away with <20 defined alerts that cover all necessary scenarios for them. Of course, there are outliers and a lot of alerting schema is based on opinion, but generally speaking, you should be able to capture all critical events with about 20 or so trigger conditions.

Without really digging into your specific scenario, I think the best advice I could give would be to first, get a report on how many times each alert has triggered in the past week so you know where your noise is, and second try to simplify your trigger conditions as much as possible so you don't have to worry about boolean logic biting you.

-ZackM

Loop1 Systems: SolarWinds Training and Professional Services

* LinkedIN: Loop1 Systems
* Facebook: Loop1 Systems
* Loop1 Twitter: @Loop1Systems
* ZackM Twitter: @ZackM