Can Solarwinds support Alerts based on trend violation?

Can NPM be configured to alert based on trend violations? This is similar to Aberrant behavior detection in RRDTool based on Holt-Winters' algorithm. It need not be based on complex curve fitting, even if the prediction is a simple linear fit it is much more useful for monitoring. In practice, having the same threshold for every device is not so useful. Lower values trigger a lot of false alarms, higher value tend to delay the fault identification.

Find more posts tagged with

Alert

trend_analysis

Accepted answers

All comments

thamizh85

Bump!

I am surprised no one is interested in this. I tried to search but couldn't find a similar question. Based on my experience in operations so far, I am convinced Trend based alerts makes more sense than an universal threshold based alerts.

byrona

I think this feature would be awesome. This type of thing has been requested many times before in the form of alerts based on a percent change or based on a tolerance level.

fcaron

We are very interested in this.

The way I view it is this:

- Completely agree with the fact that having the same threshold across many devices/elements is not good (unless we are talking SLA, but it's a different topic)

- We have a lot of thoughts about this and we call it automatic baselining. In other words detecting when a resource starts having a behavior (perf KPI) different from it's normal behavior (it's normal baseline). As you said "aaberrant" behavior

- The beauty of this it that it automatically adjusts to each resource's individual behavior and usage pattern

- When it get's funny, is that you'll see that some resources and metrics tend to have a bursty behavior and some not. You can manage that by setting them (if they send you too many alarms or not enough) some kind of sensitivity (e.g. XStandard Deviations above/below mean value, the higher X the more tolerant you are to burst and the less you'll send alarm when the behavior is outside the baseline)

- To be helpful, this needs to be aware of seasons, or at least hours of the day and days of the week, potentially days of the year.
- I view "Trend Violation" (title) as something different.

- For me Trend Violation is a planning tool: tell me when/if, in the next 90 days, any metric on any resource will reach unacceptable level. For these resources, I want to reconfigure, over-provision, increase capacity...

- Baseline violation is a real time troubleshooting tool: tell me right away, when a metric/resource has an unusual behavior (i.e. is outside of its baseline). For those, I want to look at why right now and understand what happens (nightly backup kicks-off during the day, intrusion, ...)

Anyway, we are interested and would like to see more of your reactions here.

netlogix

I have been working on something and it is working for me (somewhat). I have a SQL agent job that runs on the hour to update a custom property that has an expected average load based on the last 12 of this day (say Friday), and this hour, say 09:00, but I then add 08:00 and 10:00 to even it out a bit. I end up with 96 hour averages (24 for 09:00, 12 for 08:00 and 12 for 10:00) that I average to come up with this hours expected load. Then I have an alert that is based on a "fun" equation I came up with based on the graph of square root (parabola on it's side) and messed with it's X = Y intercept (to get X = Y at 100) and adjusted it's degree of arc, and chopped the first area out so it won't ever alert if the average is below 10%. The alert is set to require the average of the last half hour load to exceed the expected hour load and checks every 2.5 minutes. (if the load is ever below, it'll restarts the hour clock)

It works pretty well, but it isn't very nice with the load it puts on SQL (which is why I had the SQL agent update the stat every hour instead of calculating it every poll, that helped a bit).

Here's the graph:

--------------------------------------------------------------------------------------------------

SQL to create the expected Average load

--------------------------------------------------------------------------------------------------

Update Nodes

Set Nodes.ThisHourAvgCPU = Loads.ThisHourAvgCPU

From (

Select NodeID, Avg(AvgLoad) as ThisHourAvgCPU from (

select NodeID, MaxLoad, MinLoad, AvgLoad, AvgPercentMemoryUsed, 'onetime' as [Weight]

from CPULoad_Hourly

where [DateTime] between GETDATE() - 92 and datediff(dd, 0, getdate())

and DATEPART(WeekDay,[DateTime]) = DATEPART(WeekDay,GETDATE())

and DATEPART(HOUR,[DateTime]) between DATEPART(HOUR,GETDATE())-1 and DATEPART(HOUR,GETDATE())+1

union

select NodeID, Max(MaxLoad) as MaxLoad, Min(MinLoad) as MinLoad, Avg(AvgLoad) as AvgLoad, Avg(AvgPercentMemoryUsed) as AvgPercentMemoryUsed, 'onetime' as [Weight]

from CPULoad_Detail

where [DateTime] between GETDATE() - 92 and datediff(dd, 0, getdate())

and DATEPART(WeekDay,[DateTime]) = DATEPART(WeekDay,GETDATE())

and DATEPART(HOUR,[DateTime]) between DATEPART(HOUR,GETDATE())-1 and DATEPART(HOUR,GETDATE())+1

group by NodeID, DATEPART(HOUR,[DateTime]), DATEPART(WeekDay,[DateTime]), DATEPART(WEEK,[DateTime])

union

select NodeID, MaxLoad, MinLoad, AvgLoad, AvgPercentMemoryUsed, '2ndtime' as [Weight]

from CPULoad_Hourly

where [DateTime] between GETDATE() - 92 and datediff(dd, 0, getdate())

and DATEPART(WeekDay,[DateTime]) = DATEPART(WeekDay,GETDATE())

and DATEPART(HOUR,[DateTime]) = DATEPART(HOUR,GETDATE())

union

select NodeID, Max(MaxLoad) as MaxLoad, Min(MinLoad) as MinLoad, Avg(AvgLoad) as AvgLoad, Avg(AvgPercentMemoryUsed) as AvgPercentMemoryUsed, '2ndtime' as [Weight]

from CPULoad_Detail

where [DateTime] between GETDATE() - 92 and datediff(dd, 0, getdate())

and DATEPART(WeekDay,[DateTime]) = DATEPART(WeekDay,GETDATE())

and DATEPART(HOUR,[DateTime]) = DATEPART(HOUR,GETDATE())

group by NodeID, DATEPART(HOUR,[DateTime]), DATEPART(WeekDay,[DateTime]), DATEPART(WEEK,[DateTime])

) as Avgs Group By NodeID

) as Loads inner Join Nodes on nodes.NodeID = Loads.NodeID

Where Nodes.ThisHourAvgCPU <> Loads.ThisHourAvgCPU

--------------------------------------------------------------------------------------------------

Alert is a Custom Node alert with this: (The 19.97 and .606 is how I adjusted the arc and intercept)

--------------------------------------------------------------------------------------------------

Inner JOIN CPULoad_Detail on Nodes.NodeID = CPULoad_Detail.NodeID

Where CPULoad_Detail.DateTime > dateadd(MINUTE, -30, getdate())

group by Nodes.NodeID, Nodes.Caption, ThisHourAvgCPU

having avg(CPULoad_Detail.AvgLoad) > Power(19.97*ThisHourAvgCPU,.606) and avg(CPULoad_Detail.AvgLoad) > 10 and COUNT(*) > 6

--------------------------------------------------------------------------------------------------

thamizh85

Yeah, I agree we both are talking about the same thing but the nomenclature is different. In your terms, the word trend is about forecasting whereas I look at it as representing the past behavior. Anyway it is good that you guys are having serious thoughts about this. It would not hurt to refer to some ideas implemented in the Open Source domain, see this for example:

Notes on RRDTOOL implementation of Aberrant Behavior Detection

byrona

I personally think that a good place to start with this would be the ability to alert based on rate of change. While not exactly the same thing, it certainly helps detect a deviation from "normal". I imaging configuring the percent change and the number of polling cycles over which to look for the change.

I am curious to hear what others think?