Deeper questions on application baselines

We operate an environment that has a variety of teams that utilize SolarWinds, and are all responsible for the creation and implementation of their own application monitoring templates. One of our teams started using baseline thresholds for their alerting, and have gotten a lot of unwanted alerts. Additionally, they believe that the baselines are getting lost after SolarWinds maintenance.

Is there anything that would cause a baseline to be outright lost? I understand they can change, but I'm specifically asking about any condition that would result in a loss of the baseline.
What specifically causes an alert from a metric that uses baseline thresholds. My understanding is that if the statistical pool has an average (lets say 70), and a standard deviation of 5, then a value of 75+ would result in an alert. Is that a correct understanding of how this works, or is there other math at play?
It seems like there are some use cases where you don't really want to use baselines. I'm thinking things like CPU where you just might not care about utilization unless it gets above 85%+, etc. Is this an accurate assessement?
Likewise, are there some use cases where people have found baselines to be beneficial? I'm trying to suss out where they should (and should not) be used.

Find more posts tagged with

Accepted answers

mesverrum

1 ) nothing that I ever saw, the baselines are calculated as part of the db maintenance job so maybe if your maintenance log shows errors like it timing out that could be a thing. New app monitors don't get baselines until they have been in the system over a week also. Maybe if they removed a template and readded it.

2) the alert logic depends on how you set it up but the component will go into warning at avg +2 st dev and critical at +3 st dev. You may or may not actually have alerts set up to send notifications on components going critical or warning but they'll at least be yellow/red in the GUI.

3) yes, lots of scenarios where the automatic baselines are less than helpful. If you are collecting a statistic that is relatively calm and flat during the baseline capture period then you have no point of reference for what it looks like under stress. I have used the example of a cpu baseline on network gear. A router rarely changes cpu loads by more than a few percent so if I go with a baseline and my router went from 20 to 25 I could be getting a critical alert. Similarly if I have a file count monitor on a directory that processes files in batches a baseline is pointless because the average number of files in the directory has no relevance for me. Maybe if it stays over 0 for more than 24 hours that would be a concern that the batch process failed.

4) I almost never allow Orion to set its own threshold. I will look at the baseline calculations as a point of reference to give me some context but ultimately I always set the thresholds myself based on my own knowledge. If nobody on my team knows what the threshold should be then I either disable the monitor because nobody knows what it does, or I blank them out and we just collect the data for future reference with no need to alert on it until someone can justify a decision.

All comments

mesverrum

ouberlord

All great knowledge, thanks. I'd figured that baselines aren't wiped out all too often, but I really haven't used them much myself so I had to take that user report at face value. I'm also fighting with perhaps a misunderstanding on their part as to what baselines are really good for (thus some of my other questions). Really every bit of your answers to 3 and 4 are almost exactly what I had thought, but with me minimal experience with them I wanted to make sure.

This is where my understanding (or the relative lack thereof...) haunts me. Using the above scenario of an average of 70 and a standard deviation of 5, would "+2 st dev" mean 10 and "+3 st dev" mean 15? As such, the warning for that threshold would be 80 and the critical would be 85? Just making sure I understand this as well as possible.

mesverrum

You got it