Alert Clean Up Best Practice

mafaaz93

What is the best practice in doing an alert clean up?

marlief22

Hi there, 

Let me see if I can help with the most common forms of 'Messy Alerts' that I see on a regular basis:

1. Devices in multiple scopes
   We often see this with nodes and interfaces, and applications and components. By default SolarWinds has alerts for all 4 of these elements, but the status of interfaces affects the status of nodes, and the status of components affects the status of applications. So what you get is:
   
   An interface goes down, which causes an interface alert to fire, but the node status also goes into warning, which causes a node alert to fire. 
   
   or
   
   A component goes down, which causes a component alert to fire, but the application status also goes into warning, which causes an application alert to fire. 
   
2. Thresholds set incorrectly
   In this case, the alert is working fine, but thresholds are set to a point where a baseline reading from the device causes the alert to fire. Let's say that SQL reserves 95% of the memory for a server, and the memory threshold is 90%. In this case, the critical node alert would fire even though the SQL server is actually running fine. You can edit thresholds for individual nodes or even turn them off entirely. 
   
   
3. Alerts are set not to clear automatically
   For some alerts, the Reset Condition may not be set to the default which is "Reset this alert when the trigger condition is no longer true". In these cases, the alert may actually be waiting for a user to come through and clear it manually. You can do this from the active alerts page, but generally the best option is to set it to "Reset after X Minutes" if the default doesn't apply (Like for event log based alerts).
   
   
4. Alerts don't have delay before firing
   For all but your most critical devices, you generally don't want to have SolarWinds start alert based on a single failed poll. You can set alerts to wait a certain period of time before firing the alert, usually I set this to 2-4x the default interval for that metric (Status = 4-8 minutes, CPU/Memory = 20-40 minutes)

Going to a messy alerts system I usually do the following:

1. Go to the grouping drop-down and select "Alert Name" - Start with the alert with the most instances. 
2. Order by age, descending, as the oldest alerts usually give you a good understanding on what needs changing for other devices. 
3. Group by severity, informational and notice alerts should clear automatically. 

Looking forward to hearing from other users on the site however, see their processes. 

Kind regards, 

Marlie Fancourt | SolarWinds Pre-Sales Manager | Prosperon Networks