Community
- Command Central
- MVP Program
- Monthly Mission
- Blogs
- Groups
- Events
- Media Vault
Products
- Observability
- Network Management
- Application Management
- IT Security
- IT Service Management
- System Management
- Database Management
Content Exchange
- SolarWinds Platform
- Server & Application Monitor
- Database Performance Analyzer
- Server Configuration Monitor
- Network Performance Monitor
- Network Configuration Manager
- SQL Sentry
- Web Help Desk
Free Tools & Trials
Store

Best way to Condense Alerts to Avoid Growing Pains?

So, I'm pretty new to Solarwinds and so far I am trying to take small steps and get everything setup right before I start adding a large amount of server. I started with about a group of 15 servers and added them into groups depending on who the server admin was, and setup group alerts to send an email when a node in their group would go off.

I have now started adding in HTTP / HTTPS / Port monitors and I have noticed that the number of advanced alerts I have are starting to add up very fast. For each different HTTP and port application monitor for each server is its own individual alert. So with the way it is now, if I want to change the way I report these alerts, I have to go and individually change each alert. With only a test bed of 15 servers, its not that big of a deal, but if I continue like this, It seems like it can get out of hand very fast.

What I am trying to figure out is if there is a better way to set this up and condense the amount of alerts I have. A few options I have thought about doing are as follows:

Option 1 - Add all HTTP application alerts into one advanced alert so all of them are condensed onto one alert. This was my first go around, but realized I could not send out emails to particular people depending on what HTTP app went off. Since people don't want to get alerts about servers that aren't theirs, this solution was scrapped.

Option 2 - Condense the HTTP alerts based on server admin. Pretty much option 1, but not as drastic. I would condense all HTTP alerts that would effect Admin X and put them all in one single alert. Do the same thing with Admin Z HTTP alerts, Etc.

Option 3 - Add the HTTP and Port monitors applications into groups and monitor the group. This way I could add the HTTP application monitors into groups by admin and if one had an issue it would set off that group and it would send out an alert. This seems like it would work, but will this limit me on the types of variables I can use in the email since I will need to use the Group:member property rather than the APM: Application property?

I can already see the way I am currently doing it becoming very difficult to keep up with once we start adding a large amount of servers. I am just trying to get a head start on this to try to avoid as much backtracking in the future as possible, and get it setup right the first time.

Is there a better way to accomplish this?

Thanks,

Jeryn

Find more posts tagged with

Accepted answers

rjnicholson

Currently you can't add a custom property to an Application only Nodes, Interfaces, Volumes. So you are limited when it comes to Application Monitors on this. I should be more clear. You can if you wish have 1 alert for Nodes, and use CP values to send alerts out. Application monitors you either chain the alerts and have 1, but doing this if you have different groups that monitor different apps you will bomb everyone anytime 1 app goes down and this can be bad and overwhelming to the app group, or you build mulitple ones for now..

This is all on NPM 10.3.1 and SAM 5.2 Not sure if NPM 10.4 allows you to start putting CP values on Applcations yet, but I don't think you can.

All comments

aLTeReGo

This is really a personal choice, and I'm sure others will chime in offer up suggestions for what works best for them. I for one like to manage and maintain the fewest possible alerts. One trick is to chain alert triggers by chaining conditions together. This works well if the same action is required. Another option to simplifying alerts is to use groups and custom properties wherever possible. These will save you tons of future time, but take some to get setup initially. One of my favorite tricks is to use custom property macros for email addresses. This allows you to have a single alert trigger action, but based on who's responsible for managing/maintaining the node the alert will dynamically alert the appropriate person using the nodes custom property.

I'm sure there are others out there that have a system that works best for them, so feel free to chime in!

jeryn

I'm defiantly going to look into that custom property thing. It sounds like figuring out how to use that would enable me to use option 1 in my beginning post, which would be amazing. I will have to look into chaining conditions together, as I am still trying to figure out how the whole flow of the trigger conditions work. Is there a video or document out there that breaks it down and explains it? I have looked through the manual, but what I seen, it didn't seem to go to in depth about it.

Also, thanks for the help. That custom attribute sounds like it could be amazing if I can figure it out.

W1ngnu7

Can you give an example of what you mean by chaining alert triggers and conditions?

rjnicholson

You can also use a custom property to send your alerts and still use one alert.

Add E-Mail addresses to a CP Field and then in your alert you can use a Variable to call that field and populate it with the correct E-Mail address. This way you have 1 alert and when the Node/App Monitor drops it grabs the E-Mail for the person/group you have listed and shoots out the alert. You can now just update the CP fields when needed to direct E-Mails accordingly.

rjnicholson

Here ya go..

Type of Property to Monitor: APM: Application <---- Change this drop down to find other properties you want to monitor. Careful though as you can't mix some properties in the same Trigger condition as it can break the logic.

Trigger Alert when any of the following apply <---- The ANY statement is your key. ANY of the following below can be true to trigger. If you set to ALL then every condition below the alert would need to exists for this alert to trigger.

Trigger Alert when allof the following apply

Application Monitor is equal to DOTCOM HTTP

Component Status is not equal to UP

Trigger Alert when all of the following apply

Application Monitor is equal to DOTCOM HTTPS

Component Status is not equal to UP

Trigger Alert when all of the following apply

Application Monitor is equal to eStock HTTP

Component Status is not equal to UP

This can be an example of 1 alert, but it chains multiple conditions and could be watching 3 different Application Monitors. Just make sure you stay within the Logic of the Alerting Engine.

jeryn

Awesome, just started adding and populating those fields this morning and will test it out here soon. You mention I could have 1 alert for all Nodes and Apps, how would I get this to work on an application? It seems I can only add custom properties to nodes, how would it pull a node property and I setup a alert as an application alert?

Wouldn't I need a node alert that encompass all nodes and a separate application alert to handle all the applications? If that is the case, can the application alert pull a ${Node.Contact} custom property?

W1ngnu7

Thanks for the example.

Maybe I'm misunderstanding the question a bit but this is what I do:

I created an alert for whenever ANY component goes down, I get an alert with all of the variables detailing which server and which component was affected. The trigger conditions I use are just to segregate the alert for different distribution groups. If City equals Atlanta, send email to the Atlanta datacenter group, etc...

An example alert text might be:

Unexpected error occurred. The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.

Name of Alert - Alert me when a component goes down in (CITY)

Component - https://sharepoint
From Application Template - :HTTPS Critical URLs
on Node - DC3337
is - Down

For more information click the link below.
http://solarwinds01:80/Orion/View.aspx?NetObject=AM:9256

I know this is not as specific as what the OP outlined but in the spirit of keeping things simple with fewer moving parts as you grow, this is how I would approach the problem.

jeryn

My emails keep giving me an error when I try to do this. I setup a custom Node property, with the property type as text and I named is PrimaryContact. I have now tried to put ${Node.PrimaryContact}, ${PrimaryContact}, ${Nodes.PrimaryContact}, and ${Group.PrimaryContact}.. I keep getting an error saying there was no valid email.

Am I doing something wrong or is it having troubles because the Alert property is a Group Member? For this to work, does the alert property have to be Node?

UPDATE:

Just created a new alert as property Node and using the ${Node.PrimaryContact} seemed to work. So it looks like using the custom property only works when you have the alerting property set as node.. Will see if there is a way around this as I would love to get it to working for alerting for groups and applications. If anybody knows a way around this, let me know!

Thanks all for the help!

gregatkins

It would take some tweaking to get them to work, but from what I've gathered using alerts with nested conditions will help keep things at an acceptable level. The fun part is figuring out the complex and simple conditions and getting them right