How do you handle your application alerting?

Question

My org is working on restructuring our alerting; including going through every application in our environment and ensuring we have things being monitored and alerted properly.

We want to have 4 "generic" alerts - two for "testing" and two for actual alerts.  The 'test' ones would just email our SAM team when it fires; and the actual ones will open incidents in our ticketing system.

What I'm finding is that we can't find an effective and simple way to flip an application between those test/prod alerts.  For our "generic" prod alert config, we have it set so "Application Name begins with Contoso-Ticket" (name clearly changed, lol).  The generic test alert config is "Application name begins with "TEST-Contoso-Ticket".  What we have found is that if we create an App Template with the "Test" name, and then assign the monitors; the names of the assigned monitors don't change if we rename the template; and the alerting doesn't get moved over to the Prod one.   We have to delete the assigned monitors and then reassign them after renaming the template.

The alert config shows a Description field on the application that can be used - but looking at an application, it has no usable description field.  The template has a description but that doesn't appear to show up on the assigned application.  Additionally, the template has Tags - but those aren't available on the application either.  Lastly, you can create an application custom property - but it doesn't appear that this can be set from the template.

What other options might I have to do what I'm trying to do here (create template, have associated alert that emails the SAM team; tweak alerting threshholds; then shift that template's assigned monitors to our Prod alert that sends incidents)?

I'm open to changing our methods as well; this is the only org I've worked with that used SAM, so there's good chance of being other methods to accomplish this that I'm not aware of.

chucksimalchik · Answer

Yeah. I think application custom properties will be the way to go. We've got a glut of node custom properties as well, but I've found adding custom properties on other objects (Volumes/Interfaces/Applications) has given us a whole lot more flexibility.

djceola · Answer

We use a lot of custom properties on our nodes; but none yet for our applications.  We have custom scripting that integrates our application CI database with SAM and populates custom properties on the node (all nodes are auto-added from multiple sources, all we have to manually do is set the application CI name) - including the team that alerting gets routed to; so we're in a good place with regards to having one (or few) Alert configs to handle it all; we just want an easy way to set up a "test" alert that doesn't send incidents while we tweak threshholds.

chucksimalchik · Answer

I second the "Custom Property" route.

What I did is a little different use case than what you describe, but I set up custom props that indicate whether we want to send email/open incident/notify ops and other props to indicate email address(es)/ticketing groups/etc. Then I built alerts for each of those actions if the individual flag is set to Yes on a given application.

Administratively, for our enterprise-wide application monitors I have an alert that runs daily that checks if they don't have the default settings and if so, to set them. That basically happens when we add a new node and the monitors get auto-deployed without the custom props being set. Any other monitors we manage (if necessary) using the custom property editor or directly in the application monitor itself.

It took some time and design work to set up, but overall this approach makes life much easier than having to build alerts for different monitoring templates. Between the Cust Prop Editor and SWQL queries, you can easily find out who gets notified for what, rather than having to dig into alerts.