Event Alert Can't Be Found

Question

When I monitor a web page on the Internet and it goes down an event is created and we get an email. I can't find the alert linked to this event in alert manager so I can desensitize it. The node shows no link to an alert. Where can I find the alert that triggers this web page?

stuartd · Answer

I spend a fair bit of my time re-writing alert names, trigger action names, etc because the tools admin guy doesn't bother. Unfortunately it is me that ends up 99% of the time trying to track stuff down or fix bits and it is a nightmare locating the associated action with generic/default names.

Also, just how many times do you need "copy of copy of" in a name

Jfrazier · Answer

I wish I could find the thing I wrote years ago on here related to this.

I call this a breadcrumb.

All alerts need to have a breadcrumb in the alert definition that references what alert template created the alert.

So either list the alert definition name in the alert text or as part of the alert name in the template, use a unique identifier whether it be a unique number or text string. The resulting alert message needs to have that same string so you can go back and find the alert definition.  Alert names need to be unique in some obvious way or you'll spend a large amount of time searching for them.

Years ago we had a service catalog where each new alert definition had an assigned number that was referenced in the catalog where we tracked revisions to the alert.  So any alert where-ever it ended up had that string so we had a point of reference to find the source. We also listed the requesting ticket number as well as any modification request ticket number for future reference.