cancel
Showing results for 
Search instead for 
Did you mean: 
Create Post
Level 9

Advanced Alert Manager not firing alerts properly

Jump to solution

Greetings earthlings.

We have a large topology spanning several solar systems; when I say large I'm implying a few thousand elements, when I say solar systems I mean clusters of weak planets. Some of these planets are pathetic, and not worth getting alerts on, but using the Advanced Alert Manager we are not able to filter out these alerts. To make matters more complicated, the email address that the alerts are being sent from is not listed in the rule.

It feels as is there's a tremble in the force elsewhere, bypassing the strict rules we have in place.

My boss is growing weary of my incompetence, and I reach out to you with my robotic arms in hope that you can help solve this puzzle.

An example:

We have an alert setup to email us when a node's WAN link has exceeded 5k errors within an hour, node Hoth is a desolate node, and we don't want email alerts for this site. We have added to the trigger alert to not fire this email if the Node Name is equal to Hoth:

     Trigger Alert when any of the following apply

          Recv Errors - Today is greater than or equal to 5000

          Xmit Errors - Today is greater than or equal to 5000

     Trigger Alert when all of the following apply

          Vendor is equal to Cisco

          Node Name is not equal to Hoth

     Trigger actions

          Send email to darthdoodie@me.com

               from email account:

                    Name: Network Performance Monitor

                    Reply Address: solarwinds@deathstar.com

Having this setup, we are still getting emails from noreply@deathstar.com

We have tried multiple variations of this rule, nothing is working. The fact that the reply address is different than what I set it up to be is bothering me; I even edited the message sent to add "zz" at the end of the message, but it's not showing up in my email. There must be another force here interfering with my plans again.

Thanks to any assistance other than rebel scum.

0 Kudos
1 Solution

You could do -

SELECT *

FROM [dbo].[AlertDefinitions]

Where AlertDefID like 'AlertDefID_you're_looking_for'

or

SELECT *

FROM [dbo].[AlertDefinitions]

Where AlertName like 'AlertName_you're_looking_for'

View solution in original post

16 Replies

But, what I am thinking is it will let you verify that the alert you set up is the one sending the email or if there is something else sending it.

i'm having difficulty in finding this Alert Log database, could you specify where to find this? I'm thinking i'm looking in the wrong location.

0 Kudos

You can get to the AlertLog table via the Database Manger app on your Orion server(Start>All Programs>SolarWinds Orion>Advanced Features>Database Manager).  Once you open the app click on "Add default Server".  From there you can drill down to the AlertLog table, right click it, hit query table, then click execute query.  In there you'll see a log of all alert actions Orion has processed.  Once you find the one you're looking for take note of the AlertDefId field.  Copy that field to notepad, then query the AlertDefenitions table.  From there you can query for that AlertDefId and get the name of the alert that the action was executed for.

Kris

Sorry, but SQL just isn't my thing. could you give an example of the correct syntax to issue the query? I have the AlertDefID ready

0 Kudos

SELECT AlertDefID, LogDateTime, Message

FROM [dbo].[AlertLog]

Where Message like '%Email%'

Make sure that the Success Email Sent Message is associated with the correct AlertDefID or if a different Id sent it.

0 Kudos

this was a great method to pull all the logs of emails sent from notifications, and i do see the email sent from the alert in question.

is there a way to query for the name of the specific custom alert based off of the AlertDefID? I feel like this is the final step towards my domination of Advanced Alert Manager.

0 Kudos

You could do -

SELECT *

FROM [dbo].[AlertDefinitions]

Where AlertDefID like 'AlertDefID_you're_looking_for'

or

SELECT *

FROM [dbo].[AlertDefinitions]

Where AlertName like 'AlertName_you're_looking_for'

View solution in original post

Sweet Baby Palpatine!

The alert was for a downed interface, but getting emailed as an interface with 5k errors!

You two have done well for yourselves, I am pleased with your efforts.

The message will include the from address.

To help you track alerts in the Future this is what I would recommend when building alerts.

I always have my customers go through and Give ID numbers to the Alert at the beginning of the Alert name.  IE.  ID:10 - Node Down

Then in the email action or whatever action you choose at the end of the message and variables you create I have them place this ID in the message.  Kind of like this rough example I pasted below.. (Note that the variables may be wrong since I was just going off the top of my head)

Subject: Node Down

Body:  The ${NodeName} with an IP of ${IPAddress} is ${Status} at ${TimeDate}.  ID: 10

This way when you have someone ask you or question why or how an alert was triggered to them you can simple correlate the ID from the alert message with the ID of the Alert in the Advanced Alert manager.  This will help with tracking alerts and what alerts are also being triggered the most.  This is really handy when you come into an environment that has a bunch of alerts.

I have even had them go a step further before and give each Department/Group in their environment a specific number range to use for ID's to make tracking even quicker, and help you understand with out diving in to the alert to see the actions of where it should be going.

IE..

Network Team ID Range 0-39

System Team ID Range 40-79

Database Team ID Range 80-119

Application Team ID Range 120-159

Hopefully and have given you what you were looking for which I'm sure they did, but just wanted to add this in for you, so you can maybe come up with a system that helps you track these quicker and easier with out having to dive into using SQL to find alerts in the log.

Good luck!

there is great potential with you kmaxwell, you would make a formidable opponent.

Can you check the Alert Log database, query the alert time and look at the message to verify what is sending the erronious alert?

I will do this, but just for clarification the alert is legit, we do have errors on the link but do not wish to receive the email.

0 Kudos
Level 10

These are not the alerts you're looking for..    It sounds like there is another alert enabled for errors.  You'll need to audit your enabled alerts and track down the offending one.

indeed, this is the logical deduction.

however, scanning the configured alerts in the Advanced Alert Manager there is nothing listed matching this setup. There's another rule to send this alert to the syslog, so I've checked the Syslog Server settings to see if it's listed, but alas...nope.jpg

We have three additional pollers, and they're Syslog rules have also been checked.

Is there somewhere else I should look (other than Advanced Alert Manager and Syslog Server Alerts)?

0 Kudos

In this case it would have to be coming from the Advanced Alert Manager, unless you had some sort of trap or syslog forwarding going on for that type of incident that also had a trigger action to send an email.