Advanced Alert Manager not firing alerts properly

Greetings earthlings.

We have a large topology spanning several solar systems; when I say large I'm implying a few thousand elements, when I say solar systems I mean clusters of weak planets. Some of these planets are pathetic, and not worth getting alerts on, but using the Advanced Alert Manager we are not able to filter out these alerts. To make matters more complicated, the email address that the alerts are being sent from is not listed in the rule.

It feels as is there's a tremble in the force elsewhere, bypassing the strict rules we have in place.

My boss is growing weary of my incompetence, and I reach out to you with my robotic arms in hope that you can help solve this puzzle.

An example:

We have an alert setup to email us when a node's WAN link has exceeded 5k errors within an hour, node Hoth is a desolate node, and we don't want email alerts for this site. We have added to the trigger alert to not fire this email if the Node Name is equal to Hoth:

Trigger Alert when any of the following apply

Recv Errors - Today is greater than or equal to 5000

Xmit Errors - Today is greater than or equal to 5000

Trigger Alert when all of the following apply

Vendor is equal to Cisco

Node Name is not equal to Hoth

Trigger actions

Send email to darthdoodie@me.com

from email account:

Name: Network Performance Monitor

Reply Address: solarwinds@deathstar.com

Having this setup, we are still getting emails from noreply@deathstar.com

We have tried multiple variations of this rule, nothing is working. The fact that the reply address is different than what I set it up to be is bothering me; I even edited the message sent to add "zz" at the end of the message, but it's not showing up in my email. There must be another force here interfering with my plans again.

Thanks to any assistance other than rebel scum.

Find more posts tagged with

advanced_alerts

custom_properties

advanced_alerts_configuration_and_edit

npm

npm_10

Alerts

Accepted answers

kmaxwell

You could do -

SELECT *

FROM [dbo].[AlertDefinitions]

Where AlertDefID like 'AlertDefID_you're_looking_for'

SELECT *

FROM [dbo].[AlertDefinitions]

Where AlertName like 'AlertName_you're_looking_for'

All comments

kmaxwell

These are not the alerts you're looking for.. It sounds like there is another alert enabled for errors. You'll need to audit your enabled alerts and track down the offending one.

anakin

indeed, this is the logical deduction.

however, scanning the configured alerts in the Advanced Alert Manager there is nothing listed matching this setup. There's another rule to send this alert to the syslog, so I've checked the Syslog Server settings to see if it's listed, but alas...nope.jpg

We have three additional pollers, and they're Syslog rules have also been checked.

Is there somewhere else I should look (other than Advanced Alert Manager and Syslog Server Alerts)?

kmaxwell

In this case it would have to be coming from the Advanced Alert Manager, unless you had some sort of trap or syslog forwarding going on for that type of incident that also had a trigger action to send an email.

bsciencefiction.tv

Can you check the Alert Log database, query the alert time and look at the message to verify what is sending the erronious alert?

anakin

I will do this, but just for clarification the alert is legit, we do have errors on the link but do not wish to receive the email.

bsciencefiction.tv

But, what I am thinking is it will let you verify that the alert you set up is the one sending the email or if there is something else sending it.

anakin

i'm having difficulty in finding this Alert Log database, could you specify where to find this? I'm thinking i'm looking in the wrong location.

kmaxwell

You can get to the AlertLog table via the Database Manger app on your Orion server(Start>All Programs>SolarWinds Orion>Advanced Features>Database Manager). Once you open the app click on "Add default Server". From there you can drill down to the AlertLog table, right click it, hit query table, then click execute query. In there you'll see a log of all alert actions Orion has processed. Once you find the one you're looking for take note of the AlertDefId field. Copy that field to notepad, then query the AlertDefenitions table. From there you can query for that AlertDefId and get the name of the alert that the action was executed for.

Kris

anakin

there is great potential with you kmaxwell, you would make a formidable opponent.

anakin

Sorry, but SQL just isn't my thing. could you give an example of the correct syntax to issue the query? I have the AlertDefID ready