Some Trigger/Reset Actions Not Being Performed On Alerts

Hi,

Hoping someone might point me in the right direction, but I had an issue occur where an Alert was triggered, and the Trigger action worked fine, but the Reset action did not happen after receiving other indication the alert had been reset. The action is to send an Email which then gets ingested by another script I have. On the Reset Actions, I have 3 different Email Actions listed. So it should be sending 3 emails when the incident is resolved, right? When I tested the alert before, it worked fine on both Trigger and Reset.

However, on rare occasion, only a single step will be missing.

For example, I received the Trigger emails, all of them. But then when the alert was reset, only 2 of 3 emails went out. Why would that last trigger action not have been kicked off properly? This is vital for our log keeping and to avoid the mundane process of manually documenting this information, so I'm hoping this is a fault on the Solarwinds side that can be corrected.

Find more posts tagged with

trigger

email alerts

Alerts

reset

Accepted answers

bobmarley

Try looking into the events table and then the alerts table and see if you are seeing what you expect.

Events:

SELECT TOP 2000  
TOLOCAL(EventTime) AS LocalEventTime, Message, EventType, NetworkNode, NetObjectID
FROM Orion.Events 
Order by EventTime DESC

Alerts:

SELECT TOP 500
       TOLOCAL ([AH].TimeStamp) as Timestamp
     , [AH].AlertObjects.AlertConfigurations.Name AS [Alert Name]
     , [AH].AlertObjects.EntityCaption AS [Element]
     , [A].ActionTypeID AS [Action]
     , [A].Description AS [Details]
     , [A].Enabled AS [AlertEnabled]
     , [AH].Message AS [Results]
FROM Orion.AlertHistory AS [AH]
INNER JOIN Orion.Actions AS [A]
  ON [AH].ActionID = [A].ActionID
WHERE [AH].TimeStamp >= GETUTCDATE() - 30  --And [AH].AlertObjects.EntityCaption LIKE '%YOURNODENAME%'
ORDER BY [AH].TimeStamp DESC

All comments

bobmarley

You may try the query below to see if you have any alert actions that are failing. Sometimes you may find an incorrect address or the email server rejecting the email.

SELECT 
         [History].AlertObjects.AlertConfigurations.Name
       , [History].AlertObjects.AlertConfigurations.DisplayName
       , TOLOCAL ([History].TimeStamp) AS TimeStamp
--     , [History].AlertObjects.AlertConfigurations.AlertMessage
--     , [History].AlertObjects.EntityCaption
--     , [History].AlertObjects.EntityDetailsUrl
--     , [History].EventType
--     , [Actions].Description
       , [History].Message

FROM Orion.AlertHistory AS [History]
LEFT JOIN Orion.Actions AS [Actions]
     ON [History].ActionID = [Actions].ActionID
Where Message like '%ErrorMessage%' AND daydiff([History].TimeStamp ,getdate()) < 14 
ORDER BY [History].TimeStamp DESC

ConnorS68

Thanks @bobmarley , that query is incredibly helpful! I was able to see some other alerts that were having issues executing, but strangely enough, the alert I KNOW for a fact that I missed does not show up in this error log. Very strange! I appreciate your input.

And the problem seems to be reoccuring more consistently now. As I mentioned, this data gets ingested by another script. So I have received 3 more alerts since that didn't receive their corresponding "reset" action email.

I just find it super strange that I receive the email for the 1st and 2nd actions "Email [redacted] - Circuit Back Online", "Slack [redacted] channel" no problem, but the 3rd action, "Email [redacted] - Automation Reset" is having problems. Even stranger, is that when I initially set this up it worked for a few alerts, and then stopped with no changes at all to the alert that I'm aware of after that initial working period.

mesverrum

How big is your environment? How many alerts actions trigger per day?

I've definitely seen the alert engine get overwhelmed in big environments or when there is a very high number of alert actions to take. Common symptoms I've seen would be for the alert actions to not fire off at all, or if they use variables the variables will fail to render properly. So instead of the alert going to whoever is the ${{DefaultEmailRecipient}} they actually try to send an email to "${{DefaultEmailRecipient}}" and it gets errored out as an invalid email address format.

In my last big environment I would keep track of those issues and it was about one in 10,000 actions for me, but if I got an alert storm it would happen to a bunch of them all at once.

ConnorS68

Seems kind of counter-intuitive that the system would do that, considering generating alerts is about 95% of the purpose of an NPM. I definitely had a similar suspicion when setting up 3/4 separate email actions for the same Trigger event. Our environment is mid-large with a little over 20,000 total elements on the primary polling engine.

bobmarley

Try looking into the events table and then the alerts table and see if you are seeing what you expect.

Events:

SELECT TOP 2000  
TOLOCAL(EventTime) AS LocalEventTime, Message, EventType, NetworkNode, NetObjectID
FROM Orion.Events 
Order by EventTime DESC

Alerts:

SELECT TOP 500
       TOLOCAL ([AH].TimeStamp) as Timestamp
     , [AH].AlertObjects.AlertConfigurations.Name AS [Alert Name]
     , [AH].AlertObjects.EntityCaption AS [Element]
     , [A].ActionTypeID AS [Action]
     , [A].Description AS [Details]
     , [A].Enabled AS [AlertEnabled]
     , [AH].Message AS [Results]
FROM Orion.AlertHistory AS [AH]
INNER JOIN Orion.Actions AS [A]
  ON [AH].ActionID = [A].ActionID
WHERE [AH].TimeStamp >= GETUTCDATE() - 30  --And [AH].AlertObjects.EntityCaption LIKE '%YOURNODENAME%'
ORDER BY [AH].TimeStamp DESC