Device went down but did not send an alert

FormerMember

We took a device down for maintenance and it never sent an alert. I checked the history and there were no events for when it was down. the only events were for 50% packet loss. In the packet loss history though, it shows packet loss at 100% for 12 hours. Is there a reason why this would show as 100% packet loss but never get flagged as down?

tigger2

This is *not* supported in any way so perform at your own risk, but I use these queries to help track down alerting failures.  I'm running NPM 11.5.3.

1. Get the "node ID" of the host that should have alerted.  For example, I an using: '1234'. To do this, go to the node and look at the URL.  In the URL it will have the nodeID after this part:  ?NetObject=N:1234

2. Log directly into the Orion DB (with the tool of your choice, or have a DBA do it).  You need the name of the Orion DB, which for example, I will use: 'SolarWindsOrion'

3. Execute this query:

SELECT

  [MsgID]

  ,[LogDateTime]

  ,[AlertDefID]

  ,[ObjectType]

  ,[ObjectID]

  ,[ObjectName]

  ,[ActionType]

  ,[Message]

FROM [SolarWindsOrion].[dbo].[AlertLog]

where

[ObjectType] = 'node'

AND [ObjectID] = 1234

The returned rows will be all the alerts/activity for alerts sent for the node.  The word "Failed" should be at the beginning in the message field of any failed email attempts (with the reason it failed), and "Success" with successful attempts.

Notes/thoughts:

- From my experience and the forums, I believe Orion will try a single time to send an alert once it's triggered (and past the alerting rules logic). There is no retry. (Yoda might say: "Retry not. Do... or do not. There is no retry.")

- If you do not see any alerts for the node in query #1, then I believe Orion did not attempt to send them.  I'm not exactly sure where to look (there are a lot of logs in Orion)  to see if there was a failure in between the alert logging in the DB and triggering the alert.

- A long shot idea: I'm not sure if the "Solarwinds Alerting Service" and "Alerting Engine" will catch alerts during times if they are not running ( i.e. if they stop and alerts occur, will they will pick the old, unsent alerts up once they are restarted?).  It's a long shot, but maybe one or both services were down during the time the alert was triggered.  The server logs (and possibly some Orion logs somewhere) may have this info ( or absence of service restarts).

- Another long shot idea: I'm not sure of it, but I believe the MSMQ on the orion poller is what is used to transmit/load data into the Orion DB.  I could imagine a scenario where the queue was being filled up ( not loading data into the database) during your outage, then after the outage the queue was released and loaded the data.  This might? cause the alert trigger to not trigger as it never saw "the down data"

- Another long shot:  For some reason the node never registered as "down".  Orion has many states, (I'm making these up) like "unresponsive", or "unreachable", "unknown".  .Maybe it never officially went to the status "down", in Orion's mind?  I don't have a reason why it wodul do this, or know of a good way to check the past state of a node.  It should be in the events logged in the node, I would think.

On a side note:

If you just want to search for failed alerts for anything, you can do this:

SELECT

  [MsgID]

  ,[LogDateTime]

  ,[AlertDefID]

  ,[ObjectType]

  ,[ObjectID]

  ,[ObjectName]

  ,[ActionType]

  ,[Message]

FROM [SolarWindsOrion].[dbo].[AlertLog]

where  [Message] like '%Failed%'

I have the SAM module in Orion so I use this query as a 'SQL Server User Experience Monitor' to execute the query, count, and alert (multiple ways) if any alert notifications are failing (yes, there is some circular logic here), It works well enough and at least lights up red in the web console with red and I get a history and times of failed notifications.  I added 'AND [LogDateTime] >= DATEADD(minute, -10, GETDATE())'  so it only searches the time since it last checked (it polls every 10 minutes). It may miss some if the poller gets a tiny bit behind.

SELECT count([Message])

  FROM [SolarWindsOrion].[dbo].[AlertLog] with(nolock)

  where

  message like 'Failed%'

  AND [LogDateTime] >= DATEADD(minute, -10, GETDATE())