Alert logging Trigger Action multiple times for Triggered condition

stueyt12

Hi,

We have a very simple alert set up as follows:

Trigger Cond:

Machine Type is MachineType1

Node Status is Down

Condition must exist for 10 minutes to trigger

Reset Condition:

When Trigger Condition is no longer true

Trigger Action:

Log an entry to NPM log

Log an entry to Windows Event Log

Reset Action

Log an entry to NPM log

Issue:

What we're seeing is that when the trigger condition is met, multiple duplicate entries are written to both the NPM log and the Windows Application log (EventID 3003). In one of our environments it's 5 duplicate entries each time, in the other environment it's 3 entries. Then, when the node comes back up, the reset action logs the exact same amount of entries to the NPM log- 5 in the first environment, and 3 in the other. This happens with all MachineType1 nodes, not any one in particular. When we test with the Advanced Alert Manager, only 1 entry is written each time, which is what we want to happen!

The alert is not triggering and then resetting instantly by the way. For example:

7/12/2017 5:01 AM Event Type RESOLVED: CRITICAL P4 Machine1 is now back up 7/12/2017 5:01 AM

7/12/2017 5:01 AM Event Type RESOLVED: CRITICAL P4 Machine1 is now back up 7/12/2017 5:01 AM

7/12/2017 5:01 AM Event Type RESOLVED: CRITICAL P4 Machine1 is now back up 7/12/2017 5:01 AM

7/12/2017 4:57 AM Event Type Machine1 is Up

7/12/2017 4:57 AM Event Type Machine1 is responding again. Response time is 32 milliseconds.

7/12/2017 4:51 AM Event Type CRITICAL P4 Machine1 status is down 7/12/2017 4:51 AM

7/12/2017 4:51 AM Event Type CRITICAL P4 Machine1 status is down 7/12/2017 4:51 AM

7/12/2017 4:51 AM Event Type CRITICAL P4 Machine1 status is down 7/12/2017 4:51 AM

7/12/2017 4:38 AM Event Type Machine1 is Down

This really has me stumped. Anybody got any ideas what the issue might be?

ekis

Check and see if the node in question is within the scope of several "node object - related SW alerts".

Alternatively, another approach is to go to the TRIGGER ACTIONS tab of the SW Alert in question, and then take note of or take a screenshot of the name/title you gave to the trigger action in question. Then navigate to Settings > All Settings > Manage alerts > Action Manager.

From Action Manager, search for the name/title of the trigger action in question. See if it assigned to several other alerts.

Or, edit the trigger action in question by simply changing the "Name of Action" field or adding a word or two in it.

If that specific trigger action is currently in use by other SW alerts as well, it will give you a pop-up first telling you that the change you made will affect those other SW alerts as well and will ask you if you want to continue.

Once you've confirmed that there may be several other SW alerts that includes the node in question within their respective alerting scopes, maybe you can start by editing them to have some other alert action or perhaps you may want to disable those duplicates.

Just my 2 cents. Hope this helps.