Simple alert still triggers when disabled

bunjiega

I created a simple alert to e-mail me when a node goes down. The alert works fine. However, when I go to 'Alerts & Activity' --> 'Alerts' --> 'Manage Alerts' and disable the rule, it STILL e-mails me when the alert is triggered.
I was able to navigate to the 'Action Manager' where I can see the action to that rule as enabled.

Guess I am confused to why when I disable the alert does it not disable the action associated with it? Is this on purpose or a bug? It does not make sense to disable the alert but not its related actions.
Anyone else seen this.

I have a script to toggle the alert off and on, which works fine, but I do not know how to programmatically (via the API) to toggle off or on the action. If anyone knows how to do that, it would be helpful as a workaround.

Thanks!

marlief22

Hi there, 

This is certainly an interesting issue you're having, so let me set out how SolarWinds alerting should work first, then look at what is different. 

Alerts are made up of trigger/reset conditions which are queries looking for specific requirements, and trigger/reset actions that fire when those requirements are found. 

As you pointed out, you can disable an alert through the 'Manage Alerts' system to stop it from triggering, since the actions only fire when it's triggered you shouldn't receive those alert notifications. Because actions can be assigned to multiple alerts, you don't necessarily have to disable to action to stop the alert from firing, so there must be a few things that are happening that you can look into:

1. The action is assigned to another alert by accident so fires when you don't expect it.
2. The alert you are disabling isn't the one that is actually firing, this can happen a lot with duplicated alerts.
3. Check the action doesn't have "Repeat this action every X minutes until the alert is acknowledged" configured on it, this can often cause confusion with rogue emails.
4. There is something broken or stuck in the back end of SolarWinds that is causing a particular query to be repeated.

The first 3 of these you should be able to check through the message centre, event logs, SolarWinds logs, or by manually checking the actions and alerts settings. For the last point, there are a couple of SolarWinds recommended troubleshooting steps, like clearing the SWIS and Job Engine caches, that can help clear any stuck or repeating jobs:

https://support.solarwinds.com/SuccessCenter/s/article/Clear-Information-Service-Subscriptions?language=en_US

https://support.solarwinds.com/SuccessCenter/s/article/Repair-a-failing-Job-Engine-Collector-and-Information-service-in-Orion-Platform?language=en_US

If you are still having problems it would be worth submitting a support ticket with diagnostics so that the engineer can analyse platform logs. 

Hope that helps!

Marlie Fancourt | SolarWinds Pre-Sales Manager 

Prosperon Networks | SolarWinds Partner since 2006