NPM Re-triggers Old Alert Actions on Service Restart

[johnwilly]

Hi All,

Ever since i upgraded my NPM to 11.5.2, when ever i restart the Orion Services some of my very old alerts gets re-triggered and i am getting lot of incident / Emails.

Is anyone facing similar issue? Please help!!

Thanks,

Reuben

[silverbacksays]

Do all your alert triggers have the global default SMTP alias set, rather than specifying a SMTP server? If they do, you could remove the SMTP server (go to Settings>Manage SMTP Servers) and since Orion will no longer know where to send email alerts, they will not go anywhere. You'll have to add the email settings back in post reboot, however.

DISCLAIMER: I have't tested this myself, but it should work. Backup your database first, just to be on the safe side.

The other option would be to make a note of all enabled alerts with email actions, then disable them, and enable them once the reboot is complete. You'll need to note which ones you need to enable when the server is back up, however, and anything which is currently down/matches an alert trigger will trigger when the alerts are enabled.

[neomatrix1217]

Have you installed all the hotfixes?

[johnwilly]

Yes i have. This is not just related to Interface like they mentioned in Hotfix. I am getting Alerts for Power supply , Utilization ,etc etc.

Its just random. Every time it pick different old alerts which are already active in the Alerts page for a long time.

[neomatrix1217]

Do you have any other products other than NPM?

[johnwilly]

Yes I have NCM NTA UDT and VNQM along with NPM.

[neomatrix1217]

Try this.

In order to not re-trigger alerts when rebooting the Orion server,

1. Go to Settings > Alerts and Reports > Manage Alerts > Active Alerts in the top right > then in the Active Alert Page, go to the top right and click on More, select Pause Actions of all Alerts, then perform reboot.

[johnwilly]

Thanks, i tried it but it still didn't help.

Just to make things more clear when i say the alert gets re-trigged, it is not showing in the Active alerts page that it was recently generated.It shows as if it was generated long time back. However the Alert actions i configured for that particular alert say for example "send email" or "Open P2 ticket" they all gets triggered.

Solarwinds support to troubleshoot other things needs to stop and start the services every time and it is causing lot of false alerts. Since this is a Hospital infrastructure these alerts are taken into action immediately and i end up saying sorry i restarted the services.

Thanks again for all your help. Very much appreciated

Solarwinds support is not that much helpful yet as they want me to re-install re-install job engine which again requires me to stop the services. Even if i do have a maintenance window in my infrastructure to do such activity, it doesn't help as its still  re-triggers the alert actions and i cant seem to avoid it.

[neomatrix1217]

What's your ticket #

[johnwilly]

Sorry for the late reply.

The ticket # is 903523

This is the latest update. They are suspecting a bug.

Hi Reuben, It sounds like you are running into a known issue in your environment. If these alerts were over 30 days old, I can explain why this is happening. Once you have active alerts for >30 days and the alerting service is restarted, you will receive all those older alerting emails. This is because database maintenance deletes all >30days old records from the AlertsHistory table. The action execution log is deleted by default after 30 days. If the Alert Engine v2 does not find a record in the AlertsHistory table, it executes the action again because it thinks that action was not executed. Thus, you get repeated alerts. Please acknowledge if this correlates to what you are seeing. I can submit this to development, but I don't have an ETA on a fix. The best action to take is acknowledge and clear out the old active alerts."

[mlan]

I am also seeing this issue on 11.5.2 (and on previous versions), and it also re-triggers alerts that are ​less​ than 30 days old.

[thoang-marvell]

I have 11.5.2 as well and every time the services/server is restarted all active email alerts re-trigger again.  Looking for an easy way to suppress when we need to do maintenance.