This discussion has been locked. The information referenced herein may be inaccurate due to age, software updates, or external references.
You can no longer post new replies to this discussion. If you have a similar question you can start a new discussion in this forum.

NPM Re-triggers Old Alert Actions on Service Restart

Hi All,

Ever since i upgraded my NPM to 11.5.2, when ever i restart the Orion Services some of my very old alerts gets re-triggered and i am getting lot of incident / Emails.

Is anyone facing similar issue? Please help!!

Thanks,

Reuben

Parents
  • Have you installed all the hotfixes?

  • Yes i have. This is not just related to Interface like they mentioned in Hotfix. I am getting Alerts for Power supply , Utilization ,etc etc.

    Its just random. Every time it pick different old alerts which are already active in the Alerts page for a long time.

  • Do you have any other products other than NPM?

  • Yes I have NCM NTA UDT and VNQM along with NPM.

  • Try this.

    In order to not re-trigger alerts when rebooting the Orion server,

    1. Go to Settings > Alerts and Reports > Manage Alerts > Active Alerts in the top right > then in the Active Alert Page, go to the top right and click on More, select Pause Actions of all Alerts, then perform reboot.

  • Thanks, i tried it but it still didn't help.

    Just to make things more clear when i say the alert gets re-trigged, it is not showing in the Active alerts page that it was recently generated.It shows as if it was generated long time back. However the Alert actions i configured for that particular alert say for example "send email" or "Open P2 ticket" they all gets triggered.

    Solarwinds support to troubleshoot other things needs to stop and start the services every time and it is causing lot of false alerts. Since this is a Hospital infrastructure these alerts are taken into action immediately and i end up saying sorry i restarted the services. emoticons_sad.png

    Thanks again for all your help. Very much appreciated emoticons_happy.png

    Solarwinds support is not that much helpful yet as they want me to re-install re-install job engine which again requires me to stop the services. Even if i do have a maintenance window in my infrastructure to do such activity, it doesn't help as its still  re-triggers the alert actions and i cant seem to avoid it.

Reply
  • Thanks, i tried it but it still didn't help.

    Just to make things more clear when i say the alert gets re-trigged, it is not showing in the Active alerts page that it was recently generated.It shows as if it was generated long time back. However the Alert actions i configured for that particular alert say for example "send email" or "Open P2 ticket" they all gets triggered.

    Solarwinds support to troubleshoot other things needs to stop and start the services every time and it is causing lot of false alerts. Since this is a Hospital infrastructure these alerts are taken into action immediately and i end up saying sorry i restarted the services. emoticons_sad.png

    Thanks again for all your help. Very much appreciated emoticons_happy.png

    Solarwinds support is not that much helpful yet as they want me to re-install re-install job engine which again requires me to stop the services. Even if i do have a maintenance window in my infrastructure to do such activity, it doesn't help as its still  re-triggers the alert actions and i cant seem to avoid it.

Children
  • Sorry for the late reply.

    The ticket # is 903523

    This is the latest update. They are suspecting a bug.

    Hi Reuben, It sounds like you are running into a known issue in your environment. If these alerts were over 30 days old, I can explain why this is happening. Once you have active alerts for >30 days and the alerting service is restarted, you will receive all those older alerting emails. This is because database maintenance deletes all >30days old records from the AlertsHistory table. The action execution log is deleted by default after 30 days. If the Alert Engine v2 does not find a record in the AlertsHistory table, it executes the action again because it thinks that action was not executed. Thus, you get repeated alerts. Please acknowledge if this correlates to what you are seeing. I can submit this to development, but I don't have an ETA on a fix. The best action to take is acknowledge and clear out the old active alerts."

  • I am also seeing this issue on 11.5.2 (and on previous versions), and it also re-triggers alerts that are ​less​ than 30 days old.

  • I have 11.5.2 as well and every time the services/server is restarted all active email alerts re-trigger again.  Looking for an easy way to suppress when we need to do maintenance.