3 Replies Latest reply on Mar 29, 2018 1:43 AM by rohan_h

    Email Alerts with "Correlator Numbers"

    mando

      In our networking environment, we use a third-party alerting system to alert our technicians of when there is a node with issues (down node/down interface etc) in conjunction with our Solarwinds platform (when after hours and a problem arises).  This alerting system uses an email from our Solarwinds platform to create the alert but then also uses another email to "clear" the alert.  For example if an interface bounced, Solarwinds detects the down interface, sends the alert email, but then polls again and sees that the interface is up, it sends another email that states the alert has "cleared."   For our third party alerting system we had to develop a "correlator number" that will tie the clear message to the alert message this way (there is a wait time before the technician is given notice) our technicians are being bothered after hours with false alarms.  If an interface flaps multiple times it will be a different alert instance per time, otherwise by the time our alerting system gets the alert and clear messages, they could be possibly out of order and the interface could still be in alert but instead could get the clear message from a different instance.  Then the the technician wouldn't be given notice which could possibly a critical issue depending on what the interface is connected to.  The "correlator number" fixes this problem (or at least for us it did).  The number will stay the same per instance for both the "alert" email and the "clear" email tying the two together.

       

      Here are the email alert and clear message settings that we use that allows us to use our third-party alerting system:

       

      Alert MessagesClear Messages

      Interface Down Alert:

      Subject:  ${N=SwisEntity;M=Node.DisplayName} Alert: ${N=SwisEntity;M=Caption}  is ${N=SwisEntity;M=StatusDescription}

       

      Message:

      Alert: Set

       

      Entity: ${N=SwisEntity;M=Node.DisplayName}

       

      Interface: ${N=SwisEntity;M=Name}

       

      Message: ${N=SwisEntity;M=Caption}  is ${N=SwisEntity;M=StatusDescription}.

       

      Correlator: ${NodeName} ${N=SwisEntity;M=Name} ${SQL: SELECT AlertRefID FROM AlertConfigurations WHERE AlertID=${N=Alerting;M=AlertID}}${N=Alerting;M=AlertID}${N=SwisEntity;M=NodeID}

      Interface Down Clear:

      Subject:  ${N=SwisEntity;M=Node.DisplayName} Alert: ${N=SwisEntity;M=Caption}  is ${N=SwisEntity;M=StatusDescription}

       

      Message:

      Alert: Clear

       

      Entity: ${N=SwisEntity;M=Node.DisplayName}

       

      Interface: ${N=SwisEntity;M=Name}

       

      Message: ${N=SwisEntity;M=Caption} is ${N=SwisEntity;M=StatusDescription}.

       

      Correlator: ${NodeName} ${N=SwisEntity;M=Name} ${SQL: SELECT AlertRefID FROM AlertConfigurations WHERE AlertID=${N=Alerting;M=AlertID}}${N=Alerting;M=AlertID}${N=SwisEntity;M=NodeID}

      When a Node goes down Alert:

      Subject:  Alert: ${N=SwisEntity;M=Caption} is ${N=SwisEntity;M=StatusDescription}

       

      Message:

      Alert: Set

       

      Entity: ${NodeName}

       

      Message: ${NodeName} is ${N=SwisEntity;M=StatusDescription}

       

      Correlator: ${NodeName} ${SQL: SELECT AlertRefID FROM AlertConfigurations WHERE AlertID=${N=Alerting;M=AlertID}}${N=Alerting;M=AlertID}${N=SwisEntity;M=NodeID}

      When a Node goes down Clear:

      Subject:  Alert: ${NodeName} is ${N=SwisEntity;M=StatusDescription}

       

      Message:

      Alert: Clear

       

      Entity: ${NodeName}

       

      Message: ${N=SwisEntity;M=Caption} is ${N=SwisEntity;M=StatusDescription}

       

      Correlator: ${NodeName} ${SQL: SELECT AlertRefID FROM AlertConfigurations WHERE AlertID=${N=Alerting;M=AlertID}}${N=Alerting;M=AlertID}${N=SwisEntity;M=NodeID}

      When a Neighbor Goes Down Alert:

      Subject:   Alert: The routing neighbor ${SQL: SELECT Caption FROM NodesData left join NodeIPAddresses on NodesData.NodeID=NodeIPAddresses.NodeID where (NodeIPAddresses.IPAddress='${N=SwisEntity;M=NeighborIP}' OR NodesData.IP_Address='${N=SwisEntity;M=NeighborIP}')} on ${NodeName} is down.

       

      Message:

      Alert: Set

       

      Entity: ${NodeName}'s routing neighbor ${N=SwisEntity;M=NeighborIP}.

       

      Message: ${NodeName}'s routing neighbor, ${SQL: SELECT Caption FROM NodesData left join NodeIPAddresses on NodesData.NodeID=NodeIPAddresses.NodeID where (NodeIPAddresses.IPAddress='${N=SwisEntity;M=NeighborIP}' OR NodesData.IP_Address='${N=SwisEntity;M=NeighborIP}')} (IP Address: ${N=SwisEntity;M=NeighborIP}), is down.

       

      Correlator: ${NodeName} ${SQL: SELECT AlertRefID FROM AlertConfigurations WHERE AlertID=${N=Alerting;M=AlertID}}${N=Alerting;M=AlertID}${N=SwisEntity;M=NeighborID}

      When a Neighbor Goes Down Clear:

      Subject:   Alert: The routing neighbor ${SQL: SELECT Caption FROM NodesData left join NodeIPAddresses on NodesData.NodeID=NodeIPAddresses.NodeID where (NodeIPAddresses.IPAddress='${N=SwisEntity;M=NeighborIP}' OR NodesData.IP_Address='${N=SwisEntity;M=NeighborIP}')} on ${NodeName} is down.

       

      Message:

      Alert: Clear

       

      Entity: ${NodeName}'s routing neighbor ${N=SwisEntity;M=NeighborIP}.

       

      Message: ${NodeName}'s routing neighbor, ${SQL: SELECT Caption FROM NodesData left join NodeIPAddresses on NodesData.NodeID=NodeIPAddresses.NodeID where (NodeIPAddresses.IPAddress='${N=SwisEntity;M=NeighborIP}' OR NodesData.IP_Address='${N=SwisEntity;M=NeighborIP}')} (IP Address: ${N=SwisEntity;M=NeighborIP}), is up.

       

      Correlator: ${NodeName} ${SQL: SELECT AlertRefID FROM AlertConfigurations WHERE AlertID=${N=Alerting;M=AlertID}}${N=Alerting;M=AlertID}${N=SwisEntity;M=NeighborID}

       

      Note:  That on the "When a Neighbor Goes Down" alert, this is referring to a routing neighbor.  For the message and the subject to not only show an IP address, there must be a node in Solarwinds (doesn't have to be anything more than a ping meaning SNMP is not required, so the neighbor could be the ISP router as an example).  The SQL script came from another user, tnice81 that is used in the message and subject of "When a Neighbor Goes Down".  The original post on where I found his script in Thwack: Including the neighbor node's name in the "A routing neighbor went down" alert.

       

      We are currently using NPM version 11.5.2 with this configuration.

        • Re: Email Alerts with "Correlator Numbers"
          pvaysberg

          the correlator ID is awesome.  Exactly what I need to tie the whole thing together.

            • Re: Email Alerts with "Correlator Numbers"
              rohan_h

              I have also set up the "Node Down" Alert in our setup and I tried the correlator ID script in the Node Down: Correlator: ${NodeName} ${SQL: SELECT AlertRefID FROM AlertConfigurations WHERE AlertID=${N=Alerting;M=AlertID}}${N=Alerting;M=AlertID}${N=SwisEntity;M=NodeID}

               

              But after simulating the script, I get the below error:

               

              Correlator: arlrsw-02 MACRO SQL ERROR - Incorrect syntax near '='.1515

               

              Can anyone please advise what's wrong here? I am also trying the same thing, where we are trying to Integrate SolarWinds with our Ticketing portal.

               

            • Re: Email Alerts with "Correlator Numbers"
              rohan_h

              I have also set up the "Node Down" Alert in our setup and I tried the correlator ID script in the Node Down: Correlator: ${NodeName} ${SQL: SELECT AlertRefID FROM AlertConfigurations WHERE AlertID=${N=Alerting;M=AlertID}}${N=Alerting;M=AlertID}${N=SwisEntity;M=NodeID}

               

              But after simulating the script, I get the below error:

               

              Correlator: arlrsw-02 MACRO SQL ERROR - Incorrect syntax near '='.1515

               

              Can anyone please advise what's wrong here? I am also trying the same thing, where we are trying to Integrate SolarWinds with our Ticketing portal.