3 Replies Latest reply on Oct 19, 2012 1:57 PM by d09h

    False Alerts

    trinity0513
    trinity0513 Sep 11, 2012 1:57 PM

      We have alerts set to email us when a node goes down.  Lately we've been bombarded with emails about nodes going down and coming back up.  They're usually about 50 nodes all going down at once which isn't true because we are obviously still online and we can ping all the nodes.  That leaves us to think that there is a problem with the SolarWinds server itself. 

       

      My question is, if the SolarWinds server loses network connectivity, does it think the monitored nodes are down even though they're not?

        • Re: False Alerts
          d09h
          d09h Expert Sep 12, 2012 3:21 PM (in response to trinity0513)

          Of course.

           

          Run a packet capture on your SolarWinds server and you can see this in action.  From a Windows command prompt, run a ping -t to the IP of something and block pings (using host or network firewall).  Your unanswered pings will result in a device that is down from a monitoring perspective.  Easily demonstrated if you monitor the IP of the computer you are testing from.

           

          If your SolarWinds server has multiple interfaces and one can't reach a monitored device, you could also test by adding a route statement to send traffic out the "wrong" interface.

           

          The answered/ unanswered pings are reflected in this resource:

          Min/Max/Average Response Time & Packet Loss

           

          You can use "View Chart Data" for insight into when pings were unanswered.  Correlate what you see there with your packet capture.

          • Re: False Alerts
            trinity0513
            trinity0513 Sep 17, 2012 1:56 PM (in response to trinity0513)

            I tested it by killing the network connection to the SolarWinds server.  Once I got it back on the network, it sent me 110 alerts (nodes going down and up).  This is false.  There really should be a way SolarWinds can detect that it's losing connection itself and not the nodes.  Our SolarWinds server is a virtual server which doesn't make any sense on how it's losing connection.  It's on the same vSphere server as 10 other VM servers and those never go down.  Deleting/recreating the virtual NIC is my next step...

              • Re: False Alerts
                d09h
                d09h Expert Oct 19, 2012 1:57 PM (in response to trinity0513)

                Re:  "There really should be a way SolarWinds can detect that it's losing connection itself and not the nodes"

                 

                From what I understand about dependencies, you may be able to do this.  Couldn't you make everything dependent on the default gateways of the pollers?  I actually have not set up dependencies, but your use case sounds like it has dependencies written all over it.  Just monitor the default gateway of the pollers.