6 Replies Latest reply on Aug 13, 2013 10:40 AM by jbiggley

    Implicit Dependencies for Nodes and Interfaces

    jbiggley

      I've got an environment where I am preparing to roll out interface up/down status as a continuation of service where we monitor node up/down status (and other node data, but that's not important right now.)  We've been busy trying to pull together node dependencies when I realized that adding in interface up/down status is going to add a whole new layer of dependencies and wondered if anyone else had run into this before.  I've done some quick testing and, from what I can tell, if you monitor the interface and the node that if the node goes down only the node alerts.  (This is good.)

       

      My question is whether both the node and interface will show down if I go and yank the cable from the interface on the switch?  I'm going to go and test that shortly but I thought I'd throw the question out there for posterity's sake (or when I forget the answer and search for it on Thwack again in a few months!).  I suspect that the interface will show down and the node will show unreachable, especially if my theory of implicit dependencies holds true.

       

      I suspect that this only works for upstream devices (ie switches) and not for yanking cables or disabling interfaces on the monitored node as being unable to ping a node is completely dependent on the interface being up

       

      Be well,

       

      Josh

        • Re: Implicit Dependencies for Nodes and Interfaces
          jbiggley

          OK -- testing done.  Here's the outcome -- if you disconnect an interface you will get an event for both the interface and monitored node going down -- there are no implicit dependencies built into NPM.  For us, that means that the triage process is going to have to be rock solid so that we don't create incidents for nodes down and the interfaces down.

           

          Node Down.jpg

           

          Crud.

           

          Anyone have a better solution other than manually creating dependencies for all of my interfaces and monitored nodes? Any secret sauce to correlate nodes and interfaces when generating alerts?

           

          Josh

            • Re: Implicit Dependencies for Nodes and Interfaces
              stuartwhyte

              On your interface down alert, set the first criteria to be node status = up.

               

              If the node is down, the interface alert will not be triggered.

               

              Stuart

                • Re: Implicit Dependencies for Nodes and Interfaces
                  jbiggley

                  Yeah that works for interfaces directly attached to the node itself, and I believe those are actually implicit alerts, but my concern is with upstream nodes.  For example, SwitchA int0/1 is connected to Node1.  If I unplug (or administratively shut down) SwitchA int0/1 then NPM will create an event for the interface being down and the node being down.

                   

                  What I am trying to do is to filter out extraneous alerts and get the network team closer to the actual source of the event.  A node and an interface going down at the same time sends mixed messages as to the source of the problem unless you understand how alerting in Solarwinds works. 

                   

                  Unless there is something I am missing I am going to have to set up some sort of dependency for every interface or build some extra steps into my triage documentation.

                   

                  Thanks for the suggestion though.

                    • Re: Implicit Dependencies for Nodes and Interfaces
                      stuartwhyte

                      Ah yes, I see your point, and unfortunately I think dependencies are the way to go.

                       

                      As an alternative, one way I've handled this is to set a custom property "Key interface" to true for interfaces that I want alerts on and false for ones I dont, but still want statistics.  Since (generally) there are two interfaces in any one connection, one at the switch end and one and the node end, I set the switch end to true and the node end to false.

                       

                      Not a great solution, but a work around and a "noise limiter" all the same...

                      1 of 1 people found this helpful
                        • Re: Implicit Dependencies for Nodes and Interfaces
                          jbiggley

                          I ran into a bit of a problem today when trying to build a dev/test dependency -- there is no option to do interface level dependencies.  All dependencies, at least in 10.4, are for node-level only.

                           

                          Am I missing something?

                           

                          EDIT:  I found what I was missing.  When you build a dependency the inline group builder does not allow you to select interfaces.  In order to build a group with interfaces you have to build the group first and then build the dependency.  I am going to submit a feature request that allows dependencies for node up/down status to be implicitly built based on the MAC address association.  If NPM can produce a handy/dandy topology report I wonder how hard it would be to build implicit dependencies based on the same information?!?

                           

                          Message was edited by: Joshua Biggley

                           

                          EDIT2:  Scratch that -- I'm not sure how I messed that all up, but you can definitely build dependencies with interfaces as the parents, it is only when adding children that you have to either a) select a node or b) great an group of interfaces.

                           

                          Message was edited by: Joshua Biggley

                      • Re: Implicit Dependencies for Nodes and Interfaces
                        jeffness

                        I tend to write my alert criteria to use node status is not down so if it is in a warning state due another interface being down the next interface going down will still fire.