Implicit Dependencies for Nodes and Interfaces

I've got an environment where I am preparing to roll out interface up/down status as a continuation of service where we monitor node up/down status (and other node data, but that's not important right now.) We've been busy trying to pull together node dependencies when I realized that adding in interface up/down status is going to add a whole new layer of dependencies and wondered if anyone else had run into this before. I've done some quick testing and, from what I can tell, if you monitor the interface and the node that if the node goes down only the node alerts. (This is good.)

My question is whether both the node and interface will show down if I go and yank the cable from the interface on the switch? I'm going to go and test that shortly but I thought I'd throw the question out there for posterity's sake (or when I forget the answer and search for it on Thwack again in a few months!). I suspect that the interface will show down and the node will show unreachable, especially if my theory of implicit dependencies holds true.

I suspect that this only works for upstream devices (ie switches) and not for yanking cables or disabling interfaces on the monitored node as being unable to ping a node is completely dependent on the interface being up

Be well,

Josh

Find more posts tagged with

dependency

npm

implicit_dependencies

explicit_dependencies

Accepted answers

All comments

jbiggley

OK -- testing done. Here's the outcome -- if you disconnect an interface you will get an event for both the interface and monitored node going down -- there are no implicit dependencies built into NPM. For us, that means that the triage process is going to have to be rock solid so that we don't create incidents for nodes down and the interfaces down.

Node Down.jpg

Crud.

Anyone have a better solution other than manually creating dependencies for all of my interfaces and monitored nodes? Any secret sauce to correlate nodes and interfaces when generating alerts?

Josh

stuartwhyte

On your interface down alert, set the first criteria to be node status = up.

If the node is down, the interface alert will not be triggered.

Stuart

jbiggley

Yeah that works for interfaces directly attached to the node itself, and I believe those are actually implicit alerts, but my concern is with upstream nodes. For example, SwitchA int0/1 is connected to Node1. If I unplug (or administratively shut down) SwitchA int0/1 then NPM will create an event for the interface being down and the node being down.

What I am trying to do is to filter out extraneous alerts and get the network team closer to the actual source of the event. A node and an interface going down at the same time sends mixed messages as to the source of the problem unless you understand how alerting in Solarwinds works.

Unless there is something I am missing I am going to have to set up some sort of dependency for every interface or build some extra steps into my triage documentation.

Thanks for the suggestion though.

stuartwhyte

Ah yes, I see your point, and unfortunately I think dependencies are the way to go.

As an alternative, one way I've handled this is to set a custom property "Key interface" to true for interfaces that I want alerts on and false for ones I dont, but still want statistics. Since (generally) there are two interfaces in any one connection, one at the switch end and one and the node end, I set the switch end to true and the node end to false.

Not a great solution, but a work around and a "noise limiter" all the same...

jeffness

I tend to write my alert criteria to use node status is not down so if it is in a warning state due another interface being down the next interface going down will still fire.