false node down messages

Question

We have 400 nodes and 3500 elements to monitor, now and then all nodes turning yellow and alerts (with e-mail) are send out that nodes are down. After less then a minute all nodes are back up again and the alert engine is sending a mail that all nodes are back online.

Is there a way to let solarwinds check a network element twice before firing a alert?

What can be the root cause for this blinking? A fault in our network? A temporary block in the network, glich in the matrix?

We use solarwinds 7.8

Thanks for your time!

barend · Answer

Thanks for your advice. We will look into it!

jtimes · Answer

One more thing to check and also "tweak" for your environment:

In the Node Manager go to File> Advanced Settings> Node Warning Interval Tab, and make sure it's not set to low...

jtimes · Answer

The "fault" could be on or right at your Orion server itself, I'm not definitively stating it is, but could be several issues occurring.

Things to check:  Orion server NIC driver and version / configuration & corresponding switch port config (Speed/Duplex), look for layer1 through 3 issues as well as high levels of broadcast or multi-cast traffic on Orion's VLAN,  up-stream switch port (up-link) utilizations.  Look for QoS rules that could be putting your Orion traffic in the proverbial bit-bucket...

Polls Per-second Tuning, Poll success Rate, Baseline calculation method.

Also check your Orion Thresholds - Error and Warning levels, find a median setting that accommodates both WAN and LAN environments.

This might be for discovery, but check it anyway;  In the Node Manger open the Settings Tab and check the ICMP Timeout and packet size, make sure its set appropriately for your network environment, you might want to boost it up to allow for slower, or heavily used devices to respond.  ICMP is very low in network priority.

Hope these help.