Best way to alert on Down Nodes - slow to alert?

I have a question about nodes that go down. We have recently migrated from Nagios to SolarWinds and are in the stage when both products are coexisting. When a node goes offline (reboots, crashes, etc.) it seems Nagios picks it up extremely quickly, while SolarWinds seems to take much longer, and most of the time if the Node was rebooted, it comes back up before SolarWinds ever notices an issue, and we never know anything happened.

I have an alert to evaluate ever minute and alert on trigger condition:

Node Status is equal to Down

Upper management is demanding to know how a server can down and come back up and SolarWinds never report on it. Is there a better way for me to alert on this?

Find more posts tagged with

Alert

Node

solarwinds

down

slow

nagios

server

Accepted answers

bobmarley

Re: Fast Polling You can read up on it here.

All comments

bobmarley

Re: Fast Polling You can read up on it here.

osborne_graham

To cover yourself for the upper management demands of knowing that a device has rebooted, you could create an alert that will fire if the last boot time changes which would fire on the next poll no matter what. You could combine it with an additional component to check the windows syslog for unexpected reboot entries and in that way determine if the server was cleanly rebooted or if it crashed and bombed out. I find the latter alert very useful in my environment.

derik.pfeffer

tech2734,

Polling and Alerting go hand and hand.. It is all based on time. If you use the default polling interval of 2 minutes, you will get a single point in time response measurement. If the system can go down and return to a level that it responds to ICMP requests, then you might want to reduce the polling interval. Now let's move on to fast polling "Warning Status" as mentioned by bobmarley, this is a default of another 2 minutes. The default for Alert Polling is 1 minute, so if you happen to have the worst luck and it fails immediately behind a successful poll, you could be down for just 5 minutes. I like to use the Node down and the reboot alerts together to catch those systems that boot to a level that it responds to ICMP like a VM can. This process is to make sure you are getting true node downs and not just a single point in time that the end device didn't respond to an ICMP request. You might need to adjust these timers for the critical devices, and the global warning time frame to catch those instances.

Best Regards,

Derik Pfeffer

Loop1 Systems - SolarWinds Training and Professional Services

LinkIN: Loop1 Systems

Facebook: Loop1 Systems

Twitter: @Lopp1Systems

designerfx

Also as a matter of best practice; you are not going to realistically be able to respond to things sooner than that.

One other thing to watch for is a solarwinds alert for "high packetoss", as that will pick up on things during fast polling if you wish to leverage it. Warning: it can be noisy.