hosts marked offline after restarting the Solarwinds server

Question

Dear SolarWinds wizards,

Has it ever happened to you where specific hosts (mostly windows, occasionally Linux) would be marked "down" after restarting the Solarwinds server? (... and where nothing helps but to restart those "down" hosts...)

We restart the SolarWinds server about once a month after patching, and since about 6 months ago, 2-3 hosts in our environment of about 200 devices (mixed Linux, Windows, Mac, network gear and other SNMP devices) are routinely marked offline ("down"). Attempting to test WMI credentials on Windows (or SNMP community string on Linux) hosts fails with "Test Failed. Test job timeout". Restarting WMI/SNMP services on the target hosts doesn't help. SNMPWalk from the SolarWinds server to the target hosts - works. (Haven't tried deleting and re-adding the node yet - would prefer to avoid it.)

The only thing that worked so far was to restart the target host.

Given this happens only after SW server restart and only to specific hosts, it's probably safe to say the culprit is either:

* loss of network connectivity to the target host, or
* a strange SolarWinds configuration issue or a bug

... yet even the loss of network connectivity feels unlikely: why would it always coincide with restarting the SolarWinds server and always go away after restarting the host?

Thanks!

vinay.by · Answer

Thank you, will respond back if i find something.

akhasheni · Answer

Do these servers have the SW Agent installed or do they poll with WMI or SNMP normally?

WMI. (One Linux host only got marked "down" once or twice as opposed to 8-10 times for the same 2-3 Windows hosts so I should probably only focus on Windows / WMI hosts.)

akhasheni · Answer

What version of Orion are you on

2020.2.1. The issue was present on two prior versions as well (2020.2 and one prior one).

Is everything on your DB good and healthy ? I mean is there a latency issue between Orion poller and your Orion DB OR Maybe you should look for huge tables and get it reindexed and then check if Orion restart is still creating an issue.

"Database Maintenance" doesn't find any issues, completes in about 90 seconds. AppInsight for SQL on the DB server (all default thresholds) goes yellow once in a while but is mostly green. No known latency issues - else we'd likely get delayed alerts and other side effects. "Deployment Health" IDs seven "problems" (six "medium", one "low") - yet they don't seem to be directly related to the issue.

One of the flagged problems:

Ping all registered message subscribers

Last checked Nov 30, 2020 11:01:21 PM
What went wrong
Pinging recipient 'ADM' on endpoint 'd4beb8b9-6689-4399-b3ff-80fde796d471' [local] has failed because of the error: DeliveryFailedPing message received:The communication object, System.ServiceModel.Channels.ServiceChannel, cannot be used for communication because it has been Aborted.

What you can do
Ensure that the recipient 'ADM' on endpoint 'd4beb8b9-6689-4399-b3ff-80fde796d471' [local] is running.

I'll still go through them to see if I can make them go away.