SolarWinds: Fake Server has Down Alert

Almog

Hi,

im using the SolarWinds system, but sometimes i got a fake server is down alert while im using the server at the same time.

tell me what can i do about it?

rschroeder

An alert telling you a server is down--when you know the server is NOT down--is an indication of a communications problem between your Solarwinds polling engine and that server.  Your task is to discover where that problem lies.

Start with the exact alert.  Look at what it's alerting ON, and how it's trying to communicate with the server.  Is your Solarwinds polling engine using a simple ICMP ping to talk to the server? Is it using an installed software probe on the server?  Is it using WMP?

Once you know how the server is being monitored by Solarwinds, you can look into where the communication breakdown is occurring.  Some ideas to consider:

• Is your Solarwinds polling engine experiencing any Layer 1 (physical connectivity) issues?  If it is, it's possible (even probably) there are other systems showing down at the same time.
• The same question goes for the physical link to the server being reported down, but I suspect I know the answer you'll give:  you're using it, so it's not having physical issues.  But still, it's part of the Layer 1 path to troubleshoot.
• Move on up the 7-Layer Salad to the rest of the Layers in the Network OSI Model and troubleshoot each of them until you find the cause.  https://www.cloudflare.com/learning/ddos/glossary/open-systems-interconnection-model-osi/   
• Efficient analysis is aided by collecting other items that can help add up to the cause:
  • When did the problem begin?
  • When does it reoccur?  Is it on a schedule?  If it happens at the same time on a repeating basis, that's a clue.  Maybe the resources of the polling engine or the server are being consumed at that time, and the polling cycle fails because of something else considered "normal" is happening.  For example, imagine a scheduled backup process is taking place, and the server is too busy responding to that task to respond to network availability polling.  Or your System Administration team is regularly applying security patches or upgrades to the server, perhaps on a monthly basis, and your poller is catching the server rebooting or receiving large amounts of data.
  • Review the snmp information provided to Solarwinds from all of the devices and ports between the two servers.  (You ARE monitoring all of those ports, right?)  SNMP can tell you about port errors, throughput rates, physical and logical states, switch & router resources, and more.
  • What does your Netflow information reveal about the path between the two devices?  If they are communicating via routing, Netflow can be applied to the Layer 3 interfaces and it can show you if/when excessive data flow is impacting communications due to some part of the network path being too small for the data to all make the trip.  I've seen robust systems brought down to their knees by a WAN link or ISP link being too small for a demand.  Netflow can tell you how big the demand has been, whether it corresponds with the false outage alerts, and which other systems are causing the spike in demand.  It can also tell you the source and destination of excessive traffic that may be impacting the flow.  It then becomes your task to determine if the flow is appropriate or not.  If it's appropriate, then you might need to plan for bigger network connections.  If it's not appropriate, filtering or operator/device behavior changes might be needed, to ensure that business monitoring flows aren't impacted.
  • What logs can you view that correspond to the false "server down" alert times?  They can include:
    • The multiple Event Viewer logs on the servers themselves (I say "servers" because you have to be open to the idea that the polled server might not be the problem--it may be an issue with your polling engine)
    • Syslog messages from the servers.
    • Security events sent to your SIEM (Solarwinds LEM, Splunk, etc.)
    • Change Management schedules (it's important to be aware of what other teams are doing that can impact your network environment, and Change Control meetings and logged / scheduled events can correctly directly with issues you might be seeing in Solarwinds)
  • What resources on both servers ("down" server and Solarwinds polling engine) are being stressed during the down times?  CPU, memory, Network throughput, etc., all can impact the ability of the two servers to talk with each other.
• Remember that troubleshooting the OSI layers must include all of the path and devices used to connect the two servers.  I've seen WAN service providers and ISP's oversubscribe circuits several hops outside "my" network, and NetPath has been extremely helpful documenting outages and latency changes and path changes.  I've used NetPath to monitor odd outages that ISP's claim aren't caused by them, only to find the outages caused by third-party providers between ISP's.  NetPath shows that information very nicely and intuitively.
• Keep in mind you have additional non-Solarwinds resources to draw on at no charge.  For example, Wireshark.  If you can install it on one of the two servers (polling engine or target server) and having it running during the time when an outage is reported by Solarwinds, you should be able to analyze it and discover if the communication packet is being sent by one, received by the other, and responded to.  If the poller sends, but the destination server doesn't receive, there's the problem; all you have to do is discover why.  And that's where the logs and snmp information for the servers, switches, and routers become useful in explaining what's happened.

I'm confident you'll find it if you have Solarwinds deployed, good logging, Change Control information, and perseverance.  Let us know what you find.

Swift Packets!

Rick Schroeder