Solarwinds APE down due to WAN link down with Main PE and additional Web Server slow to no response

Hi All

Would like to ask a question as we have come across this a few times now already on different versions of NPM from 2019.4.2 and 2020.2.6 So we have Main Polling Engine with three additional Polling Engines with additional Web Server. Each Additional Polling Engine sites at remote WAN side with Main Polling Engine and Additional Web server at Main Location. When the one site goes down as in completely down the Additional Polling Engine does reflect as down but the nodes is still showing as up. HA is set as default. The remote Polling Engine has 2600 +/- objects while the others have about 3000-4000 objects. Now comes the interesting part. The Additional Web Server collapses with no response. You can ping the server and the services is up on the Web Server but sluggish response from Website. If you go via the main polling engine web browser it is faster but still not the same. Now once the additional polling engine is back online then services restore to normal performance. I would really appreciate guidance on this please.

Find more posts tagged with

slow_orion_performance

polling issues

Accepted answers

All comments

jm_sysadmin

Without anything collecting information on the remote nodes, nodes won't change status. And from the point of view of the polling engine for the nodes, are they actually down if you lose a link? They may not be. You did mention HA, do you have a polling engine at a different site that should be taking the work with the site down using the HA set up? That does have a timeout you can configure, the default is 5 minutes I think. During that time no polling would occur.

Websites don't have HA, just additional capacity. Typically speed issues are directly related to what's happening in the database. If the Additional web server can't hit the database, it definitely won't be up.

I really don't have a great picture of what's happening, so forgive my vague answers. If you have a single site cut off from the others, its safe to assume that Orion won't work right there. HA when correctly configured should keep it going for everybody else. I did not hear how SQL is set up, I assume that you have a SQL cluster, and one node is active, the others are passive until you fail to it. My polling engines are in active passive pools, Most things are defaulting to my main location, but 'stuff' on other nodes are kept in the remote data center when possible, and fail to the pair in the opposite location. My MS SQL cluster fails over faster than Orion does, but its also able to be used across the WAN without issues.

When I think about my additional webservers, they are completely separate from additional polling servers. I have them in each data center, and I use a GSLB (for both fault tolerance and to find the closest servers) to steer people to VIPs in each data center, and the VIPs to deal with fault tolerance in a location. There is a blog here with a picture that might explain it better. GSLB: what it is, how to use it and benefits (loadbalancer.org)

kevindk

Thanks for the info. The SQL cluster is residing with the Main PE and additional Web Server at Main site. I will look at the HA functionality to see if it can assist. Don't need HA for now on the Web services side. As we run the web on main PE for Admin and Additional Web Server for Tech Support and onsite support. So the one APE dropped due to both wan links dropping to that site as we are ISP agnostic. The Polling Engine was still operational from an onsite perspective but because no DB sync was received the nodes still reflected up but you could not poll it. But from the Additional Web Server there was literally no response and the Main Polling Engine website was still functional with little delay on it except for one APE being down. It's just curious that we picked this up from another site that was also down due to power and all nodes showed as up. Will be looking at the HA for the sites and split them rather. Also to note from the APE there is a few servers being monitored via wmi on the remote PE that was affected. Don't know if this would have affected the Solarwinds Web Server response. Will also add the auto dependencies so if the polling engine is down for a site that those nodes for the time being go into unknown state until polling engine is operational again.