Additional Web Server (AWS) - Performance Issues

We've recently ventured into moving our web console away from our main polling engine in order to provide dedicated resources for polling and web traffic. After deploying the additional web servers, we have witnessed performance issues that have rendered the web console almost useless. Once we move all web traffic back to the main polling engine, everything works flawlessly.

Our additional web servers are running on Windows Server 2016, 8vCPU, and 16 GB memory, running NPM 12.3.

The IIS Worker Process shoots up to over 20% utilization with minimal traffic on the server and at times it runs over 15% with ZERO traffic on the server. We've verified that all permissions are correct, all exclusions are present in McAfee, and have cleared ASP.net temp files a couple times.

Is anybody else experiencing issues along these lines?

Find more posts tagged with

Accepted answers

All comments

mesverrum

Just as some datapoints for you, we run a load balanced pool of 4 AWS. Nobody really touches the real web console except me. Each web server only has 2cpu/4gb and they tend to run relatively hot, ~75-100% cpu is pretty common but user experiences seem to be generally acceptable. According to another tool I monitor them with each server gets about 100-200 page requests per minute, I have a powershell script that could parse the logs to give me unique user counts but I haven't bothere to run it in a while but I would expect that across the company we usually have maybe 20-40 sessions logged in at once. We could probably consolidate it all into a pair of APE's with more resources but given that the licenses are so cheap it's convenient to have the n+1 redundancy where I can take a host out of the pool to reboot it or troubleshoot problems pretty painlessly. Also nice when something big happens for us to actually have enough excess capacity to handle everyone suddenly logging in for the first time in a year trying to chase the problem.

So in general our CPU loads sound a bit scary but the end user experience is good enough that we don't want to cough up more in cloud hosting fees to bump them up until someone complains. At this point the complaints we do got don't seem to be resource bottlenecks as much as the occasional hiccup where we need to do an unscheduled reboot (worth mentioning we pre-emptively reboot each of our SW servers on a rolling schedule throughout the week).

sreenathmp

We are also facing similar kind of issue. We have 2 additional web servers which are in load balanced faces. Often facing high CPU utilization issue in the web servers due to Solarwinds Information Service causing website inaccessible for users. We have taken this with support and they found some long running query error login in the SWIS log. But yet to identify what is causing this long running query. Applied the best practices in DB as per their suggestion, still facing the issue. Another thing is we are having some memory issue in main polling engine which we are planning to upgrade. Still the case is with support and working on it.

tony.johnson

@mbrandon32 That’s hardly normal. Something else may be going on here. If you have not already done so could you open a support case and share the case number with me?. That’s a lot of resources for an AWS to be consuming.

mbrandon32

@sreenathmpDo you recall what the long running query was? I'd be curious to check the SWIS logs on my end for something similar. What version of NPM are you running?

mbrandon32

@tony.johnsonI had a support case open (00516000) but it was recently closed due to an issue with accounts receivable stating we have an invoice past due. My account team is currently working on getting that resolved - which was why I turned to the forums. Any insight on the resources an AWS should be dishing out for around 25-30 connections at a time? I would think that the server specs are more than enough to handle that kind of load.

mbrandon32

@mesverrumYou mention that the user experience is acceptable. Is performance an issue and they just accept it? Or does the web console respond normally? When the IIS Worker Process is over 20% utilization, I hear reports of pages taking minutes to load and sometimes receiving a timeout error. What version of NPM/Orion are you running?

I too am load balancing via BIG-IP DNS GSLB, statically persisting connections and each server is on a monthly update/reboot schedule. I figured two servers was going to be enough to handle the load we were expecting. Based on what I am seeing when there is minimal traffic on the servers, I'm not certain adding more servers to the pool would do anything for me - but I do like the extra redundancy.

mesverrum

Generally pages are up in a couple seconds, but we use another tool for tracking web server data and it collects all the activities of the web server so I occasionally see requests that take >10 seconds or error out but that's maybe about 1% of the total so I can live with those. When I'm working inside the tool almost every time that I have problems its because I'm working on creating new widgets or reports and have sent something sloppy into the info service and it choked on some million row request.