Community
- Command Central
- MVP Program
- Monthly Mission
- Blogs
- Groups
- Events
- Media Vault
Products
- Observability
- Network Management
- Application Management
- IT Security
- IT Service Management
- System Management
- Database Management
Content Exchange
- SolarWinds Platform
- Server & Application Monitor
- Database Performance Analyzer
- Server Configuration Monitor
- Network Performance Monitor
- Network Configuration Manager
- SQL Sentry
- Web Help Desk
Free Tools & Trials

CPU Reporting Incorrectly

Our NPM 11.5.2 environment reported a significant change in average CPU utilization for one of our Windows Server 2008 VMs monitored via WMI (past few weeks shown below). However, if you log into this server directly it still shows <5% CPU. Also, the reported CPU increase happened immediately following the weekly scheduled reboot of our SolarWinds servers.

We have multiple NPM 11.5.2 environments, so I setup two of the others to monitor the same server - one via WMI and one via SNMP. They both accurately showed <5% CPU while our original SolarWinds environment still showed around 40%. I went back to our original SolarWinds environment and changed the polling interval from 10 minutes to 1 minute for more granularity. As soon as I made this one simple change, it started reporting the correct value (shown below just before 12:00 PM).

I'm concerned about why this happened because it calls the data for our many servers into question. Anyone else encountered this or have some theories?

Find more posts tagged with

Accepted answers

cory.cousins

I escalated the ticket through our sales rep. No luck. Everyone I've spoken with says it can't be fixed by SolarWinds, so I'll just keep "fixing" it through SolarWinds by temporarily unmanaging / re-managing nodes as needed (see example graphs below). Using SNMP or, even worse, an agent to manage the servers isn't an option.

Here's the thing... every polling cycle causes the WMI process to briefly spike CPU across all of our servers, so why do only a handful of random servers decide to start reporting that briefly spiked CPU value as the real CPU utilization of the server? Interrupting the polling cycle temporarily is enough to jostle it back into working order, but there's nothing that can be done by SolarWinds? I feel like this is one of those tickets that just didn't make it to the right person. I guess it gets real hard when the configuration wizard doesn't fix the issue.

All comments

silverbacksays

Hey cory.cousins,

On the face of it, simply changing the statistics polling interval shouldn't change the reported metric for percentage CPU used, as it's a simple number comparison. Check that you're using 64bit counters, as not using these can cause number roll-ups on large numbers, leading to SolarWinds to report wonky data. A quick Goole-fu brought up this page, which explains the difference in the two numbers, and from there you can see why not using the correct counters can cause issues. EDIT: You're using WMI, so counters are not an issue, scratch this advice

Something seemed to have happened around 25th. The first thing I would check is to ensure that the change you made was the reason for the seemingly correct polling information post change, or if it were a coincidence. Check the logs on the monitored server for anything that stands out, either on 25th when the issue started, or afterwards, when things settled back down again.

Also check the longs on your primary polling engine (or the polling engine that is polling the server in question, if you have more than one) for the times specified. See if you notice anything. I'd want to be 100% certain it's not some other issue before pointing the finger at your monitoring tool, as you'll start disbelieving everything it reports!

Incidentally, if you were to change the metrics polling interval back to 10 minutes, do the erroneous values return? Have you tried using the Windows Agent to monitor this server, and if so does this return the same result?

cory.cousins

Changing the polling interval back to the original 10 minutes did not reproduce the issue (it was only in a 1 minute state for a few hours). It's been showing accurate data since the polling interval change from 10 minutes to 1 minute. I see nothing out of ordinary in the event logs on the target server or polling engine around the times the issue started and stopped. Here's the only item that stands out - the erroneous data started being reported immediately following the weekly scheduled reboot of the polling engine responsible for monitoring this target server. No patches or updates were installed. It was just a server reboot.

I did not have an opportunity to test the Windows Agent before the counter started reporting normally again. I've had a ticket open for almost 24 hours. I'm still waiting for a response, but I'm sure it will be "run the configuration wizard and let us know if it happens again".

silverbacksays

From my perspective, now that there are Windows (and soon Linux) agents available, I use them when I can. The additional information you get on QoS from the server as part of the Agent is well worth it.

That's my advice: Install the agent. It also buffers the monitoring data for when you reboot the polling engine(s), so you don't loose any metrics.

cory.cousins

We have now noticed the WMI process (WmiPrvSE.exe) spiking CPU on some monitored servers. This only happens while the server is being polled for NPM information via WMI. In the example screenshot below, the server has been reporting 50ish% CPU utilization for the past 4 days straight. It's only hitting 50ish% because of WMI. Outside of polling, it's well below 10%.

Unlike the issue described in my original post, I can't fix this by adjusting the polling interval. I duplicated this in a remote session with SolarWinds support (case#963864) across two separate NPM environments - 11.5.2 and 11.5.3. The recommendation was to open a ticket with Microsoft.

tigger2

Bump .

I know this is an old post but....

We're on a single server install of: NPM 12.1 with Orion Platform 2017.1.3 SP3, NCM 7.6,SAM 6.4.0

And we just had this issue, and the "unmanage" + "remanage" appears to have fixed our issues. I'm posing to add some info, if it helps anyone:

1. CPU on 1 node in Orion went from "looks normal" to "looks waaaaay too high, and nothing else agrees with Orion's numbers". Basically Orion was reporting all 4 CPU cores @ 100% all the time and everything else said "~25% usage". Everything else = VMware tools not showing CPU pressure, Local perfmon on the server not showing it, SCOM not showing it either). The reason I noticed it was because it was the Orion app server.

2. I I shut down Orion services and booted the Orion server - It appeared to help for a few hours. CPU usage showed lower than all 4 cores @ 100%, and then crept back up to 100% over a few hours. All other tools did not show this CPU usage, not even close.

3. I shut down Orion services, ran the "uninstall + reinstall all Orion services" fix via the config wizard (standard procedure from SolarWinds support site) and booted the server. And it appeared to fix the problem of "100% usage" , even though CPU was still off by quite a bit (Orion reported ~75% usage, with 1 core @ 100% and the rest around 60-80%). It ran overnight and was still "off" in the morning, using the other tools as a reference.

4. The next day, unmanaged the node + remanaged: CPU looks correct now in Orion and all tools.

On another note: We have SAM and it has a custom made powershell script monitor on it assigned to the same server with the issues. Some of this data being pulled is for ephemeral port exhaustion (I've posted my SAM script in the forums) and normally shows port usage of ~2500-3000 ports used (it makes a very stable "spiky saw like wave" when graphed over time). When I had the CPU polling issue, the SAM powershell monitor graph data also multiplied by about 2x with a similar "saw wave" graph. Once I fixed the CPU issue in #4 above, the data being returned for the SAM monitor also returned to normal. So I'm betting Orion was polling certain metrics on this server more than once (and the charting/summation/statistics were affected), and the SAM monitor was also affected in the same way. Other metrics on that server appear to be unaffected. WEIRD.