CPU Issues After SAM 6.2.3 Install

So a little background. Recently I was installing an APE, and to do so I had to install SAM 6.2.3 over itself on the main poller. This was because apparently the version of 6.2.3 we had on the main poller was slightly lower than the SAM 6.2.3 version of the APE that was available for download. Anyways, ever since I did this on the main poller I have been experiencing huge issues with performance on the main poller. After about 45 minutes of the server being booted up and polling the CPU spikes and the server basically becomes unusable. Polling continues, but agents will fail to report in, and the web page takes forever to load. The task manager shows the SolarWinds.BusinessLayerHost.exe consuming the large amounts of CPU 40% - 60%.

Has anyone ever seen this happen? I'm running NPM 11.5.3 with the latest hotfixes. I'm also running the latest hotfixes for Orion Core. I've already tried the following:

Cleared out C:\Windows\Temp
Reinstalled the core collector services.
Reinstalled the JobEngines services.
Reinstalled the InformationServices services.
Cleared the information services subscribers: https://support.solarwinds.com/Success_Center/Network_Performance_Monitor_(NPM)/Clearing_Information_Service_Subscriptions
Ran the configuration wizard with services/DB/website checked.

I have an open case with support and they seem to be just as stumped as I am, which probably isn't a good sign.

Find more posts tagged with

Accepted answers

All comments

chadsikorra

I'm also seeing a lot of these in the SAM probes logs:

SolarWinds.APM.Common.Diagnostics.PerformanceCounters - Unable to get lock for initialization of performance counters SolarWinds: SAM Jobs

WARN SolarWinds.APM.Probes.ApplicationJobBase - JobEngineWorker running for '00:23:25.8220862' - requesting worker process

I rebuilt the performance counters per this KB: Performance Counters are not working or missing in PerfMon - SolarWinds Worldwide, LLC. Help and Support . Still no luck

chadsikorra

As a side note, I've been extremely disappointed with SolarWinds support on this issue. I've consistently had to reach out to them to get status updates, otherwise the ticket would go a full day or more without even a call or email from them. This is the complete opposite of the support I've received for issues I've opened with STM. It also seems like support has no idea what to do/look for towards fixing the issue. Meanwhile our main SolarWinds server is pretty much crippled a good portion of the day from this. I'm considering just rebuilding the server at this point.