High CPU usage on Solarwinds server, Solarwinds Information Service V3 using 100% CPU, empty groups

We are working on some performance changes in the next version. Until then...

Try changing:

C:\Program Files (x86)\SolarWinds\Orion\Information Service\3.0\Plugins\SolarWinds.Data.Providers.Orion.Containers.v3.dll.config

    <add key="MaxThreadPoolWorkerThreads" value ="10"/>

That is probably 100 by default.

Restart the information service v3 after the change.

let me know if it helps.

Thank you Karlo.

I believe the Groups cause the performance issue. Please see the attached screenshot.

Before I added 34 empty groups there are about 200 groups, members are servers grouped by theirs roles. Most of them are static groups and a few dynamic (less 20) groups. After I added those empty groups, CPU usage reached 100%. So I removed them and the CPU usage was back to 40-60%. Further I removed 5 dynamic groups and then the CPUI usage drop to 30-40%

Looks like Groups cause the performance issue, especially the empty groups & dynamic groups.

As the CPU usage is low now I did not apply the change you suggested yet.  But I was planning to create another 200 dynamic groups based on node locations. I will start to do it gradually and see the performance and apply the change you suggested if necessary.

Thanks again!

Message was edited by: Patrick Huang I also reducing the group refresh rate from 60 seconds to 600 seconds

I seem to have this issue after upgrading to 10.4.2, however CPU on Poller 2 seems to be the only one bouncing against the 100% ceiling on the CPU.

I have a Primary Poller and Two Additional Pollers. I am running NPM, NTA, NCM, and VQNM.

The groups do cause higher CPU utilization in SWISv3. Dynamic groups are more taxing on SWIS and the database as they require more processing, but we are trying to optimize that as we find places to improve that in each release.

Also, if you haven't done so already...

In this file "...\SolarWinds\Orion\Solarwinds.Orion.Core.Dependencies.dll.config"

Please switch option ContainersEnableCache to true

<add key="ContainersEnableCache" value="True"/>

- You need to do that on each engine.

Restart the Collector DataProcessor service after this is changed (no need to do this if the option is already true.)

I believe that the above change I recommended for the containers thread count (MaxThreadPoolWorkerThreads) would help your performance even now.

Hi M Gibson,

Depending on what services are causing your CPU to be high, my above recommendations may help your CPU as well.  If it is Job Engine or Job Engine workers, then your CPU utilization will require some more investigation from support and possibly development.

Thanks

Thank you Karlo again. I have applied the changes you suggested on both pollers. I will let you know the results next week. Thanks Patrick

Any progress here?  Just checking to see if the recommendations have helped your overall CPU utilization.

It seems no more improvement as the CPU usage is already low (30-40%) Please see attached screen shots

Another question: The additional poller CPU is quite high. Is it normal? Additional poller is polling about 900 servers Thanks

Disappointing that your main poller did not have a reduction in CPU.  What are your empty groups looking like?  Are they a dynamic query on a custom property?

What processes are using up the CPU on your additional poller?  Collector DataProcessor?  Job Engine Worker?  If the Job Engine Worker, then view the Command Line column in the Task Manager and say which workers are causing the problem (NPM Interfaces, Core, SAM,...)

Thanks for reply. Empty groups were empty, no any menbers. On additional poller, it seems SAM JobEngineWorker2 use most CPU usage, but not more than 50%. Please see the screenshots Thanks