After Update to 2022.3 main polling engine consuming more and more memory until it crashes

Hey Everyone,

we have updated our orion to 2022.3 now our main polling engine doesn't stop consuming Memory/RAM before the update 12-13 GB where enough for the whole Machine.
Now the usage climbes to the maximum of the Machine (28GB) until it crashes. I had to write a powershellskript, wich restarts the services before crashing, to keep orion alive.

we already disabled a lot of unnessesary components to delay the restarts a bit. the sql server works fine (cpu15-20%, ram80-85%)

Has anyone an ida why that could be ?

Find more posts tagged with

sam

orion

ram_utilization.

ipam

npm

main poller

Comments

hjarriell

Have you looked at task manager to see if a specific process is running high? Have you tried running a repair in the config wizard?

JayDee101

Hi Chris,

I have the same issue here since 2022.3 update. I have a case open with Solarwinds for it. For now, what they have me do, did not solve the problem. I'll keep you in touch if we figured something.

ChristophP

Hi JayDee,

good to know have a case open too but the solutions didn't helped too.

ChristophP

Hi hjarriell,

my restart skript loggs the utilisation of the processes before restarting them the last time it looked like this.
running the config wizard didn't changed anything when i know that right that is only configuring the Webpage

Count:1 RAMGB:9.39 Processname:SolarWinds.InformationService.ServiceV3
Count:1 RAMGB:1.75 Processname:SolarWinds.Collector.Service
Count:3 RAMGB:1.65 Processname:SWJobEngineSvc2
Count:2 RAMGB:0.65 Processname:SolarWinds.ServiceHost.Process
Count:7 RAMGB:1.15 Processname:SolarWinds.BusinessLayerHostx64
Count:46 RAMGB:4.54 Processname:SolarWinds.BusinessLayerHost
Count:1 RAMGB:0.21 Processname:SolarWinds.Alerting.Service
Count:11 RAMGB:1.48 Processname:SWJobEngineWorker2x64
Count:1 RAMGB:0.15 Processname:SolarWinds.Orion.LogMgmt.TrapService
Count:7 RAMGB:0.8 Processname:SWJobEngineWorker2
Count:1 RAMGB:0.14 Processname:SolarWinds.Orion.LogMgmt.PollingService
Count:1 RAMGB:0.13 Processname:SolarWinds.Orion.LogMgmt.SyslogService
Count:1 RAMGB:0.11 Processname:SolarWinds.Recommendations.Service
Count:1 RAMGB:0.1 Processname:SolarWinds.HighAvailability.Service
Count:1 RAMGB:0.03 Processname:SolarWinds TFTP Server
Count:1 RAMGB:0.02 Processname:SolarWinds.Agent.Service
Count:1 RAMGB:0.01 Processname:SolarWinds.Agent.Discovery.Plugin

ChristophP

Hi JayDee,

I don't know if you have the same problem but additionally to that problem our Database Maintanance doesn't run automatically at the moment maybe have an eye on that. Our environment didn't started polling after the last script restart. running the maintanance helped.

JayDee101

Hi Chris,

We did not experience that on our side but i'll keep an eye on it. The problem on our side is not on our main polling engine but on some of our additionnal polling engine(we have 5 more in total)

nt7943

please follow the below steps.

Please back up the Orion DB before running the SQL Query.

Open a Remote Desktop connection to the Orion Server.
Click on Database Manager in the SolarWinds Orion start menu group
Click Add default server.
Expand the Orion database.
Right-click and enter New Query.
Enter the SQL query

insert into Setting (Name, Value) Values ('Feature.Client.DefaultCacheExpiration', '01:00:00')

ChristophP

Hi nt7943,
thank you for that infomration i would like get a confirmation from the support,

can you tell me where you got this information from? do you maybe have a ticketnumber?

JayDee101

Hi Chris, Solarwinds are still working on my ticket but in the meantime, i updated to version 2022.4.0 RC2 on our main Orion Server and it seems to have resolved my issue.

Our others Poller Engine are still at 2022.3 and the issue is there. So i sent to support the logs files of both version so they maybe can figure it out what is causing the issue but i am planning to update all of our Polling Engine this week.

Regards,

JayDee101

Hi Chris,

Solarwinds dev team have identified the issue on our side and will provide a hotfix in the next few weeks to solve this problem. Here is the answer from dev team:

We believe that we identified possible root cause for the SolarWinds Information Service memory issue in the 2022.3 and 2022.4 releases connected to the finalizer thread getting stuck on some of the non-default fully en-us environments. The root cause seems to be COM interop threading configuration for SolarWinds Information Service and Business Layer Host executables. To modify the behavior of those executables a change in code is necessary. We are currently preparing buddy drops for the 2022.3 and 2022.4 releases with primary focus on the 2022.3 release first. Buddy drop preparation will take some time and we expect to have it ready during next week. Please note that the code change worked on our internal vRA machine where we were investigating the issue intensively in the last weeks, but it still may not be enough for the customer environments. Its main purpose is to verify that the fix will work and resolve the issue. We will update you once the buddy drop is available

ChristophP

Hi JayDee,

Thank you for your updates, we updated last week to 2022.4.0 RC 2 too but we had the same problem. Yesterday i had a call with the Solarwinds support and there i got a similar information.

yherrera

Has anyone had any response from solarwinds factory or other solution?

ChristophP

Hi Yherrera we switched from a German to an EN - US Operating System that fixed the RAM Issue for us.