Anyone else seeing this?
I'm seeing the the 'SolarWinds Information Service V3' service balloon in memory usage 8.7GB and runs the server out of memory and everything crashes. Reboot the server and then it happens again, over and over all day long so far today. Last night I applied HotFix 1 for 12.0.1 and ran well over night. When people are in Orion working during the day it does this. Overnight it seems to run just fine when people are not using the system.
Where do I even start to troubleshoot this?
Running NPM 12.0.1 and SAM 6.3
Server 2012 R2
VM on VMware
Solved! Go to Solution.
Weird. I can say that our deployments are nearly identical and I'm only seeing ~320 MB of memory in use right now. Same versions of both products and a similar VMWare setup. Did it start happening right after an upgrade, or kind of out of the blue? Maybe if you re-run the configuration wizard (if you can do so), otherwise you might have to schedule that.
Its been happening since we went to version 12.
After a reboot that service is around 261.7MB for a bit then it takes off and balloons and crashes quickly to where it takes the services down so we don't get a high memory usage alarm from the system.
Raise a support case, Harry Griffiths (SolarWinds Application Engineer) has identified the problem and has a fix.
This problem killed our upgrade a few weeks ago and ended in a rather hasty rollback.
The case number, with the fix in it was #1079283 "Orion very slow after upgrade, many elements not being displayed".
Support came back with this answer:
The steps are below.
Enable schema caching.
1. Find this file, C:\inetpub\SolarWinds\web.config.
2. Add this key inside the web.config <appSettings> sections(If the key already exists just change the value):
<add key="SwisEntityMacroParserPluginCacheEnabled" value="True" />
3. Hopefully this will enable schema caching which we are using for alerting service.
4. Next Open this file SolarWinds.InformationService.ServiceV3.exe.config
- Should be located in C:\Program Files (x86)\Common Files\Solarwinds\Information Service\V3.
5. Add this key to SolarWinds.InformationService.ServiceV3.exe.config inside <runtime> section (If the key already exists just change the value):
<gcServer enabled="true" />
6. Restart IIS (IISRESET) and SWISv3 to take effect.
I've implemented this on our server. We'll see how it holds up tomorrow when people are back in the Orion console.
I am having the same problem now with NPM version 12.0. The solarwinds information service v3 will suddenly start using up around 3gb memory per minute until it maxes out all of the available memory.
This is happening on our main poller and web server which has 64gb ram, so the information service v3 will use ALL of this. Only way for me to fix this at the moment is to restart the information service.
Support already sent the fix to the web.config file above, but this hasn't made any difference.
It's very frustrating as solarwinds is unusable until I resart the information service. Spent 2 1/2 hours on the phone with support yesterday, but still have the problem.
We've now traced our memory leak to a rogue custom availabillty chart. The chart was showing availabiility over last 30 days for 8 nodes. It appears the chart was somehow corrupt as creating a new chart with the same data didn't trigger the memory leak. We found this by isolating different elements of a view we'd been working on and loading each one in turn until the memory started increasing significantly.
Interestingly, closing the page showing the chart didn't release the memory back that the information service had grabbed. Only way to do this was to kill the information service v3 and let it respawn.
SolarWinds solutions are rooted in our deep connection to our user base in the THWACK® online community. More than 150,000 members are here to solve problems, share technology and best practices, and directly contribute to our product development process. Learn more today by joining now.