13 Replies Latest reply on Sep 11, 2017 7:37 PM by orioncrack

    12.0.1 running out of memory

    mjsweet130

      Anyone else seeing this?

       

      I'm seeing the the 'SolarWinds Information Service V3' service balloon in memory usage 8.7GB and runs the server out of memory and everything crashes. Reboot the server and then it happens again, over and over all day long so far today. Last night I applied HotFix 1 for 12.0.1 and ran well over night. When people are in Orion working during the day it does this. Overnight it seems to run just fine when people are not using the system.

       

      Where do I even start to troubleshoot this?

       

      Running NPM 12.0.1 and SAM 6.3

      Server 2012 R2

      VM on VMware

      12GB RAM

      4vCPU

        • Re: 12.0.1 running out of memory
          mbird

          Weird. I can say that our deployments are nearly identical and I'm only seeing ~320 MB of memory in use right now. Same versions of both products and a similar VMWare setup. Did it start happening right after an upgrade, or kind of out of the blue? Maybe if you re-run the configuration wizard (if you can do so), otherwise you might have to schedule that.

          • Re: 12.0.1 running out of memory
            yaquaholic

            Raise a support case, Harry Griffiths (SolarWinds Application Engineer) has identified the problem and has a fix.

            This problem killed our upgrade a few weeks ago and ended in a rather hasty rollback.

             

            The case number, with the fix in it was #1079283 "Orion very slow after upgrade, many elements not being displayed".

            • Re: 12.0.1 running out of memory
              steshi

              I am having the same problem now with NPM version 12.0. The solarwinds information service v3 will suddenly start using up around 3gb memory per minute until it maxes out all of the available memory.

              This is happening on our main poller and web server which has 64gb ram, so the information service v3 will use ALL of this. Only way for me to fix this at the moment is to restart the information service.

              Support already sent the fix to the web.config file above, but this hasn't made any difference.

               

              It's very frustrating as solarwinds is unusable until I resart the information service. Spent 2 1/2 hours on the phone with support yesterday, but still have the problem.

                • Re: 12.0.1 running out of memory
                  steshi

                  We've now traced our memory leak to a rogue custom availabillty chart. The chart was showing availabiility over last 30 days for 8 nodes. It appears the chart was somehow corrupt as creating a new chart with the same data didn't trigger the memory leak. We found this by isolating different elements of a view we'd been working on and loading each one in turn until the memory started increasing significantly.

                  Interestingly, closing the page showing the chart didn't release the memory back that the information service had grabbed. Only way to do this was to kill the information service v3 and let it respawn.

                  • Re: 12.0.1 running out of memory
                    shaun_9999

                    Weirdly, after making a custom chart, I have the same problem. It spiked at 97%

                    I want to end the service and then restart and see it that resolves it.

                    Urgh - why does s**t have to be so complex sometimes.  I only want a b****y chart and it's gone insane and taking 14gb up.

                  • Re: 12.0.1 running out of memory
                    cobrien

                    We have fixed a couple of memory leaks since 12.0.  The best bet is to install the latest Service Release and Hot Fixes.  If the issue continues, it's time to open a Support case and we'll dig in with you.

                    • Re: 12.0.1 running out of memory
                      orioncrack

                      Experiencing this with 12.1.