23 Replies Latest reply on Jul 1, 2019 3:30 AM by valkos

    SEM Agent Memory issues

    pwheels

      We recently upgraded our SEM appliance from LEM 6.6.0 to SEM 6.7.0. After the upgrade, the appliance went through all of the client nodes (we only use this on windows servers) and upgraded their agents to 6.7.0 as well.

       

      After this, we noticed that some of our servers were running with very high memory & the process was Javaw.exe. Removing the SEM agent (using the uninstaller) also stopped the javaw.exe process and brought the memory right back down. It seems that there might be memory issues/leak with this client? I have now uninstalled the 6.7 agent from all of our nodes (windows 2008R2 and 2012R2) and will be testing reinstalling the previous client (6.6)

       

      Has anyone else has a similar issue & is able to give any advice?

       

      Thanks!

        • Re: SEM Agent Memory issues
          dstrickland

          We are seeing the same issue.  I am not totally convinced it's 6.7 because I am fairly certain we had this issue once before with an older version.  However we did just do the same and upgrade from 6.6 to 6.7.

          1 of 1 people found this helpful
          • Re: SEM Agent Memory issues
            valkos

            Thanks for posting this issue.

            It is important for us who were about to upgrade to 6.7.0 and if it's a bug, I hope it is resolved quickly by SolarWinds.

            1 of 1 people found this helpful
            • Re: SEM Agent Memory issues
              jwckauman

              We are also having the same issue.  We just upgraded to 6.7.  Every server with that agent is having it's memory utilization spike until it hits close to 100%.  It is the javaw.exe process that is spiking. It appears to then crash and then restart again.  For a moment, memory utilization goes back to normal until the next spike. Over and over and over.  THis is a major issue.  Please send a fix.

              1 of 1 people found this helpful
              • Re: SEM Agent Memory issues
                tcox

                We are experiencing a similar issue; however, I don't believe it's directly tied to 6.7. We're seeing it on 6.6 at one site and 6.5 at another. We have been troubleshooting to no avail. An uninstall and reinstall of the agent software worked on a couple of the machines and they stopped reporting the "javaw.exe" event, but several machines are still generating several gigabytes of logs over the same repeated event.

                1 of 1 people found this helpful
                • Re: SEM Agent Memory issues
                  dcokers

                  I actually just came to this forum to see if there was any similar issues.

                   

                  We updated to 6.7 last week. We have seen this memory usage issue on a few servers and PCs. The Javaw.exe is the executable that has the memory issues and that file is located in the Solarwinds folder C:\Windows\SysWOW64\ContegoSPOP\jre6.7.0\bin

                   

                  I haven't been able to open a ticket with support yet, has anyone else gotten feed back from anyone at SW.

                   

                  2 of 2 people found this helpful
                  • Re: SEM Agent Memory issues
                    jhynds

                    Thanks to everyone for reporting this issue. We are currently looking into the issue as a matter of urgency and aiming to resolve ASAP. If you haven't already done so, I'd encourage you to raise a Technical Support ticket if you encountering high CPU on the SEM agent.

                     

                    Update: We have determined the root cause and working on a fix. Thanks for bearing with us and apologies for any inconvenience caused.

                    4 of 4 people found this helpful
                      • Re: SEM Agent Memory issues
                        cnicholas

                        We are getting hit with some vulnerabilities with Java and SEM 6.7.0 agent. Would it be safe to upgrade Java or is it going to break our clients. Will the latest agent fix (when it comes out) resolve any Java issues.

                         

                        This is the vulnerability that is coming up.

                         

                         

                        Oracle Java SE 1.7.x < 1.7.0_211 / 1.8.x < 1.8.0_201 / 1.11.x < 1.11.0_2 Multiple Vulnerabilities (January 2019 CPU)

                        Medium Nessus Plugin ID 121231

                        Synopsis

                        The remote Windows host contains a programming platform that is affected by multiple vulnerabilities.

                        Description

                        The version of Oracle (formerly Sun) Java SE or Java for Business installed on the remote host is prior to 7 Update 211, 8 Update 201, 11 Update 2. It is, therefore, affected by multiple vulnerabilities related to the following components :

                        - An issue in libjpeg 9a, a divide-by-zero error, could allow remote attackers to cause a denial of service condition via a crafted file. (CVE-2018-11212)

                        - An unspecified vulnerability in Oracle Java SE in the Networking subcomponent could allow an unauthenticated, remote attacker with network access via multiple protocols to compromise Java SE.
                        (CVE-2019-2426)

                        - An unspecified vulnerability in Oracle Java SE in the Deployment subcomponent could allow an unauthenticated, remote attacker with network access via multiple protocols to compromise Java SE.
                        (CVE-2019-2449)

                        - An unspecified vulnerability in Oracle Java SE in the Libraries subcomponent could allow an unauthenticated, remote attacker with network access via multiple protocols to compromise Java SE.
                        (CVE-2019-2422)

                        Nessus has not tested for these issues but has instead relied only on the application's self-reported version number.

                        Solution

                        Upgrade to Oracle JDK / JRE 11 Update 2, 8 Update 201 / 7 Update 211 or later. If necessary, remove any affected versions.

                        1 of 1 people found this helpful
                          • Re: SEM Agent Memory issues
                            dcokers

                            Are you sure this is related to the LEM agent.  I have the 6.7 LEM agent installed on 400+ workstations/servers but I only see this Nessus plugin (121231) as vulnerable on two machines and it is because Java is install on these machines, not because of the LEM agent.

                              • Re: SEM Agent Memory issues
                                cnicholas

                                This is the path that is shows up - the installation path for SEM

                                 

                                 

                                Plugin Output:

                                The following vulnerable instances of Java are installed on the

                                remote host :

                                 

                                  Path : C:\Windows\SysWOW64\ContegoSPOP\jre6.3.1.hotfix5\bin\java.exe

                                  Installed version : 1.8.0_131

                                  Fixed version     : 1.6.0_161 / 1.7.0_151 / 1.8.0_141

                                  • Re: SEM Agent Memory issues
                                    dcokers

                                    Interesting, we have the C:\Windows\SysWOW64\ContegoSPOP\6.3.1.hotfix5 folder on our desktops but we do not have the BIN folder.

                                     

                                    The BIN Folder for us is in the C:\Windows\SysWOW64\ContegoSPOP\jre6.7.0\bin folder. There is a java.exe file in there, but no where else in the C:\Windows\SysWOW64\ContegoSPOP directory. Our Nessus isn't showing it as vulnerable.

                                     

                                    • Re: SEM Agent Memory issues
                                      jhynds

                                      If you are running the 6.7 agent, you can safely delete the \jre6.3.1.hotfix directory. This directory is not used by the latest agent and is likely there as a result of an auto-upgrade, but no longer used. The 6.7 agent leverages OpenJDK and does not require Oracle Java. Can you remove the directory, re-run the Nessus scan and confirm that you no longer see the vulnerability?

                                • Re: SEM Agent Memory issues
                                  pabely

                                  Any news on the SEM6.7.1 release to fix the memory issue?

                                    • Re: SEM Agent Memory issues
                                      donk

                                      We are being impacted by the bug ourselves. I have had to stop the roll out of the SEM agent to over 700 Windows servers and stop the agent on about 200 others because of this. It would be nice to tell my users when this fix is expected

                                    • Re: SEM Agent Memory issues
                                      mlentoski

                                      Has there been a fix completed yet?  What is the root cause?

                                       

                                      I'd like an update, if possible, so that I do not have to roll back to 6.6

                                       

                                      Respectfully,

                                       

                                      Mike Lentoski

                                    • Re: SEM Agent Memory issues
                                      mlentoski

                                      We are seeing the same issue, and will likely have to roll back if not fixed.

                                      • Re: SEM Agent Memory issues
                                        tcox

                                        I just want to caution everyone against believing a roll back from 6.7 will fix this issue. As I mentioned above, we're seeing it at two sites on 6.6 and 6.5 as well. Waiting to hear what SolarWinds discovers, but we originally began seeing the issue when running a Windows update as part of a larger package of hotfixes and patches on our networks running SEM back in March.

                                        • Re: SEM Agent Memory issues
                                          jhynds

                                          SEM 6.7.1 is now available on your Customer Portal and includes a fix for the agent memory issue. Apologies for any inconvenience caused as a result of the issue, I understand it was frustrating. Once you upgrade the SEM appliance to 6.7.1, the update will push to your agents (provided auto-update is enabled) and no further action is required in order to resolve the issue.

                                           

                                          Thanks for bearing with us!

                                          3 of 3 people found this helpful