6 Replies Latest reply on Jul 21, 2016 8:39 AM by jaminsql

    DPA: java.lang.OutOfMemoryError: unable to create new native thread

    severi.makela@nls.fi

      Hi,

       

      We are now monitoring about 20 instances and IWC/IDC tomcat log says continously

       

      java.lang.OutOfMemoryError: unable to create new native thread

       

      and

       

      Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'waitEventQueueProducer' defined in class path resource [spring/idc-monitor.xml]: Invocation of init method failed; nested exception is java.lang.OutOfMemoryError: unable to create new native thread

       

       

      etc.

       

      I changed max heap size first from 1024 to 2048 and now to (MAX_HEAP_MEMORY OVERRIDE = 3072m,  -Xmx3072m). Still we get an errors. First it is I think like this

       

      DEBUG  (2016-07-19 14:56:01,239.EEST) [databaseManager-thread-14] [MAGENTAP_YLLAS1-ZONE-VK] MonitoredDatabaseProperties - Status on monitored database id [12] updated to [STOPPED]

      INFO   (2016-07-19 14:56:01,240.EEST) [databaseManager-thread-14] [MAGENTAP_YLLAS1-ZONE-VK] DatabaseMonitor - Monitor for database [MAGENTAP_YLLAS1-ZONE-VK] stopped.

      WARN   (2016-07-19 14:56:01,240.EEST) [databaseManager-thread-14] [MAGENTAP_YLLAS1-ZONE-VK] DatabaseMonitor - Database monitor listener failed: Internal error: destroyed a monitor that was not in the map.

      java.lang.IllegalStateException: Internal error: destroyed a monitor that was not in the map.

              at org.springframework.util.Assert.state(Assert.java:384)

              at com.confio.idc.database.DatabaseManager.databaseMonitorDestroyed(SourceFile:278)

              at com.confio.idc.database.DatabaseMonitor.destroyNotification(SourceFile:1133)

              at com.confio.idc.database.DatabaseMonitor.destroy(SourceFile:445)

              at com.confio.idc.database.job.DatabaseMonitorLifecycleJob.execute(SourceFile:104)

              at org.quartz.core.JobRunShell.run(JobRunShell.java:202)

              at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

              at java.lang.Thread.run(Unknown Source)

       

       

       

      and later it ends to out of memory.

       

      I've recently installed version 10.1.317 (upgraded from dpa_9_1_85). Finally I got this version working, it uses in some strange way ~/tomcat directory instead of real installation directory ~/DPA/dpa/iwc/tomcat (which (~/DPA/dpa) is soft link to real directory ~/DPA/dpa_10_1_317). And still it tried to start itself with java from 9.1.85 version, but finally I founded ~/DPA/dpa/iwc/tomcat/ignite_config and file java_loc.txt. After I changed that I think all things are now running from new place. That ~/tomcat we solved by creating a soft link to home directory pointing to dpa/iwc/tomcat -directory.

       

      We are running linux:

       

      Linux localhost.domani.area 2.6.32-642.1.1.el6.x86_64 #1 SMP Tue May 31 21:57:07 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

       

      Regards

      Severi Mäkelä

        • Re: DPA: java.lang.OutOfMemoryError: unable to create new native thread
          jaminsql

          It sounds like you were able to get this running in the end then but, that DPA was installing and pointing to the other version of Java. I am not sure if I followed if you still need help here or not?

          Are you still seeing the Out of memory errors? Check the system.log file in tomcat/logs to see what your start up parameters are there. It would look something like this. Though this is taken from a local windows install.

           

          -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=C:\Program Files (x86)\Confio\Ignite PI\iwc\tomcat\conf\logging.properties -Dcom.sun.management.jmxremote exit -Xms256m -Xmx1024m -Xss256k

           

          This will tell us for sure if DPA is starting with the memory settings you gave it. If you need much help on this one enter a support case also.

            • Re: DPA: java.lang.OutOfMemoryError: unable to create new native thread
              severi.makela@nls.fi

              Hi,

               

              Yes, you are right, I was able to run it with proper version etc.

               

              Still we are encountering problems, some monitors are on stopped mode and they are saying

               

              "Monitor Attempting to Start - see log"

               

              on DPA main screen.

               

              In the tomcat idc.log:

               

              WARN   (2016-07-20 08:13:20,262.EEST) [repositoryManager-thread-33] [TKESKUSP_YLLAS1-ZONE-VK] DatabaseManager - Stopping monitor for database [TKESKUSP_YLLAS1-ZONE-VK] because one or more jobs are not running.  Attemtping to retry...

              java.lang.Throwable

                      at com.confio.idc.database.DatabaseManager.updateDatabaseMonitors(SourceFile:201)

                      at com.confio.idc.repo.RepositoryMonitor.queryRepo(SourceFile:487)

                      at com.confio.idc.repo.job.RepoPropsJob.execute(SourceFile:15)

                      at org.quartz.core.JobRunShell.run(JobRunShell.java:202)

                      at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

                      at java.lang.Thread.run(Unknown Source)

               

              WARN   (2016-07-20 08:30:14,732.EEST) [repositoryManager-thread-44] TrendSummarizerManager - Unexpected exception occurred. Execution trigger (monitor-job-group-5.PublishJob).

              java.lang.OutOfMemoryError: unable to create new native thread

                      at java.lang.Thread.start0(Native Method)

                      at java.lang.Thread.start(Unknown Source)

                      at java.util.concurrent.ThreadPoolExecutor.addWorker(Unknown Source)

                      at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)

                      at org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor.execute(ThreadPoolTaskExecutor.java:241)

                      at com.confio.idc.trending.TrendSummarizerManager.jobWasExecuted(SourceFile:95)

                      at org.quartz.core.QuartzScheduler.notifyJobListenersWasExecuted(QuartzScheduler.java:1886)

                      at org.quartz.core.JobRunShell.notifyJobListenersComplete(JobRunShell.java:356)

                      at org.quartz.core.JobRunShell.run(JobRunShell.java:226)

                      at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

                      at java.lang.Thread.run(Unknown Source)

               

              From system.log:

              InputArguments: -Djava.util.logging.config.file=/users/tutti/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xss264k -Xms128m -Xmx3072m -XX:MaxPermSize

              =192m -Dcom.sun.management.jmxremote -Djava.net.preferIPv4Stack=true -Dnet.sf.ehcache.skipUpdateCheck=true -Djava.awt.headless=true -Djava.endorsed.dirs=/users/tutti/tomcat/endorsed -Dcatalina.base=/us

              ers/tutti/tomcat -Dcatalina.home=/users/tutti/tomcat -Djava.io.tmpdir=/users/tutti/tomcat/temp

               

              So it seems that it has get my new parameters for memory. We added one core more to virtual machine (now it has 4) and system memory 10G (it was 8G). This machine has other things also to do, some times top says it is fully utilized (at now: Cpu(s): 77.9%us, 20.3%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  1.9%si,  0.0%st).

               

              But I think this should not indicate with error outofmemory.

               

              Regards

              Severi

              • Re: DPA: java.lang.OutOfMemoryError: unable to create new native thread
                severi.makela@nls.fi

                Hi again,

                 

                And now hole idc has went down.

                 

                WARN   (2016-07-20 14:41:39,025.EEST) [timerFactory] ManagedRepository - IDC Heartbeat is stale - it has been 1636.0 seconds since the last one.

                 

                is in iwc.log.

                 

                WARN   (2016-07-20 14:07:50,259.EEST) [repositoryManager-thread-33] [MAGENTAT_HAMMAS] DatabaseMonitor - The job [QuickPollJob] has not been run in [609] seconds, and its tolerance is set for [600].

                WARN   (2016-07-20 14:07:50,259.EEST) [repositoryManager-thread-33] [MAGENTAT_HAMMAS] DatabaseManager - Stopping monitor for database [MAGENTAT_HAMMAS] because one or more jobs are not running.  Attemtping to retry...

                java.lang.Throwable

                        at com.confio.idc.database.DatabaseManager.updateDatabaseMonitors(SourceFile:201)

                        at com.confio.idc.repo.RepositoryMonitor.queryRepo(SourceFile:487)

                        at com.confio.idc.repo.job.RepoPropsJob.execute(SourceFile:15)

                        at org.quartz.core.JobRunShell.run(JobRunShell.java:202)

                        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

                        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

                        at java.lang.Thread.run(Unknown Source)

                DEBUG  (2016-07-20 14:07:50,275.EEST) [databaseManager-thread-8] [MAGENTAT_HAMMAS] DatabaseMonitor - Destroying monitor for database [MAGENTAT_HAMMAS]

                DEBUG  (2016-07-20 14:24:30,444.EEST) [databaseManager-thread-8] [MAGENTAT_HAMMAS] MonitoredDatabaseProperties - Status on monitored database id [15] updated to [STOPPED]

                INFO   (2016-07-20 14:24:30,444.EEST) [databaseManager-thread-8] [MAGENTAT_HAMMAS] DatabaseMonitor - Monitor for database [MAGENTAT_HAMMAS] stopped.

                 

                is in idc.log.

                 

                Regards

                Severi Mäkelä