Community
- Command Central
- MVP Program
- Monthly Mission
- Blogs
- Groups
- Events
- Media Vault
Products
- Observability
- Network Management
- Application Management
- IT Security
- IT Service Management
- System Management
- Database Management
Content Exchange
- SolarWinds Platform
- Server & Application Monitor
- Database Performance Analyzer
- Server Configuration Monitor
- Network Performance Monitor
- Network Configuration Manager
- SQL Sentry
- Web Help Desk
Free Tools & Trials
Store

DPA: java.lang.OutOfMemoryError: unable to create new native thread

Hi,

We are now monitoring about 20 instances and IWC/IDC tomcat log says continously

java.lang.OutOfMemoryError: unable to create new native thread

and

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'waitEventQueueProducer' defined in class path resource [spring/idc-monitor.xml]: Invocation of init method failed; nested exception is java.lang.OutOfMemoryError: unable to create new native thread

etc.

I changed max heap size first from 1024 to 2048 and now to (MAX_HEAP_MEMORY OVERRIDE = 3072m, -Xmx3072m). Still we get an errors. First it is I think like this

DEBUG (2016-07-19 14:56:01,239.EEST) [databaseManager-thread-14] [MAGENTAP_YLLAS1-ZONE-VK] MonitoredDatabaseProperties - Status on monitored database id [12] updated to [STOPPED]

INFO (2016-07-19 14:56:01,240.EEST) [databaseManager-thread-14] [MAGENTAP_YLLAS1-ZONE-VK] DatabaseMonitor - Monitor for database [MAGENTAP_YLLAS1-ZONE-VK] stopped.

WARN (2016-07-19 14:56:01,240.EEST) [databaseManager-thread-14] [MAGENTAP_YLLAS1-ZONE-VK] DatabaseMonitor - Database monitor listener failed: Internal error: destroyed a monitor that was not in the map.

java.lang.IllegalStateException: Internal error: destroyed a monitor that was not in the map.

at org.springframework.util.Assert.state(Assert.java:384)

at com.confio.idc.database.DatabaseManager.databaseMonitorDestroyed(SourceFile:278)

at com.confio.idc.database.DatabaseMonitor.destroyNotification(SourceFile:1133)

at com.confio.idc.database.DatabaseMonitor.destroy(SourceFile:445)

at com.confio.idc.database.job.DatabaseMonitorLifecycleJob.execute(SourceFile:104)

at org.quartz.core.JobRunShell.run(JobRunShell.java:202)

at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

and later it ends to out of memory.

I've recently installed version 10.1.317 (upgraded from dpa_9_1_85). Finally I got this version working, it uses in some strange way ~/tomcat directory instead of real installation directory ~/DPA/dpa/iwc/tomcat (which (~/DPA/dpa) is soft link to real directory ~/DPA/dpa_10_1_317). And still it tried to start itself with java from 9.1.85 version, but finally I founded ~/DPA/dpa/iwc/tomcat/ignite_config and file java_loc.txt. After I changed that I think all things are now running from new place. That ~/tomcat we solved by creating a soft link to home directory pointing to dpa/iwc/tomcat -directory.

We are running linux:

Linux localhost.domani.area 2.6.32-642.1.1.el6.x86_64 #1 SMP Tue May 31 21:57:07 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Regards

Severi Mäkelä

Find more posts tagged with

Accepted answers

All comments

jaminsql

It sounds like you were able to get this running in the end then but, that DPA was installing and pointing to the other version of Java. I am not sure if I followed if you still need help here or not?

Are you still seeing the Out of memory errors? Check the system.log file in tomcat/logs to see what your start up parameters are there. It would look something like this. Though this is taken from a local windows install.

-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=C:\Program Files (x86)\Confio\Ignite PI\iwc\tomcat\conf\logging.properties -Dcom.sun.management.jmxremote exit -Xms256m -Xmx1024m -Xss256k

This will tell us for sure if DPA is starting with the memory settings you gave it. If you need much help on this one enter a support case also.

severi.makela

Hi,

Yes, you are right, I was able to run it with proper version etc.

Still we are encountering problems, some monitors are on stopped mode and they are saying

"Monitor Attempting to Start - see log"

on DPA main screen.

In the tomcat idc.log:

WARN (2016-07-20 08:13:20,262.EEST) [repositoryManager-thread-33] [TKESKUSP_YLLAS1-ZONE-VK] DatabaseManager - Stopping monitor for database [TKESKUSP_YLLAS1-ZONE-VK] because one or more jobs are not running. Attemtping to retry...

java.lang.Throwable

at com.confio.idc.database.DatabaseManager.updateDatabaseMonitors(SourceFile:201)

at com.confio.idc.repo.RepositoryMonitor.queryRepo(SourceFile:487)

at com.confio.idc.repo.job.RepoPropsJob.execute(SourceFile:15)

at org.quartz.core.JobRunShell.run(JobRunShell.java:202)

at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

WARN (2016-07-20 08:30:14,732.EEST) [repositoryManager-thread-44] TrendSummarizerManager - Unexpected exception occurred. Execution trigger (monitor-job-group-5.PublishJob).

java.lang.OutOfMemoryError: unable to create new native thread

at java.lang.Thread.start0(Native Method)

at java.lang.Thread.start(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.addWorker(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)

at org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor.execute(ThreadPoolTaskExecutor.java:241)

at com.confio.idc.trending.TrendSummarizerManager.jobWasExecuted(SourceFile:95)

at org.quartz.core.QuartzScheduler.notifyJobListenersWasExecuted(QuartzScheduler.java:1886)

at org.quartz.core.JobRunShell.notifyJobListenersComplete(JobRunShell.java:356)

at org.quartz.core.JobRunShell.run(JobRunShell.java:226)

at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

From system.log:

InputArguments: -Djava.util.logging.config.file=/users/tutti/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xss264k -Xms128m -Xmx3072m -XX:MaxPermSize

=192m -Dcom.sun.management.jmxremote -Djava.net.preferIPv4Stack=true -Dnet.sf.ehcache.skipUpdateCheck=true -Djava.awt.headless=true -Djava.endorsed.dirs=/users/tutti/tomcat/endorsed -Dcatalina.base=/us

ers/tutti/tomcat -Dcatalina.home=/users/tutti/tomcat -Djava.io.tmpdir=/users/tutti/tomcat/temp

So it seems that it has get my new parameters for memory. We added one core more to virtual machine (now it has 4) and system memory 10G (it was 8G). This machine has other things also to do, some times top says it is fully utilized (at now: Cpu(s): 77.9%us, 20.3%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 1.9%si, 0.0%st).

But I think this should not indicate with error outofmemory.

Regards

Severi

severi.makela

Hi again,

And now hole idc has went down.

WARN (2016-07-20 14:41:39,025.EEST) [timerFactory] ManagedRepository - IDC Heartbeat is stale - it has been 1636.0 seconds since the last one.

is in iwc.log.

WARN (2016-07-20 14:07:50,259.EEST) [repositoryManager-thread-33] [MAGENTAT_HAMMAS] DatabaseMonitor - The job [QuickPollJob] has not been run in [609] seconds, and its tolerance is set for [600].

WARN (2016-07-20 14:07:50,259.EEST) [repositoryManager-thread-33] [MAGENTAT_HAMMAS] DatabaseManager - Stopping monitor for database [MAGENTAT_HAMMAS] because one or more jobs are not running. Attemtping to retry...

java.lang.Throwable

at com.confio.idc.database.DatabaseManager.updateDatabaseMonitors(SourceFile:201)

at com.confio.idc.repo.RepositoryMonitor.queryRepo(SourceFile:487)

at com.confio.idc.repo.job.RepoPropsJob.execute(SourceFile:15)

at org.quartz.core.JobRunShell.run(JobRunShell.java:202)

at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

DEBUG (2016-07-20 14:07:50,275.EEST) [databaseManager-thread-8] [MAGENTAT_HAMMAS] DatabaseMonitor - Destroying monitor for database [MAGENTAT_HAMMAS]

DEBUG (2016-07-20 14:24:30,444.EEST) [databaseManager-thread-8] [MAGENTAT_HAMMAS] MonitoredDatabaseProperties - Status on monitored database id [15] updated to [STOPPED]

INFO (2016-07-20 14:24:30,444.EEST) [databaseManager-thread-8] [MAGENTAT_HAMMAS] DatabaseMonitor - Monitor for database [MAGENTAT_HAMMAS] stopped.

is in idc.log.

Regards

Severi Mäkelä

jaminsql

Severi,

Please just enter a support case. We can then read the full logs or join you on a call and get DPA running again.

severi.makela

Hi,

I would, but your system says "Your ticket was unable to be submitted at this time due to an error. Please try again. We apologize for any inconvenience."

Do you have max length for message field "Finalize Ticket Details for Submission."?

Regards

Severi