We are now monitoring about 20 instances and IWC/IDC tomcat log says continously
java.lang.OutOfMemoryError: unable to create new native thread
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'waitEventQueueProducer' defined in class path resource [spring/idc-monitor.xml]: Invocation of init method failed; nested exception is java.lang.OutOfMemoryError: unable to create new native thread
I changed max heap size first from 1024 to 2048 and now to (MAX_HEAP_MEMORY OVERRIDE = 3072m, -Xmx3072m). Still we get an errors. First it is I think like this
DEBUG (2016-07-19 14:56:01,239.EEST) [databaseManager-thread-14] [MAGENTAP_YLLAS1-ZONE-VK] MonitoredDatabaseProperties - Status on monitored database id  updated to [STOPPED]
INFO (2016-07-19 14:56:01,240.EEST) [databaseManager-thread-14] [MAGENTAP_YLLAS1-ZONE-VK] DatabaseMonitor - Monitor for database [MAGENTAP_YLLAS1-ZONE-VK] stopped.
WARN (2016-07-19 14:56:01,240.EEST) [databaseManager-thread-14] [MAGENTAP_YLLAS1-ZONE-VK] DatabaseMonitor - Database monitor listener failed: Internal error: destroyed a monitor that was not in the map.
java.lang.IllegalStateException: Internal error: destroyed a monitor that was not in the map.
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
and later it ends to out of memory.
I've recently installed version 10.1.317 (upgraded from dpa_9_1_85). Finally I got this version working, it uses in some strange way ~/tomcat directory instead of real installation directory ~/DPA/dpa/iwc/tomcat (which (~/DPA/dpa) is soft link to real directory ~/DPA/dpa_10_1_317). And still it tried to start itself with java from 9.1.85 version, but finally I founded ~/DPA/dpa/iwc/tomcat/ignite_config and file java_loc.txt. After I changed that I think all things are now running from new place. That ~/tomcat we solved by creating a soft link to home directory pointing to dpa/iwc/tomcat -directory.
We are running linux:
Linux localhost.domani.area 2.6.32-642.1.1.el6.x86_64 #1 SMP Tue May 31 21:57:07 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux