WebSphere monitoring behind firewall

When you want monitor your WebSphere server or other Java application server which is in DMZ or behind firewall strict rules, you will get in troubles. It could be problem to setup JMX remoting in such case.

Usually you setup such parameter for JVM to enable JMX remoting:

-Dcom.sun.management.jmxremote.port=9910

RMI registry will be started and listen on the port specified, but RMI only uses the specified port for the registry connection. The actual connection will be made on a random port. This of course leads to problems as you won’t be able to open the right port on firewall.

If you use Java 7 update 25 or later you can use other JVM parameter, to fix the other port opened to not be random and to be the same as the first one:

-Dcom.sun.management.jmxremote.port=9910

-Dcom.sun.management.jmxremote.rmi.port=9910

When you can't update Java runtime to the newer version or the solution doesn't work for you for any other reason, you can use other approach to monitor your Java Application server running linux - the aproach is SNMP monitor.

There is template ready for use, it should be available in your SAM already - Java Application Server (SNMP)

In this page https://thwack.solarwinds.com/docs/DOC-142699 you can find ZIP file for donload. It contains also document - APM_Java_Application_Server_(SNMP)_Template_Pack.pdf , which describes SNMP configuration for differend kinds of Java App servers.

You need to ensure two steps:

  1. make node monitored via SNMP - by default SNMP is not enabled in most linux systems

  2. Configure WebSphere to publish vital statistics from Java Virtual Machines (JVM) via SNMP using the same security settings as in step 1

Regarding first step - guide how to enable SNMP for CentOS can be found here http://www.liquidweb.com/kb/how-to-install-and-configure-snmp-on-centos/

Don't forget to enable right ports on firewall to pass this SNMP trafic. By default SNMP uses ports 161 and 162

Then add the node into Orion with polling method "Most Devices: SNMP and ICMP".

Set community string to match what you configured in /etc/snmp/snmpd.conf

The second step is little bit more complicated:

IBM WebSphere uses its own JDK, installed at %WEBSHERE_HOME%\java (where %WEBSHERE_HOME% is the path to your WebSphere installation).Configure the IBM JDK in the following manner:

1. Go to the $JAVA_HOME/jre/lib/management folder.
2. Rename the snmp.acl.template file to snmp.acl.
3. Edit snmp.acl by replacing its content with the following:
   acl = {
   {
   communities = SAME_COMMUNITY_STRING_AS_CONFIGURED_FOR_SYSTEM_SNMP_DAEMON
   access = read-only
   managers = localhost, APM_SERVER
   }
   }
   Where APM_SERVER is the hostname of your APM server.
4. Grant access to the snmp.acl file only for the root user. Run the following commands as the root user in your shell:
   # chown root.root $JAVA_HOME/jre/lib/management/snmp.acl
   # chmod 600 $JAVA_HOME/jre/lib/management/snmp.acl
   Without this change SNMP doesn't work and fails with exception "Password file read access must be restricted"
   
5. Run the IBM WebSphere Application Server.
6. Open a web browser and then navigate to: https://hostname:9043 where hostname is the name of the IBM WebSphere server.
7. In the left panel, click Expand Servers and Server types.
8. Click WebSphere Application Servers.
9. In the main window, click your server.
10. In the Server Infrastructure section, expand Java and Process Management.
11. Click Process Definition.
12. In the Additional Properties section, click Java Virtual Machine.
13. In Generic JVM Arguments, add the following:
    -Dcom.sun.management.snmp.port=1161 -Dcom.sun.management.snmp.interface=0.0.0.0
14. Click OK.
15. Click, Save and then click OK.
16. Click Save.
17. Restart the IBM WebSphere Application Server.

Don't forget to enable the port from step 13 on firewall - here it was port number 1161

Then assign template "Java Application Server (SNMP)" to the node

You should recevie such picture about Java vital statistics on Application server - tested with WebLogic version 8.5.5

if there are some stats redundant and you are not interested in them, then you can modify template or application to fit your needs - remove components as necessary.

Troubleshooting:

If any errors occur during start up of the SNMP agent, the Java VM will throw an exception and exit.

Configuration errors include the following:

• Failure to bind to the port number - Error: Failed to start SNMP adaptor with address: /0.0.0.0:1161 (Address already in use)
• The password file is readable by anyone other than the owner.
• Invalid snmp.acl file - Error: Exception thrown by the agent : Encountered .... at line ...., column ....

Example of error message in logfile, when wrong access level to snmp.acl file granted: