JMX monitoring - timeout on initial tests?

I'm currently setting up a proof of concept/demo for Solarwinds using our development and test servers, to convince my organization of the value of the product. One of the things I want to do is track our java processes and programs, of which we have quite a few.

My first target is of course a bit special: An instance of Sonatype Nexus running in a docker container on RHEL 7.

I've been following the directions in the various articles to try and get this going.

Configure and integrate JMX monitoring in SAM - SolarWinds Worldwide, LLC. Help and Support

This step seems 100% to work. I'm able to remotely connect to the JMX instance with JConsole on my local PC, as well as with the jmxterm software.

https://thwack.solarwinds.com/docs/DOC-192291

This step is where I run into trouble. When I try to assign the application monitor template, I have left the "Java Virtual Machine (JMX)" template on its defaults. After 5 minutes, I get a message: "Test failed with 'Unknown' status [...] timeout while executing job"

Yet throughout the 5 minutes I am waiting for the timeout, my jconsole session is running and is returning statistics very rapidly.

The target is using a linux agent for monitoring the node. I've configured it to also monitor the docker container successfully. The only error I see with the node is that the status of the loopback interface is unknown. While that is annoying, I can't see it impacting this.

Is there some place I can investigate this more? Maybe some logging? I am just not sure where the breakdown might be happening.

Here's the run command for the docker image:

sudo docker run -d -p 8081:8081 --name nexus -v /nexus-data/:/nexus-data -v /nexus-root:/nexus-root -e INSTALL4J_ADD_VM_PARAMS="-Xms2073M -Xmx2073M -XX:MaxDirectMemorySize=2703M -Dcom.sun.jndi.ldap.object.disableEndpointIdentification=true -Djava.util.prefs.userRoot=/nexus-root -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.rmi.port=8686 -Dcom.sun.management.jmxremote.port=8686 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=10.110.3.123 -Dcom.sun.management.jmxremote.local.only=false" -p 8686:8686 --ulimit nofile=65536:65536 registry.connect.redhat.com/sonatype/nexus-repository-manager

Find more posts tagged with

Accepted answers

mesverrum

The kb for the JMX monitor also mentions that there are some extra steps for agent based monitoring

JMX monitor

Confirm those steps have been taken, I would also try polling the sam monitor without the agent by going to the advanced options at the top of the template and changing the preferred polling method to see if you have any better success

All comments

mesverrum

Are you able to do any other linux templates on that node, such as polling for a process or running a script monitor? Does the agent currently display all the cpu/memory metrics and such that you would expect?

I would want to kind of start from the beginning, is the agent working for just it's native functions, then narrow down what it does and does not do correctly.

mesverrum

The kb for the JMX monitor also mentions that there are some extra steps for agent based monitoring

JMX monitor

ahbrook

Right now, the node is running the Red Hat agent and is successfully polling Status & Response time, most of the volumes (it isn't monitoring the docker volumes), and all the interfaces. Everything's healthy except for loopback, which is unknown.

I'm currently listed as having 3 applications running: Linux CPU monitoring Perl, Linux Memory Monitoring Perl, and Nagios File and Directory Count Script. I'm extremely new at this, so I probably threw those on without realizing that the basic agent could collect that information.

I'm also running docker container monitoring, which has 3 active containers: Orion-OrionAggregator, Weaveworks_scope, and nexus (which holds the app I want to monitor). The monitor also picked up the 9 times I killed and restarted the docker containers trying to get the Nexus container into the state I wanted above, but I am hoping those will remove themselves from the view with time.

I can try setting up the system to monitor the host via net-SNMP. I'll report back how that goes.

Edit: Wow, that did the trick! Switching to SNMP based polling allowed the system to read the JMX of the console properly. I really, REALLY don't want to install websphere on every one of our agent-monitored linux boxes, so I'll put forward recommendations that we should use SNMP monitoring for linux going forward.