Strange JMX monitor behavior. Flapping between up and unknown.

Hi.  If you've opened a case with support, can you share the number here? 

Hi @bmrad

Yes, the case number is in the 2nd paragraph.

Case # - 01583316

Update:

We found the cause of this, our firewall rule was blocking ICMP but allowing WMI for our APE.

This leaves me with the question. Does JMX monitoring rely on a successful ping before it queries WMI?

I commend Solarwinds support for being patient and helping us with optimizing our environment.

So just an update on this, the issue was not completely resolved after allowing ICMP. We are still seeing applications flap between up and unknown, the issue is with SW engineering team as we are not the only client to report this. We are waiting patiently for feedback.

So the JMX monitors dont use ICMP or WMI.  The java application gets configured to allow JMX polling on a particular port and it sets up its own listener and the java app owners configure it for how they want it to handle authentication, could be a local user, ldap, or with an SSL cert.  So then SW uses that JMX bridge service you mentioned to handle the requests (except on linux agent based nodes where they poll jmx locally via the agent).  It's a separate thing completely than the usual WMI and ICMP polling for the rest of the server info. 

I would wager that the bug you are encountering lives inside that service and something is causing it to be inconsistent about how it polls or processes this java data.  Is there a reason you haven't upgraded to a 2024 release yet?  Its possible that whatever issue you are seeing may be fixed by now.

We did attempt upgrading to 2024 release however the JMX bridge service failed to start on the MPE, so we were forced to rollback. Haven't had the chance to clone the system and then attempt the upgrade again and then get the logs as to why the service didn't want to start but will in due to if SW support engineers don't get back to us with some info how to fix this issue.

Are you have any update of this? I just face with your same issue. As review the APE log I just found message with 2 cause as below.

1) "Unexpected error occurred. The underlying connection was closed: A connection that was expected to be kept alive was closed by the server."

2) "Unexpected error occurred. Connection doesn't exist, it may have been closed."

No update yet. Expected to have a meeting with SW support this week.

Your issue looks like it could be port related? Perhaps port exhaustion on the APE's or MPEs.

This issue is still unresolved 6 months later. We have requested a meeting with Engineering as they are saying this is an environmental issue, I am waiting for feedback regarding this.

Hi, any updates on this issue. We are experiencing the same flapping behavior in JMX components. I have noticed that it occurs much more often when some of them do not respond to queries because the service is down or similar. Looking at the jmxservice logs I see that indeed some of them do not respond because they are down.