Community
- Command Central
- MVP Program
- Monthly Mission
- Blogs
- Groups
- Events
- Media Vault
Products
- Observability
- Network Management
- Application Management
- IT Security
- IT Service Management
- System Management
- Database Management
Content Exchange
- SolarWinds Platform
- Server & Application Monitor
- Database Performance Analyzer
- Server Configuration Monitor
- Network Performance Monitor
- Network Configuration Manager
- SQL Sentry
- Web Help Desk
Free Tools & Trials
Store

Services configuration failed RabbitMQ on primary server (port 5671) is not reachable

Hi been troubleshooting this for awhile and wondering if anyone else has ran into the issue and if they were able to resolve:

We currently have a ticket open for the issue but seems none of the troubleshooting steps have helped any. We have a primary server, HA server, and 11 polling engines. We are running 2019.2HF2, NPM 12.5, SAM 6.9.1, NCM 8.0, NPM 12.5 and UDT 3.4.0 HF1.

The rabbitMQ queues are constantly incrementing well over 40k. I have reset RabbitMQ, purged messages, validate Port 5671 is open between the primary and all the pollers, will also get the error when running configuration wizard on the pollers that (Services Configuration Failed, Rabbit MQ on Primary server is not reachable)

The last action plan provided to me was below:

Action Plan: Reset RabbitMQ for Orion Platform 2019.2
1- Stop All services (Main, APE and AWS).

2- Stop RabbitMQ service on services.msc (Main poller).
Run the queries in this article
https://support.solarwinds.com/SuccessCenter/s/article/Clear-Information-Service-Subscriptions

3- Navigate to C:\ProgramData\SolarWinds\Orion\RabbitMQ,
Cut the .erlang.cookie and place it in the backup folder on the desktop

4- Navigate to C:\ProgramData\Solarwinds\Orion\RabbitMQ\log.
Cut rabbit@ServerName.log to backup folder(Create a backup folder on the desktop and paste here)

5- Navigate to C:\ProgramData\Solarwinds\Orion\RabbitMQ\db\rabbit@server name-mnesia\msg_stores\vhosts\628WB79CIFDYO9LJI6DKMI09L\queues.
Cut the folders within the queues folder to the backup folder you have created.

6- Go to C:\ProgramData\Solarwinds\Orion\RabbitMQ\db\rabbit@server name-mnesia\msg_stores\vhosts\628WB79CIFDYO9LJI6DKMI09L.
Cut out the recovery.dets to backup folder.

7- Start RabbitMQ service on services.msc (Main poller).

8- Start All services (Main Poller).

9- Start All services (APE and AWS).

But this did absolutely nothing, kind of at a loss at this point, and want to this issue fixed, any additional ideas would be greatly appreciated.

Find more posts tagged with

rabbitmq

Accepted answers

All comments

mjalden1

We had similar issue make sure your Firewall/antivirus software is not trying to scan any in use solarwinds files. We have HBSS/McAfee and run HIPS. On access scan was causing a majority of the issues. Once i fixed that no more RabbitMQ issues. I had our team put all security in Logging mode and that how I found the issues. It would either scan files or stop WEB from working completely.

justthwackit

Hey there,

just wanted to thank you for suggesting point 3 in your "Action plan". I had problems with rabbitMQ after installing Windows Updates and that did the trick.

Thank you very much for your post!

azadmin

Yep just found this because on all pollers now after upgrade to the latest 2019 platform hotfix 3 I get:

Services configuration failed:

• RabbitMQ on Primary server (port 5671) is not reachable. PubSub over MessageBus will be disabled, and some services may not function properly. See log for details.

Core server upgraded without error.

Another failed upgrade and have to get support on the line.

deiberts

I'm having the same problem and found out that it was the TLS ciphers used in the Rabbitmq.conifg file. We are in a FIPS environment and FIPS was enabled. As a temporary band-aid until Solarwinds engineers fix this (ticket is being worked), I edited out the TLS cipher suite and set FIPS to false. Then it connects up just fine.

I found this out when by checking the RabbitMQ log file with a TLS handshake error and determined a cipher mismatch. Could be something to check on your end as well. Log into the RabbitMQ local web on the server front end and validate that you at least have socket descriptors opened. RabbitMQ for Solarwinds was a nice learning curve.

azadmin

Great info! Thanks.

I am not FIPS but still will have the support agent check this since he is on the phone now.

What is your ticket ID to help them correlate why this happened after applying the hot fixes. Kind of a giant bomb.

christopher.t.jones123

One thing you may want to ask the technician to do is to provide you the list of cipher suites that are necessary to run. Then you can add those particular suites using this tool Nartac Software - IIS Crypto thats the approach we took in an environment where i was experiencing that issue.

In that environment we had to enable the cipher suites on both the main poller and any additional polling engines, we also had to go into advanced config and tell it to use the correct PubSub system, it seems that once it fails it reverts to WCF and then doesnt attempt to use it again until you change it in advanced config

azadmin

From the first post I did see Rabbitmq.conifg has ciphers and says FIPS True - to me that is wrong since we are not FIPS. Maybe not related but took some time but we got there. What cannot be explained is why everything broke after installing these, I suspect support will get a lot of calls on this.

Orion Platform 2019.2 Hotfix 3

IP Address Manager 4.9 Hotfix 1

Nartac software was used. Both of the resolutions in this were applied.

Success Center

THEN like you mentioned instead of MessageBus it was set to WCF in advanced config.

Run config wiz again and thank full this error is gone. A whole day of work down the drain after another upgrade experience. Way too many over the years.

Services configuration failed:

• RabbitMQ on Primary server (port 5671) is not reachable. PubSub over MessageBus will be disabled, and some services may not function properly. See log for details.

Thank you all for your very helpful suggestions. My case was 00407584 - maybe it will hellp someone else refer to it since it was hard to have support pay attention to advice from Thwack postings.

deiberts

My ticket number is #00398641. We are in a government environment that requires FIPS but i don't think there is a STIG that I know of for RabbitMQ. Either way, it broke the message broker. I also went into the advance settings and set the "PubSub" to "MessageBus with WCF fallback." We have to go by STIGs in our environment and that includes cipher suite order via GPO. So, if we ever update / change Windows cipher suites, it could break this product again even if patched.

Prineel

Came across a similar issue after upgrading from 2018.4 to 2019.2 HF3.

After upgrading the Main Poller Standby got a message "Performance reduction due to globally disabling PubSub over MessageBus in the Orion Platform"

RabbitMQ refused to start. Fixed by doing the following:

RESOLUTION

Stop the RabbitMQ service.

Go to \ProgramData\Solarwinds\Orion\RabbitMQ\db\

Create a backup in a separate location of the folder ending in -mnesia.

Delete -mnesia folder that you made a backup of.

Run the Configuration WIzard only on services.

https://support.solarwinds.com/SuccessCenter/s/article/RabbitMQ-failure-during-startup