RabbitMQ Password Issue

rcbarr

What seems like "out-of-the-blue" the RabbitMQ password on one of the member servers in the pool was changed.  No OS patching, No Cyber, No Firewall issues encountered.

This problem we have at the moment is when I do a rabbitmqctl cluster_status all the nodes (4) are the same.  Support is telling me it's b/c the password has changed on one of the nodes, which I confirmed with > rabbitmqctl authenticate_user orion == Invalid credentials.

We have tried multiple things remove the node, add the node back, etc.  Several iterations.

Has anyone seen this?  If so, what did you do to resolve it.

thanks.

rcbarr

Figured it out myself (got lucky I guess), anyway in two of our environments (no idea why or how) our password changed on one of our member servers in our RabbitMQ clusters.  Was not patching, cyber changes, firewall changes, I have no clue.  Regardless to fix it I ran these three commands on the problem member.  The Issue: the ORION user was removed, who knows how.

rabbitmqctl.bat add_user orion <ourPasswordHere>   {get your password from your WORKING member)
rabbitmqctl.bat set_user_tags orion administrator
rabbitmqctl.bat set_permissions -p / orion ".*" ".*" ".*"

Once we were able to log back into the UI, we rebuilt the Virtual hosts and magic (we are working again).

FYI:  After you perform these steps, if you run a rabbitmqctl cluster_status it will still not show the member servers correctly UNTIL you perform a failover.

What a day....Thanks to Marc in support for his help and sticking with me late today until we got it all put back together.