Version 2024.1 RabbitMQ Issue

rcbarr

This week we upgraded our Production HA environment to 2024.1.  Using the Centralized Installer, we ran to completion, everything looked good.  Yesterday my peer was looking in the RabbitMQ Console (version 3.12.6 / Erlang 26.1.1) on our primary poller and tried to access the queues, when he did, it produced this error.

++++++++++++++++++

ReferenceError: display is not defined ReferenceError: display is not defined at Array.process (eval at compile (localhost:15672/.../ejs-1.0.min.js:1:6654), :11:97) at EJS.render (localhost:15672/.../ejs-1.0.min.js:1:1885) at format (localhost:15672/.../main.js:1145:21) at localhost:15672/.../main.js:461:24 at with_reqs (localhost:15672/.../main.js:1127:9) at localhost:15672/.../main.js:1123:17 at req.onreadystatechange (localhost:15672/.../main.js:1204:17)

++++++++++++++++++

We have two other environments "test and QA" we have also upgraded to 2024.1.  This is what we noticed, on ALL (3) environments working (DR Pool Member in Production, test and our QA environments), you start the RabbitMQ UI and you see the "NEW" label for queues > "Queues and Streams", on the primary Production Poller (where we ran the upgrade from), we still only see the old RabbitMQ label for Queues, just that Queues, if we select it on this one server, the error above is generated.

What puzzles me is we used the same installer on both the primary and pool member dr at the same time when we upgraded. 

We tried a couple times to execute a soft rebuild of RabbitMQ.

1. Failover to DR.

2. Disable HA

3. Stop the RabbitMQ service.

4. Rename the C:\ProgramData\SolarWinds\Orion\RabbitMQ\db directory (the DB directory)

5. Run Configuration Wizard

Done

Did not work.  Have any of you guys seen this behavior post upgrading to 2024.1, specifically you guys running a HA environment.

One other thing we have seen is after you run the configuration wizard, the ORION user (sometimes) does not  get re-created, if this happens to you, perform these steps

** from C-or-D:\Program Files (x86)\SolarWinds\Orion\RabbitMQ\sbin

- rabbitmqctl add_user orion <enter your password>

- rabbitmqctl set_user_tags orion administrator

- rabbitmqctl set_permissions -p / orion ".*" ".*" ".*"

** fixed

** Note there is spaces between each (3) ".*"   ".*"    ".*" above

Any suggestions would be appreciated.

-Richard

rcbarr

OK, here was my solution (hate to admit it).  The problem for me was we had never seen this problem, additionally it was working on 3 of the 4 upgraded poller (1)s (multiple HA environments, long story).  The RabbitMQ Bug and Solution is documented here > https://community.pivotal.io/s/article/ReferenceError-display-is-not-defined-ReferenceError-display-is-not-defined-at-Array-process?language=en_US

 Unfortunately this is the ONLY place on the planet earth the "BUG" was documented and our Cyber team had BLOCKED this site, so I never found it when researching.

If you have my issue that I describe, run your browser in "Incognito", if you see "Queues and Streams" then you know you are having the same issue I did.  It is a bug with RabbitMQ version 3.12x, (which is the version you will be running after upgrading to SolarWinds v2024.1) NOT SolarWinds, in this case.

In the simplest of terms, FLUSH YOUR browser CACHE, good freaking grief.  One of those can't see the Forest for the Trees situations, ugh.

I will not repeat what Support asked me to do, I will just fall on my sword and blame myself.  ALBEIT, an Application Engineer WAS the person that found this site and subsequent solution, once the case was ESCALATED.

Hate I did not catch this, 50 demerits to me!!!!!!!