Main Polling Engine Monitor - difference between Collector Queues Folder Size and RabbitMQ Folder Size?

Greetings All.

Looking at the monitor under SAM >> Application Summary >> Orion Observability Main Polling Engine there are 2 component monitors...

Collector Queues Folder Size

RabbitMQ Folder Size

I suspect that these two queues / folders are in some way related, but I'm not sure about the connection.

Can someone provide insight in the difference between Collector Queues and RabbitMQ Folder Size?

Thanks!

Find more posts tagged with

RabbitMQ Queue

Collector Queues

Accepted answers

starlord66

Thanks Google AI :-)

In SolarWinds, Collector Queues utilize RabbitMQ, a messaging broker, to facilitate communication and data flow between various components. Specifically, RabbitMQ manages the SWIS (SolarWinds Information Service) publish-subscribe messaging, ensuring timely delivery of collected data to subscribers.

Elaboration:

1. RabbitMQ as a Messaging Broker:
SolarWinds utilizes RabbitMQ as its messaging broker, which acts as a central hub for distributing messages between different parts of the SolarWinds platform, including the collector, polling engines, and the main Orion server.
2. SWIS Publish-Subscribe:
The SWIS (SolarWinds Information Service) relies on RabbitMQ for its publish-subscribe messaging architecture. When collectors gather data, they "publish" it to RabbitMQ, and other components (like the polling engine) can "subscribe" to receive this data.
3. Collector Queues and RabbitMQ:
Collector Queues are essentially queues within the RabbitMQ system that hold messages (collected data) before they are delivered to the appropriate subscribers. These queues manage the flow of data and prevent bottlenecks or delays in delivery.
4. Monitoring RabbitMQ:
SolarWinds provides tools to monitor the health and performance of RabbitMQ, including metrics on queue size, message delivery rates, and connection status.
5. RabbitMQ Folder:
The physical location of RabbitMQ data (including queues and messages) is stored in a dedicated folder on the SolarWinds server. Monitoring the size of this folder can indicate potential issues with RabbitMQ performance, such as a backlog of undelivered messages or insufficient memory.
6. High Availability:
In a high-availability setup, RabbitMQ can be clustered across multiple servers, providing redundancy and ensuring uninterrupted data flow even if one server fails. This clustering requires specific TCP ports to be open between the servers

starlord66

Hi Robert

I don't believe the fact that the location of the RabbitMQ and Collector queue folders are have different paths should matter - as a further explanation see bellow - hope that helps - otherwise it would be more a question for a SolarWinds engineer to answer more specifically.

In SolarWinds, RabbitMQ and the Collector use different paths primarily due to their respective functionalities and the historical evolution of the system. RabbitMQ is used for Pub/Sub (publish/subscribe) messaging, a technology that handles communication between different parts of the Orion platform, especially for pushing updates and notifications. The Collector component, on the other hand, is responsible for collecting data from monitored elements. The Collector used to rely on MSMQ (Microsoft Message Queuing) and has since evolved to use SQLite in newer versions for its data storage and processing, which explains why they have separate locations.

Elaboration:

RabbitMQ:
As the primary Pub/Sub system, RabbitMQ needs a central location and configuration to manage message routing and delivery between different SolarWinds components. This includes the Orion Web Console, web resource errors, and other critical services.
Collector:
The Collector, which used to rely on MSMQ and SQLite, is now a separate component that handles data collection, including gathering information from monitored elements like nodes, agents, and web resources. Its data storage and processing logic are separate from RabbitMQ, which is why it has a distinct path.
Historical Context:
Initially, SolarWinds Orion used MSMQ for its Pub/Sub system, which meant both RabbitMQ and the Collector were linked in a different way. When RabbitMQ was introduced as an alternative, it resulted in a clear separation of the Pub/Sub messaging from the data collection process, necessitating distinct paths for the respective software.
Key Differences:
Functionality: RabbitMQ handles asynchronous communication, while the Collector focuses on data collection and processing.
Technology: RabbitMQ is a message broker, while the Collector utilizes MSMQ or SQLite for data storage and retrieval.
Location: The Collector has its own folder and configuration to manage its data storage and processing.

All comments

starlord66

Thanks Google AI :-)

Elaboration:

1. RabbitMQ as a Messaging Broker:
SolarWinds utilizes RabbitMQ as its messaging broker, which acts as a central hub for distributing messages between different parts of the SolarWinds platform, including the collector, polling engines, and the main Orion server.
2. SWIS Publish-Subscribe:
The SWIS (SolarWinds Information Service) relies on RabbitMQ for its publish-subscribe messaging architecture. When collectors gather data, they "publish" it to RabbitMQ, and other components (like the polling engine) can "subscribe" to receive this data.
3. Collector Queues and RabbitMQ:
Collector Queues are essentially queues within the RabbitMQ system that hold messages (collected data) before they are delivered to the appropriate subscribers. These queues manage the flow of data and prevent bottlenecks or delays in delivery.
4. Monitoring RabbitMQ:
SolarWinds provides tools to monitor the health and performance of RabbitMQ, including metrics on queue size, message delivery rates, and connection status.
5. RabbitMQ Folder:
The physical location of RabbitMQ data (including queues and messages) is stored in a dedicated folder on the SolarWinds server. Monitoring the size of this folder can indicate potential issues with RabbitMQ performance, such as a backlog of undelivered messages or insufficient memory.
6. High Availability:
In a high-availability setup, RabbitMQ can be clustered across multiple servers, providing redundancy and ensuring uninterrupted data flow even if one server fails. This clustering requires specific TCP ports to be open between the servers

starlord66

High Robert777 - see my previous post - cheers

robert777

Thanks! Marking this as answer. All great information.

Looking at the SolarWinds AppInsight for Main Polling Engine >> FULL DIRECTORY PATH it appears that the path is different for the RabbitMQ folder size and the Collector Queues folder size. So, it appears that these are in some way different.

If anyone in the community can provide insight into what kind of information goes into the Collector Queues verses RabbitMQ folder ("database") that would be icing on the cake.

Cheers.

starlord66

Hi Robert

Elaboration:

RabbitMQ:
As the primary Pub/Sub system, RabbitMQ needs a central location and configuration to manage message routing and delivery between different SolarWinds components. This includes the Orion Web Console, web resource errors, and other critical services.
Collector:
The Collector, which used to rely on MSMQ and SQLite, is now a separate component that handles data collection, including gathering information from monitored elements like nodes, agents, and web resources. Its data storage and processing logic are separate from RabbitMQ, which is why it has a distinct path.
Historical Context:
Initially, SolarWinds Orion used MSMQ for its Pub/Sub system, which meant both RabbitMQ and the Collector were linked in a different way. When RabbitMQ was introduced as an alternative, it resulted in a clear separation of the Pub/Sub messaging from the data collection process, necessitating distinct paths for the respective software.
Key Differences:
Functionality: RabbitMQ handles asynchronous communication, while the Collector focuses on data collection and processing.
Technology: RabbitMQ is a message broker, while the Collector utilizes MSMQ or SQLite for data storage and retrieval.
Location: The Collector has its own folder and configuration to manage its data storage and processing.

robert777

Cool!

I would encourage you to take the information provided in this thread and submit to the Thwack moderators as a technical document. I have not been able to find this kind of information on Thwack. My thoughts this should be highlighted as a banner article for the community.

Thanks.