This discussion has been locked. The information referenced herein may be inaccurate due to age, software updates, or external references.

You can no longer post new replies to this discussion. If you have a similar question you can start a new discussion in this forum.

Watch The Watcher... Or Self-Monitoring For MSMQs

AlexSoul over 9 years ago

Hi All,

This is an independent solution for monitoring MSMQs on SolarWinds application server itself. If for some reason those queues are growing - it most probably means that app server cannot update database. This is also potentially an indication (as it was in my case) that SolarWinds is not sending out any alerts... including those about high MSMQs if you have any... catch-22

Same problem has been described here and here

Clearly, we need an independent monitor to report on MSMQs on SolarWinds server itself

OK, here is what you can do:

(1)

Create scheduled task on SolarWinds app server and set it to run every XX minutes (15 for example)

(2)

Trigger attached script with the following parameters:

msmqmon.vbs server_name queue_name threshold_messages_count

example:

msmqmon.vbs your-solarwinds-server core.node.responsetimestatus.icmp 100

==============

* Script will check the number of messages in core.node.responsetimestatus.icmp MSMQ on SolarWionds serevr. If it has more than 100 messages queued - you will receive an email alert. (I hear you are asking "Why this particular queue?" ANSWER: when I had my SAM crashed this morning - this queue had the most number of massages, 72k+)

** Generally, there is nothing too fancy here - a simple VBScript to send standard CDO.Message email via remote server.

*** You only need to modify script lines 42-44 for it to work. The rest is up to you

Thanks,

Alex

Top Replies

0 fitzy141 over 9 years ago

how does differ from the Orion Template that monitors the MSMQ as well ... are we saying that is not reliable enough and just asking to ask ...
Cancel
Vote Up +1 Vote Down

Cancel
0 AlexSoul over 9 years ago

Orion will not be able to notify you about its own MSMQs, simply because your alerts will not go further than MSMQ itself. To be able to catch this - you need completely independent monitor.
Cancel
Vote Up +2 Vote Down

Cancel
0 aLTeReGo over 9 years ago

This is true if the message queue is filling and not writing out to the database at all, but if a particular message queue is simply getting backed up (like interface monitoring for example) then the Orion template would certainly notify you of that condition.
Cancel
Vote Up +2 Vote Down

Cancel
0 tigger2 over 9 years ago

Thanks! I've had similar issues and I'm going to give this script a shot. Just to add some details when it hit me, in case it helps anyone:
I'm running: Orion Platform 2014.2.1, SAM 6.1.1, QoE 1.0, IPAM 4.3, NCM 7.2.2, NPM 11.0.1, UDT 3.0.2, IVIM 1.10.0
When this occurred for me I had several of these streaming in my Application event log:
    Log Name:      Application
    Source:        Microsoft-Windows-MSMQ
    Date:          2/13/2015 8:12:40 PM
    Event ID:      2182
    Task Category: None
    Level:         Warning
    Keywords:      Classic
    User:          N/A
    Computer:      <hostname redacted>
    Description:
    Storage quota exceeded for MSMQ queue DIRECT=OS:<hostname redacted>\private$\solarwinds/collector/processingqueue/apm.blackbox.sql.applica. No more messages can be stored in the queue. You can increase a queue's storage quota by editing queue properties in Computer Management console. This event is logged at most once per 3600 seconds. To change this setting, set \HKLM\Software\Microsoft\MSMQ\Parameters\Event2182 registry value to desired time in seconds.
In addition, I had several other event log entries indicating different storage quotas (including the one in the above post). The queues that generated the above event log entry and order they came in (the first was hours before the second) were:
1. \private$\solarwinds/collector/processingqueue/apm.blackbox.sql.applica
2. \private$\solarwinds/collector/processingqueue/core.node.responsetimest
3. \private$\solarwinds/collector/processingqueue/solarwinds.interfaces.sn

Looking up the *real* queue names:
1. apm.blackbox.sql.applica = apm.blackbox.sql.application.statistics.api
2. core.node.responsetimest = core.node.responsetimestatus.icmp OR core.node.responsetimestatus.snmp
3. solarwinds.interfaces.sn = solarwinds.interfaces.snmp
Note: The first one probably queued up fastest because we have many SQL database instances and databases being monitored using the SAM/APM "Appinsight for SQL" template.
Cancel
Vote Up 0 Vote Down

Cancel