Offline SQL, MSMQ cache and missing polling data

Hi Team,

Could you confirm for me, if SolarWinds NPM has the ability to monitor/poll devices (ICMP/SNMP/WMI) even if the SQL database is unavailable for short period of time?

I was under the impression that MSMQ cache would save all polling jobs, which cannot be saved directly to the database.

I am asking, because I was performing some downtime tests of my test database and noticed that my graphs are not showing any monitoring data for even one minute of SQL downtime.

Please note that I am only disconnecting network access between SolarWinds and SQL.

I also checked below folder for MSMQ cache queues, but when my SQL is offline, the folder size is still the same:

Success Center

Thanks for your help.

Kind regards,

Marcin.

Find more posts tagged with

Accepted answers

All comments

hpstech

I believe only Agents will cache data during (24 hours) during a DB down situation.

kpmarcin

Thanks hpstech for the reply.

Yes, I am aware that Agent software is capable of storing information for 24h.

However I was under the impression that Polling Engines are using MSMQ temporary folder for storing query data, which in fact can be used for storing information during SQL downtime. It seems that there is no time limitation, only the free space on C drive.

I found information about using MSMQ cache by Additional Polling Engines in multiple places ( below quote is from mesverrum )

https://thwack.solarwinds.com/thread/127058

"The 1 hour of data is actually incorrect, the apes will stash data in msmq storage until the drive fills. It's a common complaint for people to have cases where they have intermittent issues that cause their ape to fill the drive completely. Once db service is restored you can stop/restart the Orion and msmq services and it will shoot the cached data into your db."

What do you think?

hpstech

I think I'll wait for a correct answer.

Any disruptions to the database has always caused the gaps you are seeing for me.

kpmarcin

It is also mentioned in the Scalability Engine Guide (at the bottom):

-----------------------------------------------

Scalability Engine Guidelines for SolarWinds products

What happens if the connection from a polling engine to the Orion Database Server is lost?

If there is a connection outage to the Orion Database Server, polling engines use Microsoft Message Queuing (MSMQ) to cache the polled data on the APE servers.

The amount of data that can be cached depends on the disk space available on the polling engine server. The default storage space is 1 GB. Up to one hour of data can be cached.

When the connection to the database is restored, the Orion Database Server is updated with the locally cached data, the oldest data is processed first.

If the database connection is broken for a longer than an hour, the collector queue becomes full, the newest data is discarded until a connection to the database is re-established.

kpmarcin

Thanks. Yep, let's wait for the Thwack Team to share more information on this

jm_sysadmin

I think the 1 GB MSMQ cache can be increased to buy you more time
MachineQuota | Microsoft Docs

But I have no idea if Orion will accept that.

kpmarcin

Thanks for the reply.

Yes, I am planning to test it next However the issue here is that even one minute is not stored and I am missing those monitoring statistics. Would you know the issue here?

The second arrow in the graph is for around 5 minutes SQL downtime. This is a test environment and I am monitoring only two devices, only via ICMP.

(I have increased ICMP polling to 10 seconds)