over 1 year ago

Alert for when the SQL server hosting Solarwinds goes off line

An alert is needed specifically when the SQL server or database hosing Solarwinds goes down.  When the hosting SQL server goes down Solar winds does not function and basically the environment is not monitored.  If the specific database is to busy to handle any request and queues up incoming information for a period greater than a polling period incorrect alerts will be sent.  The main reason appear to be that the Solarwinds servers are 100% reliant on this SQL server to function.  As with anything in IT everything should have a fail-safe if possible.  To fix this a simple heartbeat monitor sent from the Solarwinds node that was independent of the database could be used.  It could be made to trigger an alert in the event that there was a drop in incoming database traffic over X period.

We had an issue where the SQL server started to queue all incoming request.  This occurred for a period greater than the polling period.  At the point that the SQL server started processing the incoming request it returned information to Solarwinds.  This triggered multiple server down alerts.  The cause appeared to be that since there was not a timestamp in the database to indicate that the server was up within the last X minutes (information was queued during the period) the alert was sent.  Unfortunately the only server with an issue was the Solarwinds SQL server.