Community
- Command Central
- MVP Program
- Monthly Mission
- Blogs
- Groups
- Events
- Media Vault
Products
- Observability
- Network Management
- Application Management
- IT Security
- IT Service Management
- System Management
- Database Management
Content Exchange
- SolarWinds Platform
- Server & Application Monitor
- Database Performance Analyzer
- Server Configuration Monitor
- Network Performance Monitor
- Network Configuration Manager
- SQL Sentry
- Web Help Desk
Free Tools & Trials
Store

Alert Engine Failure Notifications (Heart Beat Monitor for Orion)

We recently encountered a problem where there was an issue with the Alert Engine had lost connection to the database due to the maximum number of connections to the database being reached. This caused all alerting for Orion to stop and meant that when we had things go down emails did not get generated to notify us that it was down. I'm currently trying to use task scheduler to work as a backup but there isn't really a good way for it to tell you whats wrong it will just flag it as an error and you have to go look at it to see if it's something fatal or not.

I would like to see some sort of feature added as a method of letting you know if there is an internal failure has occurred and alerts are not being sent out. Like a heart beat monitoring process that attempts to generate an alert and only sends an email if the alert could not be generated.

Find more posts tagged with

engine

Alerting

Status: None

Comments

adatole

In the Advanced Alert manager, look for "Alert me when a polling engine has not updated the database in 10 minutes".

techbender

Hey thanks for the feedback Leon, I went to the Advanced Alert Manager and clicked on Configure Alerts but I didn't see that anywhere in the list. I'm also not sure that this would work though because as I understand it anything that's setup in the Alert manager is still going to be using the Alerting Engine which is the very thing that we had a problem with. The service never stopped but when it had lost connection to the database it quit being able to send alerts.

I've made a really basic powershell script:

$strFileName="C:\AlertChecker\HeartBeat.txt"

If (Test-Path $strFileName){Remove-Item $strFileName}

Else{

$PSEmailServer = 'smtp.domain.com'

Send-MailMessage -From 'mail@mail.com' -To 'mail@mail.com' -Subject 'mail subject' -body 'mail body'

}

This script is set to run every hour via task scheduler. I have an alert configured with a trigger condition that is always active like Device type is equal to Windows on the node for our Orion server. This alert has a trigger action to create a log text file and the alert escalation is set to 1 hour. I've off set the time by 5 minutes so that every hour the alert makes the log and 5 minutes later the script deletes the log. If there is ever a case where the script does not have a file to delete it generates an email that tells us "Hey! Something is wrong you need to check on the alerts!" I'm still testing it but I believe this is going to be a good work around for the time being for letting us know that if for what ever reason Orion stops alerting we will know about it. I would be cool if Solarwinds could implement something like this.