Informational Alert

I have created a Component Monitor Linux script which scans through a backup log and reports different statistics found in the log. It also evaluates the info in the log to determine the backup status (Successful, In Progress, Failed, etc.). I then assigned status numbers to each status depending on which I considered successful, Warning or Critical and used those numbers in the Monitors Script Output evaluations for Warning and Critical thresholds. The monitor works great and threshold trigger events as I wanted.

As for the alert rules, I have a generic alert rule that is already built and will handle the notification if a threshold is crossed. However I also want to send an email to staff to report a backup was completed and successful. As opposed to the non-successful notifications, staff would get this notification once per day, mid-morning. So to build the rule I set the alert trigger conditions to engage if the Component Monitor state is Up. I then set the time of day to the time I want the alert rule to check and send the daily complete and successful backup email. The part I have not figured out is how to reset the alert so that it moves off of the active alert list and will be able to again trigger the following day.

What I have tried is to:

Set the time-of-day to just a couple of minutes, set the reset conditions to be the same as the trigger conditions and set the reset delay timer to one minute longer than the time-of-day end time. I would have expected this to work as the chain of events should be that the alert triggers and the reset timer starts. As long as the trigger remains active, the trigger actions should fire however they do not. The only thing that happens is that the alert appears in the active alert list and disappears with the reset timer runs out.
To ensure that the rest of the alert rule works properly, I changed the component status in the reset conditions to down and had the rule fire again. This did result in the actions being executed, but then the alert stays on the list after the actions, reset timer runs out and the time-of-day period expires because the component state is still up.
For kicks, which I did not expect to work, I also selected "Reset when trigger conditions are no longer true" in the reset conditions and of course, the trigger conditions do not change so the alert stays active and again never resets. At least not unless the backup status changes.

I can see where this would also be a helpful scenario for providing informational alerts to staff for things like successful database integrity checker sessions and so on. Has anyone done something similar to this and if so, can you guide me in the right direction?

THANKS

Mike

Find more posts tagged with

advanced_alerts

Alerts

Accepted answers

Mike_Lomax

I did also try something similar to that which I did not mention. The original test in the trigger and reset conditions was to test for Component Monitor Up. In the reset I changed the condition to not test for Component Monitor up but rather for Application Monitor Up using the same Application Monitor that contains the Component Monitor. This did not change the result. The alert never resets until, in this case, the Application Monitor state changes.

My goal was to keep the active time for the alert short so that the alert list would remain clean most of the time. For that reason, I set the reset delay timer to 2 minutes for my testing but planned to have it set to 5 minutes for my live environment since the polling period is set to 6 minutes. This would keep the rule from triggering twice for that same days notification. BTW: I also set my time-of-day window to only 1 minute for the testing.

Out of curiosity I did set up what you suggested with the reset condition testing for node up rather than for Component or Application Monitor status. The result again was the same. The Alert State stays at 3 and the trigger actions never fire.

As I was running your test I also thought about something a bit different. I first went back to my original design with both the trigger and reset conditions set to watch for Component Monitor Up. Went back to my original time settings and then moved my trigger actions to reset actions. This was the answer...

So to summarize:

Goal: Create an alert rule that would send notification of a non-event condition like a successful backup. However only keep the alert on the active alert list for minimal time.

Trigger Condition: Set to Component Monitor Up state.

Reset Condition: Set to Component Monitor Up state.

Reset Condition Delay Timer: Set to less than Application Monitor polling interval but more than the Alert Rule Time-of-Day time span listed below. I used 75 seconds.

Alert Rule (not action) Time-of-Day Setting: Set to time you want successful backup notification delivered to staff. Set time span to less than Application Monitor polling interval. I used 1 minute.

Trigger Actions: None

Reset Actions: Email with backup status and NetPerfMon Event Log entry (in my case)

Result of the above is :

the Alert triggers at the beginning of the Alert Rule Time-of-Day time span. It triggers at Alert State 3 and does not change which keeps any trigger actions from firing.
The Time-of-Day time span expires and then the Reset Condition Delay Timer starts,
The Reset Condition Delay Timer expires,
The alert disappears from the alert list,
The Reset Actions fire.

The overall process to get the alert actions out the door was the expected 2½ to 3 minutes.

Thanks for the thought provoking assistance Mike.

All comments

mdriskell

Could you do a reset if node status is equal to up but then have your reset condition have the condition exists for 60 minutes or something to that effect.

Mike_Lomax

So to summarize:

Goal: Create an alert rule that would send notification of a non-event condition like a successful backup. However only keep the alert on the active alert list for minimal time.

Trigger Condition: Set to Component Monitor Up state.

Reset Condition: Set to Component Monitor Up state.

Reset Condition Delay Timer: Set to less than Application Monitor polling interval but more than the Alert Rule Time-of-Day time span listed below. I used 75 seconds.

Alert Rule (not action) Time-of-Day Setting: Set to time you want successful backup notification delivered to staff. Set time span to less than Application Monitor polling interval. I used 1 minute.

Trigger Actions: None

Reset Actions: Email with backup status and NetPerfMon Event Log entry (in my case)

Result of the above is :

the Alert triggers at the beginning of the Alert Rule Time-of-Day time span. It triggers at Alert State 3 and does not change which keeps any trigger actions from firing.
The Time-of-Day time span expires and then the Reset Condition Delay Timer starts,
The Reset Condition Delay Timer expires,
The alert disappears from the alert list,
The Reset Actions fire.

The overall process to get the alert actions out the door was the expected 2½ to 3 minutes.

Thanks for the thought provoking assistance Mike.

aLTeReGo

The logic to reset the condition will need to be included in your script similar to how it's handled in our Log Parser templates. Fortunetly you have a Perl and PowerShell example already included in SAM. This is how the "Number of Newly Found Strings" component works within this template. Each time the script is run it remembers where it last left off and only counts new occurances. If there are none then the alert should clear. If a new instances is found on the next poll the alert will trigger again.