Advanced alert escalations + Auto Acknowledge?

We are trying to perform some automated fixes for some linux services and are trying to use alert escalations to be able to escalate an issue to someone (via email) when those automated fixes fail.

Currently I have an action to execute a powershell script which connects to the node and restarts the linux service if its down.

What i want to do is if the service is still down after attempted restart then escalate to human via email.

Basic procedure would be:

- Alert fires for component down

- Automated restart/fix deployed

- If still down after 10 minutes email human

Is there any way we can achieve this via alert escalation? I know escalations are based on acknowledgement but can we change this behavior? I am thinking of modifying the script to wait then auto acknowledge if successful.

Find more posts tagged with

alerts escalation

trigger actions

Accepted answers

neomatrix1217

This is what IanBaker is talking about.

All comments

IanBaker

In my systems the escalation can email after a reasonable time. If the script fixes the issue then the reset actions should trigger and acknowledge the alert

neomatrix1217

This is what IanBaker is talking about.

pav

Hi there,

I`m struggling to configure specific set of alerts and escalation. My network has branch network and from some time I`ve got loads of alerts, that branch device is down\warning state - while is not - and "the break" is about 1 minute in lenght. We are thinking the reason is a backup or just network congestion (most of the branches are still on adsl).

What I`m looking for is alert with escalation. First alert with message to syslog (if condition is no longer than 2min) + reset, if node comes Up in a minute, and escalation as an e-mail (if condition is longer than 2 min), if node is still Down/Warning.

Unfortunately, looks like an escalation is automatic and doent recognize change in condition (in this case node changing status back to Up), so I`ve got syslog msgs AND the e-mails anyway.

Cheers

mesverrum

Maybe have a slower timer on your escalation? How fast is your reset condition? The escalation will not happen if the alert has reset already, but if your alert is only running once every 5 minutes or requires the reset condition to be true for a while I could see how the escalation would proceed even after the node is back.

pav

Cheers mate. I have forgotten to add polling time into escalatoin. Increased to 5 min and works great. Ta!

IanBaker

Can you post what you have in Trigger and reset tabs?