Alert Flapping Issue Even Though Alert Is Configured Correctly

I configured an alert to basically tell me if a server is up or down with the following trigger:

Node category is equal to server

node percent loss is greater than 40%

I set the alert actions to send me an email first and then send an SMS through OpsGenie 5 minutes later if it is still down.

The alert works beautifully. If I shut a test server down it will cycle through all the correct steps. Once I bring the server back up it sends me a reset email. Afterwards thats when things go haywire.

The alert begins flapping sending me up and down messages for the next 5 minutes until it finally stops. Is there something I can add to the trigger to prevent this?

Find more posts tagged with

Alert

Accepted answers

wrainwater

Adjusting the evaluating trigger to change from every one minute to 5 minutes seems to have fixed the issue. thanks for all of your help.

All comments

kreases

I must admit I have never had flapping issues on a node down alert unless of course the server is bouncing. I don't know why you are using the additional condition of node percent loss is greater than 40% but I suspect that might be the cause of the flapping but you could remove that condition and test the alert to prove if it is that.

You could also try using the option at the bottom of the trigger screen where you can choose how long the condition must exist for before alerting.

Hope this helps.

i_like_eggs

you could reverse the logic here.

wrainwater

I've tried just putting the node as down. the only problem is when the server reboots, the alert never goes off. I want the alert to tell me if it's down, even if just a second. I tried setting the last boot option but that usually triggers it to go off multiple times. Servers seem to be the only thing that's causing this issue. they are virtual servers fyi

wrainwater

I think I've tried that. It gave me the same results.

As of right now I have it configured as: node category = server AND node percentage loss is greater than 5% + if node status is = unknown, down, warning or unreachable then trigger. I may have over did it but ill test and see if that works by rebooting a few servers.

wrainwater

Adjusting the evaluating trigger to change from every one minute to 5 minutes seems to have fixed the issue. thanks for all of your help.