Node Down Clear and Reboot Alerts Fire in Tandem

I have a Node Down alert with an email trigger action and an email reset action. So I get an email when a node goes down and another email when it comes back up. Pretty standard stuff.

I also have a Node Reboot alert with a trigger condition of LastBootTime has changed. This also has an email trigger action.

The problem is when a down node that comes back up, I get both the node up and node rebooted email alert. This is because both trigger conditions are met.

Is there a clever way to exclude the Node Reboot alert from triggering when a Node Down/Up?

The trigger condition I would like to define but can't is:

LastBootTime has changed

AND NOT

Node Status has Changed from Down to Up in the past 1 minute.

Find more posts tagged with

clear

lastboottime

down

Alerts

Accepted answers

All comments

pratikmehta003

Do you really need reboot alerts? If the node down/up is meeting your requirement then u can simply disable the alert.

pratikmehta003

Or I think if u put another condition stating If node is up for the node reboot condition, it might work...not sure though.

muckman

No we don't need it but my organization has fought me hard on reboot alerts. I feel they just generate noise and provide little value.

muckman

Unfortunately the node is always going to be up if NPM knows the LastBootTime has changed. So that condition would negate 100% of the alerts from firing.

jkrenzien

You could have it try to check the Last Boot event in the reset conditions. I currently can't do a full test of this, but I may be able to next week. At this point it becomes a race between the node showing up and NPM seeing the last boot has changed. You may be able to look at the system up time as well. It is odd that we can specify a duration when creating the trigger, but the the reset. Hope this helps.

node reboot filtering.JPG

john.ta

You wouldn't be able to exclude the reboot alert if recent down/up, but you can exclude the up alert if a recent reboot. If you change the alert reset condition for node down to include system uptime > 60 or whatever, wouldn't that accomplish what you are looking for?