"Alert can be triggered if" Unexpected Results

I think I have only used the "Alert can be triggered if" once in the past and now I am wondering if that one is working. Let me lay out a basic scenario and explain what I am seeing.

The Goal:

Trigger a single Alert if any or all of the Nodes are not responding to Orion.

The Configuration :

Outcome:

If 1 of the 4 Nodes is not responding, no Alert triggers.

If 2, 3 or 4 Nodes are down, Alert will trigger stating "1 Node" caused the trigger.

Question 1:

The "Alert can be triggered if more or equal 1 and then trigger single alert objects (at the same time) have met the specified condition". Why does the Alert not trigger if only 1 of the 4 Nodes is down?

Question 2:

Why, if 4 Nodes are down, does the Alert trigger stating that "4 Nodes" caused the trigger?

I also tried the same testing with SAM Component monitors and the behavior is different but still wrong. If it is 4 Component Monitors in the Alert Rule rather than 4 Nodes in the Alert Rule:

if 1 or the 4 Components down, the Alert will not trigger,

if 2 of the 4 Components are down Alert will trigger stating 1 Component caused the trigger

if 3 of the 4 Components are down Alert will trigger stating 2 Components caused the trigger

if 4 or 4 Components are down Alert will trigger stating 3 Components caused the trigger.

Why is it always 1 short of reality?

THANKS

Mike

Find more posts tagged with

sam alerting

sam

npm alerts

alert_condition

npm

Accepted answers

All comments

monitoringlife

Whats the complex alerting you have configured on the alert? If you just want a ticket if any of the 4 nodes are down I would clear the bottom two check boxes for instance and advanced.

Mike_Lomax

The goal is that any or multiple Nodes meet the condition but not all Nodes meet the condition, trigger the Alert.

I have a different Alert Definition if all Nodes meet the condition.

I realize that I can build an Alert Definition for every possible scenario but it seems to me like the "Alert can trigger if" feature was supposed to handle this but does not. Generally it looks to me like the idea behind this feature was only half baked when added and never fully developed.

Mike

monitoringlife

I think we need to adjust your alert. Remove the instance counter option.

I would personally make two separate alerts, and let both trigger. This will help communicate criticality as alerts and the node detail page will show that both this node is down, and the 'all x in this group are down' alert.

If you want to explicitly exclude this alert from the 'all 4 down'. Update the 'scope' section to include an additional AND loop that excludes if all four are present.

monitoringlife

I drafted a Alert in my environment to see if the instance counter could be switched to less than (and/or less than equal to). Looks like despite that its a drop down, its only one option (greater than or equal to).

Mike_Lomax

That doesn't fit with the requirements provided by that Team. They want 1 Warning Alert that triggers if 1 or more but not all are not responding and a Critical Alert (separate Alert Definition) if all are not responding. But thanks for that feedback.