Alert on nodes if group is not critical

turpinmaxime

Hi everybody,

I need your help for a tricky case.

We have several sites monitored (with Router, firewall, switch, AP, Controller etc) and we don't use dependencies. For a site, all equipment are stored in one group and we don't want to receive massive alerts when a site goes down.

How should we proceed to receive only one alert when a site goes down ?

Is there a way to be alerting on nodes only if the group is not in "down" ?

Many Thanks,

Maxime

cahunt

You could use a Group Down, and a Group Warning/Critical with a focus on the root cause. The Warning/Crit might trigger due to objects of less importance that might create a semi noisy scenario. The Group Down by default would trigger if only all nodes went down, and you might want that alert a little quicker if a site FW/Border fails. If using group alerts you do need to consider the other alerts that are active, as a down node could give you that alert, plus the group warning/critical. Also, enhanced node and interface polling changes your roll-up status scenarios, this may all need a little consideration.

For using Groups Specifically; 

With each site as its own full single group, each node within the group has the same ability to change the overall group status. This gives your nodes within each site the same rank or level of importance. To achieve a proper roll-up in status to the top-level group you would need to start with at least 2 subgroups. One for the main (or pair of) ingress/egress devices and one for all other devices. There might be a need for more groups depending on architecture; are things dual-homed, fully redundant, or single devices/points of failure. The Site Group setting should be 'worst status' so if your border/edge dies it rolls up to the main site group, or if (for some reason) all the other devices die and everyone at the site loses access. Anything else should show a warning to let you know something is amiss. Naming may be important for the groups (prefix or suffix details will help with alerting), which means some groups will need a delineation of the purpose of the group (i.e. node type), like -firewall or -core.

Alerting at this point could be for Down and Warning/Critical. The detail you would want in the alert is the 'Root Cause' that will highlight either the entity or the subgroup in status that is rolling up. 

You might want to consider segregating some alerts due to architecture which may fall to a point of congruency in the group name. A suffix of -Firewall or -core to highlight the purpose of creating the group will allow focus but also add a small informational detail to your alerts.

Alerting on groups can be tricky. I will say in all the years, I have seen only 1 environment use group alerts, and only group alerts. There were 4(maybe 5, but most likely only 4) active alerts. The group design and alerting had a direct correlation and every node and entity was covered in alerting with just that small set of group alerts. Honestly, it was a very intuitive design, but not something I would want to end up taking over without some strict documentation.