How To Suppress Group Down alerts when group members are muted or unmanaged

Our 24x7 monitoring team keeps an eye on change requests and they mute the affected nodes to prevent node alerts being raised. However Groups can't be muted in the GUI, so they still get Group down alerts.

I'm looking for one of two things:

1. Suggestions on how to set up something that will let our 24x7 team mute Groups

2. Guidance on setting up Group down alerts in such a way that if all nodes are muted or unmanaged the Group down alert doesn't fire. This would be my preference since it requires no extra effort from our 24x7 team.

Find more posts tagged with

nodes

Unmanaged

triggers

muted

down

groups

Alerts

Accepted answers

All comments

BaldFeegle

For testing this I have two groups with one nested inside the other. Top group contains two nodes, nested group contains one node. All three nodes are down, but the node in the nested group is muted.

So I enabled complex conditions for the alert. The first part is just a simple Group Status = Down. When the SWQL is run it returns both groups as being down as expected.

Data loss protection makes it a PITA to upload screenshots but the SWQL is:

SELECT E0.[Uri], E0.[DisplayName]
FROM Orion.Groups AS E0 
WHERE ( ( E0.[Status] = '2' ) )

Then I wrote some SWQL that only returns groups which don't contain nodes that are muted or unmanaged. When run in SWQL studio it returns the expect results. The group that has the muted node is not returned.

SELECT groups.[Uri], groups.[DisplayName]
FROM Orion.Groups AS groups 
WHERE groups.uri NOT IN
(
SELECT groups.[Uri], groups.[DisplayName]
FROM Orion.Groups AS groups 
JOIN 
   Orion.ContainerMembersNodes cm ON Groups.ContainerID = cm.ContainerID
JOIN 
    Orion.Nodes n ON cm.NodeId = n.NodeID
JOIN
    Orion.AlertSuppression a ON n.Uri = a.EntityUri
WHERE
n.Status !=9
)

When these two trigger conditions get AND'd together I expect the alert to only fire on the group that has no muted nodes, but instead it fires on both groups. I don't think it should be doing that.

chad.every

I've had a saying of "friends don't let friends use group alerts", because you end up with duplicate alerts that are challenging to logic/query out of. I've typically challenged the need to have groups alert since the underlying entity was covered by another alert. However, that might not be possible in all scenarios.

BaldFeegle

Our infrastructure benefits from group down alerts being available. I don't get why it works in Swql studio but not in the trigger.

marcrobinson

I wonder if this has to do with some of the stranger conditions that groups can show - best or worst status. Pity that you cannot leverage a dependency. Otherwise, it is custom properties to make it worse to make it work. You can use group custom properties as part of the scope for the group alert. While I have done this for volume alerts (I had to exclude a single volume on a node....), I don't see why it could not work for group alerts, but I take no blame if the SWIS service and alert engine run down the hallway screaming either. Even with the nested group - if it had the custom property set to 'ignore me' then the alert should focus on the other members and trigger accordingly. Its late for me, and I should stop trying to figure this out. G'night!

Daran

Hi @BaldFeegle ,
we had the same issue and to get around this I created a custom property for the Groups and added the following trigger condition:

As long as the engineers remember to set the custom property to Yes/True/1, the group alerts don't fire.

I also added a simple widget to the Home page so you can see at a glance which sites are set:

SWQL:

SELECT DISTINCT Groups.Name AS Name,
Groups.DetailsUrl AS [_LinkFor_Name]
from Orion.Groups AS [Groups]
WHERE [Groups].CustomProperties.Group_Under_Maintenance=1

Hope that helps.

Daran.

BaldFeegle

Hi Daran,

A group custom property is my backup plan. I've raised a support case to get to the bottom of why only the primary trigger condition is firing is my current plan. If that doesn't pan out then I'll go with this.

Cheers!

govitallen

Perhaps you could have an alert trigger on a group member being set to unmanaged and have the trigger action set the custom property on the group. Not sure if this would work, just spitballing

Lofstrand

Hi @BaldFeegle
First I have to agree with @chad.every , alerting on groups is not always such a great idea. Creating "grouping" with Custom Properties is often a better approach. (Though there are scenarios when groups are great!!)

Sound like you want to alert on Nodes within a group rather than the group itself. Hence why using Custom Properties would be better.

BUT, if you want to mute groups IT IS POSSIBLE in the GUI:

- Hoover over the group name in the "All Groups" widget.
- Click on the "Commands" menu that appears
- Click "Mute Alerts now"

On the group page though, there does not seem to be any information that the group is muted. So I created below script that can display this.

- On the "Group Details" page, add a "custom query widget"
- Edit the new widget and copy in below code
- Give the widget a suitable name and press submit

SELECT
    CASE
        WHEN Asup.EntityUri IS NULL THEN 'Group NOT Muted'
        ELSE CONCAT('Group ', C.Name, ' is MUTED from ',ToLocal(ASup.SuppressFrom),' to ',ISNULL(ToLocal(Asup.SuppressUntil),'"forever"'))
    END AS [Group Mute Status]

FROM Orion.Container AS C
LEFT OUTER JOIN Orion.AlertSuppression AS ASup ON C.Uri=Asup.EntityUri
WHERE C.ContainerID=${ID}

You will get something like this: