Is this possible - Group Alert behavior

Question

All,

I'd like to know if this can be done. Scenario:

You have a site with 4 LAN (IDF) closets and 2 switches per closet. You create a Site-LAN Group with includes all 8 switches.

When 1 switch goes down, you want a Group "Warning" alert that lists the switch that is down.

Then another switch in the same closet or another closet goes down. You get an updated "Warning" alert that now includes all the switches that are down.

If over 50% of the switches go offline, you get a "Critical" alert with all the down switches listed.

Is this something can be done? I'm looking to make my alerts more intelligent and "root cause" actionable.

Deltona · Answer

@vinay.by 's suggestion is a good one. If you need an alternative then here's how I see it being done. You will ultimately need three alert definitions; one for Warning alerts, one for resetting a timestamp and one for Critical alerts.

Prerequisites:

Start by creating a Custom Property for groups. The purpose if this Custom Property is to log the timestamp for when a change occured in the group.

New Group Warning alert:

SWQL Used:

INNER JOIN Orion.ContainerMembers cm ON Groups.ContainerID = cm.ContainerID

INNER JOIN Orion.Nodes n ON n.NodeID = cm.MemberPrimaryID

INNER JOIN Orion.Events e ON n.NodeID = e.NetworkNode

WHERE Groups.Name = 'Site-LAN Group'

AND n.Status = 2

AND e.EventType = 1

AND (datetime(Groups.CustomProperties.C_Detected_Change) < tolocal(e.EventTime)

OR Groups.CustomProperties.C_Detected_Change IS NULL)

Explanation:

n.Status = 2 (Node is currently Down)

e.EventType = 1 (A “Node Down” event occurred)

The C_Detected_Change comparison ensures the event is newer than the last time this alert fired (or first time if NULL).

SWQL Used:

INNER JOIN Orion.ContainerMembers cm ON Groups.ContainerID = cm.ContainerID

INNER JOIN Orion.Nodes n ON n.NodeID = cm.MemberPrimaryID

INNER JOIN Orion.Events e ON n.NodeID = e.NetworkNode

WHERE Groups.Name = 'Site-LAN Group'

AND n.Status = 2

AND e.EventType = 1

AND datetime(Groups.CustomProperties.C_Detected_Change) < tolocal(e.EventTime)

Variable used:

${N=Generic;M=S}

If you want to include details about the group member in any additional trigger actions, then you could use the following:

Group Name: ${N=SwisEntity;M=Name}

Timestamp: ${N=Generic;M=DateTime;F=DateTime}

Group members currently Down:

${SQL:SELECT CAST(CAST([GroupMemberFullName] AS varchar(max)) + ' - ' + CAST([GroupMemberStatusName] AS varchar(max)) + CHAR(13) + CHAR(10) AS XML)

FROM [SolarWindsOrion].[dbo].[Containers_AlertsAndReportsData]

WHERE GroupID = ${N=SwisEntity;M=ContainerID}

AND [GroupMemberStatusName] <> 'Up'

FOR XML PATH('')}

Or if you want simplicity:

${N=OrionGroup;M=GroupStatusRootCause}

—

We now need to ensure that the timestamp in the Group Custom Property C_Detected_Change, gets updated/reset once the Group member comes back up. To do this, we will need a dedicated alert definition whose only objective is to reset the timestamp on C_Detected_Change.

New C_Detected_Change reset alert:

SWQL Used:

INNER JOIN Orion.ContainerMembers cm ON Groups.ContainerID = cm.ContainerID

INNER JOIN Orion.Nodes n ON n.NodeID = cm.MemberPrimaryID

INNER JOIN Orion.Events e ON n.NodeID = e.NetworkNode

WHERE Groups.Name = 'Site-LAN Group'

AND n.Status = 1

AND e.EventType = 5

AND datetime(Groups.CustomProperties.C_Detected_Change) < tolocal(e.EventTime)

Explanation:

n.Status = 1 (Node is currently Up)

e.EventType = 5 (A “Node Up” event occurred)

Reuse the same trigger action that you used earlier.

Finally, the Group Critical alert for when 50% of the members are down. You can use the same method already mentioned in this thread, to alert on group member availability.

vinay.by · Answer

Hi @deverts

I have never tried this and I cant as well in my environment, but give it a shot, hope this helps.

Group member availability below for set of objects works as % you can set it to 25, 50, 75, 100

You will need to create 4 alerts and use 2 conditions for 2nd 3rd and 4th alert (1st alert is straight forward the way i have mentioned

2nd alert would need to be >25 and < = 50 similarly create 3rd and 4th. You should be able to alert at specific % of group status - that's what I assuming (the way this condition works) I could be wrong :)

And set your group status to worst case so that it turns red even if 1 device goes down in the group.

This might work.

deverts · Answer

Thank you, I believe this is absolutely 1 component of the solution. But you are also correct, I'm looking for something more complex. I'd like to get a new alert each time the Group status changes, i.e., another node in the group goes down.

Alert Warning Group member X1 is down.

Alert Warning Update Group member X2 is now down.

Alert Critical Group members X1, X2, X3 and X4 are down

And so on….