How to create an alert that will only alert if all the members of a group are NOT down or NOT in a group

I have a scenario that I can't wrap my brain around how to set up alert wise.

We have internet circuits that we monitor and alert on if they go down. The circuits are parent to primary firewall dependency.

For "critical" locations, there are 3 circuits in a group (with a custom property GroupType = Critical Circuits).

When all three Circuits in a Critical group go down, we want to fire Alert A saying the firewall needs to be rebooted. (I set up an alert that fires off when Group Status = Down and GroupType = Critical Circuits)

However, if a circuit goes down (node status = down) and is:

1) either NOT in a critical group, or

2) the entire critical group is NOT down

We want to send off Alert B which says the circuit itself is down.

I can't think of how to make the node alert accommodate for checking if it is in a group or checking if the group is not all down. Perhaps somehow in the advanced alerting features? I also am fine with splitting this up into another alert if that is necessary.

Thank you, any help is appreciated!

Find more posts tagged with

advanced_alert

Alert

advanced_alert_setup_question

alert_condition

custom_alerting

group_alerts

Alerts

Accepted answers

All comments

monitoringlife

I am not a fan of groups in SolarWinds and avoid them at all costs. The alert engine is fairly limited in what it can do with them, and there are not many links in and out of groups to the other tables.

I would cover this with a dependency. That way when the group is up, alerts will trigger as normal (your individual node/interface alert you want). But when the group is down, it suppresses the individual alert. You might need to modify the individual alert to exclude the suppressed state (believe its called 'unreachable').

KB: https://documentation.solarwinds.com/en/success_center/orionplatform/content/core-creating-a-new-dependency-sw1316.htm

and: https://documentation.solarwinds.com/en/success_center/orionplatform/content/core-viewing-alerts-on-child-objects-sw2354.htm

celesteflan

Thanks!

Yeah, I noticed that there wasn't really good way to link the group to the node in the alert builder which stumped me.

I think understand this setup, I think it would work. But if I understand correctly I'm going to have to create a dependency for each circuit? That'll be a lot, but doable. It'll end up with around 40-50 dependencies.

monitoringlife

It is a lot of front work, and I feel you coming from 200+ sites . Once its setup its great and makes your alerting better (and some cool dashboarding possibilities)

There is an API for it if you are handy or have a developer that you can extort a favor from.

celesteflan

Oh, when I tried to set up the dependency, I got the error that the child can't be a part of the parent group. Not sure if this will work then.

celesteflan

Ended up using my SQL knowledge to set up an alert that would detect this

monitoringlife

Can you share?

celesteflan

Sure. I've not perfected/tested this to see if this is 100% accurate, but I think I'll use some test machines and groups to make sure the logic is right ( and I can come back and edit this post with the accurate SQL). This is it so far:

SELECT Nodes.Caption, Nodes.Nodeid

FROM

DBO.Nodes

INNER JOIN [dbo].[ContainerMemberSnapshots] CMS ON Nodes.NodeID = CMS.EntityID

inner join [dbo].[Containers] C on C.ContainerID = CMS.ContainerID

inner join DBO.ContainerCustomProperties CCP ON C.ContainerID = CCP.ContainerID

INNER JOIN dbo.NodesCustomProperties ncp on ncp.nodeid = nodes.nodeid

where Nodes.Status = '2' and ncp.MaintenanceGroup = 'Circuits' and

(

(Nodes.NODEID IN

(SELECT

EntityID

FROM [dbo].[ContainerMemberSnapshots] CMS

inner join [dbo].[Containers] C on C.ContainerID = CMS.ContainerID

inner join DBO.ContainerCustomProperties CCP ON C.ContainerID = CCP.ContainerID

where

CCP.GroupType = 'Critical Circuits' AND C.Status <> 2)

)

(Nodes.NODEID NOT IN

(SELECT

EntityID

FROM [dbo].[ContainerMemberSnapshots] CMS

inner join [dbo].[Containers] C on C.ContainerID = CMS.ContainerID

inner join DBO.ContainerCustomProperties CCP ON C.ContainerID = CCP.ContainerID

where

CCP.GroupType = 'Critical Circuits')

)

ecklerwr1

This really stinks that groups aren't advanced enough to allow node down alert to not alert on nodes in a group??? I can't believe this! Which means going back to what we did in ancient Orion... using custom properties you have to set on every node and using them or keep us posted someone can come up with a generic query that can work for the alert. I decided to experiment some when I read this. Simple idea... want to monitor some clients but not alert on their node down status ie. no emails. You can't seem to add to say a group called clients and create alert logic that will alert on every except nodes in the client group.

Bill