Alert on X number of Components Down

I have a single Component applied to 4 nodes that will show an up or down status if a service is running.
I want to trigger an alert if the Component is running on less than 2 servers.
So Component "Down" < 2 then trigger.

I can't figure out what to use in the last line, looking for a way to count the number of instances that are down.

Thank you for your help,
Richard

Find more posts tagged with

SAM Alerts

Alerts

Accepted answers

Seashore

I Have been thinking on this scenario before and used versions of the options @shuth describes. Thought of another way just now.

Below is a query for "Custom SWQL Alert". It will search for a component name and alert if just 1 component of that name is up. This will actually alert on the one that is up, not the once that are down, but I guess the method can be reversed also...

Take this query:

INNER JOIN
(
SELECT 
    C.name
    ,MAX(C.ComponentID) as ComponentID
    ,COUNT(C.ComponentID) AS [QTY]
FROM Orion.APM.Component AS C
WHERE
    C.Name LIKE '%ComponentToSearchFor%'  --Component name too look for
    AND C.StatusDescription='Up'  --Only Up components, not warning or critical
GROUP BY C.Name, C.StatusDescription
HAVING COUNT(C.ComponentID)<2  --Threshold to alert on
) AS T ON Component.ComponentID=T.ComponentID</pre>and add it like below:Might need some fine tuning but hope it can work as inspiration for you.

shuth

Yep, that was the custom SWQL option I suggested. Nice query!

All comments

shuth

I don't think there is an easy count method but there a few workarounds. You say you want an alert if running on less than 2 but then say you want component down on less than 2... so I'm not 100% sure of the requirements but I'm going with you want to know if the component is stopped on more than 2 servers (i.e. 3 or 4 are stopped of the 4 components)?

Option 1 - Use a Group - only works if all 4 are down

Create a new group and put the 4 components in it
Change the group to show best status (as long as one of the 4 components is up, group will be up).
Alert on the Group status going down - all 4 components going down will change group to down

Option 2 - Use a Custom SWQL query

Change the "I want to alert on" to a custom SWQL query
You'll need to write a query that spits back a component or application or node ID if the count of components down is > 2

Option 3 - Use the complex conditions in the alert (probably the easiest?)

Tick the box at the bottom of your alert to "Enable complex conditions"
Tick the box for "alert can be triggered if" and change it to 3 - if 3 (or more) of your 4 components go down, the alert will trigger
Note: you will get 3 alerts - it will trigger an alert for each of the 3 components down

Seashore

I Have been thinking on this scenario before and used versions of the options @shuth describes. Thought of another way just now.

Take this query:

INNER JOIN
(
SELECT 
    C.name
    ,MAX(C.ComponentID) as ComponentID
    ,COUNT(C.ComponentID) AS [QTY]
FROM Orion.APM.Component AS C
WHERE
    C.Name LIKE '%ComponentToSearchFor%'  --Component name too look for
    AND C.StatusDescription='Up'  --Only Up components, not warning or critical
GROUP BY C.Name, C.StatusDescription
HAVING COUNT(C.ComponentID)<2  --Threshold to alert on
) AS T ON Component.ComponentID=T.ComponentID</pre>and add it like below:Might need some fine tuning but hope it can work as inspiration for you.

rhether

Hi @Seashore,

I will give it a try this week.
I am positive I can get this working with your SWQL statement.

The part about using SQL/SWQL in advanced alerts is it prefills part of the query as I have outlined in yellow.
I guess you have to use an "INNNER JOIN" and continue from there.

I guess the purpose of this is to only allow a SELECT query to run.

Seashore

Yes, one part that you have to include to make the alert mechanics work internally in Orion i guess. The rest I "Inner join" to have the result I want.

shuth

Yep, that was the custom SWQL option I suggested. Nice query!