How to create a rollup alert which alerts on multiple issues with the same node

I want to create an alert so that if more than 1 issue affects a certain node, those issues are rolled up into a higher level alert. The alert could have the message "Multiple alerts on node X" and then list all the issues. If a new issue comes along, that new issue is also added to the alert.

I've created the following test alert using complex conditions, but the alert is created even when the first section matches an event on node A and the second section matches an event on node B. Both should be on the same node.

Note: I'm not tied to a complex condition as a solution to this. I'm open to using groups, SQL/SWQL queries, etc. to accomplish it.

Find more posts tagged with

Alert

event correlation

complex_conditions

npm

Accepted answers

All comments

KMSigma.SWI

Totally spitballing here, but what about having an alert that looks at the AlertObject table in SWQL and if it shows more than 2 distinct alerts for an object, then you send an alert.

As far as the 'content' of the message, that might be a little trickier and I don't have a good solution for it at the moment.

A potential problem I see with this is that there might be teams who only care about alerts on an "application" but a different team cares about alerts for an "interface." You don't necessarily want them to see the other's information.

Maybe an alternative would be to have the standard alert, but add an entry in the message with the count of other alerts affecting this same device? The best thing for most people would be to go to the alert details page to see the things causing alerts because that would honor any account limitations you have built based on their account.

hebron1212

That's one of the things I was considering, assuming that the object can be defined, i.e. all alerts related to node X. So, for example, if that node has a power supply issue and also alerting on syslog errors, a summary alert is raised. We could have the NOC tech search on the node name in the alert window to see the alert details for the individual alerts.

Thanks for your input!

hebron1212

I was able to get the alert rollup to work with the following SWQL query. Thanks @KMSigma.SWI

LEFT JOIN
(SELECT COUNT(*) as AlertsPer, AlertActive.TriggeredMessage AS Message, AlertObjects.EntityCaption, AlertObjects.RelatedNodeID AS TheNode
FROM Orion.AlertObjects
INNER JOIN Orion.AlertActive (nolock=true) AlertActive ON AlertObjects.AlertObjectID=AlertActive.AlertObjectID
INNER JOIN Orion.AlertConfigurations (nolock=true) AlertConfigurations ON AlertConfigurations.AlertID=AlertObjects.AlertID
Where Message not like '%currently has more than 4 active alerts.%'

group by TheNode
HAVING COUNT(*) > 4)

AS a ON a.TheNode=Nodes.NodeID
Where AlertsPer > 4

hebron1212

To list all the alerts contributing to the rolled up alert, I tried using this in the message, which works in SWQL Studio (where I replaced "${NodeID}" with an actual nodeid).

${N=SWQL;M=SELECT
AlertActive.TriggeredMessage
FROM Orion.AlertObjects
INNER JOIN Orion.AlertActive ON AlertObjects.AlertObjectID=AlertActive.AlertObjectID
Where AlertObjects.RelatedNodeId=${NodeID}}

The actual alert message just puts the above verbatim except that the NodeID is properly populated with the nodeid from the alert.