Treat every alert/warning from particular nodes as Critical

Hey there,

I would like to make every alert/warning from particular nodes as Critical - even if the alert/warning is not Critical.

How could I make that happen? So far I only found the way to re-create every alert/warning and change the Trigger Condition / The scope of alert - but I would rather get to it from the Node side of the story.
But I dont know if that is possible in the way I want it.

It would really helpful for example critical parts of the infrastructure.

Thank you for your help!

Find more posts tagged with

critical

critical alert

krumpli

npm - network performance monitor

monitoring

Accepted answers

All comments

stuartd

Do you use priority or alert levels at all?

We have a node Custom Property call ed Priority. It has 5 levels from Priority_1 to 5. 5 effectively means ignore. Anyway, we then assign a specific P level to each and every node. We also went a step further for some clients and have P levels on interfaces.

We then output as part of the Trigger Action that variable and our collector ... in this case Zabbix processes it into SNOW at the appropriate incident priority. You could of course do this as an eMail.

bobmarley

@stuartd Great suggestion, that is the exact same way we did it as well. Another very nice Custom Property we use is Group, which is the SNOW group to assign the ticket to. All of these can be passed in the alert message using variables.

elmoceze

Hey there,

Thanks for the answer!

Yes of course we are using priority for the alerts / warnings.
What I would like to achieve, if I have an eg. Informational level alert (eg. "oh no one ping was lost" ) from a critical part of the infrastructure (eg. "Cisco Megarouter 9000-48SFP") ->
then I would like to get a nice red Critical warning that says Oh no one ping was lost on Cisco Megarouter 9000-48SFP - I would be happier to achieve that in Solarwinds itself, not in ticketing/email/webhook.
I had the same idea as you, to create a custom property, eg. IsItCritical - YES/NO - then two routes:
a) Recreate every alert / event, then add the condition IsItCritical equals YES -> then it is a Critical warning ==> most tedious and manual way to do it and least scalable, so I would like to stay away from that route
b) Create an alert, that has the condition IsItCritical equals YES AND if it has an active alert -> then create a Critical warning (hell even Ackownledged the previous alert) ==> but the " if it has an active alert " is SWQL, that im not familar with it - also, still just a workaround

Thats why I wanted to ask if there is an elegant solution to treat any Alert / Event from X;Y nodes as Critical ?

Thank you!

stuartd

Hmm, elegant. No, I don't think so but then I guess that comes down to what you define as elegant. But I digress.

It sounds to me like you already have your solution, you just need to implement it in some manner. One thing I didn't say about our Priority CP is that we have separate alerts for P2, P3 and P4 [as an aside, we never raise an alert directly at a P1 level that is left for a human to determine, and is also the correct ITIL approach]. The extra bit of SQL/SWQL can be achieved via the drop down boxes and all we essentially say is:

Is the node identified as a P2 node?
If yes, then has the condition existed for X Minutes
If yes, alert.

The X Minutes is where we see the difference. So for us, a P2 the condition exists for 2m, for P3 it is 5m and for P4 it's 10. Having created the first alert, it is simply a matter of duplicating, renaming and editing.

bobmarley

Our setup sounds exactly like what @stuartd described. Tickets are typically P2, P3, or P5. We skipped the P1 and P4 altogether as it aligns best with creating tickets in SNOW. So we have a custom property for nodes/interfaces that determines the priority when tickets get generated. That said in many of our alerts we have the setting below on the trigger condition tab set to 5 minutes to not report reboots.

However for network gear we have a single alert set up that checks for a Warning condition every minute, the field above is left blank, and an email is sent to the network team as soon as a warning triggers. So on network gear the team will get an email when it goes into warning and then a second alert creates ticket 5-9 minutes later when it goes into Critical. The second alert contains all of the SNOW automation to create the ticket.