Closed

Closed due to inactivity. Received 10 votes with last vote on 15 Mar 2019.

Granular Alerting

Just a thought I wanted to share.

We have a need to alert in ways that we cannot achieve with the limitation of the alert console. What we would like to have is for example: Setup an alert (Down alert). In this alert, create a sub level where we can section off different devices. On the action side create another sub level that would allow us to alert only the owners of said devices. To further explain, lets say in this down alert I have a section for windows servers and a section for backup servers and a section for Unix servers. Currently, it'll pick up all these alerts if you level the condition in general sense. But when it comes to action it can't for example detect if it's a unix servers send it to unix team, etc. And this is exactly what we want it to do. We want the down alert to section of devices so if said devices goes off it alerts their specific teams instead of leaving things high level and sending a network node down to all teams.

Only way I've found as a work around is to create separate alerts. But in doing so, clutter is created. And you end up with a large list of alerts to manage and monitor. It gets confusing for large environments like ours. We used to have alert central which allowed us to explore further the idea. But since it went EOL we ditched the software. OpsGenie seem like a good idea but we hated the fact that solarwinds went from a full featured software to a service that is now paid to get all the features and add the numbers of users we need. So basically solarwinds left us out to dry on this by killing the only alternative that would of worked for us.

In our situation we can't justify OpsGenie due to the price, and the fact that it's a bit overkill for what we need. However these features would of been great if they where native. It's not recreating the wheel but rather just expanding the capabilities and functions of the alert console. Giving us run to go granular several layers deep to accomplish a clean and effective way of creating alerts with out clutter but with efficient and high functionality.

I know most people deal with it. But it causes me and my team great pains currently to manage a large number of alerts and growing due to alerting demands we are getting. So this would help us streamline our processes and cut down on hours spent just to keep this in order.

Thanks!

  • I've only ever had that problem in the past when people put non-email address values in the custom property. 

  • I tried it. And I'm getting an error "The specified string is not in the form required for an e-mail address."

    What I did was I created a nodes custom property called Owner_Test_CP. I then set the alert up according to the KB and your explanation. The alert worked normally as a reboot alert. I rebooted a test server I have and it triggered the alert but no email. I checked logs and found the error I mentioned above. I check and the string is correct. I added the correct cp name there and still it didn't work. Any ideas? If not it's ok. We'll just continue as is with several alerts.

    thanks!

  • The system wouldn't 'know' to send the message to the different teams based on the device vendor on it's own. The idea is you would use your knowledge of those kind of rules to populate the custom property and the single alert will just end up going to whoever you have listed in the CP for any node that goes down.  Go to the custom property editor, group by vendor, select windows and set the owner property to WindowsAdmins@yourcompany.com, grab all your Cisco and firewalls and set them to NetworkAdmins@yourcompany.com

    Taking this a step further you can also write a separate alert to automate populating the custom property based on the vendor, and in most cases that is what I do.  Something like a rule to trigger when the Node custom property "Owner" is blank and the node vendor is net-snmp then create the action  "Set Custom Property" to set Owner = UnixAdmins@yourcompany.com

    Obviously automatic rules like that will have caveats you need to watch out for, such as how some network appliances show up as net-snmp, etc,  but I find that it tends to be pretty fast to find those exceptions and correct them when needed.

    With the rules in place you very rarely have to worry about things going to the wrong team, but obviously you would want to make sure to have feedback from the teams if they do see something misrouted toward them.

  • Marc,

    Sorry if I still sound confused. I'm trying to see it in my mind and draw it out to get an understanding but having trouble. Here is where I hit a snag. In my situation lets say I have a node down alert. This alert is to catch all devices in general that have a down status. The trigger condition is set to all devices and trigger is set to node status is equal to down.

    Now lets say a Unix system goes down. will the above know it's a unix device and only send an email to unix? If a windows system fails. Will the above know it's windows server and only email the windows team?

    Second part to the question. In building the custom property do I just create one and name it owner and for example take devices and assign the DL to those custom properties?

    I'm only trying to get a good understanding because temporarily this might actually help us as a workaround. I just need to make sure that teams won't get the wrong email like for example windows getting unix emails and so on.

    thanks,

    leandro

  • Lets say you want to do a node down alert that goes to the infrastructure support person for that node. Create a custom property for you to put those emails, in my case we will call it "Owner"

    Set the property for all the nodes so the distribution list for you linux nodes is the owner there, network stuff goes to networking, etc etc

    On the subject line of the email action it should look like this

    pastedImage_0.png

    This says that whatever object triggers this alert, send the message to whatever string is listed under it's owner custom property. 

    -Marc Netterfield

        Loop1 Systems: SolarWinds Training and Professional Services

  • Marc,

    Can you help me understand this a bit further. I've been doing some test runs using that KB but I'm still a bit confused. Do you have an alert created you could show me as an example of how it was setup?

  • Yeah, having a full if...then logic in the alert builder would probably make things easier for people to jump into the tool, since your ask boils down to being able to use the group/vendor/whatever as your "if"

  • Thank you for the KB. But the point of the feature request was basically to do the same thing but have it native and more simple so that everyone can use it. The whole point is to be able to get alerts without needing Tylenol at the end, hahaha emoticons_happy.png.

    When too much is involved for a simple end goal. I end up giving up, and start looking for alternatives. If there is a way to make it native and simplify the process that would be great.  It's why I figured it would be a good idea to throw out there.

  • The common way to address your need to route different nodes to different teams is either the way you said above, create a bunch of different alerts for each team, or what most advanced users do is to create a custom property on nodes with the email address of whoever you want to notify when that node triggers an alert.  Then on the alerts you just set the To: line to use the variable for the node owner

    This is explained in this KB Sending Alerts to a Custom Property - SolarWinds Worldwide, LLC. Help and Support

    Should only take a couple minutes in the custom property editor to set all your Unix devices to one distribution list and windows to another and network gear to theirs, etc.

    -Marc Netterfield

        Loop1 Systems: SolarWinds Training and Professional Services