Nested alerts. Can it be created?

lcsw2013

We have our devices grouped by both groups and custom properties. We have both custom properties and groups nested. For example.... 1. Americas group, 1.a state name, 1.b city, 1.c site name, 1d, group (Example: Unix, wintel, etc).

With devices organized between custom properties and groups in a nested format as above can we use it to create one alert and have it send to the proper department? For example, One disk alert for all devices. But have a condition where if it detects it's from wintel trigger an alert to the wintel team, if it detects it's from Unix send to unix team, etc. Currently we have several duplications of alerts just to get it organized by groups. So one alert for wintel, another of the same alert for unix, etc.

This creates clutter and some confusion. Where as one alert with the ability to be smart enough to email out according to the nested teams would be amazing. Efficiency is what we are aiming for. Remove clutter. Remove duplication and make the system smart. We are potentially looking at ops genie solution but we want to make our alerts better before we add more to the equation.

Management is quit upset that other solutions can do this but solarwinds cannot. This is why we are working to go around the software to try and get a workaround that works. so we can make solarwinds work for us.

Thanks!

Find more posts tagged with

Alert

custom alerts

nested alerts

Alerting

custom

nested

Accepted answers

lcsw2013

Marc,

I looked around and found a different page where Leon gave some instructions when I tried it, it worked out.

${node.CP} was the variable I used for testing and it worked. I'm still testing out variations and trying different things. But I seem to be at the end zone and almost at a workable solution after weeks of testing. haha.

Researching the error I was getting using the other variable, apparently there is known issues with NPM interpreting the email Incorrectly. I saw KB's talking about space characters. I saw other KB's mentioning things about empty space,etc. The email parser appears to have issues is my take. But when you tell NPM to look at a column and go based on values in a column....it worked!

Really really strange and weird. But then again it's solarwinds, so I shouldn't be surprised. Act of congress to get this software to do anything you want. And most often then not it's by force and workarounds because natively the functions just aren't there.

But if my tests work out between your help and the other document I might of just found my answer....finally!!

Here is the link : Custom Property for Email Address of a Volume Alert Not Working

All comments

d09h

For nodes in each group, have you set an email address as described here? https://support.solarwinds.com/Success_Center/Network_Performance_Monitor_(NPM)/Use_Custom_properties_when_sending_email_alerts

njoylif

you can put a SQL query in the alert email fields using CASE [NOTE-you cannot use () Statement] :

I use the below to BCC the ticketing system if the group is <whatever>:

sql alert.JPG

njoylif

if your Distro lists use the names of the groups, it'll be easier b/c you can ${group}+'@myco.com') for the email.

the single quotes can get hairy, but keep at it...it will work

lcsw2013

I've tried this several times could never get this to work. SolarWinds would never recognize the entry in the custom property as an email.

mesverrum

Wow I have never seen that work around for the case logic in the subject line. I always just use logic like that in the alert engine to build rules that automatically populate the custom properties using set custom property actions, but at the end of the day both methods would get you where you need to and I guess it could be easier to write the case logic once can just paste it in where you need it. Pretty slick.

govitallen

mesverrum I would like to see an example of this:

mesverrum
use logic like that in the alert engine to build rules that automatically populate the custom properties using set custom property actions

That sounds pretty slick too!

lcsw2013

This all sounds perfect. I don't know if it's me or what but I try it I still end up with errors.

Here is where I hit I snag. I go into custom property. I set a property for node. I use text format. And I use something similar to Windows server = wintel dl. So in other words anything in windows group has the wintel dl applied to it. Didn't work. So I switched it and picked a specific windows server and placed dl. Didn't work. The alert itself works because I see it in the console, but he action fails. When I checked the logs it was complaining that the custom property didn't equal a valid email or something like that.

I must of spent like a good 3-4 hours a day for two weeks messing with this and NPM wouldn't budge on me and wouldn't work. There are a few newer suggestion above I haven't tried and will do as soon as I get a free moment. But overall this appears like exactly what we are trying to achieve.

Hopefully it's not one of those things that's staring me in the face and I'm over analyzing it. I hate it when that happens. haha. Appreciate all the great input. Let me give it another go around and lets see what happens so far. As my luck has it, I'm scared my environment will just give up on me. And I have enough on my plate to be restoring everything back to normal.

Once again much appreciated. I'll update soon as I have something.

mesverrum

your dl should be wintel@yourdomain.com is that right?

mesverrum

Screenshot from my lab on how we do this, this is the variable in the to line (note that there is a glitch in the latest versions where it won't let you do mroe than one variable in the To: line, this is fixed in current hotfixes)

Then as an example this is how that property is populated for a xen server to alert the correct admin

lcsw2013

It's more like dl-ir-wintelgroup@company.com All DL's are setup in this similar format. Even individual emails are first name dot last name @ company dot com.

And in my own testing things where setup just as your screenshots shows.

It's like I mentioned the alert itself seems to work. It's the action where the failure occurs with error indicating that email was in an invalid format. I've literally sat there a whole day restructuring the custom property trying all sorts of combinations and different emails. Nothing worked. All had the exact effect. Alert worked, action did not.

My property was populated like so:

UnixGroup Dl-ir-unixteam@company.com

WintelGroup Dl-ir-wintelteam@company.com

and so on. As a test what I did was assign the property to one of the pollers and populated the property with solarwindsGroup and my email address first.last@company.com. As before the alert worked. The action didn't. And same error SolarWinds complaining about the format of the email address. I don't know if it's syntax or what, but I can't understand why the system complains. Everything is setup as suggested by the thwack suggestions and also the KB that was provided to me. Heck, I've even called solarwinds since there is a KB on this and had them try to help me and they couldn't get it work. But of course, since it's customization there was no escalation or anything else they could do.

I'm getting ready right now to try it again. I'll update when I have more information. Thank you!

mesverrum

As a quick example I set up a condition like this:

FYI You have to treat 'is blank' separately from 'is not equal to' because SQL won't compare a null cell with a cell that has a value in it, thats just DB logic you have to learn the hard way sometimes.

Anyway we reset automatically when it is no longer true, our alert action is "Change Custom Property" where I pick my contactemail property and tell them what I want it to be.

Boom, I never have to worry about a windows admin forgetting to set the contact when they import new nodes and missing their alerts. Rinse and repeat for the networking team stuff, linux admins, voip, whatever. Sometimes the rules may need to be a little more elaborate (vendor = Cisco will also catch UCS chassis, but I probably want those to go to server team not networking, etc etc)

I do a similar idea with ip ranges as the trigger condition instead of vendors. If I know a particular subnet, say 10.10.10.0/24 is prod then I can automatically tag the environment for any node with an ip that 'starts with 10.10.10.' Unfortunately the alert engine isn't aware of ip octets so ranges that aren't /24 require a little bit more manipulation but it's not rocket science to figure out the rules to get the ranges you need. Or if a particular subnet is associated with a branch office I can use that to populate my Site_Name property. Maybe your naming convention has useful information about the device that I might want to use to populate properties, HQ-LP-APPWEB01 might be at the HQ, Linux Server, Prod, web server for such and such application so I build rules to parse the names, although i'm more likely to use SQL to do those conditions since SQL substring() and charindex() help this kind of scenario a lot. So on and so forth, in big environments where we don't want to do a lot of manual babysitting of the properties I sometimes have more alert rules to set up the properties and enforce standard monitoring policies than I do to actually alert people about system issues. If your admins are comfortable in SQL its probably a bit more efficient to do all of this directly in the db but if the people I'm working with aren't comfortable we can get that same level of automation available to them through the GUI with this method, no DBA or API required.

-Marc Netterfield

Loop1 Systems: SolarWinds Training and Professional Services

LinkedIN: Loop1 Systems
Facebook: Loop1 Systems
Twitter: @Loop1Systems