16 Replies Latest reply on Oct 16, 2017 11:31 PM by mesverrum

    Nested alerts. Can it be created?

    lcsw2013

      We have our devices grouped by both groups and custom properties. We have both custom properties and groups nested. For example.... 1. Americas group, 1.a state name, 1.b city, 1.c site name, 1d, group (Example: Unix, wintel, etc).

       

      With devices organized between custom properties and groups in a nested format as above can we use it to create one alert and have it send to the proper department? For example, One disk alert for all devices. But have a condition where if it detects it's from wintel trigger an alert to the wintel team, if it detects it's from Unix send to unix team, etc. Currently we have several duplications of alerts just to get it organized by groups. So one alert for wintel, another of the same alert for unix, etc.

       

      This creates clutter and some confusion. Where as one alert with the ability to be smart enough to email out according to the nested teams would be amazing. Efficiency is what we are aiming for. Remove clutter. Remove duplication and make the system smart. We are potentially looking at ops genie solution but we want to make our alerts better before we add more to the equation.

       

      Management is quit upset that other solutions can do this but solarwinds cannot. This is why we are working to go around the software to try and get a workaround that works. so we can make solarwinds work for us.

       

      Thanks!

        • Re: Nested alerts. Can it be created?
          njoylif

          you can put a SQL query in the alert email fields using CASE [NOTE-you cannot use () Statement] :

          I use the below to BCC the ticketing system if the group is <whatever>:

           

          sql in email alert trigger

          1 of 1 people found this helpful
            • Re: Nested alerts. Can it be created?
              njoylif

              if your Distro lists use the names of the groups, it'll be easier b/c you can ${group}+'@myco.com')  for the email.

              the single quotes can get hairy, but keep at it...it will work

              1 of 1 people found this helpful
              • Re: Nested alerts. Can it be created?
                mesverrum

                Wow I have never seen that work around for the case logic in the subject line.  I always just use logic like that in the alert engine to build rules that automatically populate the custom properties using set custom property actions, but at the end of the day both methods would get you where you need to and I guess it could be easier to write the case logic once can just paste it in where you need it.  Pretty slick.

                  • Re: Nested alerts. Can it be created?
                    govitallen

                    mesverrum I would like to see an example of this:

                    mesverrum

                    use logic like that in the alert engine to build rules that automatically populate the custom properties using set custom property actions

                    That sounds pretty slick too!

                      • Re: Nested alerts. Can it be created?
                        mesverrum

                        As a quick example I set up a condition like this:

                         

                        FYI You have to treat 'is blank' separately from 'is not equal to' because SQL won't compare a null cell with a cell that has a value in it, thats just DB logic you have to learn the hard way sometimes.

                         

                        Anyway we reset automatically when it is no longer true, our alert action is "Change Custom Property" where I pick my contactemail property and tell them what I want it to be.

                         

                         

                        Boom, I never have to worry about a windows admin forgetting to set the contact when they import new nodes and missing their alerts.  Rinse and repeat for the networking team stuff, linux admins, voip, whatever.  Sometimes the rules may need to be a little more elaborate (vendor = Cisco will also catch UCS chassis, but I probably want those to go to server team not networking, etc etc)

                         

                        I do a similar idea with ip ranges as the trigger condition instead of vendors.  If I know a particular subnet, say 10.10.10.0/24 is prod then I can automatically tag the environment for any node with an ip that 'starts with 10.10.10.'  Unfortunately the alert engine isn't aware of ip octets so ranges that aren't /24 require a little bit more manipulation but it's not rocket science to figure out the rules to get the ranges you need.  Or if a particular subnet is associated with a branch office I can use that to populate my Site_Name property.  Maybe your naming convention has useful information about the device that I might want to use to populate properties, HQ-LP-APPWEB01 might be at the HQ, Linux Server, Prod, web server for such and such application so I build rules to parse the names, although i'm more likely to use SQL to do those conditions since SQL substring() and charindex() help this kind of scenario a lot.  So on and so forth, in big environments where we don't want to do a lot of manual babysitting of the properties I sometimes have more alert rules to set up the properties and enforce standard monitoring policies than I do to actually alert people about system issues.  If your admins are comfortable in SQL its probably a bit more efficient to do all of this directly in the db but if the people I'm working with aren't comfortable we can get that same level of automation available to them through the GUI with this method, no DBA or API required.

                         

                        -Marc Netterfield

                            Loop1 Systems: SolarWinds Training and Professional Services

                        1 of 1 people found this helpful
                  • Re: Nested alerts. Can it be created?
                    lcsw2013

                    This all sounds perfect. I don't know if it's me or what but I try it I still end up with errors.

                     

                     

                    Here is where I hit I snag. I go into custom property. I set a property for node. I use text format. And I use something similar to Windows server = wintel dl. So in other words anything in windows group has the wintel dl applied to it. Didn't work. So I switched it and picked a specific windows server and placed dl. Didn't work. The alert itself works because I see it in the console, but he action fails. When I checked the logs it was complaining that the custom property didn't equal a valid email or something like that.

                     

                     

                    I must of spent like a good 3-4 hours a day for two weeks messing with this and NPM wouldn't budge on me and wouldn't work. There are a few newer suggestion above I haven't tried and will do as soon as I get a free moment. But overall this appears like exactly what we are trying to achieve.

                     

                     

                    Hopefully it's not one of those things that's staring me in the face and I'm over analyzing it. I hate it when that happens. haha.  Appreciate all the great input. Let me give it another go around and lets see what happens so far. As my luck has it, I'm scared my environment will just give up on me. And I have enough on my plate to be restoring everything back to normal.

                     

                     

                    Once again much appreciated. I'll update soon as I have something.

                        • Re: Nested alerts. Can it be created?
                          mesverrum

                          Screenshot from my lab on how we do this, this is the variable in the to line (note that there is a glitch in the latest versions where it won't let you do mroe than one variable in the To: line, this is fixed in current hotfixes)

                           

                           

                           

                           

                          Then as an example this is how that property is populated for a xen server to alert the correct admin

                           

                          1 of 1 people found this helpful
                            • Re: Nested alerts. Can it be created?
                              lcsw2013

                              It's more like dl-ir-wintelgroup@company.com All DL's are setup in this similar format. Even individual emails are first name dot last name @ company dot com.

                               

                               

                              And in my own testing things where setup just as your screenshots shows.

                               

                               

                              It's like I mentioned the alert itself seems to work. It's the action where the failure occurs with error indicating that email was in an invalid format. I've literally sat there a whole day restructuring the custom property trying all sorts of combinations and different emails. Nothing worked. All had the exact effect. Alert worked, action did not.

                               

                               

                               

                              My property was populated like so:

                               

                               

                              UnixGroup      Dl-ir-unixteam@company.com

                               

                              WintelGroup    Dl-ir-wintelteam@company.com

                               

                              and so on. As a test what I did was assign the property to one of the pollers and populated the property with solarwindsGroup    and my email address first.last@company.com. As before the alert worked. The action didn't. And same error SolarWinds complaining about the format of the email address. I don't know if it's syntax or what, but I can't understand why the system complains. Everything is setup as suggested by the thwack suggestions and also the KB that was provided to me. Heck, I've even called solarwinds since there is a KB on this and had them try to help me and they couldn't get it work. But of course, since it's customization there was no escalation or anything else they could do.

                               

                               

                               

                              I'm getting ready right now to try it again. I'll update when I have more information. Thank you!

                                • Re: Nested alerts. Can it be created?
                                  mesverrum

                                  Oh that has got to be frustrating when something you need works for everyone else and you have some kind of weird issue.  I'm really at a loss, I use this technique in different sites every week.

                                   

                                   

                                  And your alerts are otherwise fine with sending emails to the DL's when you hard code them in without the properties?

                                    • Re: Nested alerts. Can it be created?
                                      lcsw2013

                                      Marc,

                                       

                                      I looked around and found a different page where Leon gave some instructions when I tried it, it worked out.

                                       

                                      ${node.CP} was the variable I used for testing and it worked. I'm still testing out variations and trying different things. But I seem to be at the end zone and almost at a workable solution after weeks of testing. haha.

                                       

                                      Researching the error I was getting using the other variable, apparently there is known issues with NPM interpreting the email Incorrectly. I saw KB's talking about space characters. I saw other KB's mentioning things about empty space,etc. The email parser appears to have issues is my take. But when you tell NPM to look at a column and go based on values in a column....it worked!

                                       

                                      Really really strange and weird. But then again it's solarwinds, so I shouldn't be surprised. Act of congress to get this software to do anything you want. And most often then not it's by force and workarounds because natively the functions just aren't there.

                                       

                                      But if my tests work out between your help and the other document I might of just found my answer....finally!!

                                       

                                      Here is the link : Custom Property for Email Address of a Volume Alert Not Working

                                        • Re: Nested alerts. Can it be created?
                                          mesverrum

                                          So the thing about needing to put the "Node." in the variable is if you are alerting on a child object such as a disk volume or interface, the "node." basically tells the alerting engine to navigate from whatever triggered the alert to its parent node and find out the cp from that.  If the alert is for a node then you will not want that "node." in the string.

                                           

                                          Memorizing that syntax stuff is not recommended as there are many variations on it, always just set up your alert, go to the message, use the inset variable button to search for your property and let the tool figure out the syntax for you.  I do this every day and I still not always get the syntax correct on my first guess for some child object types (hardware health sensors and SAM components being the worst offenders)

                                          1 of 1 people found this helpful
                                            • Re: Nested alerts. Can it be created?
                                              lcsw2013

                                              I understand,

                                               

                                               

                                              But What I was requesting was simple really. It was one single tiny alert that would trigger, then on the action it would email the team of the device. And or device owners. Setting the alert, setting the triggers, no problem. Setting actions, is where the problem happened. Where solarwinds was stubborn and would not parse the email and work as expected.

                                               

                                               

                                              I tried three KB's and all with different variations of the variable. And all gave me the same error : The specified string is not in the form required for an email address.

                                               

                                              On this thread, More KB's was shown and yet, still I couldn't get it to work. Until I ran through that article I posted above. Sure, it might not be the cleanest way to get me what I needed but it worked. It got me to the end result. So Instead of hunting down articles to figure out and troubleshoot an error I found a workaround that cleared me hours of headaches. I understand what you are saying. But again, I got to the end goal. I'm beyond exhausted chasing this and trying to find a workable solution. And I don't have patients anymore to keep chasing this down a rabbit hole.

                                               

                                              I understand that I might not want to go with the node But if I don't use it, what else can I use when nothing else works? I understand it may bring potential headaches down the line but I don't mind. I have management fed up with the system on me daily to find a solution and when I saw this is what will allow me to sleep tonight knowing that tomorrow I can start modifying the alerts and making management happy.

                                               

                                               

                                              Management is so sick of all the issues and no solution they are really bent on finding any an all reasons to ditch this software all together. Between support giving us answers that don't make since. Features that are not there. And the system not meeting our needs. They want to ditch it and I'm on their side frankly because so far the software has no met our needs.

                                               

                                               

                                              They've been dealing with headaches since version 11. And thus far even with my help they aren't convinced that we'll find what we need with this solution. Having this will allow me to at least help hold out a little bit longer showing that there are possibilities with solarwinds.

                                               

                                               

                                              I appreciate your help greatly and advice. But I'm on empty. I need to use this workaround.

                                                • Re: Nested alerts. Can it be created?
                                                  mesverrum

                                                  I'm not saying not to do that, im just letting you know that syntax will not work for certain types of alert objects so you just need to watch out and test it before you copy paste it into all your alert actions, you need to use the correct syntax for other objects.

                                                   

                                                  I hate to do the hard sell, but it seems appropriate here with all the struggling you guys seem to have been doing, sometimes it can be beneficial to pick up some assistance. If the signature I dropped into the many threads I've helped you with doesn't already give it away, Loop1 does Solarwinds 101 training courses remotely or down in Austin to help get new admins up to speed on the basic knowledge plus best practices that would likely have saved you from smashing your head against the desk so much these last few months.  And I (plus any one of our other field engineers) could more than likely whip your Solarwinds installation into tip top shape, train your admins, and deliver your exec team's wish list in a week of pro services. Maybe two if they are really big wishes

                                                   

                                                  -Marc Netterfield

                                                      Loop1 Systems: SolarWinds Training and Professional Services