25 Replies Latest reply: Feb 24, 2008 8:25 PM by SteveSW RSS

What's Wrong With This Picture?

aLTeReGo

Can anyone explain to me what's wrong with the configuration of the Advanced Alert below? I mean beside being needlessly complicated. It's actually part of a larger trigger condition that worked a while back (8.5 or 8.1 I can't remember) but doesn't work now, even when I remove the rest of the conditions (as pictured below). I got caught with my pants down this week because the alert didn't trigger even though the custom MIB poller value was not "ok".  Any help would be greatly appreciated. Thanks.

 
  • Re: What's Wrong With This Picture?
    Network_Guru

    I'm not sure why you are using multiple nested conditions.
    I would just use a single condition - Trigger when all the following apply.

    Then list the 'not OK' and then the name of the Mib Poller.
     

    • Re: What's Wrong With This Picture?
      aLTeReGo
      I'm not sure why you are using multiple nested conditions.
      I would just use a single condition - Trigger when all the following apply.


      I understand what you're saying but this is part of a larger advanced alert that was working. I copied the alert and removed all the other conditions (pictured above)   and the alert still doesn't work which is why I'm posting here looking for answers. Below is a screenshot of the original working trigger condition so you can get an idea of why I need the nested conditions.

      • Re: What's Wrong With This Picture?
        acherman

        Okay, you posted again just before I did.  I would try moving the condition of the MIB status to after the MIB matching.  I guess I'm hinting that I'm not sure what order the conditions are checked?  On all of your other lines you have the status check AFTER your MIB matching.


        Aaron

        • Re: What's Wrong With This Picture?
          aLTeReGo

          Okay, you posted again just before I did.  I would try moving the condition of the MIB status to after the MIB matching.  I guess I'm hinting that I'm not sure what order the conditions are checked?  On all of your other lines you have the status check AFTER your MIB matching.

          Ok, I'm intrigued but how would something like that look and work? Perhaps NG could chime back in since I know he's done a fair amount with Advanced Alerts.

          • Re: What's Wrong With This Picture?
            acherman

            In my mind I picture kind of a top-down, ladder style logic.


             The first line says to trigger when ALL of the following apply.  First check the status - at this point I would think "the status of what?"  If it picks a random register it may or may not match - at this point it would stop checking because not ALL conditions would match (even IF the MIB matched)......  and then check to see if the MIB matches. 


            The other way around - check to see if the MIB matches.  Then check the status of that MIB.


             And the more I look and think about this... it should be.....


            Trigger when all of the following apply


                 Trigger when any of the follwing apply


                       MIB is equal to.......


                 Status is not equal to ok


            Trigger when all of the following apply


                 ...........the rest of your conditions...........


             Aaron


  • Re: What's Wrong With This Picture?
    acherman

    And who caught you in that situation?  It's always embarassing when you have your pants down and someone runs in with a network issue.  Hope it doesn't get on YouTube!  ;-)


     I agree.  You should be able to get away with using a single "Trigger When [u]all[/u] of the following apply" with the two conditions underneath.  Other than that I don't see why it wouldn't fire.  Perhaps having the first condition match the MIB and then check it's status?  Maybe try matching the Custom Poller Name instead of the MIB?


    Aaron

  • Re: What's Wrong With This Picture?
    acherman

    Also, I'm concerned that when the Trigger occurs for matching the MIB, what is the output/result?  Does that show as a "status" for the next rule to match?  It might ****, but it might be better to break out each MIB match like the rules below it, so the Status check isn't one level above the MIB match?


    Just trying to visualize the flow in my head...  Who knows, I may be way off here...


    Aaron

    • Re: What's Wrong With This Picture?
      aLTeReGo

      If you can believe it, the advanced alert below doesn't work either. What now?

      • Re: What's Wrong With This Picture?
        aLTeReGo

        Here's the status of the custom MIB poller. As you can see the status clearly is not "ok"

        • Re: What's Wrong With This Picture?
          acherman

          Grasping at straws here.  I had a similar issue with monitoring input voltage and battery capacity of our UPS systems.  My issue also had to do with how the Custom Poller output the MIB value.


          Try this one:


          Trigger Alert when [u]all[/u] of the following apply


               MIB is equal to .....


               field Status is not equal to value ok


          Last line is a complex condition, Status is from Custom Node Pollers.


          Aaron

          • Re: What's Wrong With This Picture?
            aLTeReGo

            Try this one:

            Trigger Alert when all of the following apply

                 MIB is equal to .....

                 field Status is not equal to value ok

            Last line is a complex condition, Status is from Custom Node Pollers.

             



            Thanks for your help. You're honestly full of some great ideas but I'm afraid this one didn't work either. Sorry. got any other tricks up your sleeve?

            • Re: What's Wrong With This Picture?
              acherman

              Son of a...!!!  Don't suppose you can post shots of the custom poller config?

              • Re: What's Wrong With This Picture?
                aLTeReGo

                Son of a...!!!  Don't suppose you can post shots of the custom poller config?


                Sure thing. Part 1 of 2

                • Re: What's Wrong With This Picture?
                  aLTeReGo

                  Custom MIB Configuration Part 2 of 2

                  • Re: What's Wrong With This Picture?
                    acherman

                    Have you tried changing the "Type" to Text?  Or even None?

                    • Re: What's Wrong With This Picture?
                      aLTeReGo

                      Have you tried changing the "Type" to Text?  Or even None?

                      I haven't since there's an associated enumeration map.

                      • Re: What's Wrong With This Picture?
                        acherman

                        Ahhhhh..... right.  How about trying to match the status value of the Trigger to the matching number instead of the text?

                        • Re: What's Wrong With This Picture?
                          aLTeReGo

                          Ahhhhh..... right.  How about trying to match the status value of the Trigger to the matching number instead of the text?

                          Good idea! I tried...

                          field Status is not equal to value 3

                          but that didn't work so I tried..

                          Status is not equal to 3

                          but alas, still no joy

                        • Re: What's Wrong With This Picture?
                          acherman

                          New one:


                           Trigger Alert when all of the following apply


                               field MIB is equal to value StorageManagement-MIB:agentGlobalSystemStatus


                               field Numeric Status is not equal to 3


                          • Re: What's Wrong With This Picture?
                            aLTeReGo

                            Thanks acherman, all your suggestions have been terrific. I really appreciate your assistance. I opened a case yesterday on this and worked with an engineer into the evening. As it ultimately turned out, it seems I found another bug. I think if I find one more I win a set of steak knives.


                            One thing I neglected to mention in this thread was my alert suppression rule.


                            Suppress Alert when all of the following apply
                                  Node Status is equal to UnManaged


                            Now I didn't mention my Alert Suppression rule since I didn't think it had anything to do with why the alert wasn't triggering. After all, the node in question wasn't in an unmanaged state, so what's the problem right? Well as soon as I removed the Alert Suppression trigger, the alert triggered. Solarwinds is looking into the bug and hopefully will have this resolved in the next Orion hotfix or service pack. In the meantime, be careful where you use UnManaged status in your advanced alerts. Like me, you might find your alerts aren't triggering.


                            Thanks again acherman, and thanks Steve (You know who you are) for helping me log another bug.

                            • Re: What's Wrong With This Picture?
                              acherman

                              No worries, man.  I was just hoping to finally help someone intead of asking all the time.  haha  Maybe they should send you a t-shirt or a free year worth of support for the bugs you find?  lol


                              Good to hear you got things sorted out.  Thanks for the update.


                              Aaron

                              • Re: What's Wrong With This Picture?
                                SteveSW
                                The supression criteria is for filtering out things that are unrelated to the triggered set. The supression purposly has no knowledge of the result set of the trigger criteria. So what you had setup was supressing when ANY node was unmanaged.  The correct way to do what you want is to add that condition to the TRIGGER criteria “AND NODE STATUS NOT EQUAL TO 9”.
                                • Re: What's Wrong With This Picture?
                                  aLTeReGo

                                  Thanks for the update Steve. I guess that makes sense. I was thinking I was being clever but apparently not. If what you say is true, and I don't doubt that it is, then this logic is broken for APM advanced alerts. All of my APM advanced alerts have the Alert Suppression (Suppress Alert when all of the following apply "Node Status is equal to UnManaged") but all my APM alerts are working.

                                  • Re: What's Wrong With This Picture?
                                    SteveSW

                                    Actually, Your APM alerts DO have the same supression issue as your custom poller alerts. The supression does not know about the result set of the trigger criteria but it DOES know the net object type of the alert and filters on that. Here is what is going on under the hood for an APM supression versus a custom poller supression:
                                     


                                    SELECT Count(*) AS Supress FROM Nodes
                                    INNER JOIN APM_ApplicationAlertsData ON (Nodes.NodeID=APM_ApplicationAlertsData.NodeID)
                                    WHERE  (  (Nodes.Status = '9') )
                                     


                                    SELECT Count(*) AS Supress FROM Nodes
                                    INNER JOIN CustomPollerAssignment ON (Nodes.NodeID=CustomPollerAssignment.NodeID)
                                    WHERE  (  (Nodes.Status = '9') )
                                     


                                    If Count(*) is greater than 0 than the alert is supressed. The APM supression only looks at Nodes that have any APM applications assigned to them while the custom poller supression only looks at Nodes that have any custom poller assigned to them. Your unmanaged nodes have some custom pollers assigned to them but not any APM applications. That is why the APM alerts appear to work correctly. If you were to unmanged one of the nodes that has APM applications defined on it then any APM alerts that supressed on not being unmanaged would fail to trigger just like the custom poller ones.


                                    I would change any alert that supresses unmanaged nodes to put the unmanaged filter in the trigger criteria. The general rule of thumb is that if it CAN go in the trigger criteria it should. Supression criteria is for unrelated conditions.


                                    So this brings up the question, What would I use supression for:


                                    Supression is a way to allow users to handle topology issues until we can fully support topology.


                                    For instance you may have a node that is behind a router. You may have an ‘alert when node is down’ set up for that specific node by filtering on NodeID or NodeName. You may have the same thing set up for that router. However, if you know the router is down then you know that the node behind it will be cutoff from the system and show up as down. You already get the alert for the router so you don’t need the alert for the node.


                                    So for the node alert you set up the trigger criteria of ‘NodeName = “Name” AND Status = “DOWN”’.
                                    Then you set up the supression of: ‘NodeName = “RouterName” AND Status = “DOWN”


                                    This first checks the supression condition to see if that router is down. If it is, the alert is supressed and no further action takes place. Otherwise, the trigger condition is checked and the alert is triggered if the trigger condition is met.


                                    The UI for supression can appear like a place to put “after the trigger” exceptions. Really though, any exceptions to the trigger set can be handled in the trigger conditions.


                                    I hope this helps to clear things up for everyone. We always welcome feedback and it may be time for some feedback on supressions.
                                    --
                                    Steve