8 Replies Latest reply on Nov 7, 2014 12:02 AM by akhasheni

    alert configuration question: why did it trigger?

    akhasheni

      Here is the Trigger Condition:

      Screen Shot 2014-11-03 at 7.24.50 PM.png

       

      • The node is indeed down.
      • Its custom property "node_tier" was set to 3.
      • The alerts are designed to only fire when that value is between 1 and 4.
      • I wanted to disable the alerts and so I set "node_tier" to 5. The moment I saved it, the alert got triggered again:

       

      Tier 5 node Node_ABC is Down.

       

      Additional Information:

      - Node Tier: 5

      - Alert: [DATG-DM-VO] node down, Tier 3 escalation: email once a day

         Trigger Count: 1

         Trigger Time: 11/3/2014 4:04:40 PM

         Acknowledgement: Not Acknowledged


      Here is the alert. Note that the "Name of Alert" matches "Alert" value above. It is the same alert. "Trigger time" matches the time the node went down, "Trigger Count" seems wrong: the alert already fired once before this one, just when it happened.

      Screen Shot 2014-11-03 at 7.24.35 PM.png

      Screen Shot 2014-11-03 at 7.37.11 PM.png

      Why did this alert fire?

       

      Thanks!

        • Re: alert configuration question: why did it trigger?
          Leon Adato

          Maybe a dumb question, and it *might* not make any difference, but what kind of custom property is node_tier - text, numeric, etc?

          • Re: alert configuration question: why did it trigger?
            Leon Adato

            I got nothing. While resetting an alert WILL cause it to trigger again, you changed the value so that it SHOULDN'T have triggered again.

             

            The only other thing I can think of is to select the alert, click "reset" to ensure that all nodes for that alert are now in a "cleared" state, and see if it fires again.

             

            If so, Halloween was only a couple of days ago, so there may be a few exorcists left in town... and alive.

            1 of 1 people found this helpful
            • Re: alert configuration question: why did it trigger?
              cgregors

              Usually when troubleshooting alerts, I go back into the DB and look at the query that the alert editor wrote for me.  Quite often I find that the sql doesn't match what I intended it to do.

               

              To find the sql queries for alert triggers use this sql:

              • select alertname,triggerquery from alertdefinitions;

               

              Then go find your alert by name and grab the triggerquery column and study it.

               

              Once you've got the query, you can safely execute it by hand without worrying about it tripping anything.

               

              This is also useful when building new alerts.  Once you've designed the trigger, leave the alert disabled.  Orion will still generate the query in the db and you can study it before letting it loose.

               

              Side note:

               

              I've written > 1000 lines of php code to do this for me automagically.  It extracts alerts from the database and does an "evaluation" of them to determine what Orion targets the alert "could" trigger against and then lists the possible targets.

               

              It's surprising how many times I still write alerts that trip on too many / too few / wrong things.

               

              I do eventually plan to share the code but (like everyone else) I'm a little busy these days.

               

              Chris