Pausing alerts when a node is down

Question

Does anybody know how I can pause an alert escalation when the node status changes and then start it back up when the node returns to its previous state?

I have an alert for my UPS devices that sends out an email every 5 minutes (via Alert escalation) once the % battery remaining is below 95%.  This works great.  My problem that I discovered this weekend due to the hurricane in the Caribbean, was that when the UPS eventually shuts itself down because the battery is drained, it kept sending the alert stating that the UPS had 4% battery capacity.  Those alerts went out for ~12 hours until I cleared the alert.

My understanding of how Alert Suppression works is that it will suppress the alert from initially being triggered, but not suppress the alert escalations once the alert has been triggered and the alert actions have started.  If I'm understanding it this wrong, let me know.

I could test this out pretty easy if I had a spare UPS here, but I don't. :)

Thanks in advance.
Michael D. Jones
JetBlue Airways

michael248363 · Answer

I can do that, but part of my resolution actions are to send out a "RESOLVED" email.  If the node is down, then the low battery condition wasn't really resolved.  It's not that big of a deal, but it does "break" the standards that I've got set up.

I may be able to live with it though if that is my only option.

Thanks

Michael

qle · Answer

Unless I'm misunderstanding how alerts work, once an alert is triggered, it stays triggered until the alert is reset and the trigger conditions are not continually rechecked to see if they still "apply".  So having a condition in the trigger that checks to see if the node status is down won't make a difference if the device was up when it was triggered but later changes to a down state.

You're absolutely correct. I completely forgot about this.

The issue is that once alert escalation starts, the only way it stops is either via:

1. The reset trigger conditions being met.

Why not set the alert to reset when node status is Down, Unknown or Unmanaged? Once the UPS comes back up, wouldn't it trigger again?

michael248363 · Answer

What I want is for it to be automated, so that it keeps sending out the email alert with the battery percentage remaining as long as the node is still reachable by Orion.  As soon as it isn't reachable, I want it to stop sending the updates telling me that it has 4% (or whatever it might be) battery remaining.  I'll get a separate Node Down email alert for the device.  The device may be down for 10 minutes, it could be down for 3 days.  It shouldn't make any difference.  I don't want to keep getting email alerts every that entire time.  When it comes back up, I should get the Node UP email alert and then the battery percentage remaining alerts should resume (4%, 11%, 19%, etc.) until it reaches 100% batter remaining, at which point the reset trigger conditions are matched and the alert is resolved.

If I use the "Stop this action if the alert has been acknowledged" option, then it will stop the alerts, when I acknowledge it, but they will never start back up.  That's the problem.

Michael