Alert Suppression during power outage

kcooley7

35 Sites, not all with generators.  APC UPS' on all routers (and switches/servers/etc.)  Power goes out at remote site, the last device to run out of power is the router, 150 emails.  Power is restored, another 150 emails.  Groups and dependencies are great and work as intended, you can make your dependency based on the uplink port of the core router or the site router itself.  But, it does not help when the last thing down is the router/link.  We do have APC UPS' connected to all site routers and connected to the network and core site never goes down.  Is there a way to suppress 149 of those emails based on the OnBattery status of the UPS the site router is attached to?  I do poll this information, so it is available.  I have created a custom field Alert_Suppress, all set to false.  I have a custom field named Site (all nodes have their respective values assigned to them).  If I am going to use the Alert_Suppress field in my alert, how do I change that value prior to the 150 emails?

jkrenzien

I have been struggling with a similar situation. I think I stumbled over a possible solution when trying to help in a different post, although it will take some ground work. I don't have this working yet, but I am going to work on it for the next several days until it fits my environment. But first an explanation of the groundwork

• Have custom properties to identify each site. (I will need to work with 2 since we have over 300 remote sites, geo_id and site_id)
• Have custom property to identify power status. (I will probably just use a true/false custom prop call n_power)
• Create script that will change n_power on all nodes with the same geo_id and site_id. (This may takes some time me personally, but it will be a good learning experience)
• Modify scope of relevant alerts to require n_power not quest to false. (this way you have to worry about new nodes not having it set correctly)

So now that you have all of the groundwork here comes What I think will either be the tricky part. I don't want to take these actions if just one UPS reports a loss of input power (someone may have flipped that breaker, unplugged it for maintenance, or what have you) and I doubt anybody else would unless they only have one UPS at the site. So I am either stuck making an alert that matches each geo_id and site_id pair I have or finding some amazing miracle way to do that comparison in the alert itself. Either way I will personally alert on 2 UPS reporting inline power is out, but your mileage may vary. If there any any issues or suggestions with this somebody please let me know before I spend the next several days trying to figure this out.

I will likely start with a script like this one to create the script I need for this project.PowerShell script to modify a NODE Custom Property via Rest API/JSON