Correlate UPS Failure to Switch Down Status

Hi All,

I would like to have a way to correlate when a UPS fails because of battery exhaustion and the switch stack goes to a down status that the cause is loss of power. I can't think of a way to do this currently. We are getting traps from all APC UPS devices and we see the trap of battery is almost exhausted right before a shutdown occurs on the APC UPS but theres no good way to correlate when we see switch stack down that it's because of the UPS power failure and not because a fiber uplink failed for example. We can monitor the UPS but it has an ethernet connection to the switch stack so if we lose the switch stack because of power we also lose the ability to monitor the UPS obviously. Does anyone have ideas as to how we could correlate the two events in any way? Thanks!

Find more posts tagged with

Accepted answers

smargh

Run a regular MSSQL Agent job every x minutes to query the status of the custompoller table(s) (IF poller status = "on battery" AND node status = UP), then subsequently update the Is_On_Battery custom properties of all relevant nodes. Note that custom pollers only run I think every 10 minutes, so this is assuming that your UPSes have at least 10 minutes of runtime.

Set your current "node down" alert to only alert when Is_On_Battery = FALSE.

Copy your current "node down" alert but set it to only alert with Is_On_Battery = TRUE, and include suitable descriptive text in the email alerts to say that it may have gone down due to power failure.

That's all I can think of. It's a shame that Orion alerts can't have actions involving either custom SQL, or at least be able to sett custom propertiy values for "related" rows of other nodes/volumes/interfaces etc.

All comments

Jason.Henson

wackerk24,

This is just a brainstorm idea but one thought is that you could build a view for these switches and then use a report that provides all of the traps from your UPS equipment in a resource. This isn't a direct correlation, but if you saw a switch that was down, you could drill into it and if the device were down and you had just received an alert on it, then it should be pretty apparent the issue was the UPS because you would have a recent trap for the UPS on the Node Details view. You would have to keep in mind that the same trap list would be provided for all of these switches that you use that view for but typically I would think that you or your NOC would only be drilling into the switch that is in a 'down' status.

Any other suggestions are welcome too.

Thanks,
Jason Henson
Loop1 Systems
www.Loop1Systems.com

smargh

Set your current "node down" alert to only alert when Is_On_Battery = FALSE.

jcooler

Run a regular MSSQL Agent job every x minutes to query the status of the custompoller table(s) (IF poller status = "on battery" AND node status = UP),
.....

I have been trying to figure out a way to do this for a few weeks with Orion alerts and have not found a way to do it. I think your suggestions would work for this. Even though its not the perfered solution, it should work out to accomplish the task.

I used the tiered method to surpress alerts for any device that has a higher tier at any perticular site, but it doesnt work out when we have a power outage. This is because most of our sites have UPS for the router and core switch, but not for the distribution or access switches. So we still get alerts for all the devices that are 1 tier higher than the the core because they go down first, while the router and core are still up, on UPS.

Your suggestion just might fix the problem. Only downside is that I have to go edit every node down alert, which is around 100 of them, due to using the tiered alert method.

thanks,

Jeremy