Can a new alert be triggered when any alert is active? This may sound odd, but here's where I am going with this.
We want to leverage PagerDuty for alert escalation of those conditions for which we need our team should get roused from slumber. And the PagerDuty integration requires particular trigger actions that are not present in the out of the box alerts that are already configured. Copying and duplicating all those alerts and refining all those trigger conditions limited to just the really important stuff with the PagerDuty trigger actions will be extremely cumbersome.
I would like to add a custom property to each of my nodes like "AlertPagerDuty" and add a value of yes or true for those significant nodes which should generate alerts via PagerDuty notifications.
I would like to have an alert with the PagerDuty trigger actions included that evaluates any alert for both the "AlertPagerDuty" value and the "PDIntegrationKey" as trigger conditions. If present, the trigger action would be to forward the qualifying alert to PagerDuty. This would permit me to use this single alert and custom property value of "AlertPagerDuty" to control escalation of all other existing alerts to PagerDuty without having to reproduce/duplicate all the existing alerts with PagerDuty trigger actions and specific trigger conditions.
Another thought is that I might only evaluate the node for only the PDIntegrationKey value and not add another custom property. The PDIntegrationKey would only be applied to the "important" nodes. But this would preclude me from using other PagerDuty alerting for other purposes on less significant nodes. The two property evaluation ensures I can qualify the most significant nodes and those which are subject to the single PagerDuty alert method.
Does that make sense to anyone?
Any advice?
Thanks,
Phil