We are rolling out a blue green deployment model and I am having a hard time creating a good way to easily switch our alerts on applications/nodes depending on which tier is live in prod. I couldn't find any other questions relating to this so thought I would ask here and hopefully we are not the first to try and implement something like this.
- So far, I have two custom properties assigned to prod nodes, one that indicates "Blue" or "Green," another that indicates "Active" or "Idle."
- I have our prod alerts that page us 24/7 also looking for the custom property of "Active" to trigger.
The switch is where it gets interesting. I want to be able to change a global type flag (via api, sql, or UI) such as a custom property named "BlueGreenLive" from Blue or Green, and let that change everything. Currently, to do this, I have set up a custom property assigned to a single node with that flag, and then I have 4 alerts set up (2 for blue nodes, 2 for green nodes), that will look for that "BlueGreenLive" flag and change the nodes from "Active" or "Idle", and all alerting is happily looking at only active nodes.
My main problem with this solution is that every node in the UI (or when adding a node) now has this custom property drop down which should not be visible. It also doesn't make a ton of sense that its attached to a particular node. Same situation occurs if its a custom property for an application, group, or alert and it doesn't appear it is possible to hide a custom property.
So that is my long winded question, is there another place I could use for this blue/green flag instead of a custom property? Is there a way I am unaware of that can hide a custom property? Would love to hear how you have implemented your blue green strategy as well if you care to share, as this was the best way i could come up with.
Thanks!
Jeff
