We have built in a custom property for our servers called maintenance mode which we use in our alerts to control when we want alerts to trigger. This is used for when we are performing maintenance on the servers, such as monthly scheduled patching.
The issue i have is that when we have components go down during a period of maintenance mode (such as services not starting after automated patching) then we never get alerted once the server comes out of maintenance mode.
The latest example of this worked as follows:
- Server is put into "maintenance mode"
- Server rebooted after automated patching
- 4 automatic services fail and will not start
- services go "down" in Solarwinds, but do not trigger alert as the alert rule is configured not to send alerts during maintenance mode
- Server comes out of "maintenance mode"
- Despite services still being down, the alert never triggers
- Service Desk contact me in the morning to say that the application is down
For information, the alert in question is configured as follows (i have highlighted the section we use for maintenance mode):
Can anyone confirm for me that i should receive these alerts? It is my understanding that on the next poll after maint mode is exited these down components should trigger?