Schedule daily Alert Muting to cater for overnight server reboots.

petergwilson

We have a requirement to reboot several servers overnight.  It would be great if we could schedule a daily window either for each server or preferably for a Group where we could mute the alerts.  I'd prefer that people don't get spammed with alerts for something we already know about and really don't want service tickets being created.

isaacs

We have multiple patching groups on multiple schedules and were faced with the same issue. During the maintenance window we would get tons of alerts for the servers that were patching, rebooting etc... this would cause actual incidents to get ignored during the maintenance widow since it was assumed it was being patched. Getting 75 PagerDuty alerts on the weekend and having to figure out what was actual and whats patching was not fun. My solution was to create a mandatory Custom Property  "PatchingSchedule"  with drop downs for week 1 week 2 etc... Also by having the Custom Property in place I can send a report to impacted teams at the beginning of the week to "remind" them their servers are patching.
On the Orion Server I created a scheduled task to run a powershell script to Mute the entire group and the another to resume alerting.

Import-Module SwisPowerShell
$SwisConnection = Connect-Swis -Certificate
#Sql Stuff
$PatchingSchedule = "SELECT DISTINCT AlertObjects.entityuri
FROM Orion.AlertObjects
JOIN Orion.NodesCustomProperties ON NodesCustomProperties.Nodeid = AlertObjects.RelatedNodeid
WHERE NodesCustomProperties.patchingschedule = 'week 1 auto' AND AlertObjects.EntityType = 'Orion.Nodes'"
$Nodes = Get-SwisData -SwisConnection $SwisConnection -Query $PatchingSchedule
$Nodes = $Nodes |% {[string]$_}
ForEach ($Node in $Nodes){
    Invoke-SwisVerb -SwisConnection $SwisConnection -EntityName Orion.AlertSuppression -Verb SuppressAlerts -Arguments @($Nodes, [DateTime]::UtcNow) | Out-Null
}

You just need to change the custom property value in the WHERE clause to match the group to be muted.

The "resume" script just uses the "ResumeAlerts" verb instead of "SupressAlerts"

I'm sure there may be better ways to do this but this is what has worked for us.