Schedule Muting of Alerting on Individual Nodes Daily

Jovetic1981

I was looking for a way to schedule the muting of alerting on individual nodes between specific times every day. I couldn't find a way to achieve this on a single node without having to create multiple alerts for specific instances. I have several servers which generate alerts daily during virus scans, or backups. I found a sample on github from the SDK and have edited to achieve what I want, but I don't understand why this is not an option as default, I can only assume I am missing something obvious and the ability is there, I just missed it? Would rather have an out-of-the-box solution rather than this. I see the option to do it on a specific day, but you cant simply pick a time instead.

If not, then feel free to use the below.

node,timefrom,timeto
SERVER01,16:30:00,17:30:00
SERVER02,16:30:00,17:30:00

The script needs to be run on a schedule at the start of each day as the script imports the above from CSV format and then proceeds to build a hash table formatting the date using the current day as its date to schedule and adds on the time from the CSV.

Import-Module SwisPowerShell
$OrionServer = 'localhost'
$Username = ''
$Password = ''
$swis = Connect-Swis -Hostname $OrionServer -Username $Username -Password $Password
#Get Nodes and schedule from csv file and convert date
$Nodes = Import-Csv 'C:\Scripts\nodes-mute-schedule.csv'
$Today = Get-Date -Format dd/MM/yyyy
$Schedule = Foreach ($Node in $Nodes){
$Table = [PsCustomObject]@{
 Node = $Node.node
 entityUris = $entityUris = Get-SwisData $swis "SELECT Uri FROM Orion.Nodes WHERE Caption IN @captions" @{captions=($node.node)}
 TimeFrom = ($Today +" "+ $Node.timefrom -as [DateTime] )
 TimeTo = ($Today +" "+ $Node.timeto -as [DateTime] )
 }
 $Table
 }
# Suppress alerts for one or more entities
Foreach ($object in $Schedule){
$entityUris = $object.entityUris
$entityUris = @( $entityUris |% {[string]$_} )
Invoke-SwisVerb $swis Orion.AlertSuppression SuppressAlerts @($entityUris, $object.Timefrom.AddHours(1), $object.Timeto.AddHours(1)) | Out-Null
}
# List alert suppressions
Get-SwisData $swis "SELECT ID, EntityUri, SuppressFrom, SuppressUntil FROM Orion.AlertSuppression"

Find more posts tagged with

Accepted answers

All comments

Jovetic1981

I've actually updated this script to work without a text file and work using Custom Properties instead. This gave me the added benefit of being able to update the schedules on an event trigger alert that looked for an event where the custom property was updated.

Add 2 custom properties with the Date/Time Value. Call them:

MuteScheduleFrom

MuteScheduleTo

Next, create a ps1 file somewhere on your primary polling engine and paste the contents below and save it.

$SwisHost = "" 
$Username = ""
$Password = ""
$SecPass  = ConvertTo-SecureString -String $Password -AsPlainText -Force
$Creds    = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $Username, $SecPass


if ( -not ( Get-Command -Name Connect-Swis -ErrorAction SilentlyContinue ) )
{
    Add-PSSnapin -Name SwisSnapin -ErrorAction Stop
}

$Swis  = Connect-Swis -Hostname $SwisHost -Credential $Creds

$SwqlForCustomProperties = "SELECT [Field] FROM Orion.CustomPropertyUsage WHERE [Table] = 'NodesCustomProperties'"
$CustomProperties = Get-SwisData -SwisConnection $Swis -Query $SwqlForCustomProperties

$SwqlQueryForDevices = "SELECT [Caption], [IP_Address], [$( $CustomProperties -join '], [' )] FROM Orion.NodesCustomProperties cp INNER JOIN Orion.Nodes nd on cp.NodeID = nd.NodeID"

$Nodes = Get-SwisData -SwisConnection $Swis -Query $SwqlQueryForDevices | ?{$_.MuteScheduleFrom -ne $null} | Select Caption,MuteScheduleFrom,MuteScheduleTo

$Today = Get-Date -Format dd/MM/yyyy

$Schedule = Foreach ($Node in $Nodes){

$Table = [PsCustomObject]@{
                            Node        = $Node.Caption
                            EntityUris  = Get-SwisData $Swis "SELECT Uri FROM Orion.Nodes WHERE Caption IN @captions" @{captions=($node.caption)}
                            TimeFrom    = ($Today +" "+ $Node.MuteScheduleFrom.TimeOfDay -as [DateTime])
                            TimeTo      = ($Today +" "+ $Node.MuteScheduleTo.TimeOfDay -as [DateTime])
            }
        $Table  
        }  

Foreach ($object in $Schedule){

$entityUris = $object.entityUris
$entityUris = @( $entityUris |% {[string]$_} )

Invoke-SwisVerb $Swis Orion.AlertSuppression SuppressAlerts @($entityUris, $object.TimeFrom, $object.TimeTo) | Out-Null
}

#Remove Old Entries that have passed -1 day from current date for instances where CustomProperty has been cleared.
#$Remove = Get-SwisData $swis "SELECT ID, EntityUri, SuppressFrom, SuppressUntil FROM Orion.AlertSuppression" | ?{$_.SuppressUntil -as[datetime] -lt (Get-Date) -and $_.SuppressUntil -ne $null}
#If($Remove){
#Invoke-SwisVerb $swis Orion.AlertSuppression ResumeAlerts @(,$Remove.EntityUri) | Out-Null
#}

Get-SwisData $swis "SELECT ID, EntityUri, SuppressFrom, SuppressUntil FROM Orion.AlertSuppression"

Next create a new a new alert with the following

Then go to your trigger action and configure the "Execute Program" with the path to PowerShell and argument to call your script.

As the date time field forces the use of Date, and we are only interested in time we strip out the date within the script and replace it with the current Day. You need to back this up with a scheduled task as well, to make it sure it updates each day.

mesverrum

Responding to why it's not an ootb option, I've been pushing the idea with SW for years that there would be a use case for a lot more calendar based capabilities, but so far it hasn't made it up the priorities list.

So the muting and unmanage functions currently rely on a table that is designed to only ever have a single entry per object, even if you get crazy and start trying to insert additional rows to create a weekly schedule they tend to get cleaned out as part of the normal operations. Bummer.

So what I do on my end is kind of similar in concept to your csv, im just doing it in an extra database table. I've got scripts that read the table and it parses cron format for scheduling recurring and also one time events. If there is an upcoming maintenance that we need the script will see it, check the relevant columns in Orion and update/ insert them as necessary, I just let that run in the background every few minutes adjusting things as we go. Downside is that it will blow away anything people schedule via the GUI, so I've just had to train my users around that behavior and remind them not to use the native functions. I display the schedules in Orion by using the custom table widget and plain SQL to reach over to my table and pull in whatever we need. If I was a bit stronger in web development stuff I believe I'd be able to set up custom HTML widgets to manage this all in the Orion console to make it seem more native, but I continue to suck at webdev so we get by.

Jovetic1981

Thanks for clarifying that. It's actually been quite fun learning all this stuff. I was asked by someone today what happens if a node goes down and you have a schedule on it, how does it alert. So I created a group called Muted Alerts and I dynamically update the group based on if the time falls within the period then send the group alert to teams as informational alert via a webbook. Only in rare situations, but learned a lot doing it. I used a custom table also, works fine. Get to keep an eye on what is in there. With what I've learned over the last few days, you realise, that if its not in the system, then you can work it in to there somehow through the backend.

Seashore

Why don't use the build in tool SolarWinds Unmanage Scheduling Utility, https://support.solarwinds.com/SuccessCenter/s/article/Configure-Orion-to-unmanage-items-on-a-scheduled-basis?language=en_US

Seems it is deprecated since long but still exist and work fine.

stuartd

'Works fine' is that an oxymoron like 'Microsoft Works' or 'definite maybe'

But agreed - just use the standalone tool that SolarWinds supplied - it's klunky but in conjunction with Windows Task Scheduler it works.

Seashore

Haha, I agree with that.

I can go with "works ok" instead of "works fine".

mesverrum

To me the main limitation to that thing is it only supports unmanage, muting is not available. Also it means that any changes to the schedule have to be done by someone with admin permission on the core application server, I run a pretty large environment and push self-service capabilities as much as possible. I don't want to spend hours after every CAB meeting processing requests from 50 different teams. With my DB based method we can just set a flag on the change request, snow automation pipes the CI and scheduled windows into our db and no human needs to get involved.

Jfrazier

This is the concept I created years ago on this topic.

dgsmith80

This is something my team are currently working on as we have a massive gap in terms of Maintenance Scheduling. Like @mesverrum I only have a small team so I'm all about self-service and automation where possible, while at the same time not giving the entire staff team admin-level access.

We currently have all staff with the ability to use the native Mute & UnManage function within the UI either Now or Schedule. But we also have an ISP PEW Process (Planned Engineering Works) - This team get a list of all the circuits that have work scheduled and proceed to raise Notification Changes (NCR) in our ServiceNow instance, and then place the Interface (Or Node) into Scheduled Maintenance Mode.

The issue I have and need to resolve is that if the device has scheduled maintenance on it for PEW, but then a network engineer comes along and places the device into maintenance for an incident or small change that maintenance schedule is wiped out!

We've been working on a way to have any Audit Event for Maintenance call a Python script that places an entry into an external MongoDB which holds a maintenance schedule for each Node. Then when a Node is UnMuted or ReManaged the script is called and it checks to see if that node has any scheduled maintenance and if it does it puts it back. However, at the moment the tricky part is the message to trigger and reset the alerts. The trigger is fine as it is within Audit Event, but the reset is in the Event table which you can't use for Alerting.

I'm keen to know if you guys are hitting any similar issues, and if so how are you getting around them? My users don't have permission to edit CP's unless I add the Update Custom Property Widget, which I would need multiple's off if that were the option. Ideally, I would rather remove the GUI option and let everyone use a Microtransaction site that kicks off the script, but that would need WebDev!

mesverrum

A quick solution that I imagine would work is doing the reset via a swql query that correlates audit events to the regular events table. I'm not at my computer but I am pretty sure they both reference the netobjectid that they are associated with

IsmaelEd

It seems to me that in this situation, dumping the request is a really good idea.