I figured I would share this as I'm sure other users have ran into a similar problem. So I've been doing something a little different lately within some of my powershell scripts that is helping to eliminate some potential false positives or momentary hits on a system. What happened was we were getting alerts that a process count monitor I had written would show 30 copies of a process running which would trigger the alert. The team would go check into it and always show it was less than the alert threshold and say it was fine. Sure enough on the next poll we would get an all clear.
I could have written into the alert logic a condition of do not trigger until condition exists for X but my issue with that is that my users don't see the alert logic. They see Red on the board and often wonder why no alert fired.
What I am doing is adding a wait condition to recheck again in a certain amount of time. Typically between 10 & 60 seconds depending on what I'm monitoring. Here is an example from an NTP script I wrote. I added a sleep condition of 10 seconds to recheck before issuing a failure. Now in some cases if my first check fails but the second one passes I may issue a warning instead of a normal (I'm not doing that in this script but I am in others depending on the need).
Please forgive any scripting errors that you may see as I'm still rather new to powershell and learning as I go.
$NTPServer = $args.get(0);
$CurrentTime = $null
$CurrentTime = w32tm /monitor /computers:$NTPServer
IF
(
$CurrentTime -match "Error"
)
{
Start-sleep -seconds 10
$CurrentTime = $null
$CurrentTime = w32tm /monitor /computers:$NTPServer
IF
(
$CurrentTime -match "Error"
)
{
Write-Host Statistic:1
Write-Host Message:Consecutive NTP requests to $NTPServer failed $CurrentTime
Exit 1
}
Else
{
Write-Host Statistic:0
Write-Host Message:$CurrentTime
Exit 0
}
}
Else
{
Write-Host Statistic:0
Write-Host Message:$CurrentTime
Exit 0
}
