WSFC... alert when node(s) are not "up".

ShawD

Hello all.. first of all, I am pretty much a novice with SAM, so forgive my ignorance/stupid especially with my terminology... I am Database Administrator (DBA), so some of you will understand straight away, with me trying to explain Windows technology. 

Recently we had the situation where we were unaware that one of our nodes for a Windows Server Failover Cluster (WSFC) was paused in Failover Cluster Manager (see image below), due to an issue with Cluster Aware Updating (CAU) leaving one of the nodes paused. Unfortunately for us, the following month when CAU ran, the cluster roles stopped, impacting our SQL Server Availability Group (AG).

Understanding that we could monitor the AG listener from the point of view of knowing when SQL Server is having issues, but what we are hoping to do is being warned of any potential problem with WSFC, before it causes an outage.

In an effort to mitigate this, I am wondering if anyone can steer me in the right direction for setting up an SAM alert, that will email my team should any of the nodes of any given cluster, are not "up" for an extended period (e.g. 30mins)?

Any help appreciated.

David

Lofstrand

Hi @ShawD 

The first thing you have to do is start monitor the cluster nodes status, after that you can create an alert for that.

One way of monitor each cluster nodes status is by creating a powershell script that is ran in a SAM Powershell component. I don't have access to any WFC at the moment so I have no way of testing this but asked copilot to create a very simple script for this. By first look I think it can work. 

I would probably add something like:

# Get all cluster nodes and their status
$nodes = Get-ClusterNode | Select-Object Name, State, NodeWeight

# Initialize status counters
$upCount = 0
$downCount = 0
$pausedCount = 0

# Output for SolarWinds SAM
foreach ($node in $nodes) {
    Write-Host "Node: $($node.Name) - State: $($node.State) - Weight: $($node.NodeWeight)"

    switch ($node.State) {
        "Up"     { $upCount++ }
        "Down"   { $downCount++ }
        "Paused" { $pausedCount++ }
    }
}

# Summary metrics for SAM
Write-Host "Statistic.Up=$upCount"
Write-Host "Statistic.Down=$downCount"
Write-Host "Statistic.Paused=$pausedCount"

"Write-host "Message.Down=$DownNodeNames"

and some logic to get the names there. So you know what nodes are having issues. Then thresholds on the values and an alert rule.

If you are new to using PowerShell components, you can take a look here: Use PowerShell in SAM