Alert Escalation Level Complex Condition

richmondg

The current escalation configuration is based solely on time. It would be helpful to support more complex conditions, such as when utilization first reaches a warning threshold and later a critical threshold. For example, an alert would fire and create a priority 3 incident ticket. If the utilization then hits the critical threshold, the escalation level could have its own trigger condition—similar to the existing trigger conditions on the Alerts Trigger Condition tab—and the ticket could be updated to a priority 2.

ESCALATION LEVEL CONDITION

Trigger Action- Escalation Level Condition

ronan.tonilon

As of now, SolarWinds Alert Manager does not natively support complex, condition-based escalation levels (such as escalating based on dynamic metric thresholds like warning → critical) within a single alert’s escalation policy. Escalations are currently time-based only—not condition-driven.

Current Capability:

In SolarWinds NPM, the escalation levels (on the Alert Actions tab) operate like this:

• Alert triggers → executes Level 1 actions (e.g., email, ticket)

• If alert remains active for X minutes → execute Level 2 actions

• And so on...

 These escalation levels cannot have their own trigger logic based on changing metric severity (e.g., CPU 75% = warning, 90% = critical).

Workaround (Best Practice):

To mimic this behavior, you can create two separate alerts, each with its own threshold and action logic:

 Alert 1: Warning Threshold

• Trigger: e.g., CPU Utilization > 75% for 5 minutes

• Action: Create Priority 3 ticket

 Alert 2: Critical Threshold

• Trigger: e.g., CPU Utilization > 90% for 5 minutes

• Action: Update existing ticket or create Priority 2 ticket

• Suppress this alert if already in critical acknowledgment state or if a resolution was recently sent

You may need to use custom properties or external system integrations (e.g., ServiceNow, JIRA) to update rather than create a new ticket.

Advanced Option (if integrated with ITSM tool):

• Use alert actions with webhooks or scripts to call APIs in your ITSM (e.g., ServiceNow REST API).

• Write logic in the script to check if a ticket exists for this node or alert.

• If yes, update ticket priority.

• If no, create a new one.

Feature Request Status:

• As of 2025, it is not officially supported out-of-the-box.

• This type of condition-based escalation logic has been a frequent feature request in the SolarWinds community. Just like this Request.

Conclusion:

Capability                                                                                                    Status
Time-based alert escalation levels                                                                 Supported
Condition-based (e.g., warning → critical) escalation in same alert              Not supported natively
Workaround using multiple alerts                                                                   Feasible
Ticket priority updates via external integration                                               With customization

Sample alert logic or scrips for integration:
Practical example of how to handle dynamic alert escalation based on utilization thresholds (e.g., CPU or interface utilization) and integrate it with an ITSM tool like ServiceNow via a script or webhook.

Use Case Recap

When a device crosses a warning threshold (e.g., CPU > 75%), trigger a Priority 3 incident. If it later crosses the critical threshold (e.g., CPU > 90%), update the existing incident to Priority 2.

------------------------------------------------------------------------------------------------------------------
Option A: SWQL-Based Alert Logic (2 Alerts Setup) 

• Alert 1 – CPU Warning (P3 Incident)

                 Trigger Condition (SWQL):

                     SELECT Nodes.NodeID

                      FROM Orion.Nodes

                      WHERE Nodes.CPULoad > 75 AND Nodes.CPULoad <= 90

                 NOTE: Trigger Action: Call webhook or PowerShell script to create P3 ticket.

• Alert 2 – CPU Critical (P2 Update)

                   Trigger Condition (SWQL):

           SELECT Nodes.NodeID

           FROM Orion.Nodes

           WHERE Nodes.CPULoad > 90

       NOTE: Trigger Action: Call webhook or PowerShell script to update ticket to P2 (if one exists).

------------------------------------------------------------------------------------------------------------------

Option B: Integration Script Example (PowerShell for ServiceNow)

    This script runs as an alert action, triggered by SolarWinds.

param(
[string]$nodeName,
[string]$currentPriority,
[string]$newPriority,
[string]$incidentShortDescription,
[string]$incidentSysId
)

# ServiceNow API settings
$instance = "yourinstance"
$username = "youruser"
$password = "yourpassword"
$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes("$username:$password"))
$headers = @{Authorization=("Basic {0}" -f $base64AuthInfo); "Content-Type" = "application/json"}

# Build the API URL
$url = "https://$instance.service-now.com/api/now/table/incident&quot;

# Query for existing incident by node name
$existingIncident = Invoke-RestMethod -Uri "$url?sysparm_query=short_descriptionLIKE$nodeName^priority=$currentPriority" -Headers $headers -Method Get

if ($existingIncident.result.Count -gt 0) {
# Update to higher priority if found
$incidentSysId = $existingIncident.result[0].sys_id
$body = @{
priority = $newPriority
comments = "Escalated due to critical threshold breach."
} | ConvertTo-Json

Invoke-RestMethod -Uri "$url/$incidentSysId" -Headers $headers -Method Patch -Body $body
}
else {
# No matching incident, create new
$body = @{
short_description = "$incidentShortDescription - $nodeName"
priority = $newPriority
comments = "New critical threshold alert from SolarWinds"
} | ConvertTo-Json

Invoke-RestMethod -Uri $url -Headers $headers -Method Post -Body $body
}

Inputs You Map from SolarWinds Alert Action:

• ${Node.Caption} → $nodeName

• Priority from alert → $currentPriority and $newPriority

• Incident description → $incidentShortDescription

Alternative: HTTP POST via Webhook Action

If you prefer not to use PowerShell, you can create a SolarWinds HTTP POST alert action to send JSON to ServiceNow directly, like:

{
"short_description": "${Node.Caption} - CPU Critical",
"priority": "2",
"comments": "Auto-escalated from SolarWinds"
}

• URL: yourinstance.service-now.com/.../incident

• Method: POST or PATCH

• Auth: Basic Authentication

------------------------------------------------------------------------------------------------------------------

Summary:

Goal                                                                                                            Solution
Create ticket at warning threshold                                                                 Alert 1 with HTTP POST or PowerShell to create P3
Escalate ticket at critical threshold                                                                Alert 2 with logic to find and update to P2
Integration options                                                                                         PowerShell, HTTP POST, or external orchestration tools (e.g., Power Automate)