VictorOps - SolarWinds Integration

Overview

The VictorOps Ack-Back Application Template allows you to acknowledge and resolve alerts from the VictorOps portal. The application template consists of a single Windows PowerShell Script monitor component. The PowerShell script queries the VictorOps public API to get a current list of acknowledged and resolved incidents.

Environments Tested With

SAM 6.2.4, NPM 12.0, SolarWinds SDK 2.1.13

Requirements

  • The SolarWinds SDK must be installed on the polling engine server the node is assigned to.
  • An Enterprise Subscription to VictorOps
    • The standard subscription is limited to 500 API calls per month. You will easily go beyond this limit utilizing this template
  • API Access to VictorOps
  • Enable the REST Endpoint in VictorOps
  • A user account that can manage alerts in SolarWinds
  • SAM

If you do not own SAM it is possible to run the same PowerShell script using a windows scheduled task. You will need to update the SWIS credentials in the script manually.

#Create SWIS connection object
$Swis = Connect-Swis –Hostname $SolarWindsServer -UserName “solarwindsapi" -Password "password" -IgnoreSslErrors

Known Limitations and Issues

  • You cannot use a \ or : in the VictorOps HTTP post. You will need to use an SQL or SWQL REPLACE function like below in the SolarWinds Alert.
  • There may be other special characters not supported with the SolarWinds HTTP post not documented here.
  • You must include the SolarWinds Alert Object ID variable ${N=Alerting;M=AlertObjectID} in the HTTP Post in your Alert. This contains the alert’s unique ID.
    Note – this variable is new to NPM 12 and SAM 6.2.4
  • The VictorOps Incidents API does not return any custom fields you may be sending in your HTTP Post
  • Unable to set the AcknowledgedBy field. This will always show the service account running the script.

Setup and Configuration

Create a Limited Rights user for SolarWinds Alert Management

  1. From the Main Settings & Administration section in SolarWinds go to User Accounts > Manage Accounts
  2. Select ADD NEW ACCOUNT
  3. Select Orion individual account, click Next
  4. In the User Name field enter solarwindsapi
  5. Enter a unique strong password, click Next
  6. In the Define Settings section scroll down to ALERTS. Change the dropdown box Allow Alert Management Rights to Yes
  7. Scroll to the bottom and Click Submit

Add the Limited Rights user to the SAM Credential Library

  1. From the Main Settings & Administration section in SolarWinds go to SAM Settings > GLOBAL SAM SETTINGS > Credential Library
  2. Add the limited user account created above. For the credential name use SolarWinds API User
  3. From the Manage Nodes page in SolarWinds, Add a new Node
  4. For the Node name enter victorops.com

Create an External Node for VictorOps.com

  1. From the Manage Nodes page in SolarWinds, Add a new Node
  2. For the Node name enter victorops.com

The application template can be assigned to any node; however, this step creates a more visually appealing look in your SolarWinds instance.

Edit the VictorOps Ack-Back Application Template

  1. If you have not already import the VictorOps Ack-Back application template from thwack
  2. Select the VictorOps Ack-Back application template under Manage Application Monitor Templates, click Edit
  3. Note the Polling Frequency.  The Polling Frequency is set to 60 seconds. I would recommend leaving this custom setting to avoid ack-back delays.
  4. Expand the Component Ack-Back Get Requests
  5. Change the Credential for Monitoring to the SolarWinds API User
    pastedImage_23.png
  6. Update the PowerShell script with your VictorOps API Company ID, API Key and SolarWinds primary server name. You can find the API values from your VictorOps portal under Settings > API.  Near the top of the PowerShell script you will find two variables, $API_ID and $ApiKey. Enter your API company ID and API key from VictorOps.
    #Victor Ops Company ID
    $API_ID = "Your ID"
    
    #Victor Ops API Key
    $ApiKey = "Your API Key"
    
    #Set to your SolarWinds Primary Application Server
    $SolarWindsServer = "Your Primary SolarWinds Server Name"
  7. Click Submit to finish

Assign the VictorOps Ack-Back application template

  1. Assign the VictorOps Ack-Back application template to the victorops.com node created earlier.

Posting Alerts to VictorOps

There are several nuances to correctly posting alerts to VictorOps compared to email alerts. Below are some best practices for creating the HTTP post in SolarWinds.

Trigger and Reset Actions

Within any alert rule you will need to create a new action under the Trigger and Reset Actions sections.

  • Name of Action: Send to VictorOps
  • Action Type: Send a Get or Post Request to a Web Server
  • URL: https://alert.victorops.com/integrations/generic/20131114/YourGeneratedURL/YourRoutingKey
  • YourGeneratedURL can be found once you have enabled the REST Endpoint in VictorOps under Settings > Integrations > REST Endpoint
  • YourRoutingKey can be any routing key you have setup in VictorOps. Routing Keys are created when you create schedules in VictorOps. Below is an example of where you can find the Route Key under Settings > Schedules
    pastedImage_22.png
  • Use HTTP POST: Selected
  • Body to Post: At a minimum your body needs the required fields below. You can add in any custom fields you want. Reference the Examples further below for correct syntax. In general, you follow standard JSON formatting.

Required Fields (in order)

Field

Value

Notes

alert_rule

${N=Alerting;M=AlertName}

entity_display_name

Same line as an email subject

entity_id

${N=Alerting;M=AlertObjectID}

host_name

${NodeName}

ip_address

${Node.IP_Address}

message_type

CRITICAL or RECOVERY

CRITICAL is Used for the Trigger, RECOVERY for the Reset

monitor_name

SolarWinds

monitoring_tool

SolarWinds

state_message

Same line as an email subject

Screen Shot of a Trigger Action

pastedImage_26.png

Body Post Examples

Node Is Down :: Trigger Action

{
    "message_type": "CRITICAL",
    "monitor_name": "SolarWinds",
    "monitoring_tool": "SolarWinds",
    "alert_rule": "${N=Alerting;M=AlertName}",
    "state_message": "${NodeName} is ${Status}",
    "entity_display_name": "${NodeName} is ${Status}",
    "entity_id": "${N=Alerting;M=AlertObjectID}",
    "host_name": "${NodeName}",
    "ip_address": "${Node.IP_Address}"
}

Node Is Down :: Reset Action

{
    "message_type": "RECOVERY",
    "monitor_name": "SolarWinds",
    "monitoring_tool": "SolarWinds",
    "alert_rule": "${N=Alerting;M=AlertName}",
    "state_message": "${NodeName} is ${Status}",
    "entity_display_name": "${NodeName} is ${Status}",
    "entity_id": "${N=Alerting;M=AlertObjectID}",
    "host_name": "${NodeName}",
    "ip_address": "${Node.IP_Address}"
}

Volume Space Alert :: Trigger Action

{
    "alert_rule": "${N=Alerting;M=AlertName}",
    "entity_display_name": "${NodeName} ${SQL: SELECT REPLACE ('''${Caption}''','\',' ')} has ${VolumeSpaceAvailable} free",
    "entity_id": "${N=Alerting;M=AlertObjectID}",
    "host_name": "${NodeName}",
    "ip_address": "${Node.IP_Address}",
    "message_type": "CRITICAL",
    "monitor_name": "SolarWinds",
    "monitoring_tool": "SolarWinds",
    "state_message": "${NodeName} ${SQL: SELECT REPLACE ('''${Caption}''','\',' ')} has ${VolumeSpaceAvailable} free"
}

Volume Space Alert :: Reset Action

{
    "alert_rule": "${N=Alerting;M=AlertName}",
    "entity_display_name": "${NodeName} ${SQL: SELECT REPLACE ('''${Caption}''','\',' ')} has ${VolumeSpaceAvailable} free",
    "entity_id": "${N=Alerting;M=AlertObjectID}",
    "host_name": "${NodeName}",
    "ip_address": "${Node.IP_Address}",
    "message_type": "RECOVERY",
    "monitor_name": "SolarWinds",
    "monitoring_tool": "SolarWinds",
    "state_message": "${NodeName} ${SQL: SELECT REPLACE ('''${Caption}''','\',' ')} has ${VolumeSpaceAvailable} free"
}

Component Alert :: Trigger Action

{
    "message_type": "CRITICAL",
    "monitor_name": "SolarWinds",
    "monitoring_tool": "SolarWinds",
    "alert_rule": "${N=Alerting;M=AlertName}",
    "state_message": "${NodeName} ${N=SwisEntity;M=ComponentAlert.ComponentName} is ${N=SwisEntity;M=Status;F=Status}",
    "entity_display_name": "${NodeName} ${N=SwisEntity;M=ComponentAlert.ComponentName} is ${N=SwisEntity;M=Status;F=Status}",
    "entity_id": "${N=Alerting;M=AlertObjectID}",
    "host_name": "${NodeName}",
    "ip_address": "${Node.IP_Address}"
}

Component Alert :: Reset Action

{
    "message_type": "RECOVERY",
    "monitor_name": "SolarWinds",
    "monitoring_tool": "SolarWinds",
    "alert_rule": "${N=Alerting;M=AlertName}",
    "state_message": "${NodeName} ${N=SwisEntity;M=ComponentAlert.ComponentName} is ${N=SwisEntity;M=Status;F=Status}",
    "entity_display_name": "${NodeName} ${N=SwisEntity;M=ComponentAlert.ComponentName} is ${N=SwisEntity;M=Status;F=Status}",
    "entity_id": "${N=Alerting;M=AlertObjectID}",
    "host_name": "${NodeName}",
    "ip_address": "${Node.IP_Address}"
}

Accessing Custom Properties :: Node-based Alerts

"info":"${N=SwisEntity;M=CustomProperties.Info}",
"team":"${N=SwisEntity;M=CustomProperties.Team}"

Accessing Custom Properties :: Application-based Alerts

"info":"${N=SwisEntity;M=Application.Node.CustomProperties.Info}",
"team":"${N=SwisEntity;M=Application.Node.CustomProperties.Team}"

Attachments:

Anonymous