Overview

The VictorOps Ack-Back Application Template allows you to acknowledge and resolve alerts from the VictorOps portal. The application template consists of a single Windows PowerShell Script monitor component. The Powershell script queries the VictorOps public API to get a current list of acknowledged and resolved incidents.

Environments Tested With

SAM 6.2.4, NPM 12.0, SolarWinds SDK 2.1.13

Requirements

• The SolarWinds SDK must be installed on the polling engine server the node is assigned to.
• An Enterprise Subscription to VictorOps
  • The standard subscription is limited to 500 API calls per month. You will easily go beyond this limit utilizing this template
• API Access to VictorOps
  • This can be enabled via the settings page under Integrations > REST Endpoint
    http://victorops.force.com/knowledgebase/articles/Getting_Started/API-Getting-Started/
• Enable the REST Endpoint in VictorOps
  • http://victorops.force.com/knowledgebase/articles/Integration/Alert-Ingestion-API-Documentation/
• A user account that can manage alerts in SolarWinds
• SAM

If you do not own SAM it is possible to run the same powershell script using a windows scheduled task. You will need to update the SWIS credentials in the script manually.

#Create SWIS connection object

$Swis = Connect-Swis –Hostname $SolarWindsServer -UserName “solarwindsapi" -Password "password" -IgnoreSslErrors

Known Limitations and Issues

• You cannot use a \ or : in the VictorOps HTTP post. You will need to use an SQL replace function like below in the SolarWinds Alert.
  SQL: SELECT REPLACE ('''${Caption}''','\',' ')}
• There may be other special characters not supported with the SolarWinds HTTP post not documented here.
• You must include the SolarWinds variable ${N=Alerting;M=AlertObjectID} in the HTTP Post in your Alert. This contains the alert’s unique ID.
  Note – this variable is new to NPM 12 and SAM 6.2.4
• The VictorOps Incidents API does not return any custom fields you may be sending in your HTTP Post
• Unable to set the AcknowledgedBy field. This will always show the service account running the script.

Setup and Configuration

Create a Limited Rights user for SolarWinds Alert Management

1. From the Main Settings & Administration section in SolarWinds go to User Accounts > Manage Accounts
2. Select ADD NEW ACCOUNT
3. Select Orion individual account, click Next
4. In the User Name field enter solarwindsapi
5. Enter a unique strong password, click Next
6. In the Define Settings section scroll down to ALERTS. Change the dropdown box Allow Alert Management Rights to Yes
7. Scroll to the bottom and Click Submit
   

Add the Limited Rights user to the SAM Credential Library

1. From the Main Settings & Administration section in SolarWinds go to SAM Settings > GLOBAL SAM SETTINGS > Credential Library
2. Add the limited user account created above. For the credential name use SolarWinds API User
3. From the Manage Nodes page in SolarWinds, Add a new Node
4. For the Node name enter victorops.com

Create an External Node for VictorOps.com

1. From the Manage Nodes page in SolarWinds, Add a new Node
2. For the Node name enter victorops.com

The application template can be assigned to any node; however, this step creates a more visually appealing look in your SolarWinds instance.

Edit the VictorOps Ack-Back Application Template

1. If you have not already import the VictorOps Ack-Back application template from thwack
2. Select the VictorOps Ack-Back application template under Manage Application Monitor Templates, click Edit
3. Note the Polling Frequency
   1. The Polling Frequency is set to 60 seconds. I would recommend leaving this custom setting to avoid ack-back delays.
4. Expand the Component Ack-Back Get Requests
5. Change the Credential for Monitoring to the SolarWinds API User
   
   
6. Update the PowerShell script with your VictorOps API Company ID, API Key and SolarWinds primary server name. You can find the API values from your VictorOps portal under Settings > API
   
                 Near the top of the powershell script you will find two variables, $API_ID and $ApiKey. Enter your API company ID and API key from VictorOps.
          #Victor Ops Company ID
          $API_ID = "Your ID"
          #Victor Ops API Key
          $ApiKey = "Your API Key"
          #Set to your SolarWinds Primary Application Server
          $SolarWindsServer = "Your Primary SolarWinds Server Name"
7. Click Submit to finish

Assign the VictorOps Ack-Back application template

1. Assign the VictorOps Ack-Back application template to the victorops.com node created earlier.

Posting Alerts to VictorOps

There are several nuances to correctly posting alerts to victorops compared to email alerts. Below are some best practices for creating the HTTP post in SolarWinds.

• VictorOps has documentation on the REST endpoint in general http://victorops.force.com/knowledgebase/articles/Integration/Alert-Ingestion-API-Documentation/
• You cannot use a \ or : in the VictorOps HTTP post. You will need to use an SQL replace function like below in the SolarWinds Alert.
  SQL: SELECT REPLACE ('''${Caption}''','\',' ')}

Trigger and Reset Actions

Within any alert rule you will need to create a new action under the Trigger and Reset Actions sections.

Name of Action: Send to VictorOps

Action Type: Send a Get or Post Request to a Web Server

URL: alert.victorops.com/.../YourRoutingKey

• YourGeneratedURL can be found once you have enabled the REST Endpoint in VictorOps under Settings > Integrations > REST Endpoint
• YourRoutingKey can be any routing key you have setup in VictorOps. Routing Keys are created when you create schedules in VictorOps. Below is an example of where you can find the Route Key under Settings > Schedules
  

Use HTTP POST: Selected

Body to Post:

At a minimum your body needs the required fields below. You can add in any custom fields you want. Reference the Examples further below for correct syntax. In general, you follow the below format.

{

"field_name":"SolarWinds Data",

"field_name":"SolarWinds Data"

}

Required Fields (in order)

Field	Value	Notes
alert_rule	${N=Alerting;M=AlertName}
entity_display_name	Same line as an email subject
entity_id	${N=Alerting;M=AlertObjectID}
host_name	${NodeName}
ip_address	${Node.IP_Address}
message_type	CRITICAL or RECOVERY	CRITICAL is Used for the Trigger, RECOVERY for the Reset
monitor_name	SolarWinds
monitoring_tool	SolarWinds
state_message	Same line as an email subject

Screen Shot of a Trigger Action

Body Post Examples

Node Is Down

Trigger Action

{

"message_type":"CRITICAL",

"monitor_name":"SolarWinds",

"monitoring_tool":"SolarWinds",

"alert_rule":"${N=Alerting;M=AlertName}",

"state_message":"${NodeName} is ${Status}",

"entity_display_name":"${NodeName} is ${Status}",

"entity_id":"${N=Alerting;M=AlertObjectID}",

"host_name":"${NodeName}",

"ip_address":"${Node.IP_Address}"

}

Reset Action – Note we use RECOVERY for the message_type value

{

"message_type":"RECOVERY",

"monitor_name":"SolarWinds",

"monitoring_tool":"SolarWinds",

"alert_rule":"${N=Alerting;M=AlertName}",

"state_message":"${NodeName} is ${Status}",

"entity_display_name":"${NodeName} is ${Status}",

"entity_id":"${N=Alerting;M=AlertObjectID}",

"host_name":"${NodeName}",

"ip_address":"${Node.IP_Address}"

}

Volume Space Alert

Trigger Action

{

"alert_rule":"${N=Alerting;M=AlertName}",

"entity_display_name":"${NodeName} ${SQL: SELECT REPLACE ('''${Caption}''','\',' ')} has ${VolumeSpaceAvailable} free",

"entity_id":"${N=Alerting;M=AlertObjectID}",

"host_name":"${NodeName}",

"ip_address":"${Node.IP_Address}",

"message_type":"CRITICAL",

"monitor_name":"SolarWinds",

"monitoring_tool":"SolarWinds",

"state_message":"${NodeName} ${SQL: SELECT REPLACE ('''${Caption}''','\',' ')} has ${VolumeSpaceAvailable} free"

}

Reset Action

{

"alert_rule":"${N=Alerting;M=AlertName}",

"entity_display_name":"${NodeName} ${SQL: SELECT REPLACE ('''${Caption}''','\',' ')} has ${VolumeSpaceAvailable} free",

"entity_id":"${N=Alerting;M=AlertObjectID}",

"host_name":"${NodeName}",

"ip_address":"${Node.IP_Address}",

"message_type":"RECOVERY",

"monitor_name":"SolarWinds",

"monitoring_tool":"SolarWinds",

"state_message":"${NodeName} ${SQL: SELECT REPLACE ('''${Caption}''','\',' ')} has ${VolumeSpaceAvailable} free"

}

Component Based Alert

Trigger Action

{

"message_type":"CRITICAL",

"monitor_name":"SolarWinds",

"monitoring_tool":"SolarWinds",

"alert_rule":"${N=Alerting;M=AlertName}",

"state_message":"${NodeName} ${N=SwisEntity;M=ComponentAlert.ComponentName} is ${N=SwisEntity;M=Status;F=Status}",

"entity_display_name":"${NodeName} ${N=SwisEntity;M=ComponentAlert.ComponentName} is ${N=SwisEntity;M=Status;F=Status}",

"entity_id":"${N=Alerting;M=AlertObjectID}",

"host_name":"${NodeName}",

"ip_address":"${Node.IP_Address}"

}

Reset Action

{

"message_type":"RECOVERY",

"monitor_name":"SolarWinds",

"monitoring_tool":"SolarWinds",

"alert_rule":"${N=Alerting;M=AlertName}",

"state_message":"${NodeName} ${N=SwisEntity;M=ComponentAlert.ComponentName} is ${N=SwisEntity;M=Status;F=Status}",

"entity_display_name":"${NodeName} ${N=SwisEntity;M=ComponentAlert.ComponentName} is ${N=SwisEntity;M=Status;F=Status}",

"entity_id":"${N=Alerting;M=AlertObjectID}",

"host_name":"${NodeName}",

"ip_address":"${Node.IP_Address}"

}

Custom Fields

Node Based Alerts

"info":"${N=SwisEntity;M=CustomProperties.Info}",

"team":"${N=SwisEntity;M=CustomProperties.Team}"

Application and Component Based Alerts

"info":"${N=SwisEntity;M=Application.Node.CustomProperties.Info}",

"team":"${N=SwisEntity;M=Application.Node.CustomProperties.Team}"